Starting with satnogs-decoders


#1

This originally was intended to be a wiki article, but I didn’t find a good way to use code tags in the current wiki setup.

Here it goes, suggestions very welcome:

Writing a kaitai decoder for SatNOGS Data Warehouse

This is meant as a first document to get people started with telemtry downlink decoders and dashboards and should be improved by anyone who runs into problems or has ideas how to improve things in here! Feel free to either contact me or make those changes on your own.

Patrick, DL4PD (aka oldbug on the IRC)

What is it and why do we need it

Since we closed the loop between SatNOGS Network and SatNOGS Database all decoded frames from a satellite are availabe in SatNOGS DB, no matter if they are transferred via third party SiDS telemetry forwarders (like forwarders of Mike Rupprecht, DK3WN, the UZ7HO HS Modem, or even Dani Estevez’ gr-satellites, maybe even more), or decoded from a SatNOGS node through e.g. the internal gr-satnogs or, on a more hacky way, by decoding the data manually and feed the frames into the node that uploads them. Now there are several not human readable bytes without any human readable content in them stored in the DB. The first thing we need is to reproduce the original structure of the values stored inside these raw bytes. They might be int, float, unsigned, bitfields - what ever you can imagine! That’s where SatNOGS Decoders hook in. To get the chain complete we need another step, which is described in another article, as we do have the original structure of the data back, but it might even be human unreadable. Let me show you an example:

A frame might contain a byte sequence of 0x12 0x23 0x34 and from a telemetry description we know, the first byte is originally an unsigned eight bit integer, so we have to somehow convert this value into this. For 8 bit this is pretty easy as we do not have to meet any endianess. Now that we know it is 8 bit unsigned, there is no need to check for a sign bit and we already know that values are in the range [0…255] == 256 digits. Easy step. The next two bytes are described as 16 bit signed integer. We can now write it as 0x2334 as we know it is two bytes (== 16 bit) and we know the range is -32768 up to 32767. In kaitai this is described as a sequence, e.g.:

seq:
  - id: adc_8bit_unsigned
    type: u1
  - id: adc_16bit_signed
    type: s2

We can now access these two values as adc_8bit_unsigned and adc_16bit_signed and get the correct representation. In a real world of bytestreams one has to care about endianess as well, but please see the telemetry documentation for details and watch out for examples in the existing decoders. Now you might be getting the idea: we still do not have human readable data, because a value of e.g. 123 represented inside the adc_8bit_unsigned measurement doesn’t mean anything without a conversion rule. Yes, again, please see the satellites’ documentation to convert this values. We decided to do those calculations in the next layer - the dashboards. So we end up with raw values on the output of the satnogs-decoders.

Specialities for satnogs-decoders

As we need to somehow convert the raw output from satnogs-decoders into something we can store in the SatNOGS Data Warehouse, a timeseries database called InfluxDB we need to convert the output into JSON objects. We implemented a way to also ignore parts of the frames that might be unneccessary for a display of the interesting parts of the telemetry. We called it :field attribute and will be placed in a documenting comment in the kaitai struct file.

meta:
  id: example
  endian: be
doc: |
  :field example_obc_temperature: adc_8bit_unsigned
  :field example_battery_current: adc_16bit_signed

The part :field tells the conversion function that the adc_8bit_unsigned ID will be used as example_obc_temperature in the Data Warehouse as a measurement name. You can use this name as reference in the query you later apply in the Grafana frontend. The complete example as follows can be used to play around in the kaitai WebIDE:

meta:
  id: example
  endian: be
doc: |
  :field example_obc_temperature: adc_8bit_unsigned
  :field example_battery_current: adc_16bit_signed

seq:
  - id: adc_8bit_unsigned
    type: u1
  - id: adc_16bit_signed
    type: s2

Some examples:

As we have a lot of donwlinks using the popular AX.25 framing, a very basic decoder for these would look like this:

meta:
  id: ax25frames
  endian: be
doc: |
  :field ax25frames_dest_callsign: ax25_frame.ax25_header.dest_callsign_raw.callsign_ror.callsign
  :field ax25frames_src_callsign: ax25_frame.ax25_header.src_callsign_raw.callsign_ror.callsign
  :field ax25frames_src_ssid: ax25_frame.ax25_header.src_ssid_raw.ssid
  :field ax25frames_dest_ssid: ax25_frame.ax25_header.dest_ssid_raw.ssid
  :field ax25frames_ctl: ax25_frame.ax25_header.ctl
  :field ax25frames_pid: ax25_frame.payload.pid
  :field ax25frames_info: ax25_frame.payload.ax25_info

seq:
  - id: ax25_frame
    type: ax25_frame
    doc-ref: 'https://www.tapr.org/pub_ax25.html'

types:
  ax25_frame:
    seq:
    - id: ax25_header
      type: ax25_header
    - id: payload
      type:
        switch-on: ax25_header.ctl & 0x13
        cases:
          0x03: ui_frame
          0x13: ui_frame
          0x00: i_frame
          0x02: i_frame
          0x10: i_frame
          0x12: i_frame
          #0x11: s_frame

  ax25_header:
    seq:
      - id: dest_callsign_raw
        type: callsign_raw
      - id: dest_ssid_raw
        type: ssid_mask
      - id: src_callsign_raw
        type: callsign_raw
      - id: src_ssid_raw
        type: ssid_mask
      - id: ctl
        type: u1

  callsign_raw:
    seq:
      - id: callsign_ror
        process: ror(1)
        size: 6
        type: callsign

  callsign:
    seq:
      - id: callsign
        type: str
        encoding: ASCII
        size: 6

  ssid_mask:
    seq:
      - id: ssid_mask
        type: u1
    instances:
      ssid:
        value: (ssid_mask & 0x0f) >> 1

  i_frame:
    seq:
      - id: pid
        type: u1
      - id: ax25_info
        size-eos: true

  ui_frame:
    seq:
      - id: pid
        type: u1
      - id: ax25_info
        size-eos: true

Another good start to deal with would be the CubeSatSim decoder. It also uses AX.25 framing and has some specials in the payload, as it transmits data like the old OSCARS did:

meta:
  id: cubesatsim
  endian: be
doc: |
  :field cubesatsim_dest_callsign: ax25_frame.ax25_header.dest_callsign_raw.callsign_ror.callsign
  :field cubesatsim_src_callsign: ax25_frame.ax25_header.src_callsign_raw.callsign_ror.callsign
  :field cubesatsim_src_ssid: ax25_frame.ax25_header.src_ssid_raw.ssid
  :field cubesatsim_dest_ssid: ax25_frame.ax25_header.dest_ssid_raw.ssid
  :field cubesatsim_ctl: ax25_frame.ax25_header.ctl
  :field cubesatsim_pid: ax25_frame.payload.pid
  :field cubesatsim_data_type: ax25_frame.payload.ax25_info.data_type
  :field cubesatsim_channel_1a_val: ax25_frame.payload.ax25_info.payload.channel_1a_val
  :field cubesatsim_channel_1b_val: ax25_frame.payload.ax25_info.payload.channel_1b_val
  :field cubesatsim_channel_1c_val: ax25_frame.payload.ax25_info.payload.channel_1c_val
  :field cubesatsim_channel_1d_val: ax25_frame.payload.ax25_info.payload.channel_1d_val
  :field cubesatsim_channel_2a_val: ax25_frame.payload.ax25_info.payload.channel_2a_val
  :field cubesatsim_channel_2b_val: ax25_frame.payload.ax25_info.payload.channel_2b_val
  :field cubesatsim_channel_2c_val: ax25_frame.payload.ax25_info.payload.channel_2c_val
  :field cubesatsim_channel_2d_val: ax25_frame.payload.ax25_info.payload.channel_2d_val
  :field cubesatsim_channel_3a_val: ax25_frame.payload.ax25_info.payload.channel_3a_val
  :field cubesatsim_channel_3b_val: ax25_frame.payload.ax25_info.payload.channel_3b_val
  :field cubesatsim_channel_3c_val: ax25_frame.payload.ax25_info.payload.channel_3c_val
  :field cubesatsim_channel_3d_val: ax25_frame.payload.ax25_info.payload.channel_3d_val
  :field cubesatsim_channel_4a_val: ax25_frame.payload.ax25_info.payload.channel_4a_val
  :field cubesatsim_channel_4b_val: ax25_frame.payload.ax25_info.payload.channel_4b_val
  :field cubesatsim_channel_4c_val: ax25_frame.payload.ax25_info.payload.channel_4c_val
  :field cubesatsim_channel_4d_val: ax25_frame.payload.ax25_info.payload.channel_4d_val
  :field cubesatsim_channel_5a_val: ax25_frame.payload.ax25_info.payload.channel_5a_val
  :field cubesatsim_channel_5b_val: ax25_frame.payload.ax25_info.payload.channel_5b_val
  :field cubesatsim_channel_5c_val: ax25_frame.payload.ax25_info.payload.channel_5c_val
  :field cubesatsim_channel_5d_val: ax25_frame.payload.ax25_info.payload.channel_5d_val
  :field cubesatsim_channel_6a_val: ax25_frame.payload.ax25_info.payload.channel_6a_val
  :field cubesatsim_channel_6b_val: ax25_frame.payload.ax25_info.payload.channel_6b_val
  :field cubesatsim_channel_6c_val: ax25_frame.payload.ax25_info.payload.channel_6c_val
  :field cubesatsim_channel_6d_val: ax25_frame.payload.ax25_info.payload.channel_6d_val

seq:
  - id: ax25_frame
    type: ax25_frame
    doc-ref: 'https://www.tapr.org/pub_ax25.html'

types:
  ax25_frame:
    seq:
    - id: ax25_header
      type: ax25_header
    - id: payload
      type:
        switch-on: ax25_header.ctl & 0x13
        cases:
          0x03: ui_frame
          0x13: ui_frame
          0x00: i_frame
          0x02: i_frame
          0x10: i_frame
          0x12: i_frame
          #0x11: s_frame

  ax25_header:
    seq:
      - id: dest_callsign_raw
        type: callsign_raw
      - id: dest_ssid_raw
        type: ssid_mask
      - id: src_callsign_raw
        type: callsign_raw
      - id: src_ssid_raw
        type: ssid_mask
      - id: ctl
        type: u1

  callsign_raw:
    seq:
      - id: callsign_ror
        process: ror(1)
        size: 6
        type: callsign

  callsign:
    seq:
      - id: callsign
        type: str
        encoding: ASCII
        size: 6

  ssid_mask:
    seq:
      - id: ssid_mask
        type: u1
    instances:
      ssid:
        value: (ssid_mask & 0x0f) >> 1

  i_frame:
    seq:
      - id: pid
        type: u1
      - id: ax25_info
        size-eos: true

  ui_frame:
    seq:
      - id: pid
        type: u1
      - id: ax25_info
        size-eos: true
        type: cubesatsim_data

  cubesatsim_data:
    seq:
      - id: data_type
        type: u2
      - id: payload
        type:
          switch-on: data_type
          cases:
            _:  cubesatsim_ao_7
            0x6869:  cubesatsim_ao_7

  cubesatsim_ao_7:
    seq:
      - id: ao_7_magic
        contents: ' hi '
      - id: channel_1a_id
        contents: '1'
      - id: channel_1a_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_1a
        size: 1
      - id: channel_1b_id
        contents: '1'
      - id: channel_1b_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_1b
        size: 1
      - id: channel_1c_id
        contents: '1'
      - id: channel_1c_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_1c
        size: 1
      - id: channel_1d_id
        contents: '1'
      - id: channel_1d_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_1d
        size: 1

      - id: channel_2a_id
        contents: '2'
      - id: channel_2a_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_2a
        size: 1
      - id: channel_2b_id
        contents: '2'
      - id: channel_2b_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_2b
        size: 1
      - id: channel_2c_id
        contents: '2'
      - id: channel_2c_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_2c
        size: 1
      - id: channel_2d_id
        contents: '2'
      - id: channel_2d_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_2d
        size: 1

      - id: channel_3a_id
        contents: '3'
      - id: channel_3a_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_3a
        size: 1
      - id: channel_3b_id
        contents: '3'
      - id: channel_3b_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_3b
        size: 1
      - id: channel_3c_id
        contents: '3'
      - id: channel_3c_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_3c
        size: 1
      - id: channel_3d_id
        contents: '3'
      - id: channel_3d_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_3d
        size: 1

      - id: channel_4a_id
        contents: '4'
      - id: channel_4a_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_4a
        size: 1
      - id: channel_4b_id
        contents: '4'
      - id: channel_4b_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_4b
        size: 1
      - id: channel_4c_id
        contents: '4'
      - id: channel_4c_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_4c
        size: 1
      - id: channel_4d_id
        contents: '4'
      - id: channel_4d_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_4d
        size: 1

      - id: channel_5a_id
        contents: '5'
      - id: channel_5a_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_5a
        size: 1
      - id: channel_5b_id
        contents: '5'
      - id: channel_5b_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_5b
        size: 1
      - id: channel_5c_id
        contents: '5'
      - id: channel_5c_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_5c
        size: 1
      - id: channel_5d_id
        contents: '5'
      - id: channel_5d_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_5d
        size: 1

      - id: channel_6a_id
        contents: '6'
      - id: channel_6a_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_6a
        size: 1
      - id: channel_6b_id
        contents: '6'
      - id: channel_6b_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_6b
        size: 1
      - id: channel_6c_id
        contents: '6'
      - id: channel_6c_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_6c
        size: 1
      - id: channel_6d_id
        contents: '6'
      - id: channel_6d_val_raw
        type: u1
        repeat: expr
        repeat-expr: 2
      - id: delim_6d
        size: 1

    instances:
        channel_1a_val:
          value: ((channel_1a_val_raw[0] - 0x30) * 10 + (channel_1a_val_raw[1] - 0x30)) * 29.5
          doc: 'value * 29.5 [mA]'
        channel_1b_val:
          value: 1970 - (20 * ((channel_1b_val_raw[0] - 0x30) * 10 + (channel_1b_val_raw[1] - 0x30)))
          doc: '1970 - (20 * value) [mA]'
        channel_1c_val:
          value: 1970 - (20 * ((channel_1c_val_raw[0] - 0x30) * 10 + (channel_1c_val_raw[1] - 0x30)))
          doc: '1970 - (20 * value) [mA]'
        channel_1d_val:
          value: (channel_1d_val_raw[0] - 0x30) * 10 + (channel_1d_val_raw[1] - 0x30)
          doc: '1970 - (20 * value) [mA]'

        channel_2a_val:
          value: 1970 - (20 * ((channel_2a_val_raw[0] - 0x30) * 10 + (channel_2a_val_raw[1] - 0x30)))
          doc: '1970 - (20 * value) [mA]'
        channel_2b_val:
          value: 8 * ((1 - 0.01 * ((channel_2b_val_raw[0] - 0x30) * 10 + (channel_2b_val_raw[1] - 0x30))) * (1 - 0.01 * ((channel_2b_val_raw[0] - 0x30) * 10 + (channel_2b_val_raw[1] - 0x30))))
          doc: '8 * (1 - 0.01 * value)^2 [W]'
        channel_2c_val:
          value: 15.16 * ((channel_2c_val_raw[0] - 0x30) * 10 + (channel_2c_val_raw[1] - 0x30))
          doc: '15.16 * value [h]'
        channel_2d_val:
          value: 40 * (((channel_2d_val_raw[0] - 0x30) * 10 + (channel_2d_val_raw[1] - 0x30)) - 50)
          doc: '40 * (value - 50) [mA]'

        channel_3a_val:
          value: 0.1 * ((channel_3a_val_raw[0] - 0x30) * 10 + (channel_3a_val_raw[1] - 0x30)) + 6.4
          doc: '0.1 * value + 6.4 [V]'
        channel_3b_val:
          value: 0.1 * ((channel_3b_val_raw[0] - 0x30) * 10 + (channel_3b_val_raw[1] - 0x30))
          doc: '0.1 * value [V]'
        channel_3c_val:
          value: 0.15 * ((channel_3c_val_raw[0] - 0x30) * 10 + (channel_3c_val_raw[1] - 0x30))
          doc: '0.15 * value [V]'
        channel_3d_val:
          value: 95.8 - 1.48 * ((channel_3d_val_raw[0] - 0x30) * 10 + (channel_3d_val_raw[1] - 0x30))
          doc: '95.8 - 1.48 * value [C]'

        channel_4a_val:
          value: 95.8 - 1.48 * ((channel_4a_val_raw[0] - 0x30) * 10 + (channel_4a_val_raw[1] - 0x30))
          doc: '95.8 - 1.48 * value [C]'
        channel_4b_val:
          value: 95.8 - 1.48 * ((channel_4b_val_raw[0] - 0x30) * 10 + (channel_4b_val_raw[1] - 0x30))
          doc: '95.8 - 1.48 * value [C]'
        channel_4c_val:
          value: 95.8 - 1.48 * ((channel_4c_val_raw[0] - 0x30) * 10 + (channel_4c_val_raw[1] - 0x30))
          doc: '95.8 - 1.48 * value [C]'
        channel_4d_val:
          value: 95.8 - 1.48 * ((channel_4d_val_raw[0] - 0x30) * 10 + (channel_4d_val_raw[1] - 0x30))
          doc: '95.8 - 1.48 * value [C]'

        channel_5a_val:
          value: 95.8 - 1.48 * ((channel_5a_val_raw[0] - 0x30) * 10 + (channel_5a_val_raw[1] - 0x30))
          doc: '95.8 - 1.48 * value [C]'
        channel_5b_val:
          value: 11.67 * ((channel_5b_val_raw[0] - 0x30) * 10 + (channel_5b_val_raw[1] - 0x30))
          doc: '11.67 * value [mA]'
        channel_5c_val:
          value: 95.8 - 1.48 * ((channel_5c_val_raw[0] - 0x30) * 10 + (channel_5c_val_raw[1] - 0x30))
          doc: '95.8 - 1.48 * value [C]'
        channel_5d_val:
          value: 11 + 0.82 * ((channel_5d_val_raw[0] - 0x30) * 10 + (channel_5d_val_raw[1] - 0x30))
          doc: '11 + 0.82 * value [mA]'

        channel_6a_val:
          value: (((channel_6a_val_raw[0] - 0x30) * 10 + (channel_6a_val_raw[1] - 0x30)) * ((channel_6a_val_raw[0] - 0x30) * 10 + (channel_6a_val_raw[1] - 0x30))) / 1.56
          doc: 'value^2 / 1.56 [mA]'
        channel_6b_val:
          value: 0.1 * (((channel_6b_val_raw[0] - 0x30) * 10 + (channel_6b_val_raw[1] - 0x30)) * ((channel_6b_val_raw[0] - 0x30) * 10 + (channel_6b_val_raw[1] - 0x30))) + 35
          doc: '0.1 * value^2 + 35 [mA]'
        channel_6c_val:
          value: 0.041 * (((channel_6c_val_raw[0] - 0x30) * 10 + (channel_6c_val_raw[1] - 0x30)) * ((channel_6c_val_raw[0] - 0x30) * 10 + (channel_6c_val_raw[1] - 0x30)))
          doc: '0.041 * value^2 [mA]'
        channel_6d_val:
          value: 0.01 * ((channel_6d_val_raw[0] - 0x30) * 10 + (channel_6d_val_raw[1] - 0x30))
          doc: '0.01 * value'

As this looks like a rather complex example it is one of the most easiest to implement and already contains some conversion functions. Please simply start with your favourite satellite you have a documentation for and try to figure out how things work. If there are any questions feel free to join the chat, ask on the forums, or dm me!


#2

Hi Patrick, thanks for your offer to help with forwarder parsing.
Let me know what kind of errors you are noticing that I should filter before, uploading to SatNOGS. Thanks.
AD7NP


#3

That’s really really helpful @DL4PD. Good job! I was wondering if there’s any API to access the InfluxDB. We intend to use the decoded frames for polaris and I don’t think doing the decoding again would be too smart…