This originally was intended to be a wiki article, but I didn’t find a good way to use code tags in the current wiki setup.
Here it goes, suggestions very welcome:
Writing a kaitai decoder for SatNOGS Data Warehouse
This is meant as a first document to get people started with telemtry downlink decoders and dashboards and should be improved by anyone who runs into problems or has ideas how to improve things in here! Feel free to either contact me or make those changes on your own.
Patrick, DL4PD (aka oldbug on the IRC)
What is it and why do we need it
Since we closed the loop between SatNOGS Network and SatNOGS Database all decoded frames from a satellite are availabe in SatNOGS DB, no matter if they are transferred via third party SiDS telemetry forwarders (like forwarders of Mike Rupprecht, DK3WN, the UZ7HO HS Modem, or even Dani Estevez’ gr-satellites, maybe even more), or decoded from a SatNOGS node through e.g. the internal gr-satnogs or, on a more hacky way, by decoding the data manually and feed the frames into the node that uploads them. Now there are several not human readable bytes without any human readable content in them stored in the DB. The first thing we need is to reproduce the original structure of the values stored inside these raw bytes. They might be int, float, unsigned, bitfields - what ever you can imagine! That’s where SatNOGS Decoders hook in. To get the chain complete we need another step, which is described in another article, as we do have the original structure of the data back, but it might even be human unreadable. Let me show you an example:
A frame might contain a byte sequence of 0x12 0x23 0x34 and from a telemetry description we know, the first byte is originally an unsigned eight bit integer, so we have to somehow convert this value into this. For 8 bit this is pretty easy as we do not have to meet any endianess. Now that we know it is 8 bit unsigned, there is no need to check for a sign bit and we already know that values are in the range [0…255] == 256 digits. Easy step. The next two bytes are described as 16 bit signed integer. We can now write it as 0x2334 as we know it is two bytes (== 16 bit) and we know the range is -32768 up to 32767. In kaitai this is described as a sequence, e.g.:
seq:
- id: adc_8bit_unsigned
type: u1
- id: adc_16bit_signed
type: s2
We can now access these two values as adc_8bit_unsigned and adc_16bit_signed and get the correct representation. In a real world of bytestreams one has to care about endianess as well, but please see the telemetry documentation for details and watch out for examples in the existing decoders. Now you might be getting the idea: we still do not have human readable data, because a value of e.g. 123 represented inside the adc_8bit_unsigned measurement doesn’t mean anything without a conversion rule. Yes, again, please see the satellites’ documentation to convert this values. We decided to do those calculations in the next layer - the dashboards. So we end up with raw values on the output of the satnogs-decoders.
Specialities for satnogs-decoders
As we need to somehow convert the raw output from satnogs-decoders into something we can store in the SatNOGS Data Warehouse, a timeseries database called InfluxDB we need to convert the output into JSON objects. We implemented a way to also ignore parts of the frames that might be unneccessary for a display of the interesting parts of the telemetry. We called it :field attribute and will be placed in a documenting comment in the kaitai struct file.
meta:
id: example
endian: be
doc: |
:field example_obc_temperature: adc_8bit_unsigned
:field example_battery_current: adc_16bit_signed
The part :field tells the conversion function that the adc_8bit_unsigned ID will be used as example_obc_temperature in the Data Warehouse as a measurement name. You can use this name as reference in the query you later apply in the Grafana frontend. The complete example as follows can be used to play around in the kaitai WebIDE:
meta:
id: example
endian: be
doc: |
:field example_obc_temperature: adc_8bit_unsigned
:field example_battery_current: adc_16bit_signed
seq:
- id: adc_8bit_unsigned
type: u1
- id: adc_16bit_signed
type: s2
Some examples:
As we have a lot of donwlinks using the popular AX.25 framing, a very basic decoder for these would look like this:
meta:
id: ax25frames
endian: be
doc: |
:field ax25frames_dest_callsign: ax25_frame.ax25_header.dest_callsign_raw.callsign_ror.callsign
:field ax25frames_src_callsign: ax25_frame.ax25_header.src_callsign_raw.callsign_ror.callsign
:field ax25frames_src_ssid: ax25_frame.ax25_header.src_ssid_raw.ssid
:field ax25frames_dest_ssid: ax25_frame.ax25_header.dest_ssid_raw.ssid
:field ax25frames_ctl: ax25_frame.ax25_header.ctl
:field ax25frames_pid: ax25_frame.payload.pid
:field ax25frames_info: ax25_frame.payload.ax25_info
seq:
- id: ax25_frame
type: ax25_frame
doc-ref: 'https://www.tapr.org/pub_ax25.html'
types:
ax25_frame:
seq:
- id: ax25_header
type: ax25_header
- id: payload
type:
switch-on: ax25_header.ctl & 0x13
cases:
0x03: ui_frame
0x13: ui_frame
0x00: i_frame
0x02: i_frame
0x10: i_frame
0x12: i_frame
#0x11: s_frame
ax25_header:
seq:
- id: dest_callsign_raw
type: callsign_raw
- id: dest_ssid_raw
type: ssid_mask
- id: src_callsign_raw
type: callsign_raw
- id: src_ssid_raw
type: ssid_mask
- id: ctl
type: u1
callsign_raw:
seq:
- id: callsign_ror
process: ror(1)
size: 6
type: callsign
callsign:
seq:
- id: callsign
type: str
encoding: ASCII
size: 6
ssid_mask:
seq:
- id: ssid_mask
type: u1
instances:
ssid:
value: (ssid_mask & 0x0f) >> 1
i_frame:
seq:
- id: pid
type: u1
- id: ax25_info
size-eos: true
ui_frame:
seq:
- id: pid
type: u1
- id: ax25_info
size-eos: true
Another good start to deal with would be the CubeSatSim decoder. It also uses AX.25 framing and has some specials in the payload, as it transmits data like the old OSCARS did:
meta:
id: cubesatsim
endian: be
doc: |
:field cubesatsim_dest_callsign: ax25_frame.ax25_header.dest_callsign_raw.callsign_ror.callsign
:field cubesatsim_src_callsign: ax25_frame.ax25_header.src_callsign_raw.callsign_ror.callsign
:field cubesatsim_src_ssid: ax25_frame.ax25_header.src_ssid_raw.ssid
:field cubesatsim_dest_ssid: ax25_frame.ax25_header.dest_ssid_raw.ssid
:field cubesatsim_ctl: ax25_frame.ax25_header.ctl
:field cubesatsim_pid: ax25_frame.payload.pid
:field cubesatsim_data_type: ax25_frame.payload.ax25_info.data_type
:field cubesatsim_channel_1a_val: ax25_frame.payload.ax25_info.payload.channel_1a_val
:field cubesatsim_channel_1b_val: ax25_frame.payload.ax25_info.payload.channel_1b_val
:field cubesatsim_channel_1c_val: ax25_frame.payload.ax25_info.payload.channel_1c_val
:field cubesatsim_channel_1d_val: ax25_frame.payload.ax25_info.payload.channel_1d_val
:field cubesatsim_channel_2a_val: ax25_frame.payload.ax25_info.payload.channel_2a_val
:field cubesatsim_channel_2b_val: ax25_frame.payload.ax25_info.payload.channel_2b_val
:field cubesatsim_channel_2c_val: ax25_frame.payload.ax25_info.payload.channel_2c_val
:field cubesatsim_channel_2d_val: ax25_frame.payload.ax25_info.payload.channel_2d_val
:field cubesatsim_channel_3a_val: ax25_frame.payload.ax25_info.payload.channel_3a_val
:field cubesatsim_channel_3b_val: ax25_frame.payload.ax25_info.payload.channel_3b_val
:field cubesatsim_channel_3c_val: ax25_frame.payload.ax25_info.payload.channel_3c_val
:field cubesatsim_channel_3d_val: ax25_frame.payload.ax25_info.payload.channel_3d_val
:field cubesatsim_channel_4a_val: ax25_frame.payload.ax25_info.payload.channel_4a_val
:field cubesatsim_channel_4b_val: ax25_frame.payload.ax25_info.payload.channel_4b_val
:field cubesatsim_channel_4c_val: ax25_frame.payload.ax25_info.payload.channel_4c_val
:field cubesatsim_channel_4d_val: ax25_frame.payload.ax25_info.payload.channel_4d_val
:field cubesatsim_channel_5a_val: ax25_frame.payload.ax25_info.payload.channel_5a_val
:field cubesatsim_channel_5b_val: ax25_frame.payload.ax25_info.payload.channel_5b_val
:field cubesatsim_channel_5c_val: ax25_frame.payload.ax25_info.payload.channel_5c_val
:field cubesatsim_channel_5d_val: ax25_frame.payload.ax25_info.payload.channel_5d_val
:field cubesatsim_channel_6a_val: ax25_frame.payload.ax25_info.payload.channel_6a_val
:field cubesatsim_channel_6b_val: ax25_frame.payload.ax25_info.payload.channel_6b_val
:field cubesatsim_channel_6c_val: ax25_frame.payload.ax25_info.payload.channel_6c_val
:field cubesatsim_channel_6d_val: ax25_frame.payload.ax25_info.payload.channel_6d_val
seq:
- id: ax25_frame
type: ax25_frame
doc-ref: 'https://www.tapr.org/pub_ax25.html'
types:
ax25_frame:
seq:
- id: ax25_header
type: ax25_header
- id: payload
type:
switch-on: ax25_header.ctl & 0x13
cases:
0x03: ui_frame
0x13: ui_frame
0x00: i_frame
0x02: i_frame
0x10: i_frame
0x12: i_frame
#0x11: s_frame
ax25_header:
seq:
- id: dest_callsign_raw
type: callsign_raw
- id: dest_ssid_raw
type: ssid_mask
- id: src_callsign_raw
type: callsign_raw
- id: src_ssid_raw
type: ssid_mask
- id: ctl
type: u1
callsign_raw:
seq:
- id: callsign_ror
process: ror(1)
size: 6
type: callsign
callsign:
seq:
- id: callsign
type: str
encoding: ASCII
size: 6
ssid_mask:
seq:
- id: ssid_mask
type: u1
instances:
ssid:
value: (ssid_mask & 0x0f) >> 1
i_frame:
seq:
- id: pid
type: u1
- id: ax25_info
size-eos: true
ui_frame:
seq:
- id: pid
type: u1
- id: ax25_info
size-eos: true
type: cubesatsim_data
cubesatsim_data:
seq:
- id: data_type
type: u2
- id: payload
type:
switch-on: data_type
cases:
_: cubesatsim_ao_7
0x6869: cubesatsim_ao_7
cubesatsim_ao_7:
seq:
- id: ao_7_magic
contents: ' hi '
- id: channel_1a_id
contents: '1'
- id: channel_1a_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_1a
size: 1
- id: channel_1b_id
contents: '1'
- id: channel_1b_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_1b
size: 1
- id: channel_1c_id
contents: '1'
- id: channel_1c_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_1c
size: 1
- id: channel_1d_id
contents: '1'
- id: channel_1d_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_1d
size: 1
- id: channel_2a_id
contents: '2'
- id: channel_2a_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_2a
size: 1
- id: channel_2b_id
contents: '2'
- id: channel_2b_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_2b
size: 1
- id: channel_2c_id
contents: '2'
- id: channel_2c_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_2c
size: 1
- id: channel_2d_id
contents: '2'
- id: channel_2d_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_2d
size: 1
- id: channel_3a_id
contents: '3'
- id: channel_3a_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_3a
size: 1
- id: channel_3b_id
contents: '3'
- id: channel_3b_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_3b
size: 1
- id: channel_3c_id
contents: '3'
- id: channel_3c_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_3c
size: 1
- id: channel_3d_id
contents: '3'
- id: channel_3d_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_3d
size: 1
- id: channel_4a_id
contents: '4'
- id: channel_4a_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_4a
size: 1
- id: channel_4b_id
contents: '4'
- id: channel_4b_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_4b
size: 1
- id: channel_4c_id
contents: '4'
- id: channel_4c_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_4c
size: 1
- id: channel_4d_id
contents: '4'
- id: channel_4d_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_4d
size: 1
- id: channel_5a_id
contents: '5'
- id: channel_5a_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_5a
size: 1
- id: channel_5b_id
contents: '5'
- id: channel_5b_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_5b
size: 1
- id: channel_5c_id
contents: '5'
- id: channel_5c_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_5c
size: 1
- id: channel_5d_id
contents: '5'
- id: channel_5d_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_5d
size: 1
- id: channel_6a_id
contents: '6'
- id: channel_6a_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_6a
size: 1
- id: channel_6b_id
contents: '6'
- id: channel_6b_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_6b
size: 1
- id: channel_6c_id
contents: '6'
- id: channel_6c_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_6c
size: 1
- id: channel_6d_id
contents: '6'
- id: channel_6d_val_raw
type: u1
repeat: expr
repeat-expr: 2
- id: delim_6d
size: 1
instances:
channel_1a_val:
value: ((channel_1a_val_raw[0] - 0x30) * 10 + (channel_1a_val_raw[1] - 0x30)) * 29.5
doc: 'value * 29.5 [mA]'
channel_1b_val:
value: 1970 - (20 * ((channel_1b_val_raw[0] - 0x30) * 10 + (channel_1b_val_raw[1] - 0x30)))
doc: '1970 - (20 * value) [mA]'
channel_1c_val:
value: 1970 - (20 * ((channel_1c_val_raw[0] - 0x30) * 10 + (channel_1c_val_raw[1] - 0x30)))
doc: '1970 - (20 * value) [mA]'
channel_1d_val:
value: (channel_1d_val_raw[0] - 0x30) * 10 + (channel_1d_val_raw[1] - 0x30)
doc: '1970 - (20 * value) [mA]'
channel_2a_val:
value: 1970 - (20 * ((channel_2a_val_raw[0] - 0x30) * 10 + (channel_2a_val_raw[1] - 0x30)))
doc: '1970 - (20 * value) [mA]'
channel_2b_val:
value: 8 * ((1 - 0.01 * ((channel_2b_val_raw[0] - 0x30) * 10 + (channel_2b_val_raw[1] - 0x30))) * (1 - 0.01 * ((channel_2b_val_raw[0] - 0x30) * 10 + (channel_2b_val_raw[1] - 0x30))))
doc: '8 * (1 - 0.01 * value)^2 [W]'
channel_2c_val:
value: 15.16 * ((channel_2c_val_raw[0] - 0x30) * 10 + (channel_2c_val_raw[1] - 0x30))
doc: '15.16 * value [h]'
channel_2d_val:
value: 40 * (((channel_2d_val_raw[0] - 0x30) * 10 + (channel_2d_val_raw[1] - 0x30)) - 50)
doc: '40 * (value - 50) [mA]'
channel_3a_val:
value: 0.1 * ((channel_3a_val_raw[0] - 0x30) * 10 + (channel_3a_val_raw[1] - 0x30)) + 6.4
doc: '0.1 * value + 6.4 [V]'
channel_3b_val:
value: 0.1 * ((channel_3b_val_raw[0] - 0x30) * 10 + (channel_3b_val_raw[1] - 0x30))
doc: '0.1 * value [V]'
channel_3c_val:
value: 0.15 * ((channel_3c_val_raw[0] - 0x30) * 10 + (channel_3c_val_raw[1] - 0x30))
doc: '0.15 * value [V]'
channel_3d_val:
value: 95.8 - 1.48 * ((channel_3d_val_raw[0] - 0x30) * 10 + (channel_3d_val_raw[1] - 0x30))
doc: '95.8 - 1.48 * value [C]'
channel_4a_val:
value: 95.8 - 1.48 * ((channel_4a_val_raw[0] - 0x30) * 10 + (channel_4a_val_raw[1] - 0x30))
doc: '95.8 - 1.48 * value [C]'
channel_4b_val:
value: 95.8 - 1.48 * ((channel_4b_val_raw[0] - 0x30) * 10 + (channel_4b_val_raw[1] - 0x30))
doc: '95.8 - 1.48 * value [C]'
channel_4c_val:
value: 95.8 - 1.48 * ((channel_4c_val_raw[0] - 0x30) * 10 + (channel_4c_val_raw[1] - 0x30))
doc: '95.8 - 1.48 * value [C]'
channel_4d_val:
value: 95.8 - 1.48 * ((channel_4d_val_raw[0] - 0x30) * 10 + (channel_4d_val_raw[1] - 0x30))
doc: '95.8 - 1.48 * value [C]'
channel_5a_val:
value: 95.8 - 1.48 * ((channel_5a_val_raw[0] - 0x30) * 10 + (channel_5a_val_raw[1] - 0x30))
doc: '95.8 - 1.48 * value [C]'
channel_5b_val:
value: 11.67 * ((channel_5b_val_raw[0] - 0x30) * 10 + (channel_5b_val_raw[1] - 0x30))
doc: '11.67 * value [mA]'
channel_5c_val:
value: 95.8 - 1.48 * ((channel_5c_val_raw[0] - 0x30) * 10 + (channel_5c_val_raw[1] - 0x30))
doc: '95.8 - 1.48 * value [C]'
channel_5d_val:
value: 11 + 0.82 * ((channel_5d_val_raw[0] - 0x30) * 10 + (channel_5d_val_raw[1] - 0x30))
doc: '11 + 0.82 * value [mA]'
channel_6a_val:
value: (((channel_6a_val_raw[0] - 0x30) * 10 + (channel_6a_val_raw[1] - 0x30)) * ((channel_6a_val_raw[0] - 0x30) * 10 + (channel_6a_val_raw[1] - 0x30))) / 1.56
doc: 'value^2 / 1.56 [mA]'
channel_6b_val:
value: 0.1 * (((channel_6b_val_raw[0] - 0x30) * 10 + (channel_6b_val_raw[1] - 0x30)) * ((channel_6b_val_raw[0] - 0x30) * 10 + (channel_6b_val_raw[1] - 0x30))) + 35
doc: '0.1 * value^2 + 35 [mA]'
channel_6c_val:
value: 0.041 * (((channel_6c_val_raw[0] - 0x30) * 10 + (channel_6c_val_raw[1] - 0x30)) * ((channel_6c_val_raw[0] - 0x30) * 10 + (channel_6c_val_raw[1] - 0x30)))
doc: '0.041 * value^2 [mA]'
channel_6d_val:
value: 0.01 * ((channel_6d_val_raw[0] - 0x30) * 10 + (channel_6d_val_raw[1] - 0x30))
doc: '0.01 * value'
As this looks like a rather complex example it is one of the most easiest to implement and already contains some conversion functions. Please simply start with your favourite satellite you have a documentation for and try to figure out how things work. If there are any questions feel free to join the chat, ask on the forums, or dm me!