Building a kaitai struct for processing FOX-DUV frames

SatNOGS Software
1

Hi all together,

as the AMSAT FOX satellites are very popular these days and they produce a lot of telemetry, I am working on a kaitai struct to decode the bytestream into human readable data (e.g. display them in a grafana dashboard). I am really loosing my mind with this as the DUV stream is a bit weird to me :wink:

A DUV stream for slow speed telemetry consists of 6 bytes of header and 58 bytes of data (= 64 bytes frame, 512 bits). The bitordering is a little curious, as the on board computer seems to transfer little vs big endian values, previously generated in a shift-reg.

For now: let’s have a look at the header only. This keeps the issue small.

From the documentation, the bitstream should contain (bits MSB first, so fox_id is located at the most left 3 bits):

fox_id[0..2], reset_cnt[3..18], uptime[19..43], frame_type[44..47]

A typical frame from e.g. FOX-1A produces a header like this one:

B1 18 C0 AF 9F 10 [data ..]

This is clearly readable as “FOX-1A (first byte, 3 LSBits = 0x1), frame type 1 (6th byte, 4 MSBits = 0x1 = realtime telemetry)”. As you can see, the endianess is somehow “scrambled”: the FOX-ID is located in the bits 0, 1 and 2, but right-aligned. Bits 3, 4, 5, 6, 7 belong to the upper bits of reset_cnt. Simply exchanging the nibbles doesnt work, as this would move e.g. the 4th LSBit of the first byte into the 4 MSBits, what results in fox_id being 0x0! Maybe I am just to blind to see the issue, but how can I reproduce the correct bit-order in here?

If someone knows where to find it in the code of FoxTelem please show :slight_smile:
Any other solutions are also very welcome (maybe there’s a mathematical trick to solve this - I just can’t see).

Hope this is meaningful - if not: please feel free to ask.
On interest I can upload some files and the struct I am dealing with at the moment.

Rgds,
Patrick

P.S.: Pinging @Abraxas3d, as she is referenced in some of the docs, and @surligas as the author of the gr-satnogs DUV decoder :wink:

2

It boils down to shifting each byte part to the right position in the final value. I have basically no experience with kaitai structs so this could probably be done better:

meta:
  id: fox
  endian: le
seq:
  - id: res_1
    type: b5
  - id: id
    type: b3
  - id: res_2
    type: b8
  - id: up1
    type: b5
  - id: res_3
    type: b3
  - id: up2
    type: b16
  - id: frame_type
    type: b4
  - id: up3
    type: b4

instances:
  reset_ctn:
    value: res_1 | (res_2 << 5) | (res_3 << 13)
  uptime:
    value: up1 | (up2 << 5) | (up3 << (5+16))

I hope this helps.

1 Like
3

Maybe you’re on the right way, I can’t proof this at the moment. One hint I can give is: you can exchange the nibbles in a byte by doing a rotate right by four.

  seq:
    - id: raw_frame
      process: ror(4)
      size-eos: true
      type: fox_bitstream
[..]

Thanks for this approach! I’ll test as I have access to a PC :wink:

4

It seems that there is indeed an issue. I have talked with @concretedog about this and it seems that there is a filtering issue on https://gitlab.com/librespacefoundation/satnogs/gr-satnogs/blob/master/lib/fox_telem_mm_impl.cc#L86

So if this is the case should I proceed flipping these bits, from the gr-satnogs?

1 Like
5

@DL4PD: I didn’t know about the process/ror thing. However, circle shifting doesn’t help here. The bits within their byte parts are already in the correct order. You’ll just have to arrange those parts by ordering them from right to left.

@surligas: The bit mask which is used to get the sat id is indeed not correct. The id consists of 3 bits, so the correct mask would be 0x7. The data frames (at least the header, I haven’t looked at the payload yet) have the same format (byte,bit order) as FoxTelem uses it. So I think the demod is correct.

It took me some reads to finally get this:

1 0 | 1  0  1  0  1  0  1  0 sync
    | 1  0  1  1  0} [0  0  1]<- [FoxID] (3 bits) = 001
    | 0  0  0  0  0   0  0  1 <- {Reset count} (16 bits = 000 00000001 10110 = 54)
    | 0  1  0  1  1] {0  0  0
    | 0  0  0  0  0   0  0  1 <- [Uptime] (25 bits = 0000 00000000 00000001 01011 = 43
    | 0  0  0  0  0   0  0  0
    |{0  0  0  1}[0   0  0  0 <- {Frame type} (4 bits = 0001)

Source: https://github.com/ac2cz/FoxTelem/blob/1efad38ac25d1c2f32b05b06c1bf8610cc0122a6/src/telemetry/SlowSpeedHeader.java#L43

1 Like
6

So you think there is a sync word partially in the data frame? And these red zeros are sourced where? This format seems to be even more weird than I initially thought! :smiley:

7

Let’s discuss this a bit further, Manolis. :slight_smile:

8

Ah well, no the sync word is already stripped during the demodulation. The satnogs data frames don’t have it. The link to the source in SlowSpeedHeader.java is below the quote :wink:

1 Like
9

Hm, the example frame I mentioned above starts with 0xB1, which is 0b10110001, as you also added in that table… but it should start with the fox_id, what means: 0b001… How to concatenate further?

10

I don’t think that endianess setting applies to bit-sized integers in Kaitai. What we need to do I think is read the whole packet as a single u6le and then apply boolean and shift operations to unpack integers from bits with value instances.

11

You’re right: endianess settings count only on byte based types! That’s where the ror idea came up! As I do understand the foxtelem source, they somehow re-order the bits inside the addNext8bits() function. There must be a solution…

12

That’s the interesting/odd part. The bits for each value (sometimes spanning multiple bytes) are in the correct order. I’ll use your frame as example:

B1    10110001
18    00011000
C0    11000000
AF    10101111
9F    10011111
10    00010000
[data...]

You’re right, the first 3 bits contain the sat id, but those are the 3 LSBits, so you’ll have to take the bits from right to left. Their order however is LE and therefore correct. To workaround this with my limited kaitai fu, I parse the first (least significant) part of the reset counter first (5 bits) and then take the 3 remaining in this byte to build the sat id. So we have:

id        = 0b001
reset_ctn = 0bXXXX_XXXX_XXX1_0110

We can take the next whole byte because it’s all part of the reset counter, but we need to shift it so the reset_ctn is filled from right to left (5 bits to the left) and add it up, yielding:

reset_ctn = 0bXXX0_0011_0001_0110

The last 3 bits of the reset_ctn is at the right most position of the next byte. So we got the same issue as before. Taking the first 5 bits as the least significant part of the uptime leaves the missing 3 bits for the reset_ctn and shift those 8+3 bits to the left:

reset_ctn = 0b0000_0011_0001_0110 = 0x0316 = 790

The current reset_cnt is 790 which corresponds with the current value published here: http://www.amsat.org/tlm/health.php?id=1&port=
Same for the uptime and so on.

uptime  = 0x15F3F8 = 1438712
1 Like
13

Yeah, got it! Let me see what I can do in kaitai…

14

I think there is no point in reading bit-sized integers in the first place and then try to reassemble into full sized integers.

Since the data is fixed-size, I would just read all 6 bytes into a single unsigned integer. Then, I would isolate the values with AND and right shift.

15

Something like this (not tested):

meta:
  id: foxheader
seq:
  - id: byte
    type: u1
    repeat: expr
    repeat-expr: 6
instances:
  fox_id:
    value: byte[0] & 0x07
  reset_counter:
    value: >
      (
        (
          byte[2] << 16 |
          byte[1] << 8 |
          byte[0]
        ) >> 3
      ) & 0xFFFF
  uptime:
    value: >
      (
        (
          byte[5] << 24 |
          byte[4] << 16 |
          byte[3] << 8 |
          byte[2]
        ) >> 3
      ) & 0x01FFFFFF
  frame_type:
    value: >
      (
        byte[5] >> 4
      ) & 0x0F
16

Ok, I read a bit in the source of FoxTelem and I think it would be better and more self-descriptive, if we do it like FoxTelem: re-order the bits to match the description of the telemetry spec and convert it back to a bytestream. This is done in:

and:

Doing this in a custom process inside kaitai would hide all those tricky conversions and enables us to use the simple description in the struct.

Thanks to @wose for the link to the FoxHeader description, just could not read that on my smartphone yesterday! :wink:

17

I think the approach @Acinonyx described is much more descriptive and more natural if you’re used to C/C++. Besides using an external processor would kind of defeat the idea of a kaitai struct, wouldn’t it?

18

Hm, reading the docs it is less descriptive, as they say: e.g. first three bits are fox_id. FoxTelem does this conversion, too, and it is described in the SlowSpeedHeader.java. Sure we could do bit-shifts (and I am very well used to this in C as I wrote a lot of software for the Atmel Mikrocontrollers some years ago :wink: ), but we would not be following the docs - in my oppinion. Next thing is: we will need to do this for every value in the stream - not just the header values. And this will end up in bit shifts in instances in user types. Doesn’t sound pretty. I will try to do an example for both, maybe you get me then…

To point to the significant passage in the source, I just want to add the Link to it:

19

First example is done:

fox_masked.ksy:

meta:
  id: fox
seq:
  - id: fox_raw
    size-eos: true
    type: fox_frame
  
types:
  fox_frame:
    seq:
      - id: fox_hdr
        type: fox_hdr
        size: 6
      - id: fox_frame
        type:
          switch-on: fox_hdr.frm_type
          cases:
            0x0: fox_debug_data_t
            0x1: fox_rt_tlm_t
            0x2: fox_max_vals_tlm_t
            0x3: fox_min_vals_tlm_t
            0x4: fox_exp_tlm_t
            0x5: fox_cam_jpeg_data_t
  
  fox_hdr:
    seq:
      - id: b0
        type: u1
      - id: b1
        type: u1
      - id: b2
        type: u1
      - id: b3
        type: u1
      - id: b4
        type: u1
      - id: b5
        type: u1
    instances:
      fox_id:
        value: 'b0 & 0x7'
      reset_count:
        value: '((b2 << 16 | b1 << 8 | b0) >> 3) & 0xffff'
      uptime:
        value: '((b5 << 24 | b4 << 16 | b3 << 8 | b2) >> 3) & 0x1ffffff'
      frm_type:
        value: '(b5 >> 4) & 0xf'

  fox_debug_data_t:
    seq:
      - id: fox_debug_data
        size-eos: true

  fox_rt_tlm_t:
    seq:
      - id: fox_rt_tlm
        type: fox_rt_tlm
        size: 58

  fox_max_vals_tlm_t:
    seq:
      - id: fox_max_vals_tlm
        size: 58

  fox_min_vals_tlm_t:
    seq:
      - id: fox_min_vals_tlm
        size: 58

  fox_cam_jpeg_data_t:
    seq:
      - id: fox_cam_jpeg_data
        size-eos: true

  fox_exp_tlm_t:
    seq:
      - id: fox_exp_tlm
        size: 58

  fox_rt_tlm:
    seq:
      - id: b0
        type: u1
      - id: b1
        type: u1
      - id: b2
        type: u1
      - id: b3
        type: u1
      - id: b4
        type: u1
      - id: b5
        type: u1
      - id: b6
        type: u1
      - id: b7
        type: u1
      - id: b8
        type: u1
      - id: b9
        type: u1
      - id: b10
        type: u1
      - id: b11
        type: u1
      - id: b12
        type: u1
      - id: b13
        type: u1
      - id: b14
        type: u1
      - id: b15
        type: u1
      - id: b16
        type: u1
      - id: b17
        type: u1
      - id: b18
        type: u1
      - id: b19
        type: u1
      - id: b20
        type: u1
      - id: b21
        type: u1
      - id: b22
        type: u1
      - id: b23
        type: u1
      - id: b24
        type: u1
      - id: b25
        type: u1
      - id: b26
        type: u1
      - id: b27
        type: u1
      - id: b28
        type: u1
      - id: b29
        type: u1
      - id: b30
        type: u1
      - id: b31
        type: u1
      - id: b32
        type: u1
      - id: b33
        type: u1
      - id: b34
        type: u1
      - id: b35
        type: u1
      - id: b36
        type: u1
      - id: b37
        type: u1
      - id: b38
        type: u1
      - id: b39
        type: u1
      - id: b40
        type: u1
      - id: b41
        type: u1
      - id: b42
        type: u1
      - id: b43
        type: u1
      - id: b44
        type: u1
      - id: b45
        type: u1
      - id: b46
        type: u1
      - id: b47
        type: u1
      - id: b48
        type: u1
      - id: b49
        type: u1
      - id: b50
        type: u1
      - id: b51
        type: u1
      - id: b52
        type: u1
      - id: b53
        type: u1
      - id: b54
        type: u1
      - id: b55
        type: u1
      - id: b56
        type: u1
      - id: b57
        type: u1
    instances:
      batt_a_v:
        value: '(b1 << 12 | b0 & 0xf) & 0xfff'
      batt_b_v:
        value: '(b2 << 12 | b0 >> 4) & 0xfff'
      batt_c_v:
        value: '(b4 << 12 | b3 & 0xf) & 0xfff'
      batt_a_t:
        value: '(b5 << 12 | b3 >> 4) & 0xfff'
      batt_b_t:
        value: '(b7 << 12 | b6 & 0xf) & 0xfff'
      batt_c_t:
        value: '(b8 << 12 | b6 >> 4) & 0xfff'
      total_batt_i:
        value: '(b10 << 12 | b9 & 0xf) & 0xfff'
      batt_board_temp:
        value: '(b11 << 12 | b9 >> 4) & 0xfff'
      pos_x_panel_v:
        value: '(b13 << 12 | b12 & 0xf) & 0xfff'
      neg_x_panel_v:
        value: '(b14 << 12 | b12 >> 4) & 0xfff'
      pos_y_panel_v:
        value: '(b16 << 12 | b15 & 0xf) & 0xfff'
      neg_y_panel_v:
        value: '(b17 << 12 | b15 >> 4) & 0xfff'
      pos_z_panel_v:
        value: '(b19 << 12 | b18 & 0xf) & 0xfff'
      neg_z_panel_v:
        value: '(b20 << 12 | b18 >> 4) & 0xfff'
      pos_x_panel_t:
        value: '(b22 << 12 | b21 & 0xf) & 0xfff'
      neg_x_panel_t:
        value: '(b23 << 12 | b21 >> 4) & 0xfff'
      pos_y_panel_t:
        value: '(b25 << 12 | b24 & 0xf) & 0xfff'
      neg_y_panel_t:
        value: '(b26 << 12 | b24 >> 4) & 0xfff'
      pos_z_panel_t:
        value: '(b28 << 12 | b27 & 0xf) & 0xfff'
      neg_z_panel_t:
        value: '(b29 << 12 | b27 >> 4) & 0xfff'
      psu_temp:
        value: '(b31 << 12 | b30 & 0xf) & 0xfff'
      spin:
        value: '(b32 << 12 | b30 >> 4) & 0xfff'
      tx_pa_curr:
        value: '(b34 << 12 | b33 & 0xf) & 0xfff'
      tx_temp:
        value: '(b35 << 12 | b33 >> 4) & 0xfff'
      rx_temp:
        value: '(b37 << 12 | b36 & 0xf) & 0xfff'
      rssi:
        value: '(b38 << 12 | b36 >> 4) & 0xfff'
      ihu_cpu_temp:
        value: '(b40 << 12 | b39 & 0xf) & 0xfff'
      sat_x_ang_vcty:
        value: '(b41 << 12 | b39 >> 4) & 0xfff'
      sat_y_ang_vcty:
        value: '(b43 << 12 | b42 & 0xf) & 0xfff'
      sat_z_ang_vcty:
        value: '(b44 << 12 | b42 >> 4) & 0xfff'
      exp_4_temp:
        value: '(b46 << 12 | b45 & 0xf) & 0xfff'
      psu_curr:
        value: '(b47 << 12 | b45 >> 4) & 0xfff'
      ihu_diag_data:
        value: '(b51 << 32 | b50 << 24 | b49 << 16 | b48)'

It would be nice if one of you guys could try to validate this is correct, as this is still blowing my mind :smiley:
Simply copy&paste the above into kaitai WebIDE and upload at least one sample frame. The interesting ones are the “Type 1” frames like these below:

FOX-1A:

B1 18 B8 A1 29 10 00 00 00 68 0B 00 00 00 00 00 00 00 E6 B9 98 29 CA 9A EE A9 A1 8D FC C7 95 2C C6 A9 AC C3 52 08 00 96 69 99 1D 58 5A D0 B2 78 0B 27 64 11 6F FA 07 00 00 00 16 38 AC 00 00 20
B1 18 C8 A0 29 10 00 00 00 50 0B 00 00 00 00 00 00 00 F3 79 A1 32 3A 92 06 5A A3 90 6C C8 98 0C C6 AB 3C C3 4E 08 00 A2 49 9A 18 58 45 CF E2 71 2A F7 65 0F 7F FA 05 00 00 00 16 38 AC 00 00 20
B1 18 58 A5 29 10 00 00 00 5F 0B 00 00 00 00 00 00 00 43 9A 8A 43 EA 9F 0A DA A3 80 1C C7 83 9C C6 A5 DC C4 67 08 00 AF 19 9B 36 18 40 D1 72 82 88 88 5E 0F 6F FA 0C 15 08 00 16 38 AC 00 00 20
B1 18 E8 A4 29 10 00 00 00 76 0B 00 00 00 00 00 00 00 F0 D8 A3 00 BA A2 04 5A A3 80 3C C7 7E CC C6 A5 FC C4 63 08 00 AD B9 9A 33 58 38 D1 82 87 31 98 5C 10 6F FA 0B C1 05 00 16 38 AC 00 00 20

FOX-1B:

D2 00 D8 42 82 10 DC 88 D7 49 3D 7D C7 27 7D 13 87 77 01 60 AA D3 91 B1 F7 5A A6 00 F0 72 01 90 74 35 07 6D 85 08 00 CA EB 78 3D C7 02 A5 C8 7F A0 B8 5D 00 D9 09 11 75 0B 01 06 38 05 01 00 55
D2 00 78 41 82 10 E1 E8 D7 4F FD 7C C4 C7 7C 0E 27 77 01 40 AE ED AA 76 F2 3A B4 00 80 72 2E D7 6E 32 57 6D 7D 08 00 CF FB 77 31 B7 5E A7 62 8B 55 58 5F 01 30 6B 08 00 00 00 06 38 F4 01 00 20
D2 00 F0 46 82 10 DD A8 D7 4A CD 7D D0 C7 7D 16 17 78 01 10 5C FF 5A B0 FD 0A A7 00 B0 4C 3C C7 73 39 E7 6D 9C 08 00 C5 2B 7B 5F 67 5F AA 72 71 82 97 61 01 E0 68 04 01 01 01 06 38 F4 01 00 20
D2 00 20 45 82 10 DC 78 D7 46 8D 7D CC 77 7D 32 C7 77 01 50 67 03 1B B1 FC 7A A5 00 90 5D 3C 47 74 38 C7 6C 90 08 00 C1 4B 7A 50 F7 72 A9 F2 74 3A 18 5E 00 30 65 08 00 00 00 06 38 F4 01 00 20

As there are only a few frames from FOX-1D I didn’t add them in here. Looks like something is wrong with them :thinking:

You can convert and upload these frames copy&pasting them into an empty file, save and use xxd on the commandline to get binary files needed by the kaitai WebIDE.

$ xxd -r -p fox-1a-01.txt fox-1a-01.bin

Thanks for reading, 73,
Patrick

20

Addendum: The last 13 bits of “Realtime Telemetry” are missing for now. :crazy_face: