Important bug-fix for SatNOGS Network stations with version >2.0.0

SatNOGS
1

Hello all,

We’ve discovered a bug that may affect some of stations and it is a result of station are in version 2.0.0 or above and they are configured with a non UTC timezone.

This bug affects the timestamp that marks the demodulated data send to Network observations by the station, as it uses the local time instead of UTC time.

Stations (owners) that are affected will get an email from us as soon as we have finished the related analysis, however you can update your stations and check that you use UTC timezone to make sure that you are not affected.

How to fix the issues:

  1. If you use the default setup, you can update to the fixed version (of satnogs-flowgraphs package) by running: satnogs-setup --install --no-config
    In case you run a custom version, make sure you run the latest released version of satnogs-flowgraphs which is 2.4
  2. Additionally or in case you can not update your station, please make sure that the timezone set in your station is UTC. You can check any of the three guides below (the first two for RPi and the 3rd for generic linux system) that will guide you on how to check and change your timezone, you will just need to find UTC timezone which usually is under “other options” or “ETC/UTC” etc:

There is also a recent hotfix release in Network which from now on will not accept frames with wrong timezone timestamps and will return an error with explaining the issue.

For the already submitted frames with wrong timezones, there will be some actions for fixing them in Network, DB and Dashboards, however due to the amount of them and some other prioritized tasks it may take from some days to some weeks to be completed.

If you have any additional questions or any issues, let us know either by replying on this post or by get in touch with us in our chat room in matrix.org.

6 Likes
SatNOGS received timestamps have strange precise behaviour
Wrong Packet Timestamps in Database/Dashboard
3

Using a manual satnogs client install with my own docker-compose.yml

I forced an image pull to get the updated satnogs-flowgraphs

Inside the SatNOGS client compose directory docker compose down && docker compose pull && docker compose up

ii  satnogs-flowgraphs                      2.4-1                          all          SatNOGS GNU Radio Flowgraphs

Jan | PE0SAT

1 Like
4

got error similiar with this in satno log:
satnogs_client_1 | ERROR satnogsclient.scheduler.tasks Upload of data_13835395_2026-04-17T15-45-42_g0 for observation 13835395 failed, response status code: 400

maybe caused by satno bug, i try to update my station to latest satnogs_client, but error still same.

image
image570×153 9.45 KB

the submitted frames timezones is not wrong. and my groundstation time in utc.

examples:

$ cat data_13835395_2026-04-17T15-45-42_g0
{"decoder_name": "gr-satellites", "pdu": "pGSCnIxAYKaGZGZiaGMA8EZCAgABABkACwD/fyEa8Q6dVeJpz7YkAGAJAAAoAFAeASHtBAABADAABwAJAAcAAQABAAAAVP8AADMAlgAAACkAfQMmAzIDfQMAwAIAgB50wCQAXm65WtcU"}

$ cat data_13838410_2026-04-18T05-45-16
{
    "decoder_crc_valid": true,
    "decoder_name": "ax25",
    "decoder_version": "1.3",
    "pdu": "iqaOpmBi4IKEhoiKjOED8DoRATNFBgFGAAE7wAAAAAAJCh1AAAAAQBMhHUAofqw+Gy/dPe2evD4iIgAAAAAAAAAAAAAAwDNEAQAAAAAAAAA=",
    "sample_cnt": 708,
    "sample_start": 668777,
    "time": "2026-04-18T05:45:16.306775Z"
}
5

Issue

Issue is now fixed, was due to a typo. No need to do anything further. Thanks for reporting it.

1 Like
6

Affected stations have been notified by email.

However if you didn’t get any, make sure you have the latest update and/or you time is set to UTC.

2 Likes
7

Hi, tried updating my station 3024, but I don’t get the correct version.

I ran:

satnogs-setup --install --no-config

A couple of times. But see 2.1.1 reported in Network.

I get

ii  satnogs-flowgraphs                      2.2-1                          all          SatNOGS GNU Radio Flowgraphs

Which is not the expected 2.4.

8

try check images list with this command:
sudo docker images

1 Like
9

Hello,

Indeed, the software was not updated! We caught another bug in which we missed bumping a version. It should be fixed now. Please retry and sorry for the inconvenience.

Cheers!

3 Likes
10

Thanks for fixing! I have updated the station using the instructions of Fredy. And now it shows 2.1.2 and the correct flowgraphs version.

Will the owners of the affected station be informed by email again?

3 Likes
11

Given the short time between the fix and the sent emails, would not justify a new email (I would like to avoid spamming). However even if they performed the steps they now going to use UTC time so the problem will be fixed either if they have an updated flowgraph or not.

Also the thread here was linked in the email so they probably will see the discussion above.

1 Like
12

Can someone clarify explicitly what range of version numbers this affects? I think it’s fixed in the latest version of the client, right?

As a satellite operator, we’ll need to filter out any data from stations that are impacted by this bug for any operations that require precise timing (e.g., orbit determination, attitude control logging, etc.).

13

Hello,

Maybe @fredy can answer that since he followed the case ?

1 Like
14

Oops I missed that, thanks for pinging me.

It affected all the versions of satnogs-client from 2.0.0 to the latest and are not updated to 2.4 or latest version of satnogs-flowgraphs and are configured with a timezone different than UTC.

It was in the satnogs-flowgraphs, so it is fixed in version 2,4.
Also there were some updates in Network in order to avoid similar issues in the future, this means that even an affected station tries to upload data with wrong timestamp will be rejected by the Network.

I can get you a list of the stations that were affected or if it works better for you I can generate a list of observations of your satellite that should be ignored. This as a temporary solution until the data are fixed, unfortunately no clear estimation when this will be done.

2 Likes
15

So just to confirm, this issue affects satnogs-client 2.0.0 and satnogs-client 2.1.1 only? And then fixed in satnogs-client 2.1.2?

16

tl;dr The bug might also exist in satnogs-client 2.1.2 if the image was pulled before the bug was fixed in satnogs-flowgraphs in which case the satnogs-client image must be updated.

No it does not work exactly like that.

The bug was created from a change in satnogs-flowgraphs when we changed the decoded frame generation to use ZMQ publisher and subscriber here. This affects versions 2.0, from 11 months ago, until 2.4 where the issue was fixed 1 month ago.

What makes things a bit more complicated is that since we use docker images the satnogs-client version can be the same but the satnogs-flowgraphs version to be different, depending on whether someone downloaded the satnogs-client image from dockerhub.
This happens because when the satnogs-flowgraphs are updated we add the new version to our GNU Radio image which triggers the recreation of the GNU Radio image which triggers the recreation of the satnogs-client image without changing the version number of satnogs-client or GNU Radio image.

From what i see from the docker hub history for satnogs-client it seems that all versions before satnogs-client 2.1.2 are affected but the catch is if the satnogs-client 2.1.2 was pulled and built before the release of the satnogs-flowgraphs 2.4 will be affected also!
In that case the satnogs-client image must be pulled again. New installations are not affected.

I might have misreported some processes but i hope i give a correct general idea of what is going on. @Acinonyx could probably correct me if i say anything wrong, and explain it more in depth.

2 Likes
17

Unfortunately I don’t totally understand, but that might be fine.

Hopefully this feedback isn’t unwelcome, but an obvious suggestion is that “the version number of the ground station client should refer to exactly one version of code in use”. Dependency versions (even internally-managed dependencies, like flowgraphs) should be pinned. Tools like uv make pinning dependency versions easy.

Sharding components across repositories, implicitly pulling the latest version of components, and then not reporting it in the API decreases the useful of version numbers, and decreases the usefulness of the entire SatNOGS service. It also opens up risks like supply chain attacks and network outages from bad releases.

2 Likes
18

I agree that the version numbers should be clear, but the version of the client is not affected by the version of other components so there is no need to change it when the other components change. We are already working for the satnogs-flowgraphs codebase to run on a different container and be independent of the client. Then we will be able to update the flowgraphs independently and choose a specific version.

1 Like
19

but the version of the client is not affected by the version of other components so there is no need to change it when the other components change

This is precisely the problem I’m pointing out. There is no “need” to have version numbers on anything. Version numbers, when used, are there to create a contract between the software, its behaviour, and the users.

The contract says “Thing Version A on System B on Date C will behave the same, no matter the system or date, until upgraded to Version X, at which point it will behave differently”.

Back to the “I’m a satellite operator and I need to know how to ignore data with broken timestamps” problem — Either the flowgraph version needs to be sent to the API and stored alongside the observations, or the flowgraph version should be pinned to the client version. I strongly suggest the latter, and suggest more frequent releases of the client to bump the flowgraph version.

Another thread, where broken flowgraphs are being discussed, provides further evidence that versioning and release control of flowgraphs may be a helpful upgrade. (https://community.libre.space/t/observation-14183111-frontiersat-69015-why-is-there-no-data-demodulated)

I really don’t think the solution here is “different container” and “more dependency version sharding”.

20

@platypus thanks for your post! Let me give a quick recap and context on the current status.

Around 2-3 years ago, the development of ground station software, which can be grouped into three main components, Client software (mostly python), Radio software (mostly gnuradio) and release & deployment (CI/CD) was almost frozen, mostly due to limited resources (back then around 1.5 full-time person and some non-paid contributions for all the SatNOGS sub-projects) (maintenance, development, testing etc).

I said almost frozen as a few developments were done for some urgent and specific reasons. These developments plus some user feedback brought into surface three main issues:

  1. Install software on the host (directly on the RPi OS) create dependencies that limit user to use the device only for the SatNOGS project while in some cases people wanted to share the device with other projects too.
  2. Development more focused and based on RPi platform, adding some difficulties and needing some manual work to setup a station on other platforms
  3. Development of Client or Radio had real chance to be blocked from the other one due to conflicting dependencies in various levels (OS libraries, software libraries etc)

Thus we decided that we need to change this and introduce containers for these two components (and some additional). In order to proceed with that and with the dev team growing we decided that will left behind some of the Client features that would be added later.

Unfortunately due to different reasons (mostly due to lack of resources and focusing on other tasks), while the containerization moved forward, we didn’t manage to catch the missing features, which is something that we try these days to solve.

One of this missing features is the reporting of the version of the components both locally in Client support report and in the metadata of the observations.

I agree with you that versioning is kind of a contract and the plan is to bring back an easy way for station owners and users to view it. I can not give you an estimation for that as there are other more prioritized tasks (some of them require a lot of work as they are architectural changes) and while the dev team has grown to around 4-5 full-time people there is a lot of maintenance work that keep the progress slow.

2 Likes