[UPDATE] Fully Automated SatNOGS to STRF Pipeline Released (still 99% progress)

Hello everyone,

Following up on my previous work-in-progress automated workflow for SatNOGS waterfall tabulation, I am pleased to announce a major update. The project has evolved into a fully automated, headless pipeline integrated directly with the STRF toolkit.

The primary objective of this update is to streamline satellite tracking and orbital element (TLE) optimization, with a specific focus on the LAPAN-A2 satellite, though the methodology is applicable to other missions as well.

The source code, installation scripts, and comprehensive documentation are now available on my GitHub repository:

Key features and improvements in this release include:

  • Headless Extraction: Fully automated extraction of Doppler curves from downloaded SatNOGS waterfall images utilizing OpenCV.
  • Noise Rejection: Implementation of shape-based curve validation to automatically reject terrestrial noise and RF interference before fitting.
  • Batch Optimization: End-to-end integration with STRF’s rffit global fitting engine. A new batch mode (-B) has been introduced to rffit, allowing for automated orbital element optimization without interactive menus.
  • Automation and Scheduling: A complete background service implementation that can be scheduled via cron for continuous data management, extraction, and TLE generation.
  • Flexible Execution: The Python pipeline can be executed in a fully automatic mode (download, extract, fit, and report in a single command) or in a step-by-step mode for localized testing and validation.

The pipeline handles the entire workflow: from downloading the latest observations from the SatNOGS network, performing the image processing, executing the TLE fitting routine, to generating machine-readable metadata, debug overlays, and reports.

I would appreciate any feedback, testing, or contributions from the community. Please feel free to test the pipeline, open issues, or submit pull requests on the GitHub repository.

Thank you for your time and continued support.
help me out to improve this tool. :sweat_smile:

3 Likes

thanks to om @bali for making other version of this tools. aslo sorry ya om terlalu lama updatenya. hehe :sweat_smile:

checkout the other version of om @bali 's version automated STRF tool.

2 Likes

this cool feature not found on other project, fitting without pgplot

based on the screenshot:

the waterfall and observation id is not from lapan-a2

seen like this caused by “little glitch” when get data from satno api, that cause all good latest obs from other sat also included, not filtered by specific norad id.

i make some pr to fix this.

also some small update on install.sh that make this auto install flawless on debian based

overall i can run this app

source observation 14384250

i check the .dat file on doppler_curves/ directory, seen the dat file is produced from waterfall without unadjusted doppler first. i notice this when try open the dat file using rffit:

rffit -d 14384250.dat -c initial_40931.tle -i 40931

comparing with manual creation of dat file using satnogs waterfall tabulation helper
rffit -d 14384250_undoppler.dat -c initial_40931.tle -i 40931

if this homework is done, i’m sure the tle will be accurate…

14384250_undoppler.dat.txt (665 Bytes)
sites.txt (122 Bytes)
14384250.dat.txt (63.4 KB)
initial_40931.tle.txt (149 Bytes)

btw i’m wondering, why not produce single tle combining from all observations rather that create single tle for every obs…

Hi om @bali Thank you so much for the feedback and for diving deep into the pipeline. I really appreciate the PRs and the keen observations. Here are a few updates regarding the points you mentioned:

1. “executing the TLE fitting routine”
Thank you! Yes, we built a headless Python-native SGP4 fitter wrapper so the entire pipeline can run fully automatically on a server without needing X11 or PGPLOT. This makes deploying it as a cronjob or systemd service much easier.

2. “the waterfall and observation id is not from lapan-a2…”
Thanks for the PR on the install.sh and for looking into the API glitch! We actually noticed this API behavior as well, which is why we added a local validation step in our satnogs_api.py. Before downloading any waterfall, the script strictly checks if obs_norad != str(norad_id): continue to ensure no data from other satellites contaminates the pipeline. But fixing it at the API level is definitely the best long-term solution!

3. “seen the dat file is produced from waterfall without unadjusted doppler first”
Great catch! You are absolutely right. The pipeline is actually designed to perform an “Undoppler” correction (re-adding the theoretical Doppler shift based on the station’s active tracking) before generating the .dat file. However, there was a minor bug where it failed to parse the initial TLE from the SatNOGS metadata JSON, causing the Undoppler step to be skipped. We have just patched this bug in our repository, and the .dat files will now contain the correct absolute observed frequencies!

4. “why not produce single tle combining from all observations rather that create single tle for every obs…”
That’s an excellent suggestion and exactly how batch least-squares orbit determination should be done! We have just updated the pipeline (main.py) so that instead of looping and creating a TLE per observation, it now feeds all the new .dat files simultaneously into the SGP4 fitter to produce a single, highly accurate combined TLE.

Thank you again for testing the app and providing such valuable insights!

3 Likes

the auto track seen not too accurate to follow the center line of signal..

btw welcome to your new gs !

73!

1 Like