APIs not providing the complete set of data

I am trying to collect data on the satellites and their transmitters to provide back any valuable findings to the community. While I am trying to use the APIs for acquiring data regarding the existing and past satellites and transmitters that have been a part of SatNOGS, the json listing is only giving me 331 satellites, while the stats on the DB are showing a bigger number.

Hope someone can help me out!

1 Like

The SatNOGS DB API endpoint for satellites returns only satellite entries which fulfill the following conditions:

  1. approved (aka a suggestion by the community which was approved by a moderator)
  2. not a duplicate of some other entry (de-duplication).

Source Code: GitLab: satnogs-db/db/api/views.py#L132

The total count of satellites shown on the statistic page thus includes duplicated & non-approved entries.

I assume most of those entries are satellite suggestions waiting for approval, corresponding to a real satellite in orbit. When SatNOGS DB receives data for an unknown satellite via the SiDS data endpoint it creates a new satellite suggestion entry with the name “Unknown Satellite” (example: Satellite 47938).

I hope this answers your question sufficiently. I am happy to read about any findings you get from analyzing this data!

Sincerely,
kerel

Hi Kerel,

How will be able to access the rest of the satellite’s data too!? I will try to filter out the unknown satellites.

The goal of my exploration is to understand the interference that the transmitters are inflicting upon other co-satellites, trying to emulate this situation along with the orbital parameters. (Even a satellite I worked on was slashed with a problem)

To find any spatial or coding-based solution that can be incorporated into the amateur community for handling this ever-growing frequency allocation problem.

Any suggestions that you have for my directions will also help!

1 Like

This is an interesting topic!

There has been a Google Summer of Code project looking into it in 2020, described by this blog post.

There is no API for the “Unknown Satellite” entries. But for the studies you described the “Unknown Satellite” entries are not of interest, as they do not contain transmitter information (they can’t, as the only fact SatNOGS DB knows about them is that some user uploaded frames for the specific satellite via SiDS).

I recommend to focus on satellite entries with transmitter information only.

If you are interested in some directions, I am happy to provide them here or via private message.
Such exploration involves a lot of steps (web APIs, orbit integration, knowledge of transmitter bandwidth characteristics, data analysis & visualization). And while some of them have been pursued in the past I don’t know of a full analysis of this style yet. So feel free to ask any question you have along the way, e.g. if you want more detailed guidance on how to get Doppler data or calculate passes for many satellites we can provide it here. Many users here are happy to help, as long as the analysis is developed using Open Source Principles. So make sure to publish code and let us collaborate! :slight_smile: These principles are what makes SatNOGS and many things around it possible.

Best wishes,
kerel

2 Likes

I was bit by this as well. The API gives me 505 transmitters, while the stats page shows 2421 transmitters.

If the difference (1916 transmitters!) is non-approved transmitters and duplicates, which is the main category?

If the bulk of the “invisible” transmitters are waiting for approval, it seems we really need to step up the approval process. Does anyone know what the bottlenecks are? Is there something we can do in the community?

If the bulk of the “invisible” transmitters are duplicates, shouldn’t the stats page filter those as well? Or are we intentionally inflating the statistics to make Satnogs look better?

2 Likes

The difference, currently 83 transmitters, is that the API doesn’t show the invalid transmitters while they are, falsely, counted in transmitter stats.

Invalid transmitters are the ones that weren’t valid for various reasons and we “deleted” them. However, given that there are observations based on these transmitters in Network by using the transmitter uuid, instead of removing them we set them invalid.

These recently are visible in the satellite page by clicking to display them in the transmitters page. Anyway I’ve opened an issue in order to not calculate them in stats.

Sounds good.

But how can 2421 - 505 be 83?

(or, updated just now, 2453 - 507)

curl --silent 'https://db-dev.satnogs.org/api/transmitters/?type=Transmitter' -H 'accept: application/json' | jq -r '.[].uuid' | wc -l
507

EDIT: Found (most of) the difference: the dev db is not up to date.

curl --silent 'https://db.satnogs.org/api/transmitters/?type=Transmitter' -H 'accept: application/json' | jq -r '.[].uuid' | wc -l
2157

So only 296 missing transmitters.

Indeed db-dev is out of sync, it is just to perform testing for development.

In your request you filter the results with Transmitter type, this leaves out all the other types (Transponders and Tranceivers).

1 Like

Thanks for explaining. I was misled by the use of “Transmitters” on the stats page, I assumed it matched the transmitters on the API. But I understand the reason for the simplified language, writing Transmitters+Transponders+Transceivers on the stats page would be too verbose.