DB API: plan + idea thread

,

There are a lot of issues outstanding around the APIs for db (and much of this applies to network as well). I wanted to touch on some of the work and open a thread for API expansion ideas.

Documentation

While the default web view of our APIs are somewhat self-documenting for path browsing, they lack queries supported. My goal/proposal in this area is to have an OpenAPI spec that could then be visualized by a number of tools. If we were to push this through today, we could use drf-yasg (a swagger/openapi 2.0 generator and UI), however this would take a lot of customization given the way we have our APIs set today. It also adds a LOT of dependencies that we would need to continue tracking. Instead of going this route I suggest we stick to the django rest framework’s own schema generation. This is set for major improvement in the upcoming 3.10 release of DRF (date TBD). If 3.10 does not meet our needs for producing a good schema, we can then look to drf-yasg. Note: DRF 3.10 will be Python3 only and depend on our move to that.

The documentation improvement issue is found here.

Legacy / SiDS support

We have a very basic API endpoint for uploading data frames based on an old protocol called SiDS. It allows a client to pass the following metadata along with the decoded data frame (in hex): noradID, source (ground station name), timestamp, longitude, and latitude.

Between client decoder apps written by DK3WN and gr-satellites, we get a LOT of data this way and will need to continue supporting this method for the foreseeable future (even if we extend the API and new client apps are written, we have to consider users who do not update their apps).

Next-gen API

I propose that once our doc generation is set (and we’ve updated to django 2.2, drf 3.10) that we freeze the API as v1, inclusive of the legacy SiDS format. Then - we push new changes and development into v2, expanding our API further (v2 API urls would be prepended with /v2/). I would also propose that we define, document, and agree on the v2 API first, then develop for it when we feel it fits the needs properly. This is where I kick this into “idea thread”. What would we want in the API? The following have already been discussed and could be included:

What else should we add to the model and API for satellites in db? I would add my own ideas:

  • Dashboard URLs (multiple)
  • Tagging (“ham” sats, “weather” sats, etc)
  • Owner contact or homepage?
  • TLE export, could create custom TLE files based on tagging above (though redistribution is still a question)

tagging some interested parties: @DK3WN @EA4GPZ @fredy @pierros @kerel @Acinonyx

5 Likes

I believe that we should expose the whole database to the API. But it is not the API that needs to be rewritten at this point; it is the schema that we need to redesign/fix. Although we basically have only 3 models, Transmitter, Satellite and DemodData, the relationships are very messy. For instance, we have both satellite and transmitter fields in DemodData but the transmitter also has a satellite field (as FK) creating a circular relationship. The reason for this mess, is that we tried to use the same model (and even interface) to store verified, trusted data (SatNOGS Network) with unverified, incomplete data (e.g. SiDS). I think, we need to fix these issues first before extending the schema further.

Just to help this topic along, here is the schema for db today (from MWB):

base_telemetry can/should go, right off the bat. It is a relic of the day when the plan was to attach the decoding schema to the db (before kaitai)

I just created issue satnogs-db#339 to track the fix of this.