Network and db - "closing the loop" for data

SatNOGS
1

Hey all,

I am writing this lines to open a discussion on two different approaches to get the data-tab of an observation in sync with the db.
While talking about the SiDS interface I had the idea to simply make a SiDS request whenever ‘satnogs-client’ performs the upload of a data file. There have to be done some modifications to ‘satnogs-client’ to get some neccessary informations about the observation that produces this data - one of this informations is the satellite’s ID.

While collecting some ideas for this, @pierros told that the SiDS interface might be enhanced in the future, because it lacks some valuable informations.

@fredy suggested the second approach: let the network instance transmit the “demoded” data into db after the upload is complete. I am not very experienced with the network codebase so I don’t know about the amount of work this will produce.

Discussion open now :wink:

3 Likes
2

I think @fredy’s approach is probably the way to go - we are going to have to come up with a task that submits everything in network retroactively, so that same task can be used moving forward.

2 Likes
3

Update on this topic, we have closed the loop between network-dev and db-dev, there is a bit more work to do to make this code safe and then we’ll push it to prod.

Its a little off from the approach mentioned here, we’ll have a celery task run every minute that looks for new data to pass on. as we grow this will be something we can abstract from the main web server itself and run elsewhere to avoid resource contention.

2 Likes