An idea that we have been having for a while is to produce a definition for all byproducts of an observation. The concept would be to be able to detach an observation from its byproducts (waterfall, audio, data etc) and make sure that those byproducts (here-aft called Artifacts) are self described and atomic to the extend possible.
By proceeding with such change we will be achieving the following:
- Artifacts will be able to not be associated to a SatNOGS observation thus produced and submitted in an independent manner if needed (not necessarily by satnogs-client and gr-satnogs), thus enabling more submissions of data from stations not in SatNOGS Network. You may consider this an extension to the sids protocol (that is currently lacking all the info we need).
- We would have a flat abstract definition of all artifacts, that will enable us to deal with them in a unified way (vs the way we expect products of an observation now). As a side effect to that our databases will be able to handle the data more gracefully.
- By having an abstract definition we are future-proofing our architecture in case new types of Artifacts come along
- It will be easier to process those artifacts by researchers and people that would like to do analysis (not having to traverse the relationships with observations to obtain more info)
Such an implementation could yield the possible considerations:
- We need to decide on what to keep as attributes of observations
- We need to minimize to the extend possible the duplication of data between observations and artifacts (some is expected)
- Such an approach would require some considerable development and coordination between satnogs-client and satnogs-network projects (and down the line db and warehouse too), so it needs careful planning and input from all stakeholders
Below you can find the proposed draft definition. Please add any comments directly there or broader discussion points in this thread.