It’s a long time we had discussed that observation vetting needed a reform. Recently we had disabled auto-vetting as it affected in a negative way other parts of the project.
Today, we are moving forward to a new way to rate observations, which will help us to develop tools and smarter processes to automate the whole procedure. The main difference is that from now there will be an observation status and for all the artifacts of an observation will have a separate vetting status.
Vetting artifacts will start from Waterfall but, in the future, it will be expanded to the other artifacts too. This will give us the opportunity to improve and develop methods for vetting each artifact separately, more accurately and in some cases automatically.
Vetting waterfall will be quite similar to the previous vetting system. For more details please visit the wiki page that explains in detail waterfall vetting.
While waterfall vetting will remain for now a manual process, observation rating is fully automated. An algorithm that checks several criteria based on artifacts will try to rate an observation. This will not always be possible, so manual intervention will be needed in order to vet artifacts and help rating algorithm to decide.
Algorithm for automated observation rating is not perfect and it is still a work in progress, so if you find an inconsistency you can open an issue about it or fix it in SatNOGS Network repository. More details about rating can be found in the wiki page.
Note about auto-vetting:
A similar process to auto-vetting, is now the algorithm for rating observations. Observation with data (demodulated/decoded frames or decoded images) will be enough for rate an observation as “Good”, except if the mode of the observation’s transmitter is either CW or FM. However this may change in the future. Of course, as in the past, we will have cases, in which noise frames will be produced and will give a wrong result, in this cases manual vetting of artifacts will play a significant role, until we improve the rating algorithm and artifact vetting.