You are proposing an architectural change which is not inline with the long term goals of classifying data using machine learning. How does creating a vetting API will generate a useful training set when the rules of vetting from different clients will not be homogeneous?
This is a network of stations and not a free FTP for uploading data. You can’t define them for “your station”. This is not the proper way to achieve decentralization anyway. Within the network, and for the purpose of vetting as stated above, all stations must follow the same rules. But you do have a say on this and this is exactly what we do now on this thread; we are discussing. The final decision shall be based on facts, science and ethics, and not because someone has a “station right”. In addition to that, the observation data is there and free. So if anyone disagrees with a common approach, they can of course do their own analysis for their own stations.
Why automated FAILED flag is not implemented yet? The reason is that we have a finite number of developers, some of which are volunteers, and 422 open issue in GitLab many of them having much higher priority. So, contributions are more than welcome! Nevertheless, those which are not in the right direction or quality will not be accepted. They will definitely increase the effort to undo or maintain them in the long run.
I’m having some trouble to understand why after a simple question related to a maintenance tool I did is receiving such an irate, off point and arrogant answer.
I’m not going to continue any discussion in this tone, it’s a moot point to repeat a simple request several times and still being answered as if I’m request an “architectural change”, which is not.
Meanwhile I’ll love to have the chance to access material where to educate myself on whether this right direction and quality is defined. I willl assume this is in some written form if this is a well established acceptance criteria.
Allow vetting from API for mark an observation as failed. This will allow SatNOGS client and other user scripts to vet an observation as failed.
From the three of them, currently the easiest to implement is the second one. The first one needs more work and also as a solution needs user intervention for something that is “well-defined” and can be easily automated. The third one also needs more work on client side, which some of it is planned to be implemented, and some work on network side too. However third solution is a good idea, the client to let the network know that something has been failed, so I think we should examine it in the future.
The client part of alternative -3- of yours is something station owners can contribute with scriptings from which you can make your beauty contest on which one to endorse, so it shouldn’t drag resources from the developer team. The network side of it will of course, I can not grasp the details on how much, but I would assume is something modest.
I am sorry if my previous post was a bit harsh. I was trying to give a clear picture on the current status and break down a common misconception which is that SatNOGS Network is simply a remote control for stations. The focus should be on the project as a whole and not to individual stations.
Your opinion is and will always be respected because you are member of the community and not only because you are a station owner. Yet, there is information and discussions going almost a year back related to vetting and how inefficient it is which you are not aware of unless you have been following this project development really, really close (GitLab, dev meetings, IRC, etc.). I tried to explain the draft plan (removal of subjectivity, creating a training set for ML, etc.) and why a vetting API is pointless or even harmful at this point.
Let’s continue the discussion on technical terms on issue #724.
Tension centralized vs. de-centralized, which is far from being settled I think, much less with rude manners or assumptions on other’s mistakes and misconceptions.
To the best extend of my understanding the SatNOGS Network is a cooperative effort where a community of network managers, individual stations and general users comes to invest time, effort and money to achieve a common goal, the network itself, with some particular goals being satisfied whenever possible within the constraints of the available time, effort and money each one can devote to the endeavor. Long term views of the network managers while necessary should count as well as needs of stations should count. Or at least a civilized discussion on the opportunity or applicability of them without resorting to a blue sky long term yet to be catalized view of the future as it was a settled thing.
I’d followed some of it, probably not all of it, which I would have to say is a rather chaotic and disordered mess of threads very difficult to follow. Not completely unaware on the main issues either.
A plan is an activity, a date where it will be completed and a responsible for doing that; to the far extend of my knowledge you lack at least one, probably two and even three items from this definition. So basically you don’t have a plan, you just have a list of good intentions. I guess it’s not documented nor baselined either, which are desirables features of something being referred as a plan, even a draft one.
To the sake of the fourth, and last, time I’m making the distinction… I’m not proposing a vetting API, I could only speculate on why your understanding of my comments resonated into that direction despite having explained in the contrary several time, and why you find so threatening a request for some automation on an already existing feature of just a way to mark failed passes expending less hours on doing so. If at some foreseable future you architecturaly remove the FAILED state this API entry won’t be used any longer, but meanwhile you’d probably saved a sensible amount of maintenance time on station owners.
I am sorry, I won’t be dragged into a discussion on rude manners, offending and baseless statements, contempt of ones work. If you would like to participate in a technical discussion about the matter at hand, please comment on the issue.
SatNOGS has a method to relay information from client to network at the end of an observation, the metadata upload (satnogs-client:…/observer.py#L162-189 & satnogs-network:…/api/serializers.py#L64-68). The information needed to vet an observation as “failed” (e.g. crashed gnuradio process; no waterfall, audio or data produced) is not included yet.
This metadata upload could be extended with some fields to fix this issue, e.g.
the gnuradio process exit status
a list of files pending to be uploaded to the observation
Best wishes,
Fabian
PS: I hope we all can continue this discussion together. I myself felt hurt by some statements in this thread, but I ignore this for now and try to continue to fix the initial topic of the thread:
Maybe I didn’t define well what I mean with the “work” part. By work I mean two things, the development of the feature and the implications the feature will have in maintenance, plus the implications on and expectations of the user base. With this new definition let’s analyze again the three solutions:
Creating the new GUI means changes on frontend and backend, which means a lot of development. From my experience with the code and other similar projects in the past, having massive selection as an option will create some maintenance work when other changes are implemented in the UI but also it can be affected by changes in backend too. Finally It will help the user but as I said in my previous post, why to develop something that will still need user when you can develop something else with less effort and implications that will do the same job automatically.
Adding a task in a system that already has and runs tasks it’s easier to develop. Maintenance in this case is affected only by changes in backend and is limited/focused on the code of the task. Except that users will have less observations to vet and that they will expect failed observations to marked as such automaticaly, there isn’t any other implication or expectation. Finally doing this job on server side with a task will not increase server load with requests for (failed) vetting.
Creating an API endpoint it’s not that difficult to develop, more or less the same effort with a task. Changes on client need a little more effort right now as there is a need to change the architecture in order to implement this feature. This will happen eventually and when it happens it will be easier to implement the feature.
In case of implementation of the client side then keeping the code valid in the upcoming client architecture changes will be a maintenance task. Another one is the maintenance for the API which would be more or less the same with the task. But the maintenance of the API has a plot twist, an API endpoint is a promise with users that they will have a functionality that will not break or change, at least no without a notice and within a time period, usually long.
And here comes the user base, having an API means that users, even if there is or not client code, will write their own scripts. I leave out the discussion of what criteria these scripts will use for marking an observation as failed, and I’ll focus on the promise you will give to them with creating the API endpoint, which is that this functionality is here to stay for a long time without changes.
Knowing that several and structural changes are coming in all the SatNOGS projects (Client, Network, DB and the related ones) and the direction of moving to an automated way to vet observations and mark them as failed or not (which is going to be split from vetting) means that the API endpoint promise shouldn’t be given, at least not currently.
But if it is given for some time, when/if it goes away or change there will be reactions on why this was removed/changed or why this was implemented in the first place when it was known that it will be removed/changed. And this frustration will be justified as the users would have spent time to create scripts and code based on this promise and they may have to spend more time to adapt to these changes or the removal and I really would like to avoid this.
What I’m trying to depict with this analysis is that we need to take into account all the parts and all the trade-offs for each solution and choose the best one.
This could be a solution, I see one scenario that can fail, it is when the metadata will report a successful observation and several files to upload and for some reason client will fail to upload them.
Also architecturally it seems to me wrong to have this side effect on a request of uploading metadata. It is like the vetting as “good” when uploading data, which has been proven wrong tactic with many false positives, so I’m a little hesitant.
However I don’t find bad the idea of client letting network know that it had a failed observation, but I’m not sure if this the right way to do it or if it is necessary at all to do it. Finally I’m not currently 100% sure where these metadata will live after the completion of the artifacts changes and how network will access them but this is another discussion.
Building on your ideas I would like to propose a well known architectural pattern which is the plugin or user exit.
If the client has all the information to define when a given pass is doomed to be defined as failed and the observation itself didn’t put it into that condition perhaps a short script can either change the status prior to be sent or otherwise report the failure right away.
If the observation process prior to send the chunk of data call a well defined script (the user exit, i.e. userexitxx.py) if it exists it can pass as argument the file with the metadata and a list of pointers to the relevant files. The script might manipulate the metadata (or just observe it, as it might have other uses than just mark failed passes) and upon exit the regular process might continue.
Some must-do sanity checks needs to be introduced:
If the “userexit” doesn’t exist nothing happens.
If the “userexit” scripts dies or crashes it just should be ignored.
If the “userexit” scripts corrupt in someway the metadata it needs to be checked at the network prior to use, just the fields should be populated and with values within the ranges defined to be accepted. This can be done at the client side prior to send the data in order not to even bother the network with the checking.
If the “userexit” places status other than failed in the metadata it should be ignored at network prior to accepting the data.
The bulk of the effort is on writting the script, but some sanity checks are needed to protect de data, also a distribution of client code needs to be done with the addition.
In light of the prior discussion and the heated debate on whether this creates architectural issues I would say that this is a very controlled approach since you keep the gate with relatively simple sanity checks and you could remove the functionality in a very easy way just distribution a client code that do not call the user exit script any further.
This is a very well known approach on architecturally sound products where you want to address needs like this.