We are using the API for various purposes, including auto-scheduling and optimizations in cable unwinding. Recently, if anonymous API requests were sent, the response was something like
{"detail":"Request was throttled. Expected available in 35 seconds."}
in a few cases - which is fine, of course. I added then our API key, which solved the problem. Now the situation is a bit of opposite: with the API key the requests are throttled significantly but without the key it just works. What’s the reason for that? What’s the practical limit for such queries for registered users operating many stations (2380, 2381, 2410 in our cases)?
I’m sorry for the inconvenience, these limitations were necessary for solving some performance issues we had. They may change in the future when we improve performance and make better use of the available and possible new resources.
About the limits, this is the current status:
60 requests per hour for anonymous users and 240 requests per hour for authenticated users.
Each request returns 25 observations, which means:
60 x 25 x 24 = 36k observations per day for anonymous users
240 x 25 x 24 = 144k observations per day for authenticated users
Given that currently we have ~10k observations scheduled daily in Network, that means that anonymous users can get ~3.5 times the scheduled observations and the authenticated ones ~14.5 times.
About the auto-scheduler, given that it uses the api/observations endpoint only for getting the future jobs, it can be replaced to use api/jobs endpoint which doesn’t have any limitations.
I have planned to perform this change but unfortunately there are some other priorities in the middle, any code contributions towards that change are more than welcome. Until then I suggest you change the timings on how you use the auto-scheduler in order to limit the requests.
Thank you very much, @Fredy for the detailed explanations! Then the task is on our side to be within these limitations
Indeed, most of the queries are about purely future observations (both unwinding and scheduling).
I’ll try to optimize on the unwinding part - this is now very sub-optimal because it runs in every minute(!) if the station otherwise does not run any observation - and because of the list of future observations are spread around 3-4 pages so that might be the bottleneck what eats up effectively all of these queries available for the user. So i’m trying to add some limits in obs-time (e.g. limit it into the next ~2 hours or so) which minimizes the number of observations returned by the query. Is there any GET element for this purpose, i.e. saying “+2 hours” or something equivalent instead of computing the absolute time? Here we need only literally the next future observation (i.e. one observation or the empty set if nothing is scheduled).
The auto-scheduler is run now in every 1/2 hours, but it is not that aggressive. It fetches all of the future observations for a given station for all of the satellites of our interest (6-7 now), find the empty slots and then schedule the (either full or partial) observations. There are some auxiliary queries - e.g. fetching the station’s location - but that can easily be improved to omit via some catching mechanism since those are just constants. The db.satnogs is also queried for the recent TLEs, but that can also be catched.
