Auto scheduling

Jferns · November 12, 2018, 4:07pm

Subject : Auto-scheduling testing.
Reference:

I created a local implementation of auto scheduling on my station. I am using a transmitter priority json file and setting for minimum elevation to calculate and create json file for jobs locally. I have observations created this way with about 2 to 3 months data.

My current testing is on a setup fully autonomous and at remote location. It connects 4 times a day to the satnogs network api, gets the latest tles of all transmitters in my priority list and the priority list json file from a web page I control. All the observations captured this way are stored in a USB drive.

My code kludges involved a modified satnogs-client and other python code. Code is work in progress and embarrassing to share publicly.

Questions:

Is there any satnogs api to push these recorded observations to the network?
What is the time line/priority for creating such an api?
I am using epoch time as observation number for testing. If satnotgs api supports uploading these observations, is there any suggestions on how to handle or convert this to use unique observation number on the network side?
A satnogs API to get TLE’s used by the satnogs network using transmitter or Norad ID 's would be great. Can this be implemented?

KD9KCK · November 12, 2018, 10:04pm

Well currently you can’t upload an observation unless one already exists on the network.

I can give you a python API I made for scheduling observations by abusing the UI endpoints.

Because you could schedule things automatically and let them run as normal.

This is the only suggestion I can give. Maybe some people who know whats being worked on can give info.

I have been pushing for an API for making new observations and I hope it happens soon. (Even if its only for your own stations.) I might attempt to write a MR for it.

Here is my old script that no longer works if you think it could be interesting to see how it use to work.

gist.github.com

https://gist.github.com/wgaylord/ec4443545cdac736620b49b8e13f7475

Satnogs.py

import logging
import requests
import lxml.html
import sys

sat = "40014" #SAT norrad_cat_id
transmitter = "NSXo8tGxmxpTUMsmSH34FF" #Transmitter UUID
start="2017-12-1 17:00" #Start date and time
end="2017-12-2 03:00" #Stop date and time

This file has been truncated. show original

kerel · November 13, 2018, 7:34pm

Hi @Jferns,

Thanks for sharing, certainly a really interesting & important topic! I guess you already know the Automatic scheduling - #4 by pierros proposal formed during Hamvention 2018, or at least your implementation pretty much matches what was proposed.

Currently all observations in satnogs-network have to be scheduled via the /observations/new/ in advance. I’ve created the issue satnogs-network#543 in order to track this issue. In the meantime you could use e.g. the code by @KD9KCK to create the observation in advance (there is curently no API, the server-side implementation of the current endpoint is in satnogs-network/…/views.py#L297) and let the satnogs-client upload the observation data as usual.

When you upload the observation, the unique observation number in satnogs-network is returned. So you can continue to use the epoch number as a local identifier (at least as long as no observation was created in satnogs-network).

Most of the TLEs in satnogs-network and -db are downloaded from CelesTrak (with the exception of 8 satellites not available via CelesTrak, see this notebook). Unfortunately as of now, satnogs doesn’t have the right to redistribute TLEs downloaded from Celestrak (citation-needed).

Sincerely,
Fabian

Jferns · November 13, 2018, 10:20pm

Thanks @kerel and @KD9KCK for the replies.

I will check notebook for code to download TLE directly fom Celestrak instead of scraping tles from jobs api.

Looking forward to new observation request api feature. @KD9KCK I will check your gist code and test if it still works.

–JFerns

KD9KCK · November 14, 2018, 12:01am

I can tell you right now its broken. I am working on fixing it over this week some time.

KD9KCK · November 15, 2018, 6:06pm

I will make a simple script that if given start and stop time, The transmitter for the sat. (UUID from satnogs Database) ,the norad and your station number. Can schedule the observation. but you will have to provide the information as I can’t seem to get the UI’s internal-use api to give me prediction information.

Also be aware this could break at any time because yea all these apis are undocumented and internal use only type of apis.

cgbsat · November 15, 2018, 8:25pm

A while back I wrote some a tool, based on SatNOGS network code, to compute satellite passes given a ground station and a set of TLEs, and create firefox commands to network scheduling API to quickly schedule passes one at a time. The tool would sort the satellite passes on user defined priorities (NORAD IDs for now), or on their maximum altitude of the pass. The tool is at https://gitlab.com/cgbsat/auto-scheduler/tree/master/

The ability to automatically schedule passes on the network would allow me to use this tool to regularly, say every few hours, check if my stations have scheduled passes, and if not, fill up the schedule using user define satellite/transmitter priorities.

The tool needs to be expanded to:

automatically download TLEs using @kerel’s TLE downloading script
use the SatNOGS DB to get transmitter information
use transmitter information (frequency, success rate, status) for computing passes
use the future scheduling API to schedule passes

KD9KCK · November 16, 2018, 5:53pm

I threw together this.

gist.github.com

https://gist.github.com/wgaylord/f958f0b83b85324d79d9537c6277e8c7

newobs.py

import requests
import lxml.html


    
def MakeObservation(username,password,norad,transmitter,station,startTime,endTime):   
    loginUrl = "https://network.satnogs.org/accounts/login/" #login URL
    session = requests.session()
    login = session.get(loginUrl) #Get login page for CSFR token
    login_html = lxml.html.fromstring(login.text)

This file has been truncated. show original

Which allows one to make an observation on the network.

BUT you must provide the times in UTC in the format below as a string.

“2018-11-16 18:14:41.423”

For specifically it has to be a date in which pyephem can understand.

cgbsat · November 17, 2018, 4:22pm

Many thanks @KD9KCK!

I’m trying it out, but it doesn’t appear to work. Can the NORAD ID and sation ID be given as integers and the rest as strings?

KD9KCK · November 17, 2018, 6:07pm

Yes. I should put together some docs for that annoying function.

KD9KCK · November 18, 2018, 1:39am

I used the following.

MakeObservation(USERNAME,PASSWORD,40967,“ZyjKNJ9KqnTHBCUzAPN5G5”,41,“2018-11-19 07:38:28.000”,“2018-11-19 07:52:00.00”)
To make the following observation https://network.satnogs.org/observations/327952/

I got the times right from GPredict in the default UTC times mode.

cgbsat · November 18, 2018, 12:28pm

It works! Turns out it needed fractional seconds in the timestamps (e.g. 2018-11-18 23:04:20.0 instead of 2018-11-18 23:04:20).

Nice work!

KD9KCK · November 18, 2018, 5:20pm

I also just wrote some code to be able to get all the nextPasses displayed on the station page. Along with the overlap. (Took a bit for that)

I also made a function to pick the best transmitter that the station can receive.

I will update the file later today.

cgbsat · November 18, 2018, 6:01pm

There’s some code for that in my repository, but it might not do exactly what you need.

Incidentally, I’ve been working on fetching transmitter information given the antennas present at a station, and then fetching the TLEs for it.

KD9KCK · November 18, 2018, 6:31pm

I mean I already write the code to do that yesterday, (it gets all passes.)

And if you giver this function the NORAD the function finds the transmitters your station ad actually hear and then gives them in order of success rate.

KD9KCK · November 18, 2018, 7:45pm

On a side note I will have to rewrite this once the new OAuth changes happen.

Jferns · November 20, 2018, 4:31pm

@KD9KCK Thanks for sharing the code for creating new observations.

My station is 22 is now back online, all observations are created and scheduled locally and satnogs-client logs the observations to the network.

KD9KCK · November 20, 2018, 7:40pm

Just be aware that the code is going to break as soon as the new OAuth stuff is built. But hopefully by then I will have a MR for a public api.

cgbsat · November 29, 2018, 10:02pm

After a few days of testing my auto-scheduler tool appears to be stable.

When given a ground station ID, it obtains the antenna properties of that ground station (mostly frequency ranges) from the SatNOGS database. Next, it gets the properties of all active transmitters in the database, and downloads TLEs using @kerel’s TLE downloader. These TLEs are cached locally, and only updated if the last download was more than 24h ago. Passes for these satellites are computed for a given time frame, and these passes are given a priority based on the transmitter success rate as well as the maximum pass elevation. This list is then sorted on priority, and each successive pass that does not overlap with previously scheduled passes is scheduled on the network.

On my stations (#39 and #40) I run my auto scheduler hourly from a cronjob, scheduling passes for the coming hour. This way, the auto scheduler keeps the stations busy, while allowing other users to schedule passes further in the future.

csete · November 30, 2018, 2:17pm

Nice work @cgbsat. I noticed in the patch from @kerel that it requires matplotlib; however, at a quick glance, I can not find any references to it. Is it actually required. If yes, for what?