SIDS Submission Format?

[K4KDR]

Very sorry if this is documented, but I searched & was not able to find it.

This is VERY much out of my area of expertise, but I am attempting to create code to upload decoded bytes to the database via SIDS. I found ‘some’ help online, and by attempting to copy formatting from the submitter app in gr-satellites. However, I’m failing.

Thanks very much in advance if someone could point what I need to adjust, if I’m even close.

The upload steps that I’ve come up with look like:

  if (client.connect(server, 443)) {
    client.println("POST /api/telemetry/ HTTP/1.1");
    client.println("Host: db-dev.satnogs.org");
    client.println("Content-Type: application/x-www-form-urlencoded");
    client.print("Content-Length: ");
    client.println(SIDS_UPLOAD.length());
    client.println();
    client.print(SIDS_UPLOAD);
  }

… and that ‘SIDS_UPLOAD’ string is assembled from the fields & format that I am hoping might be correct. My test example is:

'noradID': 30776,'source': K4KDR,'timestamp': 2023-07-21T05:30:00,'locator': 'longLat','longitude': -77.61,'latitude': 37.78,'frame': 01 02 03 04 05 06,'satnogs_network': 'False','observation_id': ' '

So, am I even remotely close? Or is there a formatting guide that I have failed miserably to find in the SatNogs documentation? Or Option-C… “don’t ever try to program anything ever again”?

Many thanks,

-Scott, K4KDR

[fredy]

I guess you have seen this GitHub - janvgils/sids: Simple Downlink Share Convention (SIDS)

satnogs_network and observation_id are optional to understand if the frame came from SatNOGS Network, no need to send them.

I think an issue could be that string parameters, described in this document, are not inside quotes.

Maybe trying this:

'noradID': 30776,'source': 'K4KDR','timestamp': 
'2023-07-21T05:30:00','locator': 'longLat','longitude': -77.61,'latitude':
 37.78,'frame': '01 02 03 04 05 06'

[SA2KNG]

another example I have used:

kng/geoscan-tools/blob/main/sids_uploader.py

[K4KDR]

Many thanks for the replies!

I’ve not made progress in my actual program code yet, but in seeking to find a way to verify the exact format of a successful SIDS upload string is, I did have success w/ the “CURL” command:

curl -d 'noradID=30776&source='K4KDR'&timestamp='2023-07-21T05:30:00'&locator='longLat'&longitude='-77.61'&latitude='37.78'&frame=010203040506' https://db-dev.satnogs.org/api/telemetry/

.

… so, now that I know with certainty what format to aim for, I’ll eventually get the same string concatenated in my code.

Thanks again!

-Scott

[SA2KNG]

is this some arduino or esp thingy ?
looks like a pain to construct all the http stuff, no lib for this ?

else, make curl spit out what it actually sends and copy that ?

Debug Curl Requests (TLDR: Use -v or --trace arguments) – Curl Cookbook

[K4KDR]

Yes, that CURL statement I listed above works, so now it’s just a matter of getting uploads to work in program code.

And you guessed correctly, I’m working on something brand new (to me), an internet-connected Arduino.

To date, all my Arduino work was stand-alone. I know there are 3rd-party wifi & ethernet shields, but never investigated those.

However, now I’m in receipt of a couple of Arduino UNO R4 WiFi devices, so internet connectivity is all built in. Since I decode a lot of LoRa (and FSK) payloads on this platform, naturally I wanted to create the proper code to allow these devices to upload to SatNogs. Of course I’ll put the finished product online when it’s working correctly - seems that there’s a known issue w/ the embedded ESP32 WiFi module that requires a firmware upgrade, so working on that as we speak. Not unusual for 1st-generation products, of course.

Thanks for all the help!

[K4KDR]

On the chance that anyone was wondering about progress on getting SIDS uploads to work from the Arduino platform, after some trial and mostly error, I’ve been uploading successfully to the test database!

Initially I used entirely static, fabricated data. Today I managed to use a “real” timestamp from the Arduino UNO-R4’s real-time clock (synced to internet NTP).

… so, all that remains is to use real decode bytes instead of a static string. Hope to figure that out tomorrow.

Of course once it’s all working correctly, I’ll upload & post the URL so that others can improve on it & use it as a template for their hardware/software setup.

[PE0SAT]

Hi Scott in case you maybe missed this: GitHub - janvgils/sids: Simple Downlink Share Convention (SIDS).

There are sometimes difference in the implementation in regard to required fields.
If you want to test further, let me know I still have an experimental SIDS collector running.

[K4KDR]

Yes, I did find your page on the subject - thanks so much!

That example payload that you showed there was the key item that allowed me to proceed.

I’ll be in touch if I get stuck & need your test server - appreciate that!

[K4KDR]

Tonight (26-July-2023, 0140utc) while attempting to do some more SIDS upload testing to db-dev.satnogs.org, I’m getting “500 Internal Server Error” responses.

Could someone possible check on that, please?

If I was sending malformed data, etc., I would expect an error message indicating a client-side problem.

Thanks so much!

[K4KDR]

Please scratch that last request.

In working to move from static data to “real” hex bytes from live decodes, I neglected to include the “frame=” portion of the upload string.

I’m uploading successfully again after finding that mistake.

Thanks!

[fredy]

The request was right, it shouldn’t return a 500 error rather a 4xx error with a message of missing the frame field. I’m going to open an issue about it and fix it asap.

[K4KDR]

Another update…

dealing with the TimeStamp turned out to be challenging, but I’m sure would be much easier for anyone with more experience programming the Arduino platform.

Also the transition from a fake static payload of HEX bytes to using real over-the-air decodes was a little tricky, but not as bad as obtaining a fresh timestamp for each upload.

But the bottom line is that I’ve connected an antenna and am uploading real, live, from-outer-space LoRa decodes to the DEV database. Now for some clean-up & sanity checks before putting it online for people to use in all or part with their own hardware!

[SA2KNG]

timestamp isn’t hard really, <date>T<time>Z is the basic format, like 2023-07-28T01:54:00Z and you can have float seconds if you want.

[K4KDR]

Still corrections to be made, so not ready to put online yet - will get that done ASAP.

But an exciting night here… first ‘live’ upload of an Arduino LoRa decode to that satellite’s spot in the SatNogs production database! Didn’t know if I would get to this point. And I should add that this isn’t just a ‘LoRa thing’… the various modules available for use w/ a microcontroller such as the Arduino do several forms of FSK also. (A significant number of satellites are going up now w/ transmitters based on these hybrid radio modules, so it just made sense to figure out how to get the same family of ground station receiver hardware SatNogs-enabled).

Will get the sketch online as quickly as I can after a few more items are worked out & tested. Thanks.

[SA2KNG]

Well done.
I thought of another project that could perhaps be used here. Not sure if it’s a good idea, but TinyGS basically does this but uses MQTT. Maaaybe add optinal SiDS to that ?

[K4KDR]

Thanks. And I’ll certainly be making the tinyGS admins aware so they can review the logic & steps in this code to see if they would want to make it an option on the client or server side of their setup.

[K4KDR]

The Arduino code that I’ve been experimenting with is working well - but of course any real programmer could make endless improvements.

Nonetheless, I posted some screen shots & a link to the code if anyone would like to use parts of it for their microcontroller / RF hardware:

https://twitter.com/scott23192/status/1684953510431903745

… here is the direct link to the code:

https://www.qsl.net/k/k4kdr//satnogs/

[K4KDR]

For anyone who stumbles across this thread from a Google search, etc…

FYI, the same receive / decode / SatNogs upload mentioned above now has code that works on ESP32 hardware:

Scott Chapman

Sketch files → Index of /k/k4kdr//arduino/esp32-satnogs-sids-template

Video of a live decode → https://www.youtube.com/watch?v=SYzQ-UTqMBE

… while testing was done mostly with satellites that do not have ties to the Amateur Community, of course the goal here was to figure out how to utilize platforms like Arduino & ESP32 for when the day comes that a properly coordinated, genuine Amateur satellite goes into orbit with a downlink that only these types of receivers can decode.

[K4KDR]

After two months of working perfectly, my Arduino code now fails to complete SIDS uploads.

Since I mask the reply except for the ‘201’ response code, I removed filtering so that I could see the entire server response in hopes of seeing an error. However, to my surprise, there is NO server response.

Attempting to manually investigate, I simply put the upload URL into a browser and get the following ‘401’ error:

… has something changed in the requirements for a SIDS upload? I searched here & in the Gitlab repository but failed to find anything that I could recognize as needing to be changed on my end.

Thanks very much!

-Scott, K4KDR