Hey @0xCoto, thanks for chiming in. You raise a couple of good points, so let me clarify.
- Having the raw I/Q data recorded (instead of just the FFT samples) means your observation file is huge, so expect users’ micro SD cards to get filled quickly (since most SatNOGS ground stations operate on a Raspberry Pi). Having to upload the raw I/Q data to e.g. share it with the community implies you’re also limited by the speed of your internet connection, so how effectively can you do this?
so, regarding the storage of the raw IQ on the Pi, I certainly wouldn’t advise that. Not only is there the “filling up the card” issue, but the frequent writes tend to shorten the lifespan of the μSD card itself.
Rather I would advise anyone wanting to capture the existing post-Doppler-shifted “raw” IQ (for whatever reasons) to mount a thumb drive onto the Pi’s file system, and configure the SATNOGS client IQ_DUMP_FILENAME variable to point to that mounted path. Even USB 2.0 write speeds (with buffering) are sufficient to dump the data, and wear & tear on a thumb drive is supposedly better than that on the actual μSD card itself. Also, thumb drives are cheap, and if it turns into a yearly “consumable”, that’s fine by me, personally.
and further speaking for myself, in case anyone is curious:
- I have each Pi + SDR, sitting on a VLAN I carved out for “chatty” little IoT devices I “sort of” trust
, located at the end of a PoE/Cat5e cable, closer to the antenna outside. I didn’t want to keep an actual HDD or SSD out there, so just a “standard” 128GB thumb drive mounted on the Pi, with the IQ dumps happening there. - I use SATNOGS_POST_OBSERVATION_SCRIPT’s to rename each dump file appropriately and log any observation metadata I want along with the .raw file, although all that data is already stored on the SATNOGS DB, so just having the Obs # associated with the .raw file is good enough.
- Next, since the thumb drive would eventually fill up over time, I have cron jobs, similar to log rotations, running on the Pi to periodically push the files over the IoT network, to a general-purpose data-storage NAS I have at the house. Plenty of room there for hoarding RF samples…
- From there my normal backup processes ensure that the data doesn’t go “poof” in an unforeseen event, and if I chose, I could push the files further out across the interwebs, expose them for torrent-exchange, trade them like Pokeman cards… whatever. At this stage the data is off the Pi, and I’m more free to do whatever I want with it.
- NOTE: To keep the total process load on the Pi’s CPU at a reasonable level, and give priority to the GR Flowgraph/SDR processing, I only let the cron push files off the Pi to the home NAS during lulls in the ground station activity, and only files that are obviously a few hours old, or older. There’s plenty of room on the thumb drive to last a while, even if my station gets booked non-stop (which it probably won’t). This may not be strictly necessary but I really don’t want anything jacking with the SDR activity, and I “think” Raspbian can sometimes prioritize file I/O over other processes down at the IRQ level… someone check me on this plz!
So the thumb drive “buffers” the IQ filling up, and the NAS acts as the archive where I can “torture” the data to my hearts content. I don’t plan on keeping this stuff forever, mind you. Spring cleaning rules are in full effect, and that extends to digital cruft…
At the moment it’s mostly to accumulate enough sample data to start playing with ideas around Sat & Signal “fingerprinting”… (i.e. the ID problem), and eventually, once I have the pre-Doppler-shift IQ saved off, applying variations of @cbassa’s STRF code to extracting the actual Doppler shift, which may not ever take place on the Pi itself… As far as I’m concerned, the Pi is fine performing the role of a RF capture instrument, with some basic data forwarding thrown in to keep it lean & healthy.
The real heavy lifting on the IQ can take place, well…
In a house, With a mouse,
In a box, With a fox,
or simply
On a cloud, (or wherever huevos verdes con jamón are found)…
I’m not to proud to put it there, or anywhere.
At some point, should I attempt to “throw AI at this” similar to the work @jebba linked to above, I should have a good starting set of training data for wherever that takes me…
of course the whole point of me chiming in from the get-go was to find out if others are working on similar threads, explore possibilities of collaborating at-scale, and (maybe) help evolve the system into being able to perform these “secondary” or “tertiary” tasks if called upon to do so…
There are other posts around these forums, like @mwheeler’s post on pushing the IQ to AWS S3 buckets, or @mfalkvidd’s post expanding on that effort and widening the destination to a broader set of cloud storage. Both are excellent resources for anyone considering this to review.
I won’t make any assumptions on the upload speeds and bandwidth allotments, because everyone’s connectivity is obviously different. If you happen to be at the end of a “soda straw” ISP connection, then maybe this store-and-forward thing isn’t the right fit… Every ground station owner/operator will obviously need to make that choice.
Now, on the more complex concern about timing accuracy and clock synchronizing…
…The problem is the clocks of the SDRs need to be accurately synchronized, and have the observations begin at the exact same time (or know the very exact timestamps each sample was taken), which is a very difficult thing to accomplish.
You are absolutely right.
Getting clocks so precisely aligned so you get phase coherence on each ground station’s samples is a VERY tricky problem… if your intent is to do TDOA/PhaseDOA interferometry from multiple stations to the same bird, over the same pass, all at the same time… but that’s not the approach I was thinking about, so thanks for pointing that out.
I should have clarified my thoughts better, and the technique I’d like to explore is better described at the STRF link that I provided above.
TL;DR; version: I don’t want to combine the IQ itself. I want to extract the Doppler shift measured for each observation IQ and combine that in a more traditional OD process. The links that @jebba shared to @Noeljanes presentation also do a better job of describing what I’m looking to combine, and how.
Now, in an ideal world, I would LOVE to have phase coherence across as many of the ground stations as possible, just to allow for the opportunity to precisely derive angles to the bird over time, instead of relying on one-way Doppler, but I’ll work with what I can get… 
Hope that clears up what I was trying to say !
Cheers,
-Lucky