Now that iqzip is available, I thought I’d see if I could switch to iqzip instead of xz. I compiled and installed the iqzip library, and the iqzip example app. I then tried compressing a wav file:
-rw-r--r-- 1 pi pi 63M Dec 7 15:41 4060852_NANOCONNECT-2SAI-1.wav
-rw-r--r-- 1 pi pi 64M Dec 7 15:42 4060852_NANOCONNECT-2SAI-1.wav.iqz
-rw-r--r-- 1 pi pi 43M Dec 7 15:40 4060852_NANOCONNECT-2SAI-1.wav.xz
I expected a file that would be smaller than the original file. Maybe not as small as the xz file, but I expected some sort of compression.
You are using wav or iq_dump raw ?
I have been using zstd on iq_dump, usually compresses them down to around 25% of the original size.
50M Jan 3 17:20 iq_6975024_57600.raw
12M Jan 3 17:20 iq_6975024_57600.raw.zst
I have it running in a crontab
#!/bin/bash
# cwd to dir where the script is located
cd "$(dirname "${0}")" || exit
FILES=(*.raw)
# check if glob expanded to a file list
if [ ! -f "${FILES[0]}" ]; then exit; fi
for f in "${FILES[@]}"; do
zstd --rm -f "${f}"
done
Unfortunately, it seems like zstd is performing considerably worse than xz (at least when using the standard settings for xz and zstd), at least for the files I tried:
File
Original size
Compressed size flac
Flac ratio
Compressed size xz -6
xz ratio
Compressed size zstd
zstd ratio
Compressed size zstd -19
zstd -19 ratio
6750181_0_UVSQ-SAT.wav
89046800
47608484
53,5%
53158044
59,7%
61286795
68,8%
59189863
66,5%
6761957_0_UVSQ-SAT.wav
89955460
49148800
54,6%
54755016
60,9%
62813458
69,8%
58577089
65,1%
6915601_0_DSAT.wav
59878656
32149909
53,7%
35905036
60,0%
41314881
69,0%
38076911
63,6%
6916490_UVSQ-SAT.wav
85051672
46996343
55,3%
52051304
61,2%
59601060
70,1%
56359490
66,3%
6924496_0_NETSAT-2.wav
57384288
30086094
52,4%
33682040
58,7%
38971186
67,9%
36412881
63,5%
6929658_0_KKS-1_KISEKI.wav
68155148
34997790
51,4%
39136216
57,4%
45658157
67,0%
42512620
62,4%
6930693_0_TAURUS-1.wav
125965520
69058161
54,8%
76607548
60,8%
87767997
69,7%
82048056
65,1%
6931543_0_NEXUS.wav
51620956
28644598
55,5%
31685564
61,4%
36285008
70,3%
33540367
65,0%
6953937_0_BY70-2.wav
107504300
57741229
53,7%
64451248
60,0%
74149147
69,0%
69856033
65,0%
So far, it looks like flac is the winner. It is also considerably faster than xz. But it would be nice to use iqzip, since it includes metadata. I’m curious to why I’m getting so much worse zstd compression ratio than you do though.
Hello @mfalkvidd.
The iqzip is not a dead project, it is considered completed so there is no active development for extra features right now.
The problem is that the default options of iqzip does not correspond to the file characteristics from the iq dump e.g bits per sample, if samples are signed etc.
The command for your example should be:
So for this dataset, iqzip compresses very close to xz. flac is the winner and zstd the loser. I also noticed that iqzip is very fast, close to flac in compression speed.
I’m still curious why SA2KNG got so much better compression with zstd. Maybe the reason stems from him using plutosdr while I’m using RTL-SDR?
I’m also a bit surprised that the raw iq files are 16-bit when RTL-SDR is only 8-bit. My guess is that Satnogs uses 16-bit regardless of what SDR is used. But in that case, it should be possible to compress IQ files from RTL-SDR 50% since half the bits would be zero? Or is decimation used to increase the resolution?
FLAC was also very good in our comparisons, although Iqzip, which uses the Shorten compression algorithm, was better in our dataset for most observations. The comparisons can be found here.
Also Iqzip is a CCSDS compliant compression scheme that can be used in space missions, and that is another important aspect of our compression scheme.
I also tried to tune some parameters of the algorithm but it seems there is a bug in the application we provide. Let me fix it and i will come back with my results with tuned parameters.
Firstly i have tested it a bit and i have found that the biggest block -j that works is 128 and reference interval -r is 128. The default values are -j 64 -r 1, and the difference in size for the default and biggest parameters allowed using 1574048_AAUSAT_4.wav file for testing is 1MB (142MB original size, 80 MB and 79 MB compressed respectively). This behaviour is known as you can see here.
When i talked about a bug in our app it was a misconception on my part because i was talking about extended block size parameters and not rsi values. At some point we decided to remove the libaec as a submodule of the project and as a result changes that we made to libaec regarding the maximum block size were removed. As can be seen from our measurements, using higher values for block size than 64 does not effect compression ratio that much, hence we made that decision.
I am currently investigating why “legal” rsi values don’t work as expected. I have found a solution on compression but using an rsi of 4096 has a miniscule difference of 4 KB compared to 128.
With that in mind i would say that FLAC works better in those observations because it just does as seen in some observations on our own testing but it is not CCSDS compatible as i stated in my previous post.
Now i will try to fix the bug with rsi > 128 hopefully by the end of the week. Then i test the libaec a bit to see if we could support higher block sizes (iqzip can and should but i won’t explain it in depth in this post because i don’t remember exactly how the headers save “illegal” values) and i will update documentation and maybe measurements because even me had problems to read the charts and i have written them
Is there a way to instruct iqzip to output to stdout instead of a file?
Use case: I want to stream the output to dropbox (via rclone) to lower the amount if space required on tmpfs. Currently, tmpfs needs to store original iq file + compressed iq file.
Edit: I checked the source code it it doesn’t seem to be possible at the moment. Adding support will probably requiring changing all the error messages printed throughout the library to std::cout to use std::cerr instead but I am not sure what side effect that would have.
iqzip supports streaming compression with the use of stream_compress function that maybe could work for you. Right now there is no example of using the stream compression feature but there is an open issue here.
As i am currently working on iqzip again it might be created but i cannot give a timeline for now.
