This drift is a result of not accurate TLE set. Currently we use TLE sets based on our observations as there aren’t any from space-track.org or other source. An alternative could be to use ISS TLE sets but it would be worse.
These generated TLE sets are not very accurate due to 2 issues, the first one is that the signal isn’t so frequent, so we use less points for adjusting and calculate the orbital data. The second is that the signal doesn’t have a peak (like dots etc) that would allow us to mark points, so we manually try to find the center of each signal.
By the way there is an experiment that we are going to conduct with the QUBIK satellites with different transmissions that will try to explore the 1st issue (the one with the signal peaks), for more info about it check https://libre.space/projects/qubik/.