Network is terribly slow

This is my experience.

1 Like

Good news! (or bad news, depending on how you look at it)

Today I went into work early (5:45am local time), and in addition to setting off the alarm and being greeted by a representative of the local police (he was nice - I didn’t get tazed, bro), I was able to vet 60+ observations in about 3 minutes (a page load took 8 seconds for a single observation, which is still pretty slow - but the link bomb successfully loaded all 50 observations in less than a minute).

So the bad news is this could be related to my ISP or routing at home. But I can vet at work on my breaks - so that’s good enough for me. I appreciate everybody’s help with this - and will continue to try and figure out a solution for my home ISP issue.

–Roy
K3RLD

3 Likes

I think you’re right about the ISP being the issue. I wget'd the file you listed above, which you got at 64.9 KB/s, from some other locations. Here’s results. Note all but one connection is in a data center.

Dallas            .0343   MB/s (34.3 KB/s)
California       1.71     MB/s
Japan            1.02     MB/s
Singapore        1.62     MB/s
Germany         28.4      MB/s    
Newark           3.38     MB/s    
Atlanta          2.44     MB/s    
France           9.88     MB/s    
NL              22.0      MB/s
Romania          6.68     MB/s    
Iceland          4.30     MB/s
Denver           2.08     MB/s
Colorado (home)   .722    MB/s (722 KB/s)

All of these speeds look great, except with one outlier at the Dallas data center. It was really slow, like the connection you’re experiencing. Here’s the Dallas traceroute:

traceroute to network.satnogs.org (94.130.162.100), 30 hops max, 60 byte packets
 1  45.79.12.201 (45.79.12.201)  0.568 ms 45.79.12.202 (45.79.12.202)  1.790 ms  3.374 ms
 2  45.79.12.2 (45.79.12.2)  0.367 ms 45.79.12.6 (45.79.12.6)  0.765 ms 45.79.12.4 (45.79.12.4)  0.680 ms
 3  45.79.12.9 (45.79.12.9)  0.795 ms  0.754 ms dls-b22-link.telia.net (62.115.172.134)  0.759 ms
 4  atl-b22-link.telia.net (62.115.120.112)  28.491 ms dls-b22-link.telia.net (62.115.172.134)  0.755 ms  0.806 ms
 5  ash-bb3-link.telia.net (62.115.125.190)  116.820 ms * atl-b22-link.telia.net (62.115.120.112)  27.899 ms
 6  ash-bb2-link.telia.net (62.115.125.129)  123.867 ms prs-bb4-link.telia.net (62.115.122.158)  119.501 ms  119.346 ms
 7  prs-bb3-link.telia.net (62.115.112.243)  124.214 ms ffm-bb3-link.telia.net (62.115.123.12)  123.884 ms  123.794 ms
 8  ffm-b5-link.telia.net (62.115.114.89)  132.134 ms  131.906 ms ffm-bb2-link.telia.net (62.115.114.99)  124.031 ms
 9  ffm-b5-link.telia.net (62.115.114.89)  131.670 ms hetzner-ic-326013-ffm-b4.c.telia.net (213.248.70.3)  132.525 ms  126.166 ms
10  hetzner-ic-326013-ffm-b4.c.telia.net (213.248.70.3)  131.627 ms  126.044 ms  125.889 ms
11  ex9k2.dc4.fsn1.hetzner.com (213.239.229.134)  136.352 ms ex9k2.dc4.fsn1.hetzner.com (213.239.229.130)  128.794 ms core21.fsn1.hetzner.com (213.239.224.241)  128.846 ms
12  ex9k2.dc4.fsn1.hetzner.com (213.239.229.130)  129.054 ms srv01.libre.space (94.130.162.100)  136.349 ms !X  128.450 ms !X

I sum, I think this is just a routing issue, not the SatNOGS servers themselves.

Happy hacking,

-Jeff

Edit: I’ll note that I did the Dallas test a few times and got similar slow results. I re-ran it now and it is going at 1.64MB/sec, so it was just transient Internet routing issues there, apparently.

2 Likes

I’m 4 Mbits link, firefox reacted much better than chrome for opening the 50 tabs.

1 Like

I’ll revisit this tonight, hopefully. Meanwhile, I vetted like 1000 observations today (ISS SSTV) with little issue while I had some down time at work.

2 Likes

Here is an observation load time from my work. Also a tracert from my work.


2 Likes

Unfortunately, looks very similar to mine. Of course, I really have no clue what I’m looking at so my opinion is only worth the photons it’s printed with.

I just tried single observation here at home and load time was typical - over 30 seconds. Looks like I’ll be doing all my vetting from work. :frowning:

Funny, I just sat down at my shack computer and on a lark clicked the “load all” link and to my amazement, all 50 links loaded in about 30 seconds! Tried it for the 2nd page of links (only 10 observations) and only 2 observations loaded. The rest timed out.

[edit] Just tried Firefox… it actually seems slower, if that is even possible.

1 Like

As a further updat for any who are having similar problems, with Firefox you can modify the parameter “network.http.connection.timeout” (or something like that) in the “about:config” page to a very large time (I believe 600 seconds is the limit). Then you can just hit the link bomb and go to the bathroom or make a sandwich or something and the slow loading won’t cause the waterfall image to time out.

Just tried it an it indeed works and is a good solution for my poor network reliability.

2 Likes

Any other clues?? I apologize, I’ve had a few unfortunate life events get in the way of my time here but the fact that you and @KD9KCK both experience a slow waterfall load is bizarre, and something I’d like to see fixed for sure.

Are you using a proxy at all? vpn?

1 Like

Corey - No proxy, no VPN. Connection is via Verizon Fios home service. Can’t explain this one at all - as sometimes (like just now), I can load 35 observation in a second. Actually the last few days have been good. ??? I’ve got work arounds, so I’m happy.

@cshields I don’t think this is an issue with the servers, but with the network route to the servers. There’s a bottleneck in between not controlled by SatNOGS. The typical way to get around this is to set up a CDN or something similar. I’m NOT suggesting A*azon or any of the others.

I think @k3rld could try using VPNs to get a different route. VPNs typically slow things down a bit, but in this case may speed it up by taking a new route and bypassing the slow router.

Also, re-reading @k3rld: " just sat down at my shack computer and on a lark clicked the “load all” link and to my amazement, all 50 links loaded in about 30 seconds! Tried it for the 2nd page of links (only 10 observations) and only 2 observations loaded"

Lots of times providers will give a burst of speed at the beginning, then it tapers off. I think the motivation is to get videos playing quickly. You may be seeing that above.

I’ll also note, that from the traceroute @k3rld did and the traceroute I did from Dallas (that was slow for a bit), both went through this router: ash-bb3-link.telia.net

Blame telia :wink:

I see Linode New Jersey goes to Hetzner much more directly. If you can proxy through them, that would speed things up, I bet.

1 Like

@k3rld watch this router:

https://www.dslreports.com/routerwatch/ash-bb3-link.telia.net

I notice there’s a lot of red on there today.

I don’t use a proxy or VPN. The pervious screen caps were from my work that uses AT&T Fiber.

At home I have AT&T ADLS2+ (Basically DLS) with terrible speeds. (3 down 0.7 up) This is why my map loads so slowly for users.

Oddly its also slow loading the waterfalls on my phone. Which is also AT&T. I am wondering if AT&T is having routing problems for me.

Here is my load of an observation at home.

ANd tracert


The -w 10000 is used to set a higher timeout. (Seems not high enough for that one hop.) the -4 is for IPv4 (Tried to force IPv6 but it can;t resolve the address to one.)

I have set up a SOCKS v5 proxy, which should allow you to login and view observations. If you want to try this proxy, which is at Linode in New Jersey, set up your browser thusly:

Here’s what it looks like in Firefox:

The following domains are proxied: : satnogs.org, libre.space, auth0.com, gitlab-static.com, mapbox.com, spacecruft.org. You won’t be able to surf to anywhere else while you have the proxy enabled.

Note, this is just a temporary test server, not permanent. Info here:

5 Likes

I notice those routes reporting trouble always have telia in the path (at least that I’ve seen so far).

2 Likes

Can someone from US check also other websites hosted in Hetzner? Like hetzner.com for example. We will be opening an issue with them.

1 Like

I just dug around hetzner.com with no issues at all. But then I tried satnogs network and loaded 30 observations in 5 seconds. So whatever is happening to me is intermittent.

2 Likes