Station 3259 Online but Failing Observations

Hello everyone,

Previously, our station 3259 was based on a Ubuntu 20.04 installation on a laptop with Satnogs client. However recently, the laptop had a hardware failure. In its place, we are trying out using a RPi following the installation method found here: https://wiki.satnogs.org/Raspberry_Pi

The hardware was tested in the lab with success and observations could be made, before moving it to where the station was located. At the station, the internet network is different from the lab. Interestingly, the station can connect to the network, appear as online https://network.satnogs.org/stations/3259/, but the observations are all failing.

A dump of the journalctl log showings tests done in lab and the next boot being at the station:

>Aug 30 14:29:57 raspberrypi systemd[1]: satnogs-client.service: Consumed 19.072s CPU time.
Aug 30 14:29:57 raspberrypi systemd[1]: Started SatNOGS client.
Aug 30 14:39:18 raspberrypi satnogs-client[9081]: Found 1 device(s):
Aug 30 14:39:18 raspberrypi satnogs-client[9081]:   0:  Realtek, RTL2838UHIDIR, SN: 00000001
Aug 30 14:39:18 raspberrypi satnogs-client[9081]: Using device 0: Generic RTL2832U OEM
Aug 30 14:39:18 raspberrypi satnogs-client[9081]: Found Rafael Micro R820T tuner
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: rot_init called
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: initrots4_dummy: _init called
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: rot_register (1)
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: rot_register (2)
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: dummy_rot_init called
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: rot_open called
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: dummy_rot_open called
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: rot_get_position called
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: dummy_rot_get_position called
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: rot_get_position: got az=0.00, el=0.00
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: rot_set_position called az=20.10 el=5.23
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: rot_set_position: south_zero=0
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: dummy_rot_set_position called: 20.10 5.23
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: rig_init: rig does not have rx_range!!
Aug 30 14:39:19 raspberrypi satnogs-client[8949]: network_open: hoststr=127.0.0.1, portstr=4532
Aug 30 14:39:22 raspberrypi satnogs-client[9117]: Found Rafael Micro R820T tuner
Aug 30 14:39:22 raspberrypi satnogs-client[9117]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Aug 30 14:39:23 raspberrypi satnogs-client[9117]: Found Rafael Micro R820T tuner
Aug 30 14:39:23 raspberrypi satnogs-client[9117]: [R82XX] PLL not locked!
Aug 30 14:39:23 raspberrypi satnogs-client[9117]: [INFO] Using format CF32.
Aug 30 14:39:52 raspberrypi satnogs-client[9117]: /var/lib/satnogs/.gnuradio/prefs/vmcircbuf_default_factory: No such file or directory
Aug 30 14:39:52 raspberrypi satnogs-client[9117]: vmcircbuf_createfilemapping: createfilemapping is not available
Aug 30 14:39:52 raspberrypi satnogs-client[9117]: gr::vmcircbuf_sysv_shm: shmat (2): Invalid argument
Aug 30 14:39:52 raspberrypi satnogs-client[9117]: gr::vmcircbuf_sysv_shm: shmat (2): Invalid argument
Aug 30 14:39:52 raspberrypi satnogs-client[9117]: gr::vmcircbuf_sysv_shm: shmat (2): Invalid argument
Aug 30 14:39:52 raspberrypi satnogs-client[9117]: Allocating 15 zero-copy buffers
Aug 30 14:48:28 raspberrypi satnogs-client[8949]: netrigctl_close: done status=Command completed successfully
Aug 30 14:48:30 raspberrypi satnogs-client[9257]: Found 1 device(s):
Aug 30 14:48:30 raspberrypi satnogs-client[9257]:   0:  Realtek, RTL2838UHIDIR, SN: 00000001
Aug 30 14:48:30 raspberrypi satnogs-client[9257]: Using device 0: Generic RTL2832U OEM
Aug 30 14:48:31 raspberrypi satnogs-client[9257]: Found Rafael Micro R820T tuner
Aug 30 14:51:25 raspberrypi satnogs-client[9305]: Found 1 device(s):
Aug 30 14:51:25 raspberrypi satnogs-client[9305]:   0:  Realtek, RTL2838UHIDIR, SN: 00000001
Aug 30 14:51:25 raspberrypi satnogs-client[9305]: Using device 0: Generic RTL2832U OEM
Aug 30 14:51:25 raspberrypi satnogs-client[9305]: Found Rafael Micro R820T tuner
Aug 30 14:51:26 raspberrypi satnogs-client[8949]: rig_init: rig does not have rx_range!!
Aug 30 14:51:26 raspberrypi satnogs-client[8949]: network_open: hoststr=127.0.0.1, portstr=4532
Aug 30 14:51:29 raspberrypi satnogs-client[9314]: Found Rafael Micro R820T tuner
Aug 30 14:51:29 raspberrypi satnogs-client[9314]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Aug 30 14:51:30 raspberrypi satnogs-client[9314]: Found Rafael Micro R820T tuner
Aug 30 14:51:30 raspberrypi satnogs-client[9314]: [R82XX] PLL not locked!
Aug 30 14:51:30 raspberrypi satnogs-client[9314]: [INFO] Using format CF32.
Aug 30 14:51:31 raspberrypi satnogs-client[9314]: Allocating 15 zero-copy buffers
Aug 30 15:01:03 raspberrypi satnogs-client[8949]: netrigctl_close: done status=Command completed successfully
Aug 30 15:01:04 raspberrypi satnogs-client[9492]: Found 1 device(s):
Aug 30 15:01:04 raspberrypi satnogs-client[9492]:   0:  Realtek, RTL2838UHIDIR, SN: 00000001
Aug 30 15:01:05 raspberrypi satnogs-client[9492]: Using device 0: Generic RTL2832U OEM
Aug 30 15:01:05 raspberrypi satnogs-client[9492]: Found Rafael Micro R820T tuner
Aug 30 15:31:19 raspberrypi systemd[1]: Stopping SatNOGS client...
Aug 30 15:31:19 raspberrypi systemd[1]: satnogs-client.service: Succeeded.
Aug 30 15:31:19 raspberrypi systemd[1]: Stopped SatNOGS client.
Aug 30 15:31:19 raspberrypi systemd[1]: satnogs-client.service: Consumed 18min 56.597s CPU time.
-- Boot 36f9a3b1d32e4a9b87fddad18af3b577 --
Aug 30 15:31:24 raspberrypi systemd[1]: Started SatNOGS client.
Aug 30 15:31:35 raspberrypi satnogs-client[387]: satnogsclient.scheduler.tasks - ERROR - An error occurred trying to GET observation jobs from network
Aug 30 15:31:35 raspberrypi satnogs-client[387]: Traceback (most recent call last):
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     conn = connection.create_connection(
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/urllib3/util/connection.py", line 72, in create_connection
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
Aug 30 15:31:35 raspberrypi satnogs-client[387]: socket.gaierror: [Errno -3] Temporary failure in name resolution
Aug 30 15:31:35 raspberrypi satnogs-client[387]: During handling of the above exception, another exception occurred:
Aug 30 15:31:35 raspberrypi satnogs-client[387]: Traceback (most recent call last):
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     httplib_response = self._make_request(
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     self._validate_conn(conn)
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     conn.connect()
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     self.sock = conn = self._new_conn()
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     raise NewConnectionError(
Aug 30 15:31:35 raspberrypi satnogs-client[387]: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7232a2b0>: Failed to establish a new connection: [Errno -3] >
Aug 30 15:31:35 raspberrypi satnogs-client[387]: During handling of the above exception, another exception occurred:
Aug 30 15:31:35 raspberrypi satnogs-client[387]: Traceback (most recent call last):
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     resp = conn.urlopen(
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     retries = retries.increment(
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
Aug 30 15:31:35 raspberrypi satnogs-client[387]: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='network.satnogs.org', port=443): Max retries exceeded with url: /api/jobs/?ground_stati>
Aug 30 15:31:35 raspberrypi satnogs-client[387]: During handling of the above exception, another exception occurred:
Aug 30 15:31:35 raspberrypi satnogs-client[387]: Traceback (most recent call last):
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/satnogsclient/scheduler/tasks.py", line 191, in get_jobs
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     response = requests.get(url,
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/requests/api.py", line 73, in get
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     return request("get", url, params=params, **kwargs)
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/requests/api.py", line 59, in request
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     return session.request(method=method, url=url, **kwargs)
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     resp = self.send(prep, **send_kwargs)
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     r = adapter.send(request, **kwargs)
Aug 30 15:31:35 raspberrypi satnogs-client[387]:   File "/var/lib/satnogs/lib/python3.9/site-packages/requests/adapters.py", line 565, in send
Aug 30 15:31:35 raspberrypi satnogs-client[387]:     raise ConnectionError(e, request=request)
Aug 30 15:31:35 raspberrypi satnogs-client[387]: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='network.satnogs.org', port=443): Max retries exceeded with url: /api/jobs/?ground_st>
Aug 30 16:10:29 raspberrypi systemd[1]: Stopping SatNOGS client...
Aug 30 16:10:29 raspberrypi systemd[1]: satnogs-client.service: Succeeded.
Aug 30 16:10:29 raspberrypi systemd[1]: Stopped SatNOGS client.
Aug 30 16:10:29 raspberrypi systemd[1]: satnogs-client.service: Consumed 12.494s CPU time.
-- Boot 3348f03062f5441aaacfade8328cb64f --

Running a few network commands results in this:

root@raspberrypi:~# host network.satnogs.org
network.satnogs.org is an alias for srv01.libre.space.
srv01.libre.space has address 94.130.162.100
srv01.libre.space has IPv6 address 2a01:4f8:13b:29db::1
root@raspberrypi:~# ping network.satnogs.org
PING srv01.libre.space (94.130.162.100) 56(84) bytes of data.
64 bytes from srv01.libre.space (94.130.162.100): icmp_seq=1 ttl=46 time=175 ms
64 bytes from srv01.libre.space (94.130.162.100): icmp_seq=2 ttl=46 time=175 ms
64 bytes from srv01.libre.space (94.130.162.100): icmp_seq=3 ttl=46 time=177 ms
^C
--- srv01.libre.space ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 174.935/175.674/176.764/0.786 ms
root@raspberrypi:~# curl -v https://network.satnogs.org
*   Trying 94.130.162.100:443...
* Connected to network.satnogs.org (94.130.162.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=network.satnogs.org
*  start date: Jul  7 03:00:13 2024 GMT
*  expire date: Oct  5 03:00:12 2024 GMT
*  subjectAltName: host "network.satnogs.org" matched cert's "network.satnogs.org"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x14add70)
> GET / HTTP/2
> Host: network.satnogs.org
> user-agent: curl/7.74.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Fri, 30 Aug 2024 15:38:05 GMT
< content-type: text/html; charset=utf-8
< content-length: 6502
< content-security-policy: frame-src blob:; child-src blob:; style-src 'self' 'unsafe-inline' https://network-satnogs.freetls.fastly.net; default-src 'self' https://*.mapbox.com https://archive.org https://*.archive.org https://*.wasabisys.com https://network-satnogs.freetls.fastly.net; img-src 'self' https://*.gravatar.com https://*.mapbox.com https://*.satnogs.org https://*.wasabisys.com https://*.google-analytics.com data: blob: https://db-satnogs.freetls.fastly.net https://network-satnogs.freetls.fastly.net; worker-src blob:; script-src 'self' https://*.google-analytics.com 'unsafe-eval' https://network-satnogs.freetls.fastly.net
< strict-transport-security: max-age=31536000; includeSubDomains
< x-content-type-options: nosniff
< referrer-policy: same-origin
< cross-origin-opener-policy: same-origin
< x-frame-options: DENY
< vary: Cookie
< x-download-options: noopen
< x-permitted-cross-domain-policies: none
< x-xss-protection: 1; mode=block
< 





<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>SatNOGS Network - Home</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="https://network-satnogs.freetls.fastly.net/static/CACHE/css/output.7ab6a28e2163.css" type="text/css">

    <link rel="shortcut icon" href="https://network-satnogs.freetls.fastly.net/static/favicon.ico">

  </head>

  <body class="home">
    
    
  <div id="map" data-mapboxid="pierros.jbf6la1j" data-mapboxtoken="pk.eyJ1IjoibHNmIiwiYSI6ImNscGxvZmN6bTAyNGUyaXA2cHQ5cHFsYWMifQ.AhsGdCjsu2mVB6Zcb_4U8A" data-stations="/stations_all/"></div>


    <div class="container">
      <nav class="navbar navbar-light navbar-main navbar-expand-lg bg-light border" role="navigation">
        <button type="button" class="navbar-toggler" data-toggle="collapse" data-target="#navbarSupportedContent"
                aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
          <span class="navbar-toggler-icon"></span>
        </button>
        <a class="navbar-brand" href="
                                        /
                                      ">
          <img class="navbar-logo" src="https://network-satnogs.freetls.fastly.net/static/img/satnogs_net.png" alt="SatNOGS net">
        </a>

        <!-- Collect the nav links, forms, and other content for toggling -->
        <div class="collapse navbar-collapse" id="navbarSupportedContent">
          <ul class="navbar-nav mr-auto">
            <li class="nav-item active">
              <a class="nav-link" href="/">Home</a>
            </li>
            <li class="nav-item None">
              <a class="nav-link" href="/about/">About</a>
            </li>
            <li class="nav-item dropdown None None">
              <a class="nav-link dropdown-toggle" href="#" id="observationsDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
              Observations
              </a>
              <div class="dropdown-menu bg-light p-0" aria-labelledby="observationsDropdown">
                <a class="dropdown-item btn-sm None" href="/observations/" role="button">Browse</a>
                <div class="dropdown-divider m-0"></div>
                <a class="dropdown-item btn-sm None" href="/observations/new/" role="button">
                  Schedule
                </a>
              </div>
            </li>
            <li class="nav-item None">
              <a class="nav-link" href="/stations/">Ground Stations</a>
            </li>
            <li>
              <a class="nav-link d-none d-lg-block" href="https://community.libre.space/" target="_blank">Community</a>
            </li>
            <li>
              <a class="nav-link d-none d-lg-block" href="https://wiki.satnogs.org/" target="_blank">Wiki</a>
            </li>
          </ul>
          <span class='nav-link navbar-text d-none d-lg-block' id="current_utc">--:-- UTC</span>
          <ul class="navbar-nav">
              
                <li class="nav-item"><a class="nav-link" href="/login/auth0">Sign Up / Log In</a></li>

              
          </ul>
        </div><!-- /.navbar-collapse -->
      </nav>
    </div>

    <div class="container">

      <div id="alert-messages" class="row messages">
        
      </div>

      
  
    <div class="call2action">
      <div class="card">
        <div class="card-body">
          <h2 class="text-center">
            Crowd-sourced satellite operations
            <button type="button"
                    class="btn btn-primary"
                    data-toggle="modal"
                    data-target="#joinModal">
              Join us!
            </button>
          </h2>
        </div>
      </div>
    </div>
  

  <!-- Join Modal -->
  <div class="modal fade" id="joinModal" tabindex="-1" role="dialog" aria-labelledby="JoinModalTitle" aria-hidden="true">
    <div class="modal-dialog">
      <div class="modal-content">
        <div class="modal-header">
          <h3 class="modal-title" id="JoinModalTitle">Join SatNGOS Network</h3>
          <button type="button" class="close" data-dismiss="modal" aria-label="Close">
            <span aria-hidden="true">&times;</span>
          </button>
        </div>
        <div class="modal-body">
          <h3>SatNOGS Network</h3>
          SatNOGS Network is a global management interface to facilitate multiple ground station
          operations remotely. An observer is able to take advantage of the full network of
          SatNOGS ground stations around the world.
          <h3>Observations</h3>
          The network is open to anyone. Any observer is able to utilize all available Ground Stations
          and communicate with Satellites. All observations results are public
          and all data are distributed freely under the
          <a href="https://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative Commons Atribution-Share Alike</a> license.
          <h3>Ground Stations</h3>
          Whether you own satellite ground station equipment or you want
          to build one you can head to
          <a href="https://satnogs.org" target="_blank">SatNOGS Project site</a> or our
          <a href="https://wiki.satnogs.org" target="_blank">wiki</a>
          to get up to date documentation and info on how to build a SatNOGS ground station (still under development).
          <h3>Coding</h3>
          Fluent in Python and/or JS? We need you to help with satnogs-network web application. Take a look to our
          <a href="https://gitlab.com/librespacefoundation/satnogs/satnogs-network" target="_blank">source code</a> or visit
          our <a href="https://gitlab.com/librespacefoundation/satnogs/satnogs-network/issues" target="_blank">issues tracker</a>
          to start filling bugs, feature requests or code fixes for existing ones.
          Detailed contribution and development documentation is also availble at
          <a href="https://docs.satnogs.org/projects/satnogs-network/en/stable/" target="_blank">docs.satnogs.org</a>.
        </div>
        <div class="modal-footer">
          <button type="button" class="btn btn-outline-dark" data-dismiss="modal">Close</button>
        </div>
      </div>
    </div>
  </div>


    </div> <!-- /container -->

    

    

    <script src="https://network-satnogs.freetls.fastly.net/static/CACHE/js/output.19e5b1450501.js"></script>
</body>
</html>
* Connection #0 to host network.satnogs.org left intact

Quite clueless now as I am very sure that the RPi is on the internet, as I can remotely access it and the station’s status is online. Thanks in advance for debugging guidance!

Turns out it was a network firewall issue that caused this behavior. Changing to a different network solved the problem (:

3 Likes