ATLAS Open Data notebook can't reach opendata.mwt2.org from Binder: Network unreachable (https) / redirect limit reached (xrootd)

Binder
1

Hi Binder team!

We run an ATLAS (LHC particle physics) Open Data facility at the University of Chicago (opendata.mwt2.org), and we’re trying to get it working as a data source inside the standard ATLAS Open Data teaching notebooks, e.g.: https://mybinder.org/v2/gh/atlas-outreach-data-tools/notebooks-collection-opendata/master?labpath=13-TeV-examples%2Fuproot_python%2FHyyAnalysis.ipynb

The notebook reads .root files over xrootd/https from our dCache instance. It now works fine on Google Colab (we tracked down and fixed a certificate-chain issue on our end), but on mybinder(dot)org we’re stuck on two different errors depending on protocol.

With https://opendata.mwt2.org:2881/... (WebDAV/HTTPS door):

OSError: Multiple exceptions: [Errno 101] Network is unreachable, [Errno 111] Connect call failed ('192.170.241.218', 2881)
ClientConnectorError: Cannot connect to host opendata.mwt2.org:2881 ssl:default [None]

With root://opendata.mwt2.org:1094/... (xrootd):

OSError: File did not open properly: [FATAL] Redirect limit has been reached

The xrootd case gets further. We see that it connects and starts redirecting, but never lands on a pool it can actually read from. The https case looks like the outbound connection from the Binder container to port 2881 never gets out at all, which made us wonder if this is a similar situation to the CERN EOS case we found:

i.e. an explicit allowlist/egress exception was needed for CERN’s host/ports, and something equivalent may be needed for opendata.mwt2.org (ports 1094/xrootd and 2881/https, both open to the world from our side).

A few indicators that leads us to believe the issue is on the binder side, though we haven’t been able to confirm any of this from within MyBinder’s own environment:

  • The ports are open externally (we’ve verified with openssl s_client and curl from outside our network).
  • It isn’t the certificate chain issue we hit with Colab (that’s fixed, and doesn’t explain the network-unreachable or redirect-limit errors anyway).
  • It works from Colab, so the failure looks specific to Binder’s network environment rather than the client code or notebook itself — but we’re happy to be pointed at something on our end if we’re missing it.

Would it be possible for someone to take a look at whether opendata.mwt2.org needs a similar egress exception to what CERN EOS got, and/or whether there’s a known cause for the xrootd redirect-limit behavior in the Binder network setup? We’d really like to offer MWT2 as a supported Open Data source for people running these notebooks on Binder.

Many thanks!

Best, Giordon (obo the admin team)

2

Ports 1094 and 1095 are allowed for xroot:

What’s port 2881 used for?

3

Hi @manics - port 2881 is used for WebDAV (davs). Open Data is expected to be available/accessible over xrootd and davs.

4

We’re reluctant to open more ports on mybinder.org. If anything we’d like to reduce outgoing traffic due to abuse, which unfortunately is inevitable given it’s a free unauthenticated compute service. Can you run your webdav service on a standard https port instead?

5

That is a reasonable protection to limit the egress. Unfortunately for us, the majority of xrootd servers that support davs / WebDAV all generally use port 2881. We can go back to our colleagues at CERN and let them know that mybinder will not work for Open Data (https://opendata.cern.ch/) with davs and they will need to find an alternative. I’m not sure we can swap to a different port without breaking other things that need that port.

6

Hi @manics - in the meantime, since port 1095 is open, we will use that port since we’re using 1094 for root. If in the future this gets changed, we can swap.