Investigate iRODS for EMSL collaboration, PlanetMicrobe

[jeff-cohere]

We've been asked to consider using iRODS to facilitate a data sharing arrangement amongst EMSL, JGI and ARM as part of a BER Unified Data Framework RFI.

Turns out Globus has a connector for iRODS. Maybe we should consider using it for this collaboration?

[jeff-cohere]

Another angle on this: we're looking at data from PlanetMicrobe to move to the next-gen data lakehouse. PlanetMicrobe uses a data store backed by Cyverse, which also uses IRODS.

The Globus IRODS connector looks premium, which means it costs money that we probably don't have in our budget. Fortunately, there's a Golang IRODS client actively maintained by the Cyverse people. This seems like a good way to proceed.

[jeff-cohere]

Looking at the source code for the Globus iRODS connector client, I noticed that Globus uses GridFTP as its underlying transfer protocol. I don't know if this gives us anything practical, but it doesn't hurt to know.

[jeff-cohere]

Another observation: the KBase lakehouse doesn't use Globus at all. It's based on MinIO, which provides an Amazon S3-compatible API, so maybe it's more interesting to think about supporting an iRODS/S3. iRODS seems to have a client that presents an S3 API, and MinIO has released a Go client for S3-compatible storage.

Finally, it looks like CyVerse also has a Go module that provides an S3 API for an iRODS host.

Lots of puzzle pieces! This is encouraging.

[jeff-cohere]

I think we can layer the iRODS -> S3 proxy (third-to-last link in the above comment) on top of an S3 endpoint to get what we need here.

[mattldawson]

This containerized iRODS provider might be useful for testing

[jeff-cohere]

PlanetMicrobe data is stored at the path /iplant/home/shared/planetmicrobe/centrifuge within the CyVerse Data Store, reachable here. The /iplant/ root path is mentioned in the GoCommands data transfer documentation, which describes the iRODS (Golang) command line tool. The CyVerse iRODS host info is here. Maybe this is all the info we need?

[jeff-cohere]

The IRODS -> S3 proxy is a mess to build, but there is actually a Docker image on DockerHub. If we can configure this image to run in a container alongside the DTS, we can try out transfers between CyVerse and a minio instance.