LDAP-Backed Service Clusters

Please note that this post is in draft. It may contain entirely incorrect opinions.

1 May 2020

I’ve had a regular tech hacking/tinkering group that I’ve spent a large portion of my time around in recent years. Recently, we decided to migrate off the disparate services we had into one self-contained cluster that allowed us full control. The primary push for this in the end was storage: we sometimes have some pretty large files we want to pass between each other, and it’s annoying to find a decent way to do that. Every solution is bad. Ours isn’t much better, but it’s ours.

Players

• Storage: A mix of SMB, FUSE, and an S3-like.
• Authentication: LDAP. Yes, that LDAP. We ended up using openldap as our central authentication server.
• Version Control: Gitea. It’s the best self-hosted Git solution available at the moment. We considered self-hosting sr.ht, but Gitea is far easier to set up.
• Communications: Matrix. We run our own homeserver with varying permission levels for federated access to channels.

Storage and SMB

Storage was the prime driver for this little experiment so we wanted to make sure we got it right. We wanted a cheap-ish way to be able to store files of an arbitrary size for arbitrary lengths of time. Doing a bit of quick math, it appeared that using an S3-like object storage solution would work out cheaper than straight up cloud block storage, with the downside of increased write times (read times don’t matter, I’ll get to this later) and latency. The problem is, uploading to S3-likes is annoying at the best of times and impossible at the worst. So, we’ll have to abstract it! Given those files may be used by other services, we also have to make sure that abstraction is available to those services. Enter s3fs-fuse. It’s a clever little application that exposes an S3-like bucket as a volume that can be treated like a regular storage volume.

Setting this up alone is trivial, but we wanted to make it work with some hip new-age technologies that you might not have heard of yet.

I’m sure somebody will wet themselves in a fit of excitement while scrambling to tell me it’s insecure, but until Microsoft stop doing it with Azure, I won’t stop either.

The secret sauce to getting all of this working in a way that I didn’t have to worry about storage was through our S3-like. We created a simple Docker image that creates the s3fs volume then, by blindly twiddling some Docker knobs at random, found a way to easily expose that back to the host in a place where it would become useful later via Docker volume binds:

docker run -d --privileged --device=/dev/fuse --rm --name s3storage -v /s3storage:/data:rshared -e S3_BUCKET=my_bucket_name -e S3_URL="https://my_bucket_url" -e DFAULT_ACL=public-read -e S3_KEY=my_s3_key -e S3_SECRET="my_s3_secret" misbehaviouronline/s3storage:latest

There’s a few key pieces here:

• --privileged allows the container to access MKNOD, which is required for FUSE.
• --device=/dev/fuse passes the host’s /dev/fuse device to the container, which is required to make sure the volume actually functions properly on the host.
• -v /s3storage:/data:rshared allows two-way propagation of this bind.rshared is described in the Dockermentation as The same as shared, but the propagation also extends to and from mount points nested within any of the original or replica mount points.

This allowed us to get some access to our s3fs-fuse volume on the host. Now that this is treated as a regular mount on the host, it can be passed to other containers!

• We pass it through to the Nginx container for homedir emulation. For instance, files from https://misbehaviour.online/~tecuane are served via the S3-like service. A to-do here is to actively redirect requests to those from the S3-like itself.
• We pass it through to the Samba container to allow users to “mount” their S3-likes as SMB shares.
• We pass it through to the Matrix container to serve assets on that storage, instead of serving it off the local disk.

Given this all runs on a single host, we don’t need Kubernetes(!) Instead we rely on good old-fashioned Docker. However, we did experiment briefly with a Kubernetes cluster until I saw the invoice for what equated to about $15 in a virtual machine and some expansible storage and cancelled it. It’s possible to use a DaemonSet to expose that S3-like bucket as a volume on every host node in the cluster, providing access for any pods that require it.

LDAP

Gitea

Matrix

Overall

Overall, we’re pretty happy with the setup. It’s proved resilient enough for us to mess about with. My one criticism of the setup is Matrix. I’m not a tremendous fan of how insane Synapse’s memory requirements are, and Synapse 2021 Remastered: Go Edition seems to be moving slowly enough that it can almost be declared a dead project. We looked at Mattermost and RocketChat, but neither of these fulfilled our needs and wants quite as well as Matrix.