LDAP-Backed Service Clusters

Please note that this post is in draft. It may contain entirely incorrect opinions.

1 May 2020

I’ve had a regular tech hacking/tinkering group that I’ve spent a large portion of my time around in recent years. Recently, we decided to migrate off the disparate services we had into one self-contained cluster that allowed us full control. The primary push for this in the end was storage: we sometimes have some pretty large files we want to pass between each other, and it’s annoying to find a decent way to do that. Every solution is bad. Ours isn’t much better, but it’s ours.


Storage and SMB

Storage was the prime driver for this little experiment so we wanted to make sure we got it right. We wanted a cheap-ish way to be able to store files of an arbitrary size for arbitrary lengths of time. Doing a bit of quick math, it appeared that using an S3-like object storage solution would work out cheaper than straight up cloud block storage, with the downside of increased write times (read times don’t matter, I’ll get to this later) and latency. The problem is, uploading to S3-likes is annoying at the best of times and impossible at the worst. So, we’ll have to abstract it! Given those files may be used by other services, we also have to make sure that abstraction is available to those services. Enter s3fs-fuse. It’s a clever little application that exposes an S3-like bucket as a volume that can be treated like a regular storage volume.

Setting this up alone is trivial, but we wanted to make it work with some hip new-age technologies that you might not have heard of yet.

I’m sure somebody will wet themselves in a fit of excitement while scrambling to tell me it’s insecure, but until Microsoft stop doing it with Azure, I won’t stop either.

The secret sauce to getting all of this working in a way that I didn’t have to worry about storage was through our S3-like. We created a simple Docker image that creates the s3fs volume then, by blindly twiddling some Docker knobs at random, found a way to easily expose that back to the host in a place where it would become useful later via Docker volume binds:

docker run -d --privileged --device=/dev/fuse --rm --name s3storage -v /s3storage:/data:rshared -e S3_BUCKET=my_bucket_name -e S3_URL="https://my_bucket_url" -e DFAULT_ACL=public-read -e S3_KEY=my_s3_key -e S3_SECRET="my_s3_secret" misbehaviouronline/s3storage:latest

There’s a few key pieces here:

This allowed us to get some access to our s3fs-fuse volume on the host. Now that this is treated as a regular mount on the host, it can be passed to other containers!

Given this all runs on a single host, we don’t need Kubernetes(!) Instead we rely on good old-fashioned Docker. However, we did experiment briefly with a Kubernetes cluster until I saw the invoice for what equated to about $15 in a virtual machine and some expansible storage and cancelled it. It’s possible to use a DaemonSet to expose that S3-like bucket as a volume on every host node in the cluster, providing access for any pods that require it.





Overall, we’re pretty happy with the setup. It’s proved resilient enough for us to mess about with. My one criticism of the setup is Matrix. I’m not a tremendous fan of how insane Synapse’s memory requirements are, and Synapse 2021 Remastered: Go Edition seems to be moving slowly enough that it can almost be declared a dead project. We looked at Mattermost and RocketChat, but neither of these fulfilled our needs and wants quite as well as Matrix.