Why? Why the hell not!

One of the things that has always fascinated me as a full-stack developer is the database tooling landscape. For some reason, it's always felt a bit archaic to me. Old UIs, new applications that die after a few seasons (looking at you, Azure Data Studio), plugins to let you perform functionality you'd think would be included in the IDE, and don't even get me started on import and export functionality.

Granted, it's probably my fault for always sticking to MSSQL instead of branching out to something shiny and modern like PostgreSQL (it's on the TODO list).

With that said, migrations have always fallen into the category of things I consider fundamental to the lifecycle of a database. Yet a lot of industry projects I've seen tend to follow very similar patterns. Either you use some form of proprietary software that your developers have to learn the ins and outs of, or you roll out your own novel in-house project that is built in a rush and then lost to time, with the original developer leaving and no one knowing how to work with it.

The third option I've seen is to go with a FOSS solution, but that often only lasts a few months before limitations are reached or frustrations arise due to a lack of support for certain features.

Now, I've been a fan of Redgate's Flyway for a while and have always wanted to get my hands on the paid-tier features (schema generation, drift reports, etc.). Unfortunately, it's not in the budget this year... or the next.

But also we need something ASAP because our current tool was written five years ago, needs an update, and we don't have the source code anymore.

So let's build something that will put proprietary software to shame!

The problem

What are we actually trying to solve?

At its core the problem seems simple. I have a database in state A and i want it to be in state B, then state C and so on. We need this process to be repeatable and predictable and ideally simple to use. Pretty straight forward, right?

Well... Specs change, we introduce bugs and sometimes we may incidentally run a script or two directly on a production server to fix urgent and forget about scripting that change out. Oops...

And what about stored procedures, views, functions, triggers and Schema binds? What about all the other objects we can create within SQL? Do we track them too? Or are we going to limit ourselves to just tables? (boring)

And what about data? There's plenty of times where we might need to drop a column, a table, an index or rename things? Should we stop the process? email a developer? require input for confirmation?

What if i want to keep track of a schema over time instead just migrations files. Keeping track of what the current schema is and being able to see it change over time without having to inspect every commit or migration folder would be good.

Where will we run? do we have users? or are we tooling in a CI/CD process? Maybe we need both incase something goes wrong?

What happens if developers create migrations touching the same object? How is it merged and in what order? Can we neatly tie code for a feature to a migration?

Our solution needs to account for all of these possibilities and or eventualities, while making sure the integrity of the database.

Requirements

With most of the problem established, lets build our specs and features

• Migration identity: Each migration needs a unique identity and predictable execution order.

• Migration tracking: We need to be able to see who or what ran a migration, what scripts did it run, how long it took and ideally what objects might have changed.

• Migration atomicity: Each migration should be atomic. It either all succeeds or the whole thing fails.

• Migration validation: We need a way to validate the SQL before it gets run, and some rules around what we consider to be a valid migration folder, migration file and schema change.

• Controlled execution: We need to be to avoid re-running migrations, select only pending, select by a date range and other selection methods.

• Failure handling: If a migration fails, we need useful error information and a clearly defined strategy for handling the resulting database state.

• Structured logging: We will need a lot of logging to make sure things are working correctly and so that we can trace why things go wrong.

• Configuration: To make some things easier, we should allow users to store connection string, behavioral settings and tweaks.

• Schema view: I want to be able to view a pure schema definition and its changes over time without searching every folder. Something like a git history.

Extra features

The above is what i would consider the foundation to the tool, this is what most FOSS and community editions essentially cover. Below are some of the more advanced features id like to get to:

• Schema drift detection: Compare two schemas and find differences.

• Schema script generation: Automatic migration file generation for comparisons or automatic flow.

• Rollback or recovery flow: Compare schema snapshots and generate a possible path back to an earlier state. This would be a recovery aid, not a guarantee of a safe rollback.

• UI: for developer flow integration

Some of these might turn out to be more complicated than they're worth, while others might become essential once we start using the tool in practice. I know I've probably missed some things too.

For now, the priority is getting a reliable migration engine working before worrying about all the shiny extras.

The libraries

I spent a bit of time doing research and found some useful libraries. Lucky for us Microsoft already provides much of the SQL Server tooling we need, so I don't have to write a parser or build a schema-comparison system from scratch. I have chosen to avoid this, you however may want to own the whole stack so i will list them below so you can look them up and see the complexity of these projects yourself.

The heavy lifters

• Microsoft.SqlServer.DacFx: the same engine behind SSDT and SqlPackage. A .sqlproj builds into a dacpac, which is a complete model of the database. DacFx can compare two dacpacs, or a dacpac against a live database, and spit out the script to get from one to the other. This is the library that gets us the "paid-tier" features: schema generation, drift reports and deploys.

• Microsoft.SqlServer.TransactSql.ScriptDom: Microsoft's T-SQL parser. It turns a script into a syntax tree you can walk with a visitor. We use it to validate SQL before it runs, work out which object a script touches, split generated scripts into one file per object, and safely rewrite scripts on the fly.

• git: not a library, but it's doing a lot of the work. The schema project lives in source control anyway, so instead of storing snapshots ourselves we let git be the schema history.

• git archive pulls the schema project out at any commit without touching your working tree, and git ls-tree tells us which migration folders existed at a given commit. That's what lets us generate a migration (or a rollback) by comparing two commits, no database needed.

Talking to the database

• Microsoft.Data.SqlClient: the official SQL Server driver.

• Dapper: a thin layer over ADO.NET for the journal queries. No ORM, no magic, just SQL I can read.

The CLI

• Spectre.Console.Cli: command parsing, help text, and the pretty tables, spinners and colours in the terminal.

• Serilog: structured logging to the console and to rolling log files, so we can trace why things went wrong after the fact.

• Microsoft.Extensions.*: dependency injection and layered JSON/environment configuration, the same stuff you'd use in an ASP.NET app.

Testing and the IDE

• xUnit v3: regression tests over Core.

• Microsoft.VisualStudio.Extensibility: the new out-of-process VS extension model, used for the tool window.

Project structure

Ill be using C# for this project. I wanted to use C++ or Rust but i couldn't find equivalent SQL graphing or parsing libraries or bindings.

Below is going to be my initial project structure:

• MigDb.Core: the main library that can implemented by whatever fronted we want to wrap it around.

• MigDb.CLI: so that it can be invoked without blocking in CI/CD.

• MigDb.Tests: for regression testing

• MigDb.VSExtension: This is a choice i know.

I want to explain the choice of using VS Extension before i go further. Our previous workflow used Git Hooks more precisely pre-commit hooks. You'd finish your commit and every single time regardless of whether you had database changes or not a form would be triggered where you were asked if database changes were made and you had to respond and fill out if you did.

I wanted to change the workflow to be more passive. If i knew i had database changes i could just click a button on my IDE and i would invoke it. I feel the responsibility should be on the developer to seek the tool instead of it being shoved in their face. So this little extension will be a window that can be pinned to the side or moved around until you need it.

MigDb
├── MigDb.CLI/
├── MigDb.Core/
├── MigDb.Test/
└── MigDb.VSExtension/

Conclusion

So that's the plan. We have a problem that looks simple until you start poking at it, a list of requirements that's probably longer than it should be, and a set of libraries that means we don't have to reinvent the hard parts. DacFx and ScriptDom do the heavy lifting on the SQL side, and git gives us the schema history for free.

The goal isn't to clone Flyway feature for feature. It's to build something small enough that the next developer can pick it up and understand it, but solid enough that we trust it with production databases.

In the next part we will dive deeper into our migration strategy and possible folder structures and how we can solve some of the problem questions.

If youve made it this far, thank you for taking the time to read this and ill catch you in the next one!

Context

Recently I've had the opportunity to work on a few long-form media projects. These projects required us to collect what I would consider a fair bit of media for what are relatively small theatre companies that wanted to try their hand at increasing their social media exposure for shows. To give you an idea, one project required multiple photo shoots, which resulted in hundreds of photos, and then required full recordings of a tech run and a final recording of the full show on closing night. The stills and video needed to be processed like normal, but the videos needed to be taken further to create trailers and social media content, and also cut into clips so that actors could show off their performances in their portfolios.

To give you a sense of how the data scaled, a single 3-hour show run in 4K ProRes 422 HQ comes to roughly 800GB. Multiply this by two because we have to do closing night too, and we have ~1.6TB of footage. The 600 Fuji RAW files then add another ~10GB.

After a few days of editing we can shrink this down to a few ~25GB renders, each with different colour grades and feels, and about 4GB of stills. This is great, that's a lot of space saved. Final project files end up being ~80GB give or take. Oh, and those renders are H.265. Yes, I know AV1 exists, but we have to stay compatible with client hardware. And nobody is building new PCs with the current price of hardware.

The problem

Typically my projects don't require 3 hours of straight recording, so I've been able to get away with juggling cloud services. If the final render is under 15GB I can just use Google Drive and let Google handle security and links. If not, maybe we need to split project files and put the stills on Dropbox to make it all work.

But free tiers only go so far, and growing them 500MB at a time with referrals isn't a plan. And what happens when a client asks for the raw footage?

If you've made it this far, thank you, and you can probably guess where this is going: how do I get these files to clients without mailing or lending out my 2TB NVMe?

The obvious answer is yet another monthly subscription for 1–2TB of cloud storage. The responsible answer might be Azure Storage, handing out SAS URLs to blobs, and wrapping everything in password-protected zips.

But that's boring, and I've got a Dell R730xd under my desk that's spent far too long as a footrest.

Part 1 - The setup

The server I received already came with ESXi installed and a few TBs that are managed through an internal RAID controller. I won't go into too much detail about the server itself, since the same result can be achieved on bare metal, a VM or containers. The main requirement is a place you can install OpenMediaVault (OMV) and a big storage pool. I was lucky enough that the server came with an HDD1 store of 12TB, and I've added HDD2 and HDD3 as supplementary storage for apps, config backups and other things.

I've chosen to separate the data by using HDD1 as the main storage location for FileBrowser Quantum and HDD2 for Docker components and any other configs, because I prefer the physical boundary. You don't have to do it this way. You may well only have one giant pool; you'll still be able to separate the locations using OMV's shared folder functionality.

Installing the OS

OMV has a lot of useful documentation, and the installation process is pretty straightforward with their installer. You burn the ISO to a USB and boot into the installer. If you're on Windows, Rufus is the tool I use; otherwise Etcher or dd can be used on Linux.

Through ESXi we can skip all that and upload the ISO image directly to one of our datastores, so that we can bind it to the VM during the creation step. We then run through the VM creation steps, choosing Linux as the guest OS family, Debian as the OS version, and binding the ISO.

Another great tweak is to enable the VM to start on boot, so if the server restarts or something happens the VM automagically comes back online.

You should now be able to follow the standard installation guide from OMV and end up with a similar-looking terminal displaying an IP that hosts the web frontend, and some credentials to use on the login page.

Install done.

Configuring OpenMediaVault

At this stage you should be able to log in to OMV through your browser and click around to familiarise yourself with the interface. I would also strongly recommend you check for updates under

System -> Update Management -> Updates

so that you have the latest security fixes and features. Oh, and make sure all your disks show up under

Storage -> Disks

We should now be able to start creating Shared Folders and set up SMB. I will create three Shared Folders:

1. DockerAppData - Where we store our Docker compose files

2. FileBrowserShares - Where we store what we want to share

3. Media1 - The original location of all our data (optional)

I have Media1 here because I was already using OMV for storing the giant RAW footage. Everything can work off a single share, but then all of it will be exposed to the container.

Now we just need to enable a way to move files over the network to OMV. Personally I picked SMB for this, since NFS permissions are IP-based and file ownership uses user IDs. Rsync is also a really good option for large files.

Head down to

Services -> SMB/CIFS -> Settings

and enable it there. You should now be able to create SMB shares under

Services -> SMB/CIFS -> Shares

Another optional step is to create a user or a group if you want to limit actions. You should only need one or two shares depending on your setup. Point them at the correct Shared Folders from the previous step:

1. FileBrowserShares - Where we drop our final render

2. Media1 - Where we store the raw footage

Add it to your file browser of choice or your fstab file and test.

Setting up OMV Docker

With OMV Shared Folders done and SMB set up, we now need to set up Docker and omv-extras, a set of community-maintained packages that add extra functionality.

In a terminal, either directly or through SSH, we need to run the install script.

wget -O - https://github.com/OpenMediaVault-Plugin-Developers/packages/raw/master/install | bash

You should now have a new option in your sidebar under

System -> omv-extras

Tick the Docker repo box and click the big blue Save button. Then run apt clean and wait until it says done.

With omv-extras enabled we should now be able to install openmediavault-compose, which will let us create our Docker containers from compose files. Install the plugin under System -> Plugins and refresh the page. If you get a connection lost message, it's because OMV restarts its web UI server; just refresh the browser.

In your sidebar you should now see a new

Services -> Compose -> Settings

page, which we will need to set up. The first thing you'll want to change is your compose files shared folder. Point this to your DockerAppData shared folder and hit Save. Then hit Install or Reinstall Docker.

DON'T FORGET TO HIT INSTALL OR REINSTALL DOCKER AT THE BOTTOM

Part 2 - Installing FileBrowser and cloudflared

At this point you should have OMV installed with SSH or direct terminal access, a few Shared Folders set up, SMB or some other way to sync files over the network, and omv-extras with Docker enabled and configured for compose files.

One hard requirement for this second part is a domain using Cloudflare DNS (the free plan is fine), which gives you access to Cloudflare Zero Trust tunnels. With that said, we're going to start this section by getting the tunnel set up before continuing with OMV. So go ahead and open a new tab or browser window.

We're going to start by creating a tunnel through the Cloudflare dashboard. You should be able to find the option under

Protect and Connect -> Zero Trust

This should redirect you to a new page with a different nav bar that should have

Networks -> Tunnels

Here you should be able to create a new tunnel of type Cloudflared, then for your connector make sure you select Docker.

IMPORTANT: Save the token displayed on this page, as it will be used in our compose file later.

Finally, at the publish step you should be able to set a subdomain and a path. I've chosen files as my subdomain, so I get files.[my-domain].com.

The other crucial step here, if you want to narrow access to just the generated links, is to set the Path field to ^/public/.*. This stops intruders from spamming your login page and scraping anything that's outside of FileBrowser's public folder. The tradeoff is that you also won't be able to access the login page or application externally. For me this isn't an issue, as 90% of the time I will be accessing it locally.

Lastly, the service section is the connection between our cloudflared container and FileBrowser Quantum container. We want to set the type to HTTP because FileBrowser serves plain HTTP and this hop never leaves the VM. Visitors still get HTTPS via Cloudflare, and the tunnel itself is encrypted. Set the URL to filebrowser:80 (we can use the container names here since Docker provides name resolution between containers; alternatively you could use [server-ip]:8080).

With the tunnel all set up, we can go back to OMV and start writing our compose file!

Final steps - Back to OMV

OK, we're nearly done. Hopefully you haven't run into any issues! With the tunnel all set up, we can write our configs and compose files and try to spin up our containers!

For this step, make sure you have that tunnel connector token handy.

First we're going to create the FileBrowser config file, so that when the container spins up it's ready to go. This typically lives under a /data folder in FileBrowser. We're also going to create a separate volume in our container called /deliveries that will be the publicly exposed directory.

Create a new folder under DockerAppData/filebrowser/data:

mkdir -p "/srv/dev-disk-by-uuid-[disk-uuid]/DockerAppData/filebrowser/data"

Now create a new config.yaml file inside this folder using a text editor such as nano or vim. Make sure you set your external URL correctly.

http:
  externalUrl: "https://files.[domain].com"
  trustProxyHeaders: true
  disableWebDAV: true

server:
  cacheDir: /home/filebrowser/data/tmp
  sources:
    - path: /deliveries
      name: Deliveries
      config:
        defaultEnabled: true

frontend:
  name: "MySiteName"
  description: "Client video delivery"

The path here needs to match the compose file's volume from our next step, so take note.

Lastly, we need to set the owner to the container's user so it can read and write to its database.

chown -R 1000:1000 "/srv/dev-disk-by-uuid-[disk-uuid]/DockerAppData/filebrowser/data"

Now for our compose file. Back in OMV's web UI we can create compose files directly by navigating to

Services -> Compose -> Files

You should get a form where you can copy and paste the following YAML, substituting your disk UUIDs and your token from Cloudflare's tunnel setup.

services:
  filebrowser:
    image: ghcr.io/gtsteffaniak/filebrowser:2.0-stable
    container_name: filebrowser
    restart: unless-stopped
    ports:
      - "8080:80"
    volumes:
      - /srv/dev-disk-by-uuid-[disk-uuid]/FileBrowserShares:/deliveries:ro
      - /srv/dev-disk-by-uuid-[disk-uuid]/DockerAppData/filebrowser/data:/home/filebrowser/data

  cloudflared:
    image: cloudflare/cloudflared:latest
    container_name: cloudflared
    restart: unless-stopped
    command: tunnel --no-autoupdate run --token [MyBigToken]
    depends_on:
      - filebrowser

You'll notice here that we mount /deliveries and /data from our previous step. The rest of the YAML is pretty self-explanatory. Hit that Save button.

The :ro at the end of the /deliveries volume mounts it read-only. FileBrowser can show and share your files, but it can never modify or delete them, even if something goes wrong. I've done this because I'm testing and didn't want to accidentally delete files. Once you're happy you can remove this

Back in the main list you should be able to see a new row containing the new FileBrowser compose file. Select the row and click the "Up" button in the toolbar. This will download the images and spin up our containers!

Part 3 - The fun part!

If everything went well, you should see a green "Up" status and the bound ports on the right. You can click the URLs or navigate to your server's IP on port 8080 and see a login screen!

You'll need to grab the credentials from the Docker logs, as they are auto-generated by default. You can do this in the terminal using:

docker logs filebrowser 2>&1 | grep -i password

Log in as admin using that password (you'll be asked to set a new one straight away), and you should be greeted with an empty dashboard! You can now prep some files and drag them through SMB, or whatever network method you chose before, into your FileBrowserShares shared folder and they will appear!

Back in Cloudflare Tunnels you should now be able to see a Healthy status and one active connector, which confirms everything is hooked up correctly.

Testing Links

The final step is to right-click a directory or file and hit Share to generate a link! Give it an expiration time and a password and test it internally. You can then send it to someone, or test it on your phone, to make sure it works externally.

You should also test that hitting files.[domain].com directly results in a 404, as we don't want direct traffic.

If you've reached the end, everything should be working, and your FileBrowser instance should be tunnelled and only allow publicly served content and links in and out.

Conclusion

This was quite a journey, I'm not going to lie. It was my first time working with Cloudflare and its many features, and frankly I'm not convinced it's the absolute best solution to my problem. Maybe blob storage in a cloud provider would have been way easier, who knows. But it was an interesting mini project that taught me a lot, and I'll be testing it out for a few months.

Ive heard Cloudflare could potentially throttle this solution since serving large video is restricted and has to be routed through their paid services like Stream or R2 but the writing on the wall is that it only applies to 'disproportionate' usage. it will be interesting to see if this applies to my case of a few files a month.

If anybody knows of any other similar solutions, let me know! I'd love to see how other people have solved this.

References

• https://www.openmediavault.org/install.html

• https://docs.openmediavault.org/en/stable/installation/index.html

• https://github.com/gtsteffaniak/filebrowser

• https://forum.openmediavault.org/index.php?thread/5549-omv-extras-org-plugin/

• https://www.cloudflare.com/service-specific-terms-application-services/

• https://blog.cloudflare.com/updated-tos/