I expect the interested audience for this post to be pretty small. I encourage you to stop reading as soon as that seems right to you.

I just finished writing NixOS modules + packages for flexiprocity, e.g. config.nix. I haven't quite switched the "live" version over to the new host, but I think the only thing left is the data migration, which I'm expecting to not be a big deal.

My aim has been to have all the configuration in code, so that I don't have to remember what I did to set up the box, and I can easily recreate the box (e.g. when I move host, as I am doing). A few sore points:

• I'm using pulumi's hcloud provider for deploying the server itself. You can boot servers directly into Ubuntu, but they don't prepackage images for NixOS. However you can mount a NixOS ISO and boot from there. (Why do they natively support the ISO image but not a normal OS image? I don't know). The process of installing NixOS from ISO is pretty manual: I have to configure the server to mount the ISO, go into the console and run a bunch of commands to get it all set up correctly, (spend a bit of time confused about how to install GRUB an so on, have a few false reboots...), shut down, reconfigure the server to unmount the ISO, and boot it again. I could automate much of this with the API, but automating running the install process from the console means typing stuff into a VNC session, which sounds possible but not necessarily fun. (There's also nixos-infect which tries to turn a running Ubuntu (or etc.) host into a NixOS one, but that sounds a bit scary, and also maybe I do want to be able to set up my filesystems, which really sounds like I'd need to boot from ISO to do it).
• The smallest Hetzner server instance (4G RAM) can run nixos-rebuild, but it OOM'd at AWS (0.5G). If you want to use NixOS, you're not going to get away with as tiny a memory or disk footprint than if you didn't, I think. (Unless you do some fancy thing where you use Nix to build a Docker image or OS image, but I can't be bothered with that).
• It seems like packaging JavaScript stuff on NixOS is in a not-great state right now, compounded by the fact that I don't fully understand the interactions between node, npm, and e.g. yarn myself. For now I'm using npm exec for postgraphile instead of making a "real" package for it, which seems to work fine.
• I don't have a great solution for "state initialisation" (e.g. applying the initial database schema, creating the jwt key). My postgres server systemd unit just has a postStart hook that checks if the database exists and runs the schema if not. I'm not certain that postgraphile waits for this to happen before trying to connect.
• I don't have a great solution for secrets (e.g. my Google secret key for implementing login with Google). Currently I just do it manually over ssh.
• Getting the elm compiler to work in a network- and filesystem-isolated environment is surprisingly annoying, because it wants to download package metadata from the website. Luckily, the people who wrote elm2nix have already worked around this to some extent, but there's more I want from them.
• Writing a package that puts git revision information into the program was pretty painful. nix asks you what the source directory for your build is, and makes a copy before building in it, so you have to specify that to be the whole repo so you can pick up the .git dir and get git information from it. Worse, if you're submoduled in a larger repo, you don't even know what the whole repo is! I worked around this by letting the owner of the "supermodule" tell the package what's going on, but the "real fix" is probably going to be just not using git version information anymore.

All that said, I'm pretty happy that I can write config that manages a pretty disparate collection of languages, build processes, and system configurations, and I can put it all in one repo. I learned a bunch about how NixOS modules and packages worked, though I still have more to figure out.

I guess technically three repos: I'm still unsure how / if to publish my NixOS configuration.nix and Pulumi infra configuration, but it's not where most of the meat is, I think, so I'm not losing sleep over it.