Some experiments with Erlang and NixOS
Crafing a development environment for a monorepo with Zig, Erlang, Nix and Postgres
Updated at : I've changed the way we package our erlang project with Nix, mostly for the better.
Recently I've been collaborating with friends on Lyceum, an MMO game with an Erlang backend and a Zig + Raylib client (as if erlang wasn't a weird enough of a choice). Now, this is an unusual combination, but that's the whole reason our pesky group exists in the first place (if you want to know more check my friend's Lemos post).
There is also a couple of standards we try to follow when doing this project, all of the team works with microservices all day in their normal jobs, so whenever we want to do something we try follow some simple rules:
- Can we develop all of the project parts locally? Preferably with no networking as well (besides pulling dependencies).
- Can we do so by leveraging a couple handy tools to their limit?
One can imagine that setting up such a development environment might be
nightmarish, but thankfully the 21st century brought us some interesting tools
that make Unix less of a mess to deal with, and yes, I'm talking about Nix. My
goal here is to show people what our development experience looks like and maybe
convince a few souls dealing with more normal tools (brew, asdf, <insert random
linux package manager>, …) to at least give Nix a try.
Devenv
We use devenv to setup our development shell, think of it as your favorite
programming language's envinroment and dependency manager (pip, poetry, nvm,
rvm, …) but capable of installing anything availiable on nixpkgs and
much more.
A unified development shell for Erlang and Zig
No one is expected to have Erlang, Zig and Postgres installed, nor are they
expected to have any of the environment variables needed for this project to
work, the development shell already does all of that boring stuff for
you. Here's a snippet of what it looks like:
# (...)
devShells =
let
linuxPkgs = with pkgs; [
inotify-tools
xorg.libX11
xorg.libXrandr
xorg.libXinerama
xorg.libXcursor
xorg.libXi
xorg.libXi
libGL
];
darwinPkgs = with pkgs.darwin.apple_sdk.frameworks; [
CoreFoundation
CoreServices
];
in
{
# (...)
# `nix develop`
default = devenv.lib.mkShell {
inherit inputs pkgs;
modules = [
(
{ pkgs, lib, ... }:
{
packages =
with pkgs;
[
just
raylib
]
++ lib.optionals stdenv.isLinux (linuxPkgs)
++ lib.optionals stdenv.isDarwin darwinPkgs;
languages.erlang = {
enable = true;
package = erlangLatest;
};
languages.zig = {
enable = true;
package = zigLatest;
};
env = mkEnvVars pkgs erlangLatest erlangLibs raylib;
scripts = {
build.exec = "just build";
server.exec = "just server";
client.exec = "just client";
db-up.exec = "just db-up";
db-down.exec = "just db-down";
db-reset.exec = "just db-reset";
};
enterShell = ''
echo "Starting Development Environment..."
just deps
'';
services.postgres = {
package = pkgs.postgresql_16.withPackages (p: with p; [ p.periods ]);
enable = true;
initialDatabases = [ { name = "mmo"; } ];
port = 5432;
listen_addresses = "127.0.0.1";
initialScript = ''
CREATE USER admin SUPERUSER;
ALTER USER admin PASSWORD 'admin';
GRANT ALL PRIVILEGES ON DATABASE mmo to admin;
'';
};
}
)
];
};
};
# (...)
Let's try building the Zig client:
$ just client-build $ just client
Running Postgres
As you may have noticed, not only are we installing Erlang and Zig, some
madlad even put dbeaver there for God knows what reason, but hey, that's the dev
shell, just do whatever you want. We also have a local postgres setup and the
workflow mimics what you usually have with docker-compose or podman. By running:
devenv up
inside the shell, a local Postgres 16 with custom extensions will be
spinned. The list of services supported by devenv keeps growing and you can
check them here.
Direnv
As if thigs weren't awesome enough, I need to talk about direnv, a simple tool
that can make wonders (and it comes with nix integrations for free), with a
single .envrc in your project's repo you can jump inside a certain development
shell just by cd-ing into the directory. Here's an example of my
.envrc:
use flake . --impure
followed by a direnv allow in my shell:
$ direnv allow direnv: loading ~/Code/Personal/lyceum/.envrc direnv: using flake . --impure direnv: nix-direnv: Renewed cache Starting Development Environment... rebar3 get-deps ===> Verifying dependencies... rebar3 nix lock ===> Verifying dependencies... # (...)
That's it. Now every time I cd <lyceum-directory>, I'll immediatly load the
whole development shell and be ready to work on it. This section is optional but
it really simplifies my life, as I don't need to remember to activate/deactivate
an environment.
The CI environment
Since we are already went to the trouble of setting up a whole dev environment for Erlang and Zig, we should just make another one for when we need to run builds and test suites on CI.
# `nix develop .#ci`
# reduce the number of packages to the bare minimum needed for CI
ci = pkgs.mkShell {
env = mkEnvVars pkgs erlangLatest erlangLibs raylib;
buildInputs = with pkgs; [
erlangLatest
just
rebar3
rsync
zigLatest
raylib
];
};
If you use Github Actions, now you can leverage both the Install Nix and Magic Nix Cache actions.
The full devshell
You can check what the full devshell looks like here.
Nix Build
In the previous section I've showed you our impure environment, there's no way (as of now) to make things 100% pure while developing, specially because we need to have a postgres service running to debug and test locally. However, things change when we talk about releases, we need to find a way to properly build the server.
A pure build of the Erlang server
This is the original reason I've decided to write this, it took me some time to go through the NixOS BEAM manual and I've yet to know how to properly build this project with the buildRebar3 Tools (it seems it's used more inside Nixpkgs itself than to integrate with Erlang projects). Nevertheless, you can properly package this with the abstractions plain Nix already gives you:
# Leverages nix to build the erlang backend release
# nix build .#server
server =
let
deps = import ./server/rebar-deps.nix {
inherit (pkgs) fetchHex fetchFromGitHub fetchgit;
builder = pkgs.beamPackages.buildRebar3;
};
in
pkgs.beamPackages.rebar3Relx {
pname = erl_app;
version = "0.1.0";
root = ./server;
src = pkgs.lib.cleanSource ./server;
releaseType = "release";
profile = "prod";
beamDeps = builtins.attrValues deps;
};
This is a derivation, a meta-package, a recipe containing every step and every
dependecy I need to satisfy and properly build our server. Now, as for the
deps.nix file, it was auto-generated with rebar3-nix, which itself has a rebar3
plugin. So everytime someone adds a BEAM dependency in our current flow, we
automatically generate a nix lockfile to match the rebar3 lockfile as
well. Here's what we needed to add in our rebar3 config to benefit from the Nix
integration:
{plugins, [
{ rebar3_nix, ".*", {git, "https://github.com/erlang-nix/rebar3_nix.git", {tag, "v0.1.1"}}}
]}.
now let's see if this really works:
$ nix build .#server # (...) # We now have a `result` directory in the project's root... $ ls ./result/ bin database erts-13.2.2.10 lib releases # Now try running the server we've just build and... $ ./result/bin/server foreground Exec: /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/erts-13.2.2.10/bin/erlexec -noinput +Bd -boot /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/releases/0.0.1/start -mode embedded -boot_var SYSTEM_LIB_DIR /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/lib -config /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/releases/0.0.1/sys.config -args_file /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/releases/0.0.1/vm.args -- foreground Root: /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server Connecting to: "127.0.0.1" Connected to "127.0.0.1" with USER = "admin" Finding migration scripts... Migration Path: "/nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/database/migrations" Running DB migrations. Migrations completed successfully. # (...) it works
Containers
There is a treasure trove of examples in Nixpkgs, I've decided to go with the simplest one. This what a container for the backend looks like in Nix:
# nix build .#dockerImage
dockerImage = pkgs.dockerTools.buildLayeredImage {
name = erl_app;
tag = "latest";
created = "now";
# This will copy the compiled erlang release to the image
contents = [
server
pkgs.coreutils
pkgs.gawk
pkgs.gnugrep
pkgs.openssl
];
config = {
Volumes = {
"/opt/${erl_app}/etc" = {};
"/opt/${erl_app}/data" = {};
"/opt/${erl_app}/log" = {};
};
WorkingDir = "/opt/${erl_app}";
Cmd = [
"${server}/bin/${erl_app}"
"foreground"
];
Env = [
"ERL_DIST_PORT=8080"
"ERL_AFLAGS=\"-kernel shell_history enabled\""
"NODE_NAME=${erl_app}"
];
ExposedPorts = {
"4369/tcp" = {};
"4369/ucp" = {};
"8080/tcp" = {};
"8080/udp" = {};
};
};
};
It doesn't really look like most Dockerfiles you see around the net. Notice that
I'm using the server derivation from the previous step, the hard work required
to make it work the first time is immediatly rewarded because now we can keep
composing the previous solutions into more complex flows. To test this, let's
build the image:
$ nix build .#dockerImage # Now load the build image in docker (or podman) $ docker load < ./result # Make sure you have `devenv up` running $ docker container run --network=host --rm lyceum:latest Exec: /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/erts-13.2.2.10/bin/erlexec -noinput +Bd -boot /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/releases/0.0.1/start -mode embedded -boot_var SYSTEM_LIB_DIR /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/lib -config /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/releases/0.0.1/sys.config -args_file /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/releases/0.0.1/vm.args -- foreground Root: /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server server[1] Starting up Connecting to: "127.0.0.1" Connected to "127.0.0.1" with USER = "admin" Finding migration scripts... Migration Path: "/nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/database/migrations" Running DB migrations. Migrations completed successfully. # (...)
Conclusion
As I wanted to show here, we've used Nix all the way from defining a common development environment for the developers, reused some of the stuff in CI, to later repurpose some of the flows for pure builds, that later got shoved into our containers, all by leveraging the same tool. I wish modern devops was more about that, but it seems it'll take time for people to realize that immutability, composition and functional programming can go hand in hand and give us a better experience than one can find in most other solutions (built by trillion dollar companies who want you to manage infra with YAML). Luckilly, Nix is gaining some traction and more people are talking about it.
I've been using it for the past 6 years in my workstations and don't regret doing so, its a tool worth learning (and there's still so much to learn about it), it makes my life dealing with Unix systems less painfull.
TODO
There is still much to do, and it can be left for a part II later.
[ ]I have yet to learn how to deploy a production-ready erlang system. Add (Cesarini and Vinoski 2016) to my readlist.[ ]Properly build the client, it seems that non2nix breaks with the format for zon files, I'm not familiar with Zig toolig and ill take a look at this later