Learning Nix - Nix language basics

10/04/2023

A quick summary of the first chapters of the Nix Pills tutorial

If you are new to Nix, Nix Pills is a great tutorial to begin with, and to learn the main concepts of Nix/Nixpkgs from the ground up. In this post I will summarize the concepts I found important in the first chapters, plus some information I found relevant.

Why Nix ?

Being tired of waiting for brew commands to execute, and seeing how painful it can be to properly maintain a system in prod or even to do debian packaging, I recently started to take a look at Nix. It looked promising for multiple reasons, namely:

• It is declarative and reproducible ¹: your whole environment, be it a development environment on your laptop or a production on a server, is specified in .nix files that can be version controlled. In case there is any issue, resetting the entire environment or rolling back to a previous version is trivial.
• It seems robust and flexible: each package declares its own dependencies and is isolated from the others. This means that if package A depends on postgres < 14 and package B depends on postgres >= 16, I can install both without fearing any conflict 🎉
• It is cross-platform: I can use it on a debian laptop, on a macbook, or even on a raspberry pi.

All this does not come for free however, and will require some work: a new functional programming language to learn, many new concepts and vocabulary (derivations, channels, overlays), some trick commands ², and a whole new experimental API quickly getting adopted.

In these series of blog posts I will try to document my learning journey, as a newcomer in 2023.

Of course there are already plenty of resources online to learn Nix, so why another series of blog posts? Well, maybe it will only be useful to me. But I found this series – How to learn Nix by Ian Henry – to be extremely useful, so why not following the example?

Installation

Although the official installation script should work fine, the (unofficial) nix-installer seems an interesting alternative (Nix Flakes enabled by default and uninstalling is made easier).

Overview

The Nix language is a functional language. You write Nix expressions in .nix files, which you can compose and combine, in order to produce derivations.

Figure 1: Nix core concepts

Nix language syntax

The first step is to be able to read and write nix expressions, using the Nix language. You can run nix repl to enter an interactive environment and play with the language.

Data structures

Nix supports basic types and operations that can be found in other languages:

nix-repl> builtins.typeOf "abc"
"string"

nix-repl> "a" + ''bc''  # you can enclose strings with double quotes (") or 2 single quotes ('')
"abc"

nix-repl> builtins.typeOf 2
"int"

nix-repl> builtins.typeOf 2.3
"float"

nix-repl> 1 + (2 * 4) - 0.5
8.5

nix-repl> builtins.typeOf ["a" 1.3 [4 5 6]]
"list"

Attribute sets are a key data structure in Nix. Similar to a dictionary in python for instance, they map string keys to values:

nix-repl> { "a" = 1; b = 2; "1/2" = [4 5]; }
{ "1/2" = [ ... ]; a = 1; b = 2; }

nix-repl> rec { a = 1; b = a+c; c = 3; }  # the 'rec' keyword allows defining sets recursively
{ a = 1; b = 4; c = 3; }

Functions

Nix functions can take one or multiple positional parameters, and make it easy to define partial applications.

nix-repl> (x: 2*x) 1.5  # call lambda function 'x: 2*x' on float '1.5'
3

nix-repl> (a: b: 2 * a + b) 1 2  # call the function with a=1, b=2
4

nix-repl> (a: b: 2 * a + b) 1  # partial application, only missing parameter b
«lambda @ (string):1:5»

Using attribute sets, it is also possible to have named parameters:

nix-repl> mul = { a, b }: a * b  # expect an attribute set with at least 'a' and 'b'

nix-repl> mul { a = 1; b = 3; }
3

nix-repl> mul = s: s.a * s.b  # allow other attributes in the input set

nix-repl> mul { a = 1; b = 3; c = "toto" }
3

# allow other attributes in addition to expected parameters (similar to **kwargs in python)nix-repl> mul = s @ { a, b, ... }: a * b + s.c  # also equivalent to: `{ a, b, ... } @ s`

nix-repl> mul { a = 1; b = 3; c = 0.5; d = 0.2; }
3.5

nix-repl> mul = { a, b ? 1 }: a * b  # define default values

nix-repl> mul { a = 1; }
1

A number of builtin functions already exist inside the builtins attribute set, and are accessible directly in nix repl, or inside .nix files.

nix-repl> builtins.sort builtins.lessThan [ 3 1 2 ]
[ 1 2 3 ]
nix-repl> :doc builtins.sort
Synopsis: builtins.sort comparator list
    Return list in sorted order.
    ...

Additional functions are available as part of nixpkgs, inside the lib attribute set.

Special keywords

Just above, in nix-repl, we assigned a value to the variable mul, and simply used it right after. But actual Nix expressions in .nix files do not allow these imperative statements ! In Nix, everything is immutable, so there is no need for assignments statements.

Instead, you can use the let ... in syntax to define variables inside a precise scope (a.k.a. the inner expression):

# in default.nix
let
    a = 1;
    b = a + 1;
in
    { c = b + 0.5; }  # this is the inner expression, in this case also the return value

$ nix-instantiate --eval -E '(import ./default.nix).c'
2.5

Another keyword often used is with, which basically adds the symbols of an attribute set into the scope of the inner expression.

# in default.nix
let
    a = -1;
    s = { a = 1; b = 2; };
    s' = with s; { c = b + 1; };  # the `with s` syntax simply avoids doing `s.b`
in
    with s; { d = a + b + s'.c; }  # d = -1 + 2 + (2+1)

$ nix-instantiate --eval -E '(import ./default.nix).d'
5

Note that the with keyword does not override existing variables in the scope.

The inherit keyword is also a convenient keyword, often used when creating attribute sets. It basically avoids repeating a = a or a = s.a inside the set definition:

# in default.nix
let
    x = 1 + 2;
    y = 2;
    s = { a = 4 / 0; b = "d"; };
in
    {
        inherit x y;  # equivalent to `x = x; y = y;`
        inherit (s) a b;  # equivalent to `a = s.a; b = s.b;`
    }

$ nix-instantiate --eval -E '(import ./default.nix)'
{ a = <CODE>; b = <CODE>; x = <CODE>; y = 2; }

Also, note how a, b and x attributes are not evaluated when you instantiate the attribute set: Nix is lazy and only evaluates expressions when needed. This is why defining a = 4 / 0 is fine, until you try to access it.

What is a derivation?

Remember Figure 1 ? The main reason we write .nix files is not to create general purpose programs, but to be able to describe and build packages. This is done thanks to the derivations.

But what is a derivation? It is a data structure (could be a json) stored in a .drv file inside the Nix store, containing all the information necessary to build a specific package. It is an intermediate representation between your source files (.nix) and your store outputs (programs and binaries in /nix/store).

This is a bit similar to object files (.o) in C for instance: they are an intermediate representation that you obtain from source files (.c), before linking them to obtain an executable.

Creating a derivation

You can obtain a derivation using the built-in derivation function (see the manual), that takes as input an attribute set, and outputs another attribute set:

$ nix-instantiate --eval -E 'derivation { name = "mypkg"; builder = "builder"; system = "mysystem"; }'
{ all = <CODE>; builder = "builder"; drvAttrs = { builder = "builder"; name = "mypkg"; system = "mysystem"; }; drvPath = <CODE>; name = "mypkg"; out = «repeated»; outPath = <CODE>; outputName = "out"; system = "mysystem"; type = "derivation"; }

This attribute set (with type = "derivation") uniquely defines a package. This is more or less what you will find inside the giant nixpkgs attribute set, when defining your package dependencies, or when combining packages together in an environment for instance.

An interesting thing to note is that, if you try to coerce a derivation (or any attribute set with an outPath attribute) to a string, you will obtain the outPath attribute:

$ nix-instantiate --eval -E 'let d = derivation { name = "mypkg"; builder = "builder"; system = "mysystem"; }; in "${d}"'
"/nix/store/12ly8qc4601sgnhjlygss43yinw6ahdj-mypkg"
$ nix-instantiate --eval -E 'let d = derivation { name = "mypkg"; builder = "builder"; system = "mysystem"; }; in d.outPath'
"/nix/store/12ly8qc4601sgnhjlygss43yinw6ahdj-mypkg"

Creating a working derivation

If you try to build the dummy derivation above, it will fail. Here is an example of a working derivation:

# hello.nix
let
    pkgs = import <nixpkgs>{};
in derivation {
    # --- Required attributes
    # Derivation name
    name = "mypkg";
    # The executable called to build the package
    builder = "${pkgs.bash}/bin/bash";
    # System on which the derivation can be built (nix-build will refuse to work otherwise)
    # e.g. "i686-linux" or "x86_64-darwin"
    system = builtins.currentSystem;

    # --- (Some) optional attributes
    # This list of args is passed to the builder executable above
    args = [ ./hello.sh ];
    # Any additional attributes is cast to a string passed
    # as an environment variable to the builder
    coreutils = pkgs.coreutils;
    mymsg = ./msg.txt;
}

This time, if we remove the --eval option, nix-instantiate will actually create a .drv file with the contents of our derivation:

$ nix-instantiate hello.nix
/nix/store/0smhrqz82a49vkkxfb5i4xplkggzkcw1-mypkg.drv
$ cat /nix/store/0smhrqz82a49vkkxfb5i4xplkggzkcw1-mypkg.drv
# not easily readable...
$ nix show-derivation /nix/store/0smhrqz82a49vkkxfb5i4xplkggzkcw1-mypkg.drv
{
    "/nix/store/0smhrqz82a49vkkxfb5i4xplkggzkcw1-mypkg.drv": {
        "args": [
            "/nix/store/2cx9h7pdb75hvm8a8bmsl9017aw2gjq2-hello.sh"
        ],
        "builder": "/nix/store/2198gb5ws3cyma9cxrx3clq6p83781kc-bash-5.1-p16/bin/bash",
        "env": {
            "builder": "/nix/store/2198gb5ws3cyma9cxrx3clq6p83781kc-bash-5.1-p16/bin/bash",
            "coreutils": "/nix/store/bwrmdh64v9b0ygl3bv3ys0x7fw4pykg5-coreutils-9.1",
            "mymsg": "/nix/store/gfis1dcq9sjplx85nfygi5bnb0qyanal-msg.txt",
            "name": "mypkg",
            "out": "/nix/store/rw75b9bvpn2f9id6j411xjcllpl9wxjy-mypkg",
            "system": "x86_64-darwin"
        },
        "inputDrvs": {
            "/nix/store/6wmsjv6zvq3ahdp42f5by31va2457zdy-coreutils-9.1.drv": [
                "out"
            ],
            "/nix/store/ng18jzn95n9fdiw2i334i6wjsffcfs4f-bash-5.1-p16.drv": [
                "out"
            ]
        },
        "inputSrcs": [
            "/nix/store/2cx9h7pdb75hvm8a8bmsl9017aw2gjq2-hello.sh",
            "/nix/store/gfis1dcq9sjplx85nfygi5bnb0qyanal-msg.txt"
        ],
        "outputs": {
            "out": {
                "path": "/nix/store/rw75b9bvpn2f9id6j411xjcllpl9wxjy-mypkg"
            }
        },
        "system": "x86_64-darwin"
    }
}

A couple interesting things to note here:

• Whenever you specify a path in a derivation, the file gets copied to the nix store and this is the path you can use in the build step.
• The builder attribute can be:
  • a string referring to an absolute path
  • a path referring to an executable
  • or a derivation (and the outPath will be used as the executable).
• This builder executable will executed as is, so if you tried to put ./hello.sh directly here, it would be executed with the default shell on your system (e.g. /bin/sh), this is not what we want.
• This one is tricky, but the derivation name may be different from the name you will use when referring to other packages. ³

Let’s put ‘hello’ in the msg.txt file, and have the following hello.sh file:

# Add mkdir, chmod and other GNU coreutils to the PATH
export PATH="$coreutils/bin"
mkdir -p $out/bin
# Create our executable: a simple script that always outputs the content of mymsg
echo "$coreutils/bin/cat $mymsg" > $out/bin/hello
chmod +x "$out/bin/hello"

We can now build the package:

$ nix-build hello.nix  # or 'nix-store --realise /nix/store/0smhrqz82a49vkkxfb5i4xplkggzkcw1-mypkg.drv'
/nix/store/rw75b9bvpn2f9id6j411xjcllpl9wxjy-mypkg
$ /nix/store/rw75b9bvpn2f9id6j411xjcllpl9wxjy-mypkg/bin/hello
hello

Bingo ! 🎉🥳

Build and runtime dependencies

Using the --references option, we can query the immediate dependencies of an object in nix store:

$ nix-store -q --references /nix/store/0smhrqz82a49vkkxfb5i4xplkggzkcw1-mypkg.drv
/nix/store/2cx9h7pdb75hvm8a8bmsl9017aw2gjq2-hello.sh
/nix/store/ng18jzn95n9fdiw2i334i6wjsffcfs4f-bash-5.1-p16.drv
/nix/store/6wmsjv6zvq3ahdp42f5by31va2457zdy-coreutils-9.1.drv
/nix/store/gfis1dcq9sjplx85nfygi5bnb0qyanal-msg.txt
$ nix-store -q --references /nix/store/rw75b9bvpn2f9id6j411xjcllpl9wxjy-mypkg/
/nix/store/bwrmdh64v9b0ygl3bv3ys0x7fw4pykg5-coreutils-9.1
/nix/store/gfis1dcq9sjplx85nfygi5bnb0qyanal-msg.txt

Since the derivation is used to build the package, the dependencies of our .drv files correspond to our build dependencies. Inside, we can find all the store objects in the inputDrvs and inputSrcs.

The dependencies of our output package is more interesting: it contains only the store paths that we need to run our hello executable, i.e. our runtime dependencies. And we did not even have to specify them !

By looking inside the content of our build outputs (here, the /nix/store/rw75b9bvpn2f9id6j411xjcllpl9wxjy-mypkg/bin/hello executable), Nix was able to find references to nix store paths, and register them as runtime dependencies.

Here, /nix/store/rw75b9bvpn2f9id6j411xjcllpl9wxjy-mypkg/bin/hello is a simple text file, but it also works with binaries for instance. No matter the file format ⁴, Nix will scan the content of the file to find the runtime dependencies.

Conclusion

The Nix language allows us to declare derivations, that can then be used to build packages (or other outputs) in the nix store. A derivation is simply an attribute set containing all the required build inputs (builder executable, source files, environment variables, other dependencies), and is stored as a .drv file in the nix store once instantiated.

Resources

• Nix Pills
• How to learn Nix, by Ian Henry
• Nix Reference Manual