Playing with Nix - building a package in practice

05/05/2023

Trying to observe how packages and derivations are built in practice, using Nix CLI

In the previous post, we saw the typical steps to build a package with Nixpkgs stdenv, using the stdenv/setup.sh script and the mkDerivation function.

Let’s see how this works in practice, by looking at the jq derivation.

The nix show-derivation command (still experimental at time of writing)

$ # using the latest version in nixpkgs
$ nix show-derivation nixpkgs#jq
$ # or, if you installed it already (e.g. with `nix-env -iA jq`), you can run:
$ nix show-derivation $(readlink $(which jq))

{
    "/nix/store/wjk2va74dzv4phvwqhryaarn1pn8svij-jq-1.6.drv": {
        "args": [
        "-e",
        "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
        ],
        "builder": "/nix/store/2198gb5ws3cyma9cxrx3clq6p83781kc-bash-5.1-p16/bin/bash",
        "env": {
            "__darwinAllowLocalNetworking": "",
            "__impureHostDeps": "/bin/sh /usr/lib/libSystem.B.dylib /usr/lib/system/libunc.dylib /dev/zero /dev/random /dev/urandom /bin/sh",
            "__propagatedImpureHostDeps": "",
            "__propagatedSandboxProfile": "",
            "__sandboxProfile": "",
            "bin": "/nix/store/a1xg4nlfrp7cr0fx3k4bv4pv1az65d5d-jq-1.6-bin",
            "buildInputs": "/nix/store/i7dz0378s499bp07933kk2n7jj62jrlg-onig-6.9.8",
            "builder": "/nix/store/2198gb5ws3cyma9cxrx3clq6p83781kc-bash-5.1-p16/bin/bash",
            "cmakeFlags": "",
            "configureFlags": "--bindir=${bin}/bin --sbindir=${bin}/bin --datadir=${doc}/share --mandir=${man}/share/man",
            "depsBuildBuild": "",
            "depsBuildBuildPropagated": "",
            "depsBuildTarget": "",
            "depsBuildTargetPropagated": "",
            "depsHostHost": "",
            "depsHostHostPropagated": "",
            "depsTargetTarget": "",
            "depsTargetTargetPropagated": "",
            "dev": "/nix/store/i9nbwq5jx9q7vkk3slpbf0521knyzqx5-jq-1.6-dev",
            "doCheck": "",
            "doInstallCheck": "1",
            "doc": "/nix/store/3mwsnwrhadxc8439gsrc4a6k054280nj-jq-1.6-doc",
            "installCheckTarget": "check",
            "lib": "/nix/store/acwv21d3npq439k04rfmqq39g5s4401m-jq-1.6-lib",
            "man": "/nix/store/g3inz0gzqr5dc965ksr2iad30z10pcw7-jq-1.6-man",
            "mesonFlags": "",
            "name": "jq-1.6",
            "nativeBuildInputs": "/nix/store/83f8wi09ravyv2r29qyw734g3vpnyg1y-hook",
            "out": "/nix/store/rgsbps29bfdnffac1bb5973p1dm6gdnr-jq-1.6",
            "outputs": "bin doc man dev lib out",
            "patches": "/nix/store/s0nsdqgd0x6ivb2kzgdzxz700irvvi69-fix-tests-when-building-without-regex-supports.patch",
            "pname": "jq",
            "postInstallCheck": "$bin/bin/jq --help >/dev/null\n$bin/bin/jq -r '.values[1]' <<< '{\"values\":[\"hello\",\"world\"]}' | grep '^world$' > /dev/null\n",
            "preBuild": "rm -r ./modules/oniguruma\n",
            "preConfigure": "echo \"#!/bin/sh\" > scripts/version\necho \"echo 1.6\" >> scripts/version\npatchShebangs scripts/version\n",
            "propagatedBuildInputs": "",
            "propagatedNativeBuildInputs": "",
            "src": "/nix/store/6ncacqfpl29xv67cilz4axhygyyz1brr-source",
            "stdenv": "/nix/store/f9ldh0bv3vrxfksfxjm1p2q9m2ii5nna-stdenv-darwin",
            "strictDeps": "",
            "system": "x86_64-darwin",
            "version": "1.6"
        },
        "inputDrvs": {
            "/nix/store/07a2865d207q1pj03q5qbx3jwj9drygp-stdenv-darwin.drv": [
            "out"
            ],
            "/nix/store/4i0yip7rh5cjr9gcrwqqm0v4nr1fw8fh-fix-tests-when-building-without-regex-supports.patch.drv": [
            "out"
            ],
            "/nix/store/bjxdqc615i11gpks5ni8vm8grl2jjnfl-source.drv": [
            "out"
            ],
            "/nix/store/dsi19kfm859kif207ash7247z92ja7yr-hook.drv": [
            "out"
            ],
            "/nix/store/n819whb33s41qsg5rqb85rnl349c41vz-onig-6.9.8.drv": [
            "out"
            ],
            "/nix/store/ng18jzn95n9fdiw2i334i6wjsffcfs4f-bash-5.1-p16.drv": [
            "out"
            ]
        },
        "inputSrcs": [
        "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
        ],
        "outputs": {
            "bin": {
                "path": "/nix/store/a1xg4nlfrp7cr0fx3k4bv4pv1az65d5d-jq-1.6-bin"
            },
            "dev": {
                "path": "/nix/store/i9nbwq5jx9q7vkk3slpbf0521knyzqx5-jq-1.6-dev"
            },
            "doc": {
                "path": "/nix/store/3mwsnwrhadxc8439gsrc4a6k054280nj-jq-1.6-doc"
            },
            "lib": {
                "path": "/nix/store/acwv21d3npq439k04rfmqq39g5s4401m-jq-1.6-lib"
            },
            "man": {
                "path": "/nix/store/g3inz0gzqr5dc965ksr2iad30z10pcw7-jq-1.6-man"
            },
            "out": {
                "path": "/nix/store/rgsbps29bfdnffac1bb5973p1dm6gdnr-jq-1.6"
            }
        },
        "system": "x86_64-darwin"
    }
}

We can note a couple interesting things:

Now that we have seen the base derivation, let’s try to explore the build process, using nix-shell:

The command nix-shell will build the dependencies of the specified derivation, but not the derivation itself. It will then start an interactive shell in which all environment variables defined by the derivation path have been set to their corresponding values, and the script $stdenv/setup has been sourced. This is useful for reproducing the environment of a derivation for development.

So let’s try to get the path of the .drv file (aka the deriver) for our installed package jq:

$ nix-store -qd $(readlink $(which jq))
/nix/store/fbqn0ci8r70v9p0i370xydg8b804kaw1-jq-1.6.drv

And enter the build environment for this derivation:

$ nix-shell $(nix-store -qd $(readlink $(which jq)))
error: path '/nix/store/fbqn0ci8r70v9p0i370xydg8b804kaw1-jq-1.6.drv' does not exist and cannot be created

Humm, so apparently there is a bug with the -d (aka --deriver) option of nix-store --query ? Not sure why? In any case, if we look at the content of the derivation above, the .drv is indeed a different one

$ nix-shell "<nixpkgs>" -A jq

Example: mkDerivation applied on a simple package - look at the actual derivation produced - drop into nix-shell and do the steps manually

Oooops messing with the store

$ # this is bad, but just to try
$ sudo rm /nix/store/wjk2va74dzv4phvwqhryaarn1pn8svij-jq-1.6.drv
$ # see the result:
$ nix-store --verify
reading the Nix store...
checking path existence...
path '/nix/store/wjk2va74dzv4phvwqhryaarn1pn8svij-jq-1.6.drv' disappeared, but it still has valid referrers!
warning: not all store errors were fixed
$ # this should work? but does not for some reason
$ sudo nix-store --verify --repair
warning: $HOME ('/Users/alexandre') is not owned by you, falling back to the one defined in the 'passwd' file ('/var/root')
reading the Nix store...
checking path existence...
path '/nix/store/wjk2va74dzv4phvwqhryaarn1pn8svij-jq-1.6.drv' disappeared, but it still has valid referrers!
warning: cannot repair path '/nix/store/wjk2va74dzv4phvwqhryaarn1pn8svij-jq-1.6.drv'
warning: not all store errors were fixed
a
$ # tried: removing jq by uninstalling + `nix-store --gc`, but not enough since it still has referrers
$ # drastic solution: remove everything
$ nix-env -e '*'
$ # remove all previous generations as well, since there are references in there
$ nix-env --delete-generations +1
$ nix-store --gc
$ nix-store -iA nixpkgs.jq
$ nix-store --verify
OK !!