Building MozJS with Nix

From Rust Community Wiki
(Redirected from Nix MozJS Tutorial)

Nix is a package manager and build system for Linux and macOS. It is designed to be able to self-host an operating system (called NixOS) and to be able to run alongside an existing UNIX system (such as Ubuntu or macOS).

This tutorial shows how to use Nix to build a somewhat complicated Rust project.

Getting started[edit]

Each step in this tutorial can be found in the mozjs-example GitHub repository. You can follow along with the commits in the repository, since each one corresponds to a step here.

Start out by installing the nix package manager.

Set up a crate[edit]

View on GitHub

We'll start by putting some files and Cargo dependencies. Why are we even going to bother with Nix? Because this crate depends on SpiderMonkey (that's MozJS, or the Firefox JavaScript engine), and that has very specific dependencies on its C compiler and available libraries.

You can "cargo build" if you want, but unless you have libclang lying around, it won't build.

As is normal for Rust projects, this will generate a Cargo.toml and a source file]. The example code has added some code, plus a couple of dependencies.

# Cargo.toml
[package]
name = "mozjs-example"
version = "0.0.1"
authors = ["Michael Howell <michael@notriddle.com>"]
edition = "2018"

[dependencies]
smol = "0.1.5"
mozjs = { git = "https://github.com/servo/rust-mozjs" }

Set up niv[edit]

View on GitHub

This step is going to depend on niv, the nix version manager and the tool we use to handle auto-updating dependencies.

$ nix-env -i niv

Once that's installed, we'll use niv to generate a dependency file for the project repository itself. By making the nix repository a fixed constant, we can promise that as long as you have a spec-compliant nix interpreter, you'll be able to build the app itself.

$ niv init
$ niv update nixpkgs -o NixOS -b master -r nixpkgs

The result will be two files: nix/sources.json and nix/sources.nix. The nix file is a script for importing the json file. The nix file is identical across almost all nix projects, since its purpose is just to bootstrap it, while the json file (since it's json) is easy to automatically update.

The step where we update to a newer nixpkgs is needed because we depend on some features that are currently only in nightly.

Set up crate2nix and nix-shell[edit]

View on GitHub
$ niv add kolloch/crate2nix

The result of running the niv command will be a new couple of lines in the dependencies file (and also switch to a newer version of nixpkgs itself, because the Rust compiler in the old version of nixpkgs is really, really old).

To really use it, though, we'll want to create a nix-shell file. Let's do a couple of things to make everything easier for us.

# nix/dependencies.nix
# This is a file we'll write ourselves to make it easier to ensure that configurations stay the same between each environment, both build and dev.

let
  sources = import ./sources.nix;
  pkgs = import sources.nixpkgs { };
in
{
  # This will import our pinned instance of the nixpkgs upstream repository.
  inherit pkgs;
  # This will contain all of the CLI tools that our shell uses.
  devDeps = [
    (import sources.crate2nix {inherit pkgs;})
    pkgs.llvmPackages.lld
    pkgs.yasm
    pkgs.binutils
    pkgs.cargo
    pkgs.llvm
    pkgs.autoconf213
    pkgs.python3
    pkgs.python2
    pkgs.which
    pkgs.perl
  ];
  # Pinned versions of other tools.
  libclang = pkgs.llvmPackages.libclang;
}

Then let's write the shell.nix file, which actually defines the shell proper.

# shell.nix

let
  # Import dependencies (this preamble will be very common)
  dependencies = import ./nix/dependencies.nix;
  pkgs = dependencies.pkgs;
in
  # A "shell", in nix, defines the environment variables and other dependencies
  # for doing development. The "clangStdenv" line makes it use LLVM's C compiler
  # instead of the default GNU C Compiler, because that's what SpiderMonkey
  # needs.
  pkgs.mkShell.override { stdenv = pkgs.clangStdenv; } {
    # This name is pretty much arbitrary.
    name = "mozjs-example";
    # This makes it pull in crate2nix and other tools.
    buildInputs = dependencies.devDeps;
    # This is needed to build SpiderMonkey.
    LIBCLANG_PATH = "${dependencies.libclang.lib}/lib";
  }

Now, if you run nix-shell, it should pull in crate2nix and let you use it.

Generate Cargo.nix and write default.nix[edit]

View on GitHub

First, run nix-shell, then use cargo to generate a lockfile, then run crate2nix:

$ nix-shell
nix$ cargo update
nix$ crate2nix generate

And then, finally, let's write our main build script.

# default.nix
# Allows you to specify if this is a release build on the CLI
# We're going to modify this file later
{release ? true}:
let
  dependencies = import ./nix/dependencies.nix;
  pkgs = dependencies.pkgs;
  # Import Cargo.nix with supplied arguments.
  cargoNix = pkgs.callPackage ./Cargo.nix {
    # Pass the "release" flag on.
    inherit release;
    # Use our version if nixpkgs.
    inherit pkgs;
    nixpkgs = pkgs;
    # Tell it to build with clang (because mozjs needs it).
    stdenv = pkgs.clangStdenv;
  };
in
cargoNix.rootCrate.build

Fix the mozjs-sys build[edit]

View on GitHub

Try running nix-build. It won't work, instead failing with a message like this:

Building build.rs (mozjs_sys)
Running rustc --crate-name build_script_build build.rs --crate-type bin -C opt-level=3 -C codegen-units=8 --edition 2015 --cfg feature="default" --out-dir target/build/mozjs_sys --emit=dep-info,link -L dependency=target/buildDeps --extern bindgen=/nix/store/vf1w3q8svms49bcjhr4ssj57bb5np4aw-rust_bindgen-0.53.3-lib/lib/libbindgen-9df4e066ea.rlib --extern cc=/nix/store/jraman5ssgjs33a92gy6ky6hxr1pidl2-rust_cc-1.0.54-lib/lib/libcc-9e4ac1bad6.rlib --extern walkdir=/nix/store/7qmf2mlfgnjwvj96kvzlaxnr7qjd1jsv-rust_walkdir-2.3.1-lib/lib/libwalkdir-6344f26ad1.rlib --cap-lints allow -L native=/nix/store/0wq80m7c42wbynpxxdzhg0pxh2jqv6ph-rust_libloading-0.5.2-lib/lib/libloading.out -l dl -l static=global_static --color always
make: /usr/bin/env: Command not found
make: /usr/bin/env: Command not found
make: *** [/build/mozjs-9a6d8fc/makefile.cargo:190: maybe-configure] Error 127
thread 'main' panicked at 'assertion failed: result.success()', build.rs:177:5
stack backtrace:
   0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
   1: core::fmt::write
   2: std::io::Write::write_fmt
   3: std::panicking::default_hook::{{closure}}
   4: std::panicking::default_hook
   5: std::panicking::rust_panic_with_hook
   6: std::panicking::begin_panic
   7: build_script_build::main
   8: std::rt::lang_start::{{closure}}
   9: std::panicking::try::do_call
  10: __rust_maybe_catch_panic
  11: std::rt::lang_start_internal
  12: main
  13: __libc_start_main
  14: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
builder for '/nix/store/5w87199rvl60ksj5b7a05fw4g03hdjcs-rust_mozjs_sys-0.68.1.drv' failed with exit code 101
cannot build derivation '/nix/store/mzny9101sixpg4lb11a0r7wyv33wm5id-rust_mozjs-example-0.0.1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/mzny9101sixpg4lb11a0r7wyv33wm5id-rust_mozjs-example-0.0.1.drv' failed

To fix this, we need to give mozjs-sys access to its non-Rust dependencies, and patch its build script. Here's the changed version of dependencies.nix:

# default.nix
# Allows you to specify if this is a release build on the CLI
{release ? true}:
let
  dependencies = import ./nix/dependencies.nix;
  pkgs = dependencies.pkgs;
  # Import Cargo.nix with supplied arguments.
  cargoNix = pkgs.callPackage ./Cargo.nix {
    # Pass the "release" flag on.
    inherit release;
    # Use our version of nixpkgs.
    inherit pkgs;
    nixpkgs = pkgs;
    # Tell it to build with clang (because mozjs needs it).
    stdenv = pkgs.clangStdenv;
  };
in
cargoNix.rootCrate.build.override {
  crateOverrides = pkgs.defaultCrateOverrides // {
    encoding_c_mem = {src, ...}: {
      # Copy .h files from target directory to output directory.
      # encoding_c_mem assumes that its include file in target/ will be
      # available to crates that depend on it. This is not true in Nix.
      postInstall = ''
        echo "export DEP_ENCODING_C_MEM_INCLUDE_DIR=$lib/include" > $lib/env
        cp include -rP $lib/include
      '';
    };
    mozjs-example = {src, ...}: {
      # Expose libclang.
      extraRustcOpts=["-C" "linker=${pkgs.llvmPackages.clang}/bin/clang" "-C" "link-arg=-fuse-ld=${pkgs.llvmPackages.lld}/bin/ld.lld"];
      LIBCLANG_PATH = "${dependencies.libclang.lib}/lib";
      # Need to make the location of the system libraries available to `bindgen`, which reads the CLANGFLAGS env variable to get at this data.
      CLANGFLAGS="-isystem ${pkgs.clangStdenv.lib.getDev pkgs.clangStdenv.cc.cc}/lib/clang/${pkgs.clangStdenv.cc.cc.version}/include -isystem ${pkgs.clangStdenv.cc.cc.gcc}/include/c++/${pkgs.clangStdenv.cc.cc.gcc.version}/${pkgs.hostPlatform.config} -isystem ${pkgs.clangStdenv.cc.cc.gcc}/include/c++/${pkgs.clangStdenv.cc.cc.gcc.version} -isystem ${pkgs.clangStdenv.lib.getDev pkgs.clangStdenv.cc.libc}/include";
      # Need to make sure everybody uses Clang, because mozjs doesn't really support building with GCC.
      CC="${pkgs.llvmPackages.clang}/bin/clang";
      CXX="${pkgs.llvmPackages.clang}/bin/clang";
      LD="${pkgs.llvmPackages.lld}/bin/ld.lld";
      # Tell it to pull in all our specified deps (like autoconf).
      buildInputs = dependencies.devDeps;
    };
    mozjs = {src, ...}: {
      # Expose libclang.
      extraRustcOpts=["-C" "linker=${pkgs.llvmPackages.clang}/bin/clang" "-C" "link-arg=-fuse-ld=${pkgs.llvmPackages.lld}/bin/ld.lld"];
      LIBCLANG_PATH = "${dependencies.libclang.lib}/lib";
      # Need to make the location of the system libraries available to `bindgen`, which reads the CLANGFLAGS env variable to get at this data.
      CLANGFLAGS="-isystem ${pkgs.clangStdenv.lib.getDev pkgs.clangStdenv.cc.cc}/lib/clang/${pkgs.clangStdenv.cc.cc.version}/include -isystem ${pkgs.clangStdenv.cc.cc.gcc}/include/c++/${pkgs.clangStdenv.cc.cc.gcc.version}/${pkgs.hostPlatform.config} -isystem ${pkgs.clangStdenv.cc.cc.gcc}/include/c++/${pkgs.clangStdenv.cc.cc.gcc.version} -isystem ${pkgs.clangStdenv.lib.getDev pkgs.clangStdenv.cc.libc}/include";
      # Need to make sure everybody uses Clang, because mozjs doesn't really support building with GCC.
      CC="${pkgs.llvmPackages.clang}/bin/clang";
      CXX="${pkgs.llvmPackages.clang}/bin/clang";
      LD="${pkgs.llvmPackages.lld}/bin/ld.lld";
      # Tell it to pull in all our specified deps (like autoconf).
      buildInputs = dependencies.devDeps;
    };
    mozjs_sys = {src, ...}: {
      # Expose libclang.
      extraRustcOpts=["-C" "linker=${pkgs.llvmPackages.clang}/bin/clang" "-C" "link-arg=-fuse-ld=${pkgs.llvmPackages.lld}/bin/ld.lld"];
      LIBCLANG_PATH = "${dependencies.libclang.lib}/lib";
      # Need to make the location of the system libraries available to `bindgen`, which reads the CLANGFLAGS env variable to get at this data.
      CLANGFLAGS="-isystem ${pkgs.clangStdenv.lib.getDev pkgs.clangStdenv.cc.cc}/lib/clang/${pkgs.clangStdenv.cc.cc.version}/include -isystem ${pkgs.clangStdenv.cc.cc.gcc}/include/c++/${pkgs.clangStdenv.cc.cc.gcc.version}/${pkgs.hostPlatform.config} -isystem ${pkgs.clangStdenv.cc.cc.gcc}/include/c++/${pkgs.clangStdenv.cc.cc.gcc.version} -isystem ${pkgs.clangStdenv.lib.getDev pkgs.clangStdenv.cc.libc}/include";
      # Need to make sure everybody uses Clang, because mozjs doesn't really support building with GCC.
      CC="${pkgs.llvmPackages.clang}/bin/clang";
      CXX="${pkgs.llvmPackages.clang}/bin/clang";
      LD="${pkgs.llvmPackages.lld}/bin/ld.lld";
      # Tell it to pull in all our specified deps (like autoconf).
      buildInputs = dependencies.devDeps;
      prePatch = ''
        # Remove reference to /usr/bin/env from build scripts,
        # because that app doesn't actually exist in the Nix build sandbox.
        substituteInPlace makefile.cargo --replace '/usr/bin/env bash' '${pkgs.bash}/bin/bash'
      '';
      # Copy .h files from target directory to output directory.
      # mozjs_sys assumes that its include file in target/ will be
      # available to crates that depend on it. This is not true in Nix.
      postInstall = ''
        echo "export DEP_MOZJS_OUTDIR=$lib/lib/mozjs_sys.out/build/" > $lib/env
        cd $lib/lib/mozjs_sys.out/build/
        xd=$(pwd)
        find . | grep -v virtualenv | while read path; do
          target=`readlink $path || true`
          if [ -n "$target" ]; then
            dirname=`dirname $path`
            filename=`basename -s "" $path`
            cd "$dirname"
            echo "$target => $path"
            rm -f "$filename"
            cp -f "$target" "$filename"
            cd "$xd"
          fi
        done
      '';
    };
  };
}

Troubleshooting[edit]

It says I'm out of disk space?[edit]

On my Ubuntu-based computer, the build ran in /run/user/1000/, in a tmpfs folder that didn't have room enough to build mozjs. I expanded it by running this command as root, since I actually do have enough memory to store all the temporary files:

# mount -o remount,size=8613616k /run/user/1000/

You can also change where build files are put using the TMPDIR environment variable.