Natively run OCaml from Rust

Using ocamlopt

Ocamlopt has a flag for static libraries, -output-obj. However this will not export OCaml’s runtime, leading to more linking mess. There is anotherIt’s not in the man page, but ocamlopt --help yields “output-complete-obj: Output an object file, including runtime, instead of an executable”

 flag that does the job, namely -output-complete-obj.

$ ocamlopt -output-complete-obj math.ml -o libmath.o
$ file libmath.o 
[...] ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), \
      with debug_info, not stripped

This creates a static library that contains OCaml’s runtime.

Using dune — formally known as “jbuilder”

Using dune can prove useful for big project with several dependencies.

It took us a while to understand why we couldn’t get it to produce the same thing as above. The trick is that it relies on the output name to know what to put in the file — having a correct dune file is not enough.

One needs to:

• Target an object: that’s in the config file and in the extension.
• Include the runtime. For dune this requires an executable target.

Here’s a possible dune file:

(executables
  (names math)
  (modes object)
)

This discussion and this part of the documentation got us to understand that to do that a working extension is .exe.o, so build it with dune build math.exe.o. Inspecting the file gives the following output:

$ dune build math.exe.o
$ cp _build/default/math.exe.o libmath.o
$ file libmath.o
[...] ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), \
      with debug_info, not stripped

The C wrapper

A C file that wraps the external functions into C functions is then required. It has to handle the memory layout of values, plus some other plumbing. This is where the exposed string name — “twice” — will matter, beware!

#include <stdio.h>
#include <string.h>
#include <caml/mlvalues.h>
#include <caml/callback.h>

int twice(int n)
{
    static value *twice_closure = NULL;
    if (twice_closure == NULL) twice_closure = caml_named_value("twice");
    return Int_val(caml_callback(*twice_closure, Val_int(n)));
}

Val_int and Int_val are provided by mlvalues.h to interface the memory layout. We will have to reimplement them later on for Rust interfacing.

#define Val_long(x)     ((intnat) (((uintnat)(x) << 1)) + 1)
#define Long_val(x)     ((x) >> 1)
#define Val_int(x) Val_long(x)
#define Int_val(x) ((int) Long_val(x))

This needs to be compiled to a static library too. One way to do this is to run ocamlc -c mathwrap.c which will produce a mathwrap.o static libraryThis can be taken care of either by ocamlopt or by dune directly.

. ocamlc takes care of the caml headers, which by default gcc wouldn’t — though in the end gcc gets called.

$ ocamlc -c mathwrap.c
$ file mathwrap.o
[...] ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

The main.c file

The main.c has little more to do. It initializes the OCaml runtime and must be told about the functions — they are not extern because they are already imported by the wrapper:

#include <stdio.h>

int twice(int n);
int caml_startup(char ** argv);

int main(int argc, char ** argv)
{
  caml_startup(argv);
  printf("twice(2)        = %d\n", twice(2));
  printf("twice(twice(2)) = %d\n", twice(twice(2)));
  return 0;
}

It must be compiled against both the C wrapper object and the OCaml object. It also needs to know where OCaml’s header files are — and it needs two additional libraries on our machinesThis may change from version to version? I’ve seen curses used elsewhere too.

.

$ ocamlopt -output-complete-obj math.ml -o libmath.o
$ ocamlc -c mathwrap.c
$ gcc -lm -ldl -I `ocamlc -where` libmath.o mathwrap.o main.c
$ ./a.out
twice(2)        = 4
twice(twice(2)) = 8

Below is a complete Makefile to summarize the required steps:

ocaml-dune: math.ml mathwrap.c main.c dune
    dune build math.exe.o
    ocamlc -c mathwrap.c
    cp _build/default/math.exe.o libmath.o
    gcc -lm -ldl -I `ocamlc -where` libmath.o mathwrap.o main.c
    ./a.out

ocamlopt: math.ml mathwrap.c main.c
    ocamlopt -output-complete-obj math.ml -o libmath.o
    ocamlc -c mathwrap.c
    gcc -lm -ldl -I `ocamlc -where` libmath.o mathwrap.o main.c
    ./a.out

ocamlc: math.ml mathwrap.c main.c
    ocamlc -output-complete-obj math.ml -o libmath.o
    ocamlc -c mathwrap.c
    gcc -lm -ldl -I `ocamlc -where` libmath.o mathwrap.o main.c -lcurses
    ./a.out

clean:
    dune clean
    rm -f *.o *.cm* a.out dune-project

Note: we added an ocamlc version for completeness, which requires the libcurses library to be passed on to gcc.

A remark about OCaml libraries

Note that adding OCaml libraries to your source, say the Unix module, can be taken care of by dune. It will go as far as adding the symbols in the output .o file when it canIf it can’t shared objects are required, see below

. As far as we can tell, ocamlopt will not.

If one wants to stay with ocamlopt and add unix.cmxa to the ocamlopt instruction, then unix.a must be passed to gcc. However some functions may be defined twice, both in unix.a and in libmath.o hence errors will occur. The workaround is to use the ar command to add libmath.o to unix.a and avoid the repetitions of symbols — something like this:

$ cp `ocamlc -where`/unix.a .
$ ar rs unix.a libmath.o

If you want to add more libraries, think about switching to dune. Otherwise the stack gets worse: ar x <lib>.a to get its internal .o files, then ar qs <...> to archive the whole thing together — it gets messy quickly and we’re on shaky grounds.

The main Rust file

We’ll give the file and then go through it:

mod ocamlthings {
    use std::ptr;

    #[link(name = "math")]
    extern "C" {
        // translated from caml/callbacks
        fn caml_startup(argv: *mut *mut u8);
        fn caml_named_value(name: *const u8) -> *const Value;
        fn caml_callback(closure: Value, arg: Value) -> Value;
    }

    // translated from caml/mlvalues
    type Value = usize;
    macro_rules! val_int {
        ($x: expr) => {
            (($x as usize) << 1) + 1
        };
    }
    macro_rules! int_val {
        ($x: expr) => {
            ($x as usize >> 1) as i32
        };
    }

    pub fn initialize_ocaml() {
        let mut ptr = ptr::null_mut();
        let argv: *mut *mut u8 = &mut ptr;
        unsafe {
            caml_startup(argv);
        }
    }

    pub fn twice(n: i32) -> i32 {
        static TWICE_NAME: &str = "twice\0";
        static mut TWICE_CLOSURE: *const Value = ptr::null();
        unsafe {
            if TWICE_CLOSURE.is_null() {
                TWICE_CLOSURE = caml_named_value(TWICE_NAME.as_ptr());
            };
            return int_val!(caml_callback(*TWICE_CLOSURE, val_int!(n)));
        }
    }
}

fn main() {
    ocamlthings::initialize_ocaml();
    println!("twice(2)        = {}", ocamlthings::twice(2));
    println!("twice(twice(2)) = {}", ocamlthings::twice(ocamlthings::twice(2)));
}

The first half is a port of OCaml’s header files. For your own project you may either want to check the raml project that already ports a few of the structures or read the relevant OCaml header files.

The #[link(name = "math")] line tells the compiler that the symbols are defined in an object named libmath.o.

Finally it declares Rust functions that call the OCaml counterparts, with the relevant memory layoutization. This is where the name must match what OCaml exported, beware!

The build process

$ dune build math.exe.o && cp _build/default/math.exe.o ./libmath.o
$ ar qs libmath.a libmath.o
$ rustc -L . src/main.rs
$ ./main
twice(2)        = 4
twice(twice(2)) = 8

use std::env;
use std::process::Command;

fn main() {
    let out_dir = env::var("OUT_DIR").unwrap();
    let dune_dir = "_build/default";

    Command::new("dune")
        .args(&["build", "math.exe.o"])
        .status()
        .expect("Couldn't run builder clean. Do you have dune?");
    Command::new("cp")
        .args(&[
            &format!("{}/math.exe.o", dune_dir),
            &format!("{}/libmath.o", out_dir),
        ])
        .status()
        .expect("File copy failed.");
    Command::new("ar")
        .args(&[
            "qs",
            &format!("{}/libmath.a", out_dir),
            &format!("{}/libmath.o", out_dir),
        ])
        .status()
        .expect("ar gave an error");

    println!("cargo:rustc-link-search={}", out_dir);
}

$ cargo run  # this is all we need!
twice(2)        = 4
twice(twice(2)) = 8

With rustc

Change are needed the main.rs similar to what was needed for main.c — details in the source archive — and then the previous dune -> cp libraries -> Rust sequence should build the binary.

As expected, ./main fails with a shared library loading error, but as above the LD_LIBRARY_PATH should solve thatsetting it to the shared library’s path

:

$ dune build draw.so && cp _build/default/draw.so .
$ rustc -L . src/main.rs
$ LD_LIBRARY_PATH=. ./main
$ file output.png
output.png: PNG image data, 120 x 120, 8-bit/color RGBA, non-interlaced

Success.

With cargo

The build.rs file is pretty straightforward once the previous case is solved:

use std::env;
use std::process::Command;

fn main() {
    let out_dir = env::var("OUT_DIR").unwrap();
    let dune_dir = "_build/default";

    Command::new("dune")
        .args(&["build", "draw.so"])
        .status()
        .expect("Couldn't run builder clean. Do you have dune?");
    Command::new("cp")
        .args(&[
            &format!("{}/draw.so", dune_dir),
            &format!("{}/libdraw.so", out_dir),
        ])
        .status()
        .expect("File copy failed.");

    println!("cargo:rustc-link-search={}", out_dir);
}

And indeed if we try:

$ cargo run  # this is all we need!
$ file output.png
output.png: PNG image data, 120 x 120, 8-bit/color RGBA, non-interlaced

For the time being we don’t know how to tell rustc/cargo to change the rpath of the binary so we don’t know how you can move it around with its friend the shared library and we have to rely on LD_LIBRARY_PATH.