OCaml: Issues linking C and OCaml-CodePudding

I am able to wrap C code and access it from the OCaml interpreter, but cannot build a binary! I'm 98% sure it is some linking problem, but can't find the tools to explore the linkage.

Getting even to this point was a chore, (endless quantities of Error: The external function is not available messages) so I'll document everything I did.

A 'system' file stuff.c

#include <stdio.h>
int fun(int z)  // Emulate a "real" subroutine
{
        printf("duuude whoa z=%d\n", z);
        return 42;
}

Compile above as

cc -fPIC -c stuff.c
ld -shared -o libstuff.so stuff.o

An OCaml wrapper around above, in ocmstuff.c:

#include <caml/mlvalues.h>
CAMLprim value yofun(value z) {
    return Val_long(fun(Long_val(z)));
}

Build above as

cc -fPIC -c ocmstuff.c
ld -shared -o dllostuff.so ocmstuff.o -L . -lstuff -lc -rpath .

Yes, the rpath really is needed, else the next steps suffer. (Edit: If you don't use rpath, you'll need to use LD_LIBRARY_PATH=. instead. For the final 'production' version, you'd change the rpath to the actual library path, or do ld.so.conf trickery or install into 'standard' locations, or tell your users about LD_LIBRARY_PATH. This is just like what you'd do for any other system. The rpath solution seems to be the most stable and reliable solution.)

Next, a module declaration, stored in fapi.mli

module Fapi : sig
        external ofun : int -> int = "yofun" ;;
end

Build above as:

ocamlc -a -o fapi.cma -intf fapi.mli -dllib -lostuff

Does it work? Yes it does:

$ rlwrap ocaml fapi.cma
        OCaml version 4.11.1
open Fapi ;;
Fapi.ofun 33 ;;
duuude whoa z=33
- : int = 42
#

So the wrapper works fine. Now lets compile with it. Here's myprog.ml:

open Fapi ;;
Fapi.ofun 33 ;;

Compile it:

ocamlc -c myprog.ml
ocamlc -o myprog myprog.cmo fapi.cma

The very last command spews:

File "_none_", line 1:
Error: Required module `Fapi' is unavailable

I am 98% sure the above error is due to some silly linking error, but I cannot track it down. Why do I think this? Well, here's a related problem that provides a hint.

$ rlwrap ocaml
open Fapi ;;
# Fapi.ofun 33 ;;
Error: The external function `yofun' is not available
#

Well, that's odd. It clearly must have found fapi.cma because that is the only way it can know about yofun. But somehow, it doesn't know it needs to dig into dllostuff.so for that. Or possibly dllostuff.so is failing to correctly link/load libstuff.so ? Or maybe libc.so to get printf ? I'm pretty sure its one of these last few, but I just can't get it to work, and don't have the tools to debug it. (nm and ldd -r look healthy. Are there some similar tools for the assorted cma,cmo,cmi,cmx files?)

CodePudding user response：

Interfacing with C is much easier if you use dune. You don't need to know the low-level details it is all handled for you.

Now, back to your example. This is definitely not how OCaml users are interfacing with C, but if you really want to learn about it here are a few notes.

The reason why you have the error is that:

you specified modules in an incorrect order, it should be topological, not reverse topological order, i.e., the dependency comes before dependent
you do not have the .ml file (the -intf option means very different)

The reason why the last snippet doesn't work is because you're not loading the library. The ocaml binary obviously doesn't have any fapi units linked into it, so you have to explicitly load it using either #load directive or by passing it in the command line.

Also the following line is not necessary,

ld -shared -o dllostuff.so ocstuff.o -L . -lstuff -lc -rpath .

First of all, there is no need to link a stub file into a shared library. It is counterproductive and doesn't really bring you anything. Second, passing -rpath . will render the end executable unusable, unless the shared objects are stored in the same folder as the executable. Just remove this.

Just to complete your exercise, here is how it could be built and run. First, let's fix the stub file. We need the ml file and we also need to remove an extra module definition,

$ cat fapi.{ml,mli}
external ofun : int -> int = "yofun" ;;
external ofun : int -> int = "yofun" ;;

Yes, they are the same. The mli file is not really needed here, but let's keep it for the sake of completeness.

The way how you build the pure C part is fine, as long as you get a relocatable .so file it works.

Now to build the ocstuff.c (which we conventionally call stubs) you just need to do,

ocamlc -c ocstuff.c

Don't turn it into a shared library, don't do anything else with it. Now let's build the fapi library,

ocamlc -c fapi.mli
ocamlc -c fapi.ml

Now let's build the library that contains both OCaml and C code,

ocamlmklib -o fapi fapi.cmo ocstuff.o -lstuff -L.

Now we can finally build the executable,

ocamlc -c myprog.ml 
LD_LIBRARY_PATH=. ocamlc -o myprog fapi.cma myprog.cmo

and run it,

LD_LIBRARY_PATH=. ./myprog 
duuude whoa z=33

Notice that we have to use the LD_LIBRARY_PATH to tell the system dynamic loader where to look for the external dependency libstuff.so. You can, of course, use rpath to specify its location (pass it to ocamlmklib via -ccopt) but in general it is assumed that the external library is installed at some location that the system loader knows.

Again, unless you're developing your own build system, please use dune or oasis for building OCaml programs. These systems will handle all low-level details in the best possible way.

P.S. It is also worth mentioning that you're not building a binary, but a bytecode executable. For binaries, you will have to use the ocamlopt compiler. And this would be a completely different story. Again, dune is the solution.

CodePudding user response：

There is a lot to take in here, but these lines are suspicious:

ocamlc -c myprog.ml
ocamlc -o myprog myprog.cmo fapi.cma

OCaml expects modules in topologically sorted order, with a module appearing on the command line before the modules that refer to it.

So it would seem the last line should be this:

ocamlc -o myprog fapi.cma myprog.cmo

I hope this helps, it's just a quick response.

CodePudding user response：

The answer provided by ivg works. It also provides enough hints to retrofit the original question to get the correct behavior. The changes to the original recipe are:

Create fapi.mli and fapi.ml which both have the same content: external ofun : int -> int = "yofun" ;;
Compile both the above with ocaml -c. The mli must be compiled first: it yields an interface file cmi which is needed before the ml file can be compiled into it's object file cmo.
The name dllostuff.so was wrong: it must be dllfapi.so to maintain naming consistency.
Build the cma archive/library as ocamlc -a -o fapi.cma fapi.cmo -dllib -lfapi

That's it! Other than these, the original instructions work. The answer from ivg suggests using

ocamlmklib -o fapi fapi.cmo ostuff.o -L. -lstuff

instead of

ld -shared -o dllfapi.so ostuff.o -L. -lstuff

Either of these work. The primary difference is that ocamlmklib also creates a static-linked library libfapi.a. Other than that, it creates the dllfapi.so as before. (That version also contains a motley assortment of typical gcc symbols, for handling exceptions, library ctors, etc. It's not clear why these are needed here, since they'll show up sooner or later anyway.)