Using rcopy and sexp methods to create a .Call interface to a compiled Julia library

[dmbates]

Development versions of Julia-1.12 and 1.13 provide compilation of Julia code to a shared library and ways to shrink the size of the binary (juliac with the --trim option). One attractive application of this capability is to write compute-intensive parts of R packages in Julia instead of going the C/Fortran/C++ route. I think the preferred interface from R for such Julia code would be the .Call interface or the .External interface (https://cran.r-project.org/doc/manuals/R-exts.html#Interface-functions-_002eCall-and-_002eExternal-1).

For each of these interfaces the arguments are passed as pointers to a C struct called SEXPREC (the pointer type is called SEXP) and the return value must also be an SEXP. This the basic boxing/unboxing strategy in R. Doing the unboxing in Julia is straightforwardly accomplished by the rcopy methods from this package (i.e. RCall.jl). The boxing in RCall.jl is accomplished by methods for sexp. But these methods use functions from libR that manipulate locations in the R environment to do things like allocate storage within R and mark it as protected from garbage collection. An example in https://juliainterop.github.io/RCall.jl/dev/custom/#Julia-to-R-direction is

import RCall: sexp, protect, unprotect, setclass!, RClass

function sexp(::Type{RClass{:Bar}}, f::Foo)
    r = protect(sexp(Dict(:x => f.x, :y => f.y)))
    setclass!(r, sexp("Bar"))
    unprotect(1)
    r
end

which causes allocation in the inner calls to sexp, pushes and pops from a stack in the R environment in the protect and unprotect calls, and requires access to certain global constants in setclass!. These Julia function all eventually end up calling C functions in libR. Especially the allocation and the protect/unprotect must manipulate global locations in the environment of the R process calling the Julia code.

If a library was compiled from Julia code that contained code similar to src/types.jl, src/Const.jl and files in src/convert, and that library was opened by an R process, would the calls to libR functions be resolved against the copy of libR in use by the R process? It seems from the way that src/Const.jl contains both @load_const and @load_const_embedded macros that the answer is yes because @load_const_embedded doesn't specify libR.

So would creating Julia code to unbox the arguments from .Call or .External and boxing the result be as simple as creating a RconversionBase package with some of the code from src/types.jl, src/Const.jl, src/methods.jl and the files in src/convert then using that package to provide boxing/unboxing for RCall.jl and for Julia code to be called from R packages?

[dmbates]

I tried using juliac as described in https://github.com/TheCedarPrince/InteroperableJuliaBinaries but I am on an M1 Mac and ran into the problems described in Issue 1 of that repository.

[palday]

A few months ago, I started working on closing #444. Some of that work made it into various PRs, but I got stuck at some point and never shared the rest. I'll try to go ahead and do that at some point this week. I mention that in this thread because that would create a cleaner line between the very thin sexp/pointer interface and higher level interface and also be necessary for splitting off a RConversionBase package.

In terms of the R process ... I think that's quite possible and somewhat related to what JuliaCall does. I would be happy to help in this effort, though I must admit I'm much better at promising help for interesting ideas than actually finishing implementing them....

[dmbates]

I am trying to get the creation of a shared library with juliac.jl and dyn.loading that library in R working. I can load the shared library in Julia and ccall a function from Julia. When I try to dyn.load that shared library in R I get a failure due to an undefined symbol. This may be due to conflicting versions of libunwind.

I'm doing this on a Linux computer (thanks @palday). First step is to create symbolic links to .jl files in the nightly Julia version.

$ ls -l *.jl
lrwxrwxrwx 1 dmbates dmbates  76 Mar 10 17:10 juliac-buildscript.jl -> /home/dmbates/.julia/juliaup/julia-nightly/share/julia/juliac-buildscript.jl
lrwxrwxrwx 1 dmbates dmbates  64 Mar 10 17:10 juliac.jl -> /home/dmbates/.julia/juliaup/julia-nightly/share/julia/juliac.jl
lrwxrwxrwx 1 dmbates dmbates  70 Mar 10 17:10 julia-config.jl -> /home/dmbates/.julia/juliaup/julia-nightly/share/julia/julia-config.jl
-rw-r--r-- 1 dmbates dmbates 357 Mar 10 12:01 simple.jl

Then compile libsimple.so

$ julia +nightly ./juliac.jl --output-lib libsimple.so --compile-ccallable --experimental --trim=unsafe-warn simple.jl

The output shared object, libsimple.so, links against the version of libunwind shipped with Julia

$ ldd libsimple.so 
	linux-vdso.so.1 (0x00007ffe698c9000)
	libjulia.so.1.13 => /home/dmbates/.julia/juliaup/julia-nightly/lib/libjulia.so.1.13 (0x00007adcb28dc000)
	libjulia-internal.so.1.13 => /home/dmbates/.julia/juliaup/julia-nightly/lib/julia/libjulia-internal.so.1.13 (0x00007adcb2296000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007adcb2000000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007adcb226a000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007adcb2265000)
	libunwind.so.8 => /home/dmbates/.julia/juliaup/julia-nightly/lib/julia/libunwind.so.8 (0x00007adcb1a00000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007adcb225e000)
	libz.so.1 => /home/dmbates/.julia/juliaup/julia-nightly/lib/julia/libz.so.1 (0x00007adcb1400000)
	libatomic.so.1 => /home/dmbates/.julia/juliaup/julia-nightly/lib/julia/libatomic.so.1 (0x00007adcb2254000)
	libstdc++.so.6 => /home/dmbates/.julia/juliaup/julia-nightly/lib/julia/libstdc++.so.6 (0x00007adcb1d3d000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007adcb1c54000)
	libgcc_s.so.1 => /home/dmbates/.julia/juliaup/julia-nightly/lib/julia/libgcc_s.so.1 (0x00007adcb2224000)
	/lib64/ld-linux-x86-64.so.2 (0x00007adcb3558000)

Using

$ R --version
R version 4.4.3 (2025-02-28) -- "Trophy Case"
Copyright (C) 2025 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu

and trying to dyn.load this shared object produces an error

$ LD_LIBRARY_PATH=. LD_PRELOAD=libsimple.so R
/usr/lib/R/bin/exec/R: symbol lookup error: /home/dmbates/.julia/juliaup/julia-nightly/lib/julia/libjulia-internal.so.1.13: undefined symbol: unw_ensure_tls

but that symbol is defined in Julia's version of libunwind.so.8

$ nm -g ~/.julia/juliaup/julia-nightly/lib/julia/libunwind.so.8 | fgrep unw_ensure_tls
000000000000438b T unw_ensure_tls

I'm not all that familiar with how the dynamic linker goes about resolving symbols. Could the R executable already have links to an incompatible libunwind? It doesn't seem that libR.so has any links to libunwind.so

$ ldd /usr/lib/R/lib/libR.so 
	linux-vdso.so.1 (0x00007ffd85d7e000)
	libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007e3258a00000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007e325992a000)
	libreadline.so.8 => /lib/x86_64-linux-gnu/libreadline.so.8 (0x00007e32598d5000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007e3259366000)
	libdeflate.so.0 => /lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007e32598c2000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007e325988e000)
	libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007e325987a000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007e325934a000)
	libtirpc.so.3 => /lib/x86_64-linux-gnu/libtirpc.so.3 (0x00007e325931d000)
	libicuuc.so.74 => /lib/x86_64-linux-gnu/libicuuc.so.74 (0x00007e3258600000)
	libicui18n.so.74 => /lib/x86_64-linux-gnu/libicui18n.so.74 (0x00007e3258200000)
	libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007e32592c7000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e3257e00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007e3259a3c000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007e3259873000)
	libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007e3259293000)
	libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007e325923f000)
	libicudata.so.74 => /lib/x86_64-linux-gnu/libicudata.so.74 (0x00007e3256000000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007e3255c00000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007e325920f000)
	libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007e3258937000)
	libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007e32591e3000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007e32591db000)
	libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007e32591ce000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007e32591c7000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007e32591b4000)

For reference, simple.jl is

module JuliaTest

    Base.@ccallable function add_r(a::Ptr{Csize_t}, b::Ptr{Csize_t}, out::Ptr{Csize_t})::Csize_t
        a = unsafe_load(a)
        b = unsafe_load(b)
        out = unsafe_wrap(Array, out::Ptr{Csize_t}, 1::Int)
        out[1] = a + b
    end

    Base.@ccallable function add_julia(x::Cint, y::Cint)::Cint
        return x + y
    end
end

[dmbates]

In a slack discussion a similar issue in the JuliaCall repository, JuliaInterop/JuliaCall#238 (comment), was referenced. So part of the problem is that an older version of libunwind.so.8 is being accessed because R is overwriting the LD_LIBRARY_PATH.

$ LD_LIBRARY_PATH=. R --quiet
> Sys.getenv("LD_LIBRARY_PATH")
[1] "/usr/lib/R/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/default-java/lib/server:."

The suggestion there was to preload the libunwind from the Julia lib directory but that encounters problems further down the line.

$ LD_PRELOAD="/home/dmbates/.julia/juliaup/julia-nightly/lib/julia/libunwind.so.8" R 
R version 4.4.3 (2025-02-28) -- "Trophy Case"
Copyright (C) 2025 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> dyn.load("libsimple.so")
ERROR: could not load symbol "jl_system_image_size":
/usr/lib/R/bin/exec/R: undefined symbol: jl_system_image_size

[palday]

Would Sys.setenv help? It seems pretty problematic that R is overwriting environment variables instead of just extending them.

[dmbates]

I have the feeling that trying to retroactively patch up environment variables after R has overridden them will lead to a "twisty maze of passages" from which there is no escape while retaining one's sanity.

[palday]

I have the feeling that trying to retroactively patch up environment variables after R has overridden them will lead to a "twisty maze of passages" from which there is no escape while retaining one's sanity.

Given that R already melts brains, we should probably preserve what little sanity remains available to us. 😂

[dmbates]

I have a proof-of-concept using juliac to create a shared library, dyn.load()'ing it from an R session and evaluating Julia code on R objects.  At present it is just using the .C interface, passing pointers and ignoring the return value (which means that you must pass storage for the result and modify those locations in the Julia code).  So it is not the .Call or .External interface yet but it is encouraging,  The example uses the convolve function that is used extensively in the Writing R Extensions manual.  The Julia function is

Base.@ccallable function jc_convolve(
    ra::Ptr{Cdouble},
    rna::Ptr{Cint},
    rb::Ptr{Float64},
    rnb::Ptr{Cint},
    rab::Ptr{Cdouble},
)::Nothing
    # extract lengths from pointers
    na = unsafe_load(rna)
    nb = unsafe_load(rnb)
    # convert pointers and lengths to 1-dimensional Julia arrays
    a = unsafe_wrap(Array, ra, na)
    b = unsafe_wrap(Array, rb, nb)
    ab = fill!(unsafe_wrap(Array, rab, na + nb - one(na)), 0.0)
    for i in eachindex(a)
        ai = a[i]
        for j in eachindex(b)
            ab[i + j - 1] += ai * b[j]
        end
    end 
    return nothing
end

The compilation call is

% juliac --output-lib build/libmylib --project libtest ./src/libtest.jl --trim=safe --experimental --bundle            

and the call in R looks like

> dyn.load("build/lib/libmylib.dylib")
> is.loaded("jc_convolve")
[1] TRUE
> a <- c(1, 2, 3, 4, 5)
> typeof(a)
[1] "double"
> b <- c(5,4,3,2,1)
> 
> conv <- function(a, b)
+     .C("jc_convolve",
+        as.double(a),
+        as.integer(length(a)),
+        as.double(b),
+        as.integer(length(b)),
+        ab = double(length(a) + length(b) - 1))$ab
> conv(a, b)
[1]  5 14 26 40 55 40 26 14  5

[dmbates]

@simonbyrne, @randy3k Do you think it is feasible to extract the code that handles conversion from/to SEXPRECs into a separate package that is included by RCall.jl? The purpose would be to allow for Julia packages that are to be compiled to shared libraries that will be dyn.loaded into R and called with the .Call or .External interfaces to also import methods from this package.

If so, can you speculate on which RCall.jl source files should be moved to such a package?

Finally, do you think that RObjects.jl would be a good name for such a package? Do you have a better name?

[dmbates]

My guess at which files should be in a package RObjects.jl, if such a package were to be created, is just the first 12 files included in src/RCall.jl. That is, types.jl, Const.jl, methods.jl and all the source files in src/convert. Am I off-base somewhere?

[dmbates]

I'm running into a wall here trying to implement the .Call interface in Julia. The basic problem is in calling C functions from libR in the Julia code. This runs pretty deep - if you pass any numeric, integer or logical object from R to Julia as a Ptr{Sxp} you need to pull apart the type of object, a pointer to its contents, its length, and sometimes the attributes, which are stored as a named cons list. Even getting the length is not simple once "long" vectors in R became available. There is a 32-bit integer field in the struct that contains the length if it is less than typemax(Int32) but longer vectors use a sentinel value in that position and squirrel away the length in another location as in Int64. The recommended way to get the length is to call a C function Rf_xlength in libR.

So somehow I need to find the libR that is in use by the calling R instance. I don't think it is just a matter of dlopen'ing a path name in that some functions that are used extensively - to allocate and protect/unprotect storage in a vector (and R only has vectors - it doesn't have scalars) manipulate storage locations in the caller's memory image. So somehow the version of Rf_protect that is called from Julia needs to have access to the caller's memory image. I'm getting beyond my knowledge of how functions like dlopen work and would be grateful for any advice.

[dmbates]

I managed to get an example of the .Call interface from R working after reading the documentation for Julia's ccall function. (Pro tip: read the documentation.) The first argument to ccall can be a Symbol or String which will be resolved by traversing the list of currently loaded dynamic libraries. That is, all my fussing about finding the desired version of libR is unnecessary.

I modified some of the code from RCall.jl/src to use only the symbol name, not the (symbol_name, library_name) tuple and now have

> dyn.load("build/lib/libmylib.dylib")
> is.loaded("jc_convolve_Call")
[1] TRUE
> .Call("jc_convolve_Call", c(1,2,3,4,5), c(5,4,3,2,1))
[1]  5 14 26 40 55 40 26 14  5

where the Julia code is

Base.Base.@ccallable function jc_convolve_Call(
    ra::Ptr{RealSxp},
    rb::Ptr{RealSxp},
)::Ptr{RealSxp}
    a = unsafe_vec(ra)
    b = unsafe_vec(rb)
    rab = allocArray(RealSxp, length(a) + length(b) - 1)
    ab = fill!(unsafe_vec(rab), 0.)
    for (i, ai) in enumerate(a)
        for (j, bj) in enumerate(b)
            ab[i + j - 1] += ai * bj
        end
    end
    return rab
end

and the definitions of unsafe_vec and allocArray come from RCall.jl with the ccall calls just using the symbols like Rf_allocArray as the first argument.

[dmbates]

After some further tinkering I can produce a slightly more general version using Rf_coerceVector to coerce the arguments into numeric vectors if possible. The Julia code looks like

Base.@ccallable function jc_convolve_Call(ra::Ptr{Sxp}, rb::Ptr{Sxp})::Ptr{RealSxp}
    a = unsafe_vec(protect(coerceVector(ra, RealSxp)))
    b = unsafe_vec(protect(coerceVector(rb, RealSxp)))
    ab = zeros(length(a) + length(b) - 1)
    for (i, ai) in enumerate(a)
        for (j, bj) in enumerate(b)
            ab[i+j-1] += ai * bj
        end
    end
    unprotect(2)
    return sexp(ab)
end

Note that the result is allocated in Julia with the zeros function then converted by sexp to a Ptr{RealSxp} to be returned to R.

This version allows more flexibility in the arguments

> .Call("jc_convolve_Call", c(1,2,3,4,5), c(5,4,3,2,1))
[1]  5 14 26 40 55 40 26 14  5
> typeof(1:5)
[1] "integer"
> .Call("jc_convolve_Call", 1:5, c(5,4,3,2,1))
[1]  5 14 26 40 55 40 26 14  5

Is anyone else interested in this? There hasn't been a whole lot of discussion other than posts by me.

My objective would be to offer Julia as a back-end computational engine to R, similar to the use of C++ through Rcpp. The answer to "why do this?" is obvious to anyone who has programmed in both C++ and Julia.

One current issue is that I can test this on MacOS and on Linux but not on Windows and it may not work on Windows. Also I haven't had any responses on whether it would be worthwhile creating a package that just provides the boxing/unboxing of R objects in Julia so that such a utility package could be included by RCall.jl and by Julia code that is intended to be compiled to a library and called from R.

[biona001]

I'm extremely interested in this and have been following this thread closely. But to be honest, this seems a lot more technical than I'd be comfortable developing, so I'm not sure how I can help 😭

My objective would be to offer Julia as a back-end computational engine to R, similar to the use of C++ through Rcpp

This is exactly what I'm hoping for. Sooo glad to see you are thinking (and actually doing!) the same