Foreign Function Interfaces (FFI's) can be intimidating. For those of you unfamiliar with the term, an FFI serves as a facility in a language that acts as a bridge to other languages, often C. The landscape of programming has changed over decades, and I imagine I'm not alone in having learned Python and JavaScript first; so it's interesting to basically use the FFI as a spring board for understanding C itself. In this particular article, we'll look at bringing the mpg123 audio library to Chez Scheme.
The mpg123 library is a cross-platform audio decoding and encoding library for mpeg formats including mp3's. It has a fairly straightforward API and has a nice range of challenges to gracefully introduce features of Chez's FFI.
Getting Started
The only prerequisites are the mpg123 development and header files. Using apt, this was as simple as:
$ sudo apt-get install libmpg123-devAn Introduction to the Chez FFI
Loading a library
The first step in interoperating with C code is to load a shared library. In the case of the mpg123, we obtained this with the download of the development files. This could of course be a shared object of your own creation instead. We load this into Chez with load-shared-object.
(define lib-mpg123 (load-shared-object "libmpg123.so"))Of course, with mpg123 being cross-platform, we could bring in mpg123.dll or better yet, conditionally load the right file based on the OS, but for now we'll keep things simple.
Calling Foreign Procedures Let's take a look at what can be described as mpg123's "first" function, mpg123_init. The header file is a good place to familiarize ourselves with the functions signature:
MPG123_EXPORT int mpg123_init(void);We can safely ignore the MPG123_EXPORT prefix, for the reasons given here. We have a return type, int, a procedure name, mpg123_init and no arguments, expressed as void.
foreign-procedure from the (chezscheme) library is the key to calling C procedures from Scheme. It takes the foreign procedures name, the foreign arguments, enclosed in parenthesis, and the return type.
The following defines a scheme function that calls into the library:
(define mpg123_init (foreign-procedure "mpg123_init" () int))That's it! There's a few things to elaborate on, but regardless, (mpg123_init) now calls mpg123_init from the C Library! The last argument int looks just like the C return type, it's important to note however, that this is a Chez foreign return type, an ftype, for the most part they look just like their C counterparts. There are differences however: void is not a type in the ftype system, to pass void, as in the case of the foreign arguments, you simple pass nothing, () and pointers to ftypes are expressed via (* ftype). ftypes are very powerful and we'll explore them more as we continue.
A helpful macro The following macro I used extensively in wrapping the library.
(define-syntax define-function
(syntax-rules ()
((_ ret name fpname args)
(define name
(foreign-procedure (symbol->string 'fpname) args ret)))))This allows the previous function to be defined:
(define-function int initialize mpg123_init ())This puts the return type before the name, allows me to pass a new name, and then accepts the foreign entries name and arguments, structuring it much more like the C signature.
Schemeification
Therein lies the first question, what should wrapping a C library look like. To me, the goal was a Scheme-like interface to the library. Instead of having a bunch of functions prefixed mpg123_, I decided that I would rely on the R6RS library import features to allow users to choose their own prefix. I also chose to use snake-cased function names, preferring full words over abbreviations and expanding short names like open, read and write (some of which conflict with existing symbols) to more specific names: open-stream, read-stream, and write-stream respectively. This, of course, is all personal preference.
Working with C Strings
Let's take a look at the mpg123_init function again. It returns an int. Per the documentation that int it returns is either MPG123_OK (0) or otherwise, an error number. Would be nice to get a string of that error... Let's do just that, by moving on to mpg123_plain_strerror. Again, let's start by examining the function signature.
const char* mpg123_plain_strerror ( int errcode )const char*, in this case, represents a C string, this is a char sized pointer to the start of a string that is null terminated. Here the ftype system extends standard C types by providing a string ftype. This makes our definition look like this:
(define-function string get-error-string mpg123_plain_strerror (int))Now a call to (get-error-string -1) will return "A generic mpg123 error." Hopefully, by now the rich ecosystem of C libraries should feel more accessible to users new to the FFI.
A bit about mpg123
mpg123's API functions in a fairly straight forward manner. To put it simply, you create a handle to a chunk of memory, and you pass that handle around to functions that operate on it. Creating a handle is done with the mpg123_new function:
mpg123_handle* mpg123_new ( const char * decoder, int * error )Both the arguments in this function are optional and NULL can be passed instead. For simplicity's sake we'll do just that. However, even this simplified function has a few things worth examining.
(define-function void* %new-handle mpg123_new (void* void*))
(define (new-handle) (%new-handle 0 0))Of note, for now, we're working with all the pointers opaquely. I don't really care that mpg123_handle* is the type of that pointer, void* is more than enough. Likewise with the pointers as arguments. Chez is handing those off to C and the C library is going to assume you're handing it the expected pointers. Of course, (string (* int)) could have been used as the argument signature and a custom ftype could be provided as the return. In fact, it's likely we'd do just this as we increase the abstraction of the API, but for now let's proceed. Another reason to proceed is to show that not every interaction with the FFI needs to be "complete", you don't need to wrap whole libraries and type definitions, wrap what you need as opaquely or explicitly as your needs dictate. The other thing to point out is the % prefix, the Scheme API layer will get the 'clean' name,
Handling Errors Many of the functions simply return an integer error state (or success state in the case of 0), and this works fine at the C level, but with R6RS there are more idiomatic ways to handle these things.
Let's implement a function that can get passed any function, returns an int success or error code, and handles exemptions accordingly.
(define (handle-mpg123-error err-code)
(cond [(zero? err-code) err-code]
[(member err-code '(-12 -11 -10))
(begin
(raise (condition (make-warning)
(make-message-condition (get-error-text err-code))))
err-code)]
[else
(begin
(raise (condition
(make-error)
(make-message-condition (get-error-text err-code))))
err-code)]))We return the err-code regardless, this keeps the return values in-line with the API, but now errors and warnings are appropriately dispatched. To move things along, -12 -11 -10 are hard-coded as warning returns, this is per the documentation.
Now our initialize function should be renamed %initialize and a new initialize should utilize the error handler:
(define (initialize) (handle-mpg123-error (%initialize)))Working with Memory
It's not always so straightforward. For example, if we're going to decode an audio file we need to know what kind of audio file we're working with and what attributes this possesses. Enter mpg123_getformat:
int mpg123_getformat ( mpg123_handle * mh, long * rate, int * channels, int * encoding )So, applying what's been used so far we have the following two functions:
(define-function int %get-format mpg123_getformat (void* (* long) (* int) (* int)))
(define (get-format handle rate channels encoding) (%get-format handle rate channels encodings))While this would accurately map to the underlying C API, it would make for a less than ideal API for scheme users. rate,channels, and encoding are all passed as pointers, mpg123 then goes to that address to write that value in place. Let's write get-format in a continuation passing style that is more familiar to Scheme users:
(define (get-format handle k)
(let ([rate-ptr (foreign-alloc (ftype-sizeof long))]
[channels-ptr (foreign-alloc (ftype-sizeof int))]
[encoding-ptr (foreign-alloc (ftype-sizeof int))])
(handle-mpg123-error
(%get-format handle rate-ptr channels-ptr encoding-ptr clear))
(let ([rate (foreign-ref 'long rate-ptr 0)]
[channels (foreign-ref 'int channels-ptr 0)]
[encoding (foreign-ref 'int encoding-ptr 0)])
(foreign-free rate-ptr)
(foreign-free channels-ptr)
(foreign-free encoding-ptr)
(k rate channels encoding))))This might seem like a lot, but it's rather straightforward taken piece-by-piece. foreign-alloc allocates and returns a pointer (the memory address) to the requested amount of allocation space, received from passing ftype-sizeof the desired ftype. So in essence, only a lambda needs to be passed to get-format, the other needed arguments are created in the outer let.
After we call %get-format we extract our values from those addresses using foreign-ref and passing it the ftype we want (as a symbol), the memory address, and any offset bytes (0 in this case).
Once we have our values we can free up the allocated memory using foreign-free and call the passed lambda, passing in our values.
Let's take a look at using our new get-format function:
(get-format my-handle (lambda (rate channels encoding)
;; rate, channels, and encoding are available here
))One area that can get a bit tricky is dealing with arrays. A common scenario in mpg123 is getting back an array of null-terminated strings. This C data type is of little use in the Scheme side, and returning it isn't quite idiomatic. In Scheme, I feel like it would make more sense to return a list of strings.
The following function takes that pointer to a memory location (for example, perhaps one returned from mpg124_decoders) and starts to scan it, building up a list of strings to return. In this instance the foreign-alloc isn't needed, as the allocation of memory has already occurred, and we are merely recursing over that memory space, but note how foreign-ref is used to extract the current value at the index.
(define (c-string-array->list-of-strings ptr)
(let ([mem-block-ptr ptr])
(define (inner index list-of-strings string-value prev-value)
(let ([value (foreign-ref 'char mem-block-ptr index)])
(if (eq? value #\nul)
(if (eq? prev-value #\nul)
(reverse list-of-strings)
(begin
(inner (+ 1 index)
(cons string-value list-of-strings) "" value)))
(inner (+ 1 index) list-of-strings (string-append string-value (string value)) value))))
(inner 0 '() "" #\nul)))Which makes calls to (get-decoders) return lists that look like '("AVX" "NEON64" "NEON" "ARM" "x86-64" "SSE"), which is of far more use to the Scheme API users than an address to C-strings.
Final Thoughts
Interacting with C Libraries is surprisingly straightforward, for many use-cases it feels almost as simple as calling the function. That next step, making the API feel more natural to the conventions of the targeted runtime is a challenge in and of itself and I hope the examples provided give some insight into some of the many approaches one can undertake when shaping the API.
I want to end by reiterating that this is by no means a tutorial on the expert use of the FFI, it's more of a journal of my progress in exploring the FFI. I'd also like to thank the users of the Scheme discord as always for answering the many question's I've had along the way, especially Sam, Erkin and shakdwipeea.
The WIP for chez-mpg123 lives here. A gist that uses the this library to play an mp3 from the command line can be found here.
There are many, many, more things to be said about the FFI but we'll save that for the future.