Implement new FFI system as in the proposal by davazp · Pull Request #583 · jscl-project/jscl

davazp · 2026-01-31T00:06:16Z

Done

#j"foo" reader macro for JS string literals
(jsstring x) builtin for converting Lisp strings to JS strings
Printer support: JS strings print as #j"...", raw with princ
Printer support for +true+, +false+, +null+, +undefined+
Rename new → object, make-new → new, remove object!
Remove lisp_to_js conversion from object
Ensure Lisp strings work for property access in oget/oset
Fix disassemble to work with lambda forms
Multiple values side-channel (internals._mv instead of values argument)
unwind-protect save/restore of _mv
Comprehensive multiple-values test suite
(clstring x) convert JS string to CL string
(jsbool x) convert t/nil to +true+/+false+
(clbool x) convert JS booleans to t/nil
Remove redundant oget*/oset* variants
Remove js_to_lisp conversion from oget (make it behave like oget*)
Remove lisp_to_js conversion from oset (make it behave like oset*)

Not included:

LispString representation for CL strings (wrapping JS strings instead of arrays)

function (values, ...args) { } Now CL functions take a general JS signature fucntion (...args) { ... } but can write multiple values to a internals._mv variable. The compiler has been adjusted to carefully set and reset this side channel variable in multiple places. mv contain all values, not just the secondaries That way we can differentiate between (values) and (values nil)

The _mv side channel is cleared too early in compile-funcall. When a function call is compiled with *multiple-value-p* = t, the compiler generates: (internals._mv = null, func.fvalue(arg1, arg2, ...)) The _mv = null clearing happens before argument evaluation (due to JS comma operator semantics - the left side executes first, then the arguments are evaluated as part of the function call expression on the right). But if any argument contains a sub-expression that calls a function returning multiple values (like intern, which returns (values symbol was-present-p)), that sub-expression will set _mv to a non-null value after the clearing but before the outer function call executes. When the outer multiple-value-call then checks _mv, it finds the stale values from the inner intern call instead of the actual multiple values from the outer call. This causes it to use the wrong values. In the defstruct case: 1. (setf (dsd-constructors dd) (or constructors (list (make-dsd-constructor :name (%symbolize "MAKE-" (dsd-name dd)) :boa :default)))) 2. The setf is wrapped in a multiple-value-call by the compiler 3. Inside the argument evaluation, %symbolize calls intern which sets _mv = [MAKE-ES, NIL] 4. The outer multiple-value-call picks up this stale _mv value 5. Instead of passing the correct list (#S(dsd-constructor ...)) to the setf function, it passes MAKE-ES 6. So (dsd-constructors dd) becomes the symbol MAKE-ES instead of a list of constructor structs 7. Later, mapcar over this "list" tries caar on MAKE-ES, producing the TYPE-ERROR The fix would need to ensure _mv is cleared after all arguments are evaluated but before the function body begins executing, or the code generation needs to evaluate arguments into temporaries first and clear _mv after.

github-actions · 2026-01-31T21:31:14Z

Deploy preview ready!
https://deploy-preview-583--jscl-preview.netlify.app

Initialize the symbols earlier as uninterned symbols, and intern them into the CL package later.

kchanqvq · 2026-02-01T00:03:22Z

I don't think +true+ etc should be printed as +true+. This will print both '+true+ and +true+ as +true+, but the former is a Lisp symbol, the latter is a JS primitive value.

davazp · 2026-02-01T01:11:45Z

I don't think +true+ etc should be printed as +true+. This will print both '+true+ and +true+ as +true+, but the former is a Lisp symbol, the latter is a JS primitive value.
Good point. Hm, any other ideas of how we could make it readable? Read it /print it as #j+true+ or similar perhaps?

It would be very convenient if any js object is printed in a way that would be readable too.

src/ffi.lisp

davazp · 2026-02-01T01:30:08Z

The ergonomics are not great yet:

["foo", "bar"].map(x => x.toUpperCase())

can be done like

  (let ((arr (vector #j"foo" #j"bar")))                                                                                                                                                     
      ((oget arr "map")                                                                                                                                                                     
       (lambda (x) ((oget x "toUpperCase")))))

A syntax for calling methods would be nice

  (let ((arr (vector #j"foo" #j"bar")))                                                                                                                                                     
      (jsmethod arr "map"
        (lambda (x) (jsmethod x "toUpperCase"))))

would be even nicer if #j allowed for dynamic root more easily, then

  (let ((arr (vector #j"foo" #j"bar")))                                                                                                                                                     
      (#j:(arr):map (lambda (x) (#j:(x):toUpperCase))))

kchanqvq · 2026-02-01T01:32:44Z

I think the current oget is fine. I don't like the idea of adding too many/involved/complicated custom syntax. This include the #{...} mentioned elsewhere, I think we should leave this to the user to define it if they find this useful. And they would be responsible for using something like named-readtable to manage collision (although we don't have read table yet, but we will).

kchanqvq · 2026-02-01T01:36:27Z

Good point. Hm, any other ideas of how we could make it readable? Read it /print it as #j+true+ or similar perhaps?

It would be very convenient if any js object is printed in a way that would be readable too.

A common solution found in the wild is #.+true+. Do we really want to make this readable? Note e.g. in SBCL, something like NaN is not printed readably.

Use special UNBOUND marker instead of undefined.

kchanqvq

Kudos for this ambitious undertaking!

Besides the inline review/discussion, two high-level things:

I think a lot of complexity is added to emulate JS values in the host because we use #j:... literal during bootstrap.

How strongly do you want to use DEFSTRUCTs for the emulation? We could use some magic symbol markers, which will remove the need for MAKE-LOAD-FORM. But you could say this is one step backward.
The option I like better is to not use #j:... literal in the host at all. I think this is possible: just write (jsstring "cl-string"), and our compiler is smart enough the elide the conversion. If this works I think this can massively simplify. We're currently only emulating these literals as opaque datum anyway and I think it's unlikely we really need them.

I think representing NIL with undefined will simplify lots of things and is more ergonomic. But this probably goes into another PR. Let's discuss!

docs/ffi.md

src/clos/describe.lisp

kchanqvq · 2026-02-03T07:08:25Z

src/compiler/codegen.lisp

+           (output-arg (cadr args))
+           (dolist (operand (cddr args))
+             (js-format ",")
+             (output-arg operand))))


I think this loop can be written cleaner with a first-arg-p flag. Something like
(let ((first-arg-p t)) (dolist ... (when first-arg-p ...output-comma... (setq first-arg-p nil)) ...)). And there would be no need for an flet (and cddr etc to peek into later part of the list).

src/compiler/compiler.lisp

src/boot.lisp

src/documentation.lisp

src/ffi.lisp

src/compiler/compiler.lisp

davazp · 2026-02-03T09:37:03Z

I think a lot of complexity is added to emulate JS values in the host because we use #j:... literal during bootstrap.
How strongly do you want to use DEFSTRUCTs for the emulation? We could use some magic symbol markers, which will remove the need for MAKE-LOAD-FORM. But you could say this is one step backward.

The option I like better is to not use #j:... literal in the host at all. I think this is possible: just write (jsstring "cl-string"), and our compiler is smart enough the elide the conversion. If this works I think this can massively simplify. We're currently only emulating these literals as opaque datum anyway and I think it's unlikely we really need them.

I had this before. But the thing is now the reader does not return (jsstring "foo"), it does return a raw JS string directly. We'd need some extra conditionals in reader and compiler.

We could make the reader return logic conditional on being the host or not. But even if now they are only tokens, it'd be useful to dispatch based on js types in the host. So I think we'd end up replicating defstructs on top of s-expressions.

We might use defstruct lists to remove the make-load-form to this also? we have to be careful with consp when implementing the compatibility typeof but that's okay I guess.

davazp · 2026-02-03T09:39:44Z

I think representing NIL with undefined will simplify lots of things and is more ergonomic. But this probably goes into another PR. Let's discuss!

yeah. undefined is a bit prefer for NIL over null and false now I think. oget and oget? would return nil.

But undefined can't do the nice jscl_car trick we did to make (car nil) = nil!

I think we should still experiment with it, but I'd leave it for another branch.

kchanqvq · 2026-02-03T09:47:19Z

I had this before. But the thing is now the reader does not return (jsstring "foo"), it does return a raw JS string directly.

IIUC my proposal is different. My proposal is never use #j"literal" in any code that is read in the host. This way, no JS value literal will ever get into the source forms processed by the host, thus no emulation/dumping etc is required at the host stage.

I speculate this is possible, and wherever we use #j"literal" in the host right now can probably be rewritten to use (jsstring "..."), thus embedding no JS primitive value literal in the code (processed by the host).

kchanqvq · 2026-02-03T09:52:40Z

But undefined can't do the nice jscl_car trick we did to make (car nil) = nil

We could also use ?.$$jscl_car for this. I don't know will this bring more overhead/are JS engines specifically optimized for ?.. I'll experiment this in another branch!

davazp · 2026-02-03T17:54:00Z

IIUC my proposal is different. My proposal is never use #j"literal" in any code that is read in the host. This way, no JS value literal will ever get into the source forms processed by the host, thus no emulation/dumping etc is required at the host stage.

I speculate this is possible, and wherever we use #j"literal" in the host right now can probably be rewritten to use (jsstring "..."), thus embedding no JS primitive value literal in the code (processed by the host).

I see. That could work. So to make sure I understand:

Provide jsstring and jsbool as primitives
Compiler does both:
- (eq (typeof X) (jsstring "string")) (for target mostly)
- jsstring primitive (for host mostly)

So we do not dump structs.

I'll still have to keep the structs. And make jsstring a constructor for it. But at least we remove the make-load-form.

Use jsstring and jsbool in key bootstrap places to make `make-load-form` unnecessary.

davazp · 2026-02-03T18:56:18Z

I'm happy with the state of the PR now.

So if not objections, I'd leave extra improvements for future PRs.

kchanqvq · 2026-02-03T21:25:24Z

I'll still have to keep the structs.

I think we can get rid of any emulation on host stage, but I'll have to experiment, only way to know!

Otherwise LGTM! There're some cosmetic simplification on the compiler code that I wanted (like the loop for generating call), but we can leave that for later.

docs/ffi.md

davazp · 2026-02-04T09:09:25Z

Merging! 🎉 Thank you @kchanqvq and @vlad-km for the help.

We continue with next iteration if needed on #586.

davazp added 6 commits January 31, 2026 16:01

Implement JS strings according to the new FFI system

ac12140

Rename new to object and make-new to new according to new FFI

a186afc

Fix disassemble to work with lambda forms

6ccc8a1

Print JS FFI constants as readable forms

1bee84e

Ensure Lisp strings can be used to access object properties

ef324b0

Avoid any conversions in object. remove object!

cb1e01c

davazp force-pushed the ffi-improvements branch from 57313f3 to 9a361ae Compare January 31, 2026 15:02

davazp added 2 commits January 31, 2026 17:48

Fix macroexpand to return a secondary value

63c060f

davazp force-pushed the ffi-improvements branch from 9a361ae to 63c060f Compare January 31, 2026 16:52

davazp added 7 commits January 31, 2026 18:38

Fix nested function call and multiple value bug

38b0342

Remove unnecessary detail from doc

55c267e

More readable withMV() by using an object

b436a3d

Add clstring, clbool and jsbool functions

68b4459

Add preview deployment for branches

c19b988

Add .nvmrc file with the node version

24669ec

davazp force-pushed the ffi-improvements branch from c8bbaba to 24669ec Compare January 31, 2026 21:30

davazp added this to the FFI System milestone Jan 31, 2026

davazp added 2 commits February 1, 2026 00:33

OGET and OSET do not make type conversions

b3ccfa8

Move NIL/T initialization earlier

7d68afd

Initialize the symbols earlier as uninterned symbols, and intern them into the CL package later.

Do not convert undefined to NIL in OGET

6b52388

davazp force-pushed the ffi-improvements branch from 50eec1e to 6b52388 Compare February 1, 2026 01:07

davazp commented Feb 1, 2026

View reviewed changes

src/ffi.lisp Show resolved Hide resolved

davazp added 5 commits February 2, 2026 19:12

Fix SLEEP

b8653e5

Export more FFI functions

ffbc822

Fix indentation

46d9005

Fix bound-ness of symbols bound to #j:undefined

cc42d23

Use special UNBOUND marker instead of undefined.

Add back js-null primitive type

652f8d6

davazp mentioned this pull request Feb 2, 2026

This #j:Fs and #j:FsPath is a bit hacky. #585

Open

kchanqvq reviewed Feb 3, 2026

View reviewed changes

davazp added 8 commits February 3, 2026 19:05

Remove TODO comment

bccbc41

Implement useful ?? JS operator for FFI

e59f314

Simplify condition

b3004ea

Simplify unnecessary conditional

d0ad01f

Simplify code with new ?? expression

9a13a6f

Use find-function helper

797634e

Prevent dumping JS compat values in host

ca8b2c3

Use jsstring and jsbool in key bootstrap places to make `make-load-form` unnecessary.

Make spread operator a bit more readable

c2e1319

davazp added 2 commits February 3, 2026 20:00

Define clone a regular function instead of compiler primitive

1738435

Export FFI symbols from JSCL/FFI

bd30e21

davazp force-pushed the ffi-improvements branch from 7215fba to bd30e21 Compare February 3, 2026 20:01

kchanqvq reviewed Feb 3, 2026

View reviewed changes

docs/ffi.md Outdated Show resolved Hide resolved

Export IN and udpate doc

8c0ae04

davazp merged commit dbbb15e into master Feb 4, 2026
1 check passed

davazp deleted the ffi-improvements branch February 4, 2026 09:09

Conversation

davazp commented Jan 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 31, 2026

Uh oh!

kchanqvq commented Feb 1, 2026

Uh oh!

davazp commented Feb 1, 2026

Uh oh!

Uh oh!

davazp commented Feb 1, 2026

Uh oh!

kchanqvq commented Feb 1, 2026

Uh oh!

kchanqvq commented Feb 1, 2026

Uh oh!

kchanqvq left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kchanqvq Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davazp commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davazp commented Feb 3, 2026

Uh oh!

kchanqvq commented Feb 3, 2026

Uh oh!

kchanqvq commented Feb 3, 2026

Uh oh!

davazp commented Feb 3, 2026

Uh oh!

davazp commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kchanqvq commented Feb 3, 2026

Uh oh!

Uh oh!

davazp commented Feb 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

davazp commented Jan 31, 2026 •

edited

Loading

davazp commented Feb 3, 2026 •

edited

Loading

davazp commented Feb 3, 2026 •

edited

Loading