Compatibility with non-ESM node usage?

[borkdude]

There are some users who want to write code (libraries) that should run both in shadow or vanilla CLJS (with target node) and nbb, but they run into the $default thing as the only difference. The $default is necessary because nbb operates in an ES modules context whereas other compilation targets don't (the only CLJS option for target ESM is currently shadow).

I recommend that they could use .cljc files and make a reader conditional branch for nbb, but to be honest this gets old quite fast.

What if nbb could support some way to facilitate this use case. E.g.:

(ns foo
  (:require
   ["puppeteer" :as ^:default puppeteer]))

;; in nbb this means "puppeteer$default" :as puppeteer

Then again, if people would switch to :target :esm in shadow (or in the future CLJS) then they would add $default and nbb would read $default.default? This is probably asking for trouble...

[borkdude]

thheller yeah and don't add anything "smart" to work around it. that it what got us to this mess in the first place. webpack and babel trying to hide the interop issues until they realized they really can't 😛

[thheller]

FWIW I am looking into something related for shadow-cljs and learned something new. Apparently it is both valid to use import * as React from "react" and import React from "react" for commonjs packages from node and related tools such as rollup.

// test.mjs
import * as R1 from "react";
import R2 from "react";

console.log("R1", R1.createElement("div", {}))
console.log("R2", R2.createElement("div", {}))

// node test.mjs
R1 {
  '$$typeof': Symbol(react.element),
  type: 'div',
  key: null,
  ref: null,
  props: {},
  _owner: null,
  _store: {}
}
R2 {
  '$$typeof': Symbol(react.element),
  type: 'div',
  key: null,
  ref: null,
  props: {},
  _owner: null,
  _store: {}
}

Note that both behave exactly the same despite react only being available as commonjs. However if you do the same for puppeteer that is not true. I tested a few other commonjs packages and they behave like react. I don't yet know why puppeteer is different but maybe something to be aware of when coming up with a solution.

[borkdude]

Interesting. Thanks for sharing and keep me posted if you have any new info on this!

[borkdude]

One possible solution without converting your entire project to .cljc:

Make one or more namespaces that only deal with importing your deps:

(ns deps
  (:require #?(:org.babashka/nbb
               ["puppeteer$default" :as p]
               :cljs
               ["puppeteer" :as p])))

#?(:cljs
   (def puppeteer p))

And then use those namespace(s) from your scripts:

(ns script
  (:require [deps :refer [puppeteer]]))

(.launch puppeteer #js {:headless false})

The above works with both lumo and nbb.

[borkdude]

Text posted in cljs-dev channel:

A while ago I discussed here some of the issues around targeting ES modules (which nbb does) and the necessity of having people add $default to their namespace names. There seemed to be no way around this since the JS world imposed this breakage and it would be highly disputable if a tool like nbb would have to "compensate" for that. Also when targeting ES modules with shadow (via :target :esm ) you end up with the exact same $default stuff, so at least it's compatible from that perspective (although vanilla CLJS currently doesn't emit ES modules).

So with that settled, someone wrote a multi-namespaced project which consumes node libraries, but he (@chris McCormick) tried to make it work with both shadow-cljs (target node-script) and nbb, but then this CJS vs. ESM stuff came up again.

I believe I've found a workaround for this situation by

• introducing reader conditionals for nbb :
  #?(:org.babashka/nbb ["cookie-parser$default" :as r-cookies]
  :cljs ["cookie-parser" :as r-cookies])
• moving all dep imports to an isolated .cljc namespace (so you don't have to deal with this boilerplate in every ns and you don't have to turn all your .cljs files into .cljc files)
• re-export the deps using vars:
  (def cookies r-cookies)
• and then referring to the deps as (:require [deps :refer [cookies]) , (.foobar cookies ...)

Full example:
https://github.com/borkdude/sitefox/blob/nbb-example/src/sitefox/deps.cljc

Just sharing this FYI. Perhaps reader conditionals can be used for similar purposes in the future to mitigate these incompatibilities in the JS ecosystem.

[borkdude]

I'm inclined to add something to the metadata part of the namespace to make this easier:

(ns example
  {:nbb/auto-default #{"handlebars"}} ;; automatically selects $default for
                                      ;; these libs for compatibility
  ;; {:nbb/auto-default true} ;;=> does it for every lib that doesn't already have $property
  (:require ["handlebars" :as handlebars]))

(def template (.compile handlebars "Hello {{name}}!"))

(prn (template #js {:name "world"}))

/cc @thheller: perhaps shadow is also interested in a feature like this and we could come up with some standard?

[thheller]

As I have said before I'm completely against doing anything regarding this on the tool side. The transition to ESM needs to happen cleanly and working arround it from the tool side is what got us to this mess in the first place. Frankly I don't even know what is going on anymore because of all the tools trying to make things easier instead of just being strict.

If you have some time try to figure out why this is fine for react but not for handlebars or puppeteer.

import * as R1 from "react";
import R2 from "react";

// why do both variants work here?
console.log("R1", R1.createElement("div", {}))
console.log("R2", R2.createElement("div", {}))

import * as H1 from "handlebars";
import H2 from "handlebars";

// but not here?
console.log("H1", H1.compile("test"))
console.log("H2", H2.compile("test"))

react is also just commonjs so why can I use two variants here to access createElement while I cannot do the same for the handlebars? Does this only apply for node or what do webpack or rollup do here?

I can't investigate this too much myself currently but that is the question that needs an answer before any "workarround" should be considered. The answer might be that this is just a problem on the package side and might be fixed on that end at some point, which may then end up breaking any possible workarround implemented on the tool side. And so the circle repeats ...

[borkdude]

So when I log R1 I see that it is an object that has all the functions including createElement and a default key with again all the functions including createElement.
When I log H1 I see it's an object with only a default key. So the question is: why does React have all the functions merged on the module object itself as well as an default key. Going to look further into this.

[borkdude]

Found out the following:

When I edit the index.js of React and add the flag mod.__esModule = true then the merging of functions into the top level of the module disappears:

'use strict';
if (process.env.NODE_ENV === 'production') {
  module.exports = require('./cjs/react.production.min.js');
} else {
  var mod = require('./cjs/react.development.js');
  mod.__esModule = true;
  module.exports = mod;
}

Then this prints only default for both libraries:

import * as R1 from "react";
console.log(Object.keys(R1));

import * as H1 from "handlebars";
console.log(Object.keys(H1));

The handlebars library sets __esModule = true for some reason.

[borkdude]

Some more data:
When I first assign react to a const and then export that const, I also only get the default, regardless of __esModule.

'use strict';
if (process.env.NODE_ENV === 'production') {
  module.exports = require('./cjs/react.production.min.js');
} else {
  const mod = require('./cjs/react.development.js');
  // console.log(Object.keys(mod));
  // mod.__esModule = true;
  module.exports = mod;
}

[thheller]

This Module Namespace Exotic Object can be directly observed either when using import * as m from 'cjs' or a dynamic import:

I guess I misread this line from the documentation as always providing this "mirrored" object, but I guess thats not what they meant. So it is actually still relying on dynamic inspection of the code as mentioned in

Named exports detection covers many common export patterns, reexport patterns and build tool and transpiler outputs. See cjs-module-lexer for the exact semantics implemented.

So whenever an export cannot be discovered statically it won't work and you need to use the default export. I really wish they went with the default-only way but I guess we'll have to live with this mess much longer then when packages aren't forced to upgrade (or provide two variants).

Since you are using dynamic import and can look at what that returns you could maybe just infer what you need and adjust how :as works accordingly?

[borkdude]

@thheller Just spitballing here, but what if CLJS chose default as the default property when targeting ES6 (so ["term-size"] instead of ["term-size$default]. If you really wanted to have * you could write ["term-size$*" :as term-size-all] or so.

[thheller]

import * as X from "thing" syntactically exactly matches (:require ["thing" :as X]) and technically X/default is valid as a regular property. Absolutely no on thing$ as the "all".

[borkdude]

Yeah, got it. But how do you feel about existing projects/libraries that use library foobar, that want to target both ES6 and CJS outputs and either have ["foobar" :as foobar] or ["foobar$default" :as foobar] but you can't have both at the same time?

[thheller]

As I said I'm absolutely against doing anything on the tool side. I absolutely hate this situation, but I don't see a clean way out of it. I expect the problem to get worse as more code moves to ESM. I absolutely despise that the user has to care about how the library is published, but thats the way the JS community went and we can hardly change that.

[borkdude]

OK, then this approach:

https://github.com/chr15m/sitefox/blob/main/src/sitefox/deps.cljc

is the only workaround I can come up with for now.