Node.js default interop case

[guybedford]

I'm posting this here, which is a discussion that came up between Node.js and RollupJS but has grown to include members from other projects as well. See rollup/plugins#635 for the original issue.

@evanw I would really value your feedback here as interop is one of those things where it happens too quickly in a brief moment, and then we are stuck with it for years. Even if the action is do nothing, we need to make sure it is a considered do nothing.

If you have a chance to read that thread or have any questions I would value your input. Perhaps we may just have to live with some thrashing. I still hope not. Best case we get alignment between bundlers.

[evanw]

It's very unfortunate that the situation with default is such a mess. I sympathize with the struggle for compatibility, although I'm not sure what the best answer is. Thank you for reaching out about this.

Side note: If we're talking about better interop between bundlers, one thing I'd advocate for is at least behaving consistently regardless of syntax. Right now Webpack, Rollup, and Parcel all expose different exports depending on how you access them syntactically. I did a brief survey of this here: #472 (comment). This makes it difficult for me to know what the most compatible behavior is supposed to be.

I scanned the thread but I'm not too sure what the proposal is. The discussion is about how the behavior differs from Rollup, but I'm not too familiar with Rollup's behavior. If the proposal is just aligning with Node's semantics, that would look like this for esbuild I think:

• Exports from CommonJS objects are no longer live (they currently are in esbuild)
• The Babel __esModule property is no longer respected (it is currently respected in esbuild)

But wouldn't that break compatibility with the existing package ecosystem? I'm trying to make esbuild compatible with the ecosystem so it has a high chance of "just working" with existing packages, as long as it still follows ECMAScript module semantics (e.g. module namespace objects must not be callable). If a package works with Webpack, I'm inclined to also have it work with esbuild, even if it doesn't work with Node. The package may not even be intended for consumption with Node so enforcing strict compatibility with how Node does things even if it breaks stuff seems like it makes unnecessary hassle for users of bundlers.

I could see having esbuild do something like the union of the two approaches. If __esModule is present you would both have a default export that points to module.exports and also have named exports for every property on module.exports at the moment in time when the initial import is complete. I think that would have compatibility with both Webpack and Node. Do you have any thoughts about that?

To fully specify the behavior you'd want to describe what is done with non-enumerable properties and properties on the prototype of module.exports. People expect both of those to be present on import module namespace objects because of legacy transpilers that incorrectly convert import statements to bare require calls.

[guybedford]

It's very unfortunate that the situation with default is such a mess.

Interestingly it was actually created and specified to specifically support module.exports for Node.js compat, yet through inadequate planning and coordination we find ourselves here yes.

If __esModule is present you would both have a default export that points to module.exports and also have named exports for every property on module.exports at the moment in time when the initial import is complete. I think that would have compatibility with both Webpack and Node. Do you have any thoughts about that?

This is exactly what Node.js has now implemented with named exports extraction as described in https://nodejs.org/dist/latest-v15.x/docs/api/esm.html#esm_commonjs_namespaces. Node.js does this via a static analysis because it cannot execute CJS modules before the ES module execution phase and in the ES module specification named exports must be known at linking time, before execution.

As of the next 12.x release this month this named exports support will be fully backported and be the stable legacy target support base by early next yea. So it is fine for bundlers to extract named exports, and for modules without __esModule, but as per the documentation there yes they should not be live bindings and should be plucked while also offering the module.exports object as the default. Bundlers can even support more named exports than Node.js, as the compat subset is the really important aspect here.

But the compat subset breaks down on the default export when there is also an __esModule export. That's the specific problem being discussed in that thread where the alignment question comes in as it's an either or situation where compat thrashing between tools and environments may be the unfortunate situation.

[evanw]

Here's a quick compatibility matrix I just generated to get a better idea of the situation:

• Importing module.exports without the __esModule property:

  	Node 15.2	Webpack 5.4	Rollup 2.33	Parcel 1.12	esbuild 0.8
  Normal property	🚫	✅	✅	🚫	✅
  Non-enumerable property	🚫	😐	😐	🚫	✅
  Property from prototype	🚫	✅	😐	🚫	✅
  __esModule property present	🚫	😐	🚫	🚫	✅
  default export present	✅	✅	✅	✅	✅

• Importing module.exports with the __esModule property set to true:

  	Node 15.2	Webpack 5.4	Rollup 2.33	Parcel 1.12	esbuild 0.8
  Normal property	🚫	✅	✅	✅	✅
  Non-enumerable property	🚫	😐	😐	✅	✅
  Property from prototype	🚫	✅	😐	✅	✅
  __esModule property present	🚫	✅	😐	✅	✅
  default export present	✅	✅	✅	🚫	🚫

Here's what each icon means:

✅ = Always imported (never undefined)
😐 = Sometimes imported (either undefined or not depending on the import syntax used)
🚫 = Never imported (always undefined)

The normal, non-enumerable, and prototype properties were all set to true:

let o = Object.create({ inherited: true })
Object.defineProperty(o, '__esModule', {value: true})
o.normal = true
Object.defineProperty(o, 'nonEnumerable', {value: true})
module.exports = o

The test is a .mjs file in a package with "type": "module" that imports a .js file (the code above) in a package with "type": "commonjs".

It looks like esbuild should be generating a default import if __esModule is present to match Webpack, Node, and Rollup. I can make that change.

For the discussion, it seems like tools seeking to maximize compatibility will end up just doing the union of what other tools do. So that's where my preference would be and I'm imagining that's where things will end up.

[guybedford]

@evanw I'd suggest using a Node.js statically analyzable test case as well for the Node.js tests:

exports.name = 'value';

I'd also suggest checking the shape of import * as m from 'cjs' and whether it has a default as module.exports with a named exports shell, or whether it has just the exports.

It looks like esbuild should be generating a default import if __esModule is present to match Webpack, Node, and Rollup. I can make that change.

The default should always be module.exports, regardless of __esModule. The question is just what to do when it clashes with a default export with __esModule.

For the discussion, it seems like tools seeking to maximize compatibility will end up just doing the union of what other tools do. So that's where my preference would be and I'm imagining that's where things will end up.

Agreed, but the thrashing issue I'm talking about is when there is mutual exclusivity in the solution space with the case I describe above.

[guybedford]

And I will still be pushing very hard to get RollupJS to change its default behaviour here for the edge case to match Node.js as well, as I think very slightly worse compat now is better than a future of bad compat. The greedy algorithm of software evolution that usually doesn't do too badly breaking down exactly why I'm having this discussion with you.

[evanw]

The default should always be module.exports, regardless of __esModule. The question is just what to do when it clashes with a default export with __esModule.

Ah, I finally understand. That is tricky. Now that I think about it, I'm pretty sure that's why I didn't make esbuild generate a default export in this case.

Here's a simple test file (statically analyzable this time):

Object.defineProperty(module.exports, '__esModule', { value: true })
module.exports.default = 123

This is how the different environments behave on this one as far as the value of the import default:

	Node 15.2	Webpack 5.4	Rollup 2.33	Parcel 1.12	esbuild 0.8
{default: 123}	✅	✅
123			✅	✅	✅

Node and Webpack are the majority of the ecosystem so if that's the case, I think that should be the behavior to converge to.

[evanw]

Actually, after thinking about it more I'm not totally sure. Doing that means that converting ESM to CJS can no longer be done correctly, which seems like a mistake.

In particular, that would mess up esbuild's automatic ESM-to-CJS transform that happens in certain scenarios. For example, if an ESM module has a direct call to eval in it, esbuild will convert the ESM to CJS during bundling to ensure that the names in the top-level scope can be preserved (since eval may reference them) without causing collisions with names from other modules in the bundle due to scope hoisting. This works fine now but would break if this change was implemented.

I could always work around this by making esbuild's automatic ESM-to-CJS transform special somehow and excluding files transformed by it from this rule. It'd be great to not have to do that since doing that is even more complexity but I'm not sure what the alternative is.

[guybedford]

eval is a really interesting case, and I don't believe RollupJS makes any allowance for it currently. In theory under a strict optimization, eval should never merge its bindings to ensure they are always the same scope names. In reality it really depends on the patterns that dominate what rules make sense. Even the conversion might break down on cases like imported namespaces or live bindings or this access (exports in CJS, undefined in ESM), within the eval. Out of interest, what sort of use cases did eval support come up for?

[evanw]

Out of interest, what sort of use cases did eval support come up for?

It's mainly just me trying to implement a correct bundler and minifier. For example, I've been looking at test cases from the UglifyJS and Terser projects and there are lots of test cases about direct eval.

Of the real-world uses of direct eval I've seen:

• A lot of the use of it at my workplace is for code that is run during tests, although that is also bundled and needs to work correctly
• Some appear to be only trying to access globals such as Object, in which case of course global eval is sufficient (and better even)
• Some appear to be accidental uses by people who don't know what direct eval is or why it's bad to publish code that uses it
• Some of it is deliberate but meant to access identifiers that aren't declared within that file but that also aren't declared as globals such as eval('require')
• Some of it is deliberate but is intended to access local variables declared inside a function scope (we have a test helper that uses this to set up some data-driven tests)
• None of it attempts to access top-level variables in the module

There are other cases where esbuild does ESM-to-CJS in addition to eval:

• I have decided to have esbuild support both CJS and ESM within the same module so users don't have to deal with getting the script vs. module goal wrong. If you use CJS features (exports or module or a top-level return) any ESM syntax will be converted to CJS. This works pretty seamlessly at the moment since CJS bindings are live when imported into ESM and the default export from the original ESM before the conversion to CJS is respected.

• Another case is when an ESM module does a re-export of a CJS module using export * from. Since exports in CJS are dynamic at run-time, the re-exports are computed at run-time too. That requires the re-exporting module to also be converted from ESM to CJS.

[sokra]

Yeah, another interop discussion...

Note that webpack has two different modes. One that's compatible with the ecosystem and one that's compatible with Node.js.

Node.js compat mode is used when import originates from a .mjs file (or type: module). Otherwise the normal mode is used.

That are the main differences:

• __esModule + default export is handled correctly in normal mode and Node.js compatible in Node.js compat mode (default is module.exports).
• Normal mode auto-detects ESM vs CommonJs, while Node.js mode only supported ESM.
• import * as x behaves like x = require(...) in normal mode (Babel compat), while it correctly creates a namespace object in Node.js mode.

And btw. commonjs imports are always live in webpack. Not having live-bindings would break modules that are transpiled from ESM to CJS, so it doesn't really make sense to me why it is this way in Node.js...

[evanw]

Note that webpack has two different modes. One that's compatible with the ecosystem and one that's compatible with Node.js.

Node.js compat mode is used when import originates from a .mjs file (or type: module). Otherwise the normal mode is used.

Thanks for pointing this out. It didn't occur to me that this could be the case. This is interesting. Perhaps this could be the approach esbuild ends up taking too.

And btw. commonjs imports are always live in webpack. Not having live-bindings would break modules that are transpiled from ESM to CJS, so it doesn't really make sense to me why it is this way in Node.js...

Good to know. I plan on keeping them live in esbuild too, since that would obviously break ESM code that has been converted to CJS (including code that esbuild auto-converts itself which needs to continue to work correctly, as described above).