Synthetic Modules

[sgammon]

This is a tracking issue for a mechanism for injecting "synthesized" modules which are implemented host-side.

Rationale

Elide provides its own ESM and CJS modules (for example, elide:sqlite). Elide also implements core Node API modules (i.e. node:fs). All of these implementations exist either in Kotlin or in native code with a Kotlin/JVM type-safe layer on top.

To facilitate module overrides/injection in JavaScript, we have to perform a backflip in the code:

(1) Modules are implemented in Kotlin as a ProxyObject instance, manually, with @Polyglot annotations wherever language borders are permeable.

For example:

// Binder / factory
@Intrinsic internal class SomeNodeModule : AbstractJsIntrinsic() {
  @Inject private lateinit mod: SomeNodeModuleFacade
  override fun install(bindings: MutableBindings) {
    bindings["SomeSymbol".asPublicJsSymbol()] = Value.asValue("here is a constant")
    bindings["SomeOtherSymbol".asPublicJsSymbol()] = ProxyInstantiable { /* ... call a factory ... */ }
    bindings["some_node_module".asJsSymbol()] = mod
  }
}

// Node module API
public interface SomeNodeModule {
  // host-side types
  @Polyglot public fun someFunction(prop: HostType)

  // value types
  @Polyglot public fun someFunction(prop: Value?) = someFunction(HostType.from(prop))
}

// Constant name for a prop or method
private const val SOME_FUNCTION = "someFunction"

// Node module facade implementation
@Singleton public class SomeNodeModuleFacade : SomeNodeModule, ProxyObject {
  override fun getMemberKeys(): Array<String> = arrayOf(SOME_FUNCTION )
  override fun hasMember(key: String): Boolean = key in memberKeys
  override fun putMember(key: String, value: Any?): Unit = Unit
  override fun getMember(key: String): Any? = when (key) {
    SOME_FUNCTION  -> ProxyExecutable { someFunction(it.getOrNull(0)) }
    else -> null
  }

  @Polyglot override fun someFunction(prop: HostType) {
    /* ... */
  }
}

(2) Modules are additionally mounted in the DI context, and rely on are allowed to rely on the DI context (generally speaking), since they may need shared resources. Module dependencies typically use DI for resolution.

(3) Additionally, a JavaScript module layout file is written in TypeScript, in the runtime codebase. This file provides finalized module layout, which is importable in JavaScript, and merely calls the intrinsic object's exports uniformly:

/**
 * Intrinsic: Zlib.
 *
 * Provides access to the Zlib compression and decompression library.
 */

const { node_zlib } = primordials;
if (!node_zlib) {
  throw new Error(`The 'zlib' module failed to load the intrinsic API.`);
}

// ...

/**
 * Calculates a CRC32 checksum for the given data.
 *
 * @param data The data to checksum
 * @param value The initial value for the checksum (optional)
 * @returns The CRC32 checksum for the given data
 */
export function crc32(data: string | Buffer | DataView | any, value?: number): number {
    return intrinsic().crc32(data, value);
}

(4) The TypeScript layouts are built by Bazel in the runtime codebase, then embedded within a js.modules.tar archive which is held in classpath resources.

(5) The JS modules are mounted as a VFS bundle at runtime, at the path /__runtime__/<module>. This of course means all modules are unpacked from their tarball resource into an in-memory VFS at startup.

(6) Mappings are provided to the JavaScript context at startup time for each module; these mappings override named modules with VFS-exposed paths, for example:

mapOf(
  "node:fs" to "/__runtime__/fs",
  "fs" to "/__runtime__/fs",
  // ...
)

(7) VFS wiring is set up to intercept these paths and return corresponding content from the js.module.tar.

(8) Boom, working ESM and CJS imports.

---

Instead, a mechanism to synthesize named JavaScript modules with code, and then inject them into the JavaScript context, would immediately result in better performance, smoother internal DX, and much better compiler visibility into builtin modules.

We already have the mechanism for this via JSRealmPatcher, which sets up the TypeScriptModuleLoader.