Idea: hijack fs / require API to pre-cache all dynamic / conditional requiring

[songkeys]

Here's an idea to solve some general dynamic / conditional requiring issues.

The current common issues are when the code is hard for a static analyzer we are using to parse, the required modules will not be bundled into the target file. E.g.

// ./main.js
const paths = ['./a.js', './b/js', './c.js']
paths.forEach(p => {
  require(p); // this won't be analyzed correctly with our current webpack
})

To solve this in my project, I hijacked apis like fs.readdir fs.readfile require glob etc. with a "pre-loader" to load these files (i.e. run the project to make "dynamic" works) before using ncc. All the dynamic file paths will be passed through the loader and put into a "bridge" file generated like this:

// ./bridge.js
module.export = {
  "dir": {
    "a.js": () => require("/path/to/a.js"),
    "b.js": () => require("/path/to/b.js"),
    "c.js": () => require("/path/to/c.js"),
  }
}

Then running ncc will go through the hijacked require's so that all the dynamic files will be bundled.

To make this more clear, here is a simple demo implementation of this "bundle helper":

import * as fs from 'fs'

/**
 * A bundle helper to make ncc happy
 */
class BundleHelper {
  constructor() {
    this.initBridge()
  }

  public IS_BUNDLING_MODE() {
    return process.env.BUNDLING === 'true'
  }

  public IS_BUNDLED_MODE() {
    return process.env.BUNDLED === 'true'
  }

  /**
   * registers a file to require when bundling
   * @param filePath file path to require
   * @param customKey you can also use this key to require when in bundled mode; default: filePath
   */
  public registerFile(filePath: string, customKey?: string) {
    const key = this.encodeFilePath(filePath)
    // always exports normal filePath so we can get a complete file list
    let data = `export const ${key} = () => require('${filePath}')\n`
    if (customKey) {
      data += `export const ${customKey} = () => require('${filePath}')\n`
    }
    fs.appendFileSync('./bridge.ts', data)
  }

  /**
   * requires a file and cache it for bundling
   * @param filePath file path or (custom key if provided) to require
   * @param customKey you can also use this key to require when in bundled mode; default: filePath
   * @returns file content
   */
  public require(filePath: string, customKey?: string) {
    if (this.IS_BUNDLED_MODE()) {
      const key = customKey ?? this.encodeFilePath(filePath)
      return this.getBridgeContent()[key]()
    } else if (this.IS_BUNDLING_MODE()) {
      this.registerFile(filePath)
      return require(filePath)
    } else {
      return require(filePath) // normally just require
    }
  }

  /**
   * gets registered file list when bundled
   * this is helpful when your are using libraries like "glob"
   * @returns registered file list
   */
  public getFileList() {
    const exports = this.getBridgeContent()
    const fileList = Object.keys(exports)
      .filter((k) => k.includes('_slash_'))
      .map((k) => this.decodeFilePath(k))
    return fileList
  }

  private getBridgeContent() {
    return require('./bridge')
  }

  private encodeFilePath(filePath: string) {
    return filePath.replace(/\//g, '_slash_').replace(/\./g, '_dot_')
  }

  private decodeFilePath(filePath: string) {
    return filePath.replace(/_slash_/g, '/').replace(/_dot_/g, '.')
  }

  private initBridge() {
    if (!this.IS_BUNDLED_MODE()) return
    fs.writeFileSync('./bridge.ts', '')
  }
}

export default new BundleHelper()

To use it, replace all dynamic require's with bundleHelper.require or just hijack them. And run ncc using:

process.env.BUNDLING = 'true' // enable for bundling
main() // start your app - preload for caching
execSync('ncc main.ts -o output'); // run ncc

// don't forget prepend "process.env.BUNDLED = 'true'" to your output/main.js file.

I'm using this in a couple of my project so far. I think this could be more general and can be integrated into ncc to solve some common dynamic requiring cases.

Any ideas or suggestions?

[guybedford]

@songkeys moving the analysis down to this level is a nice idea, there are two main issues though:

1. Code coverage is hard to achieve. The "integration run" needs to cover all dynamic code paths which is difficult to achieve and also provide as a library feature at the same time.
2. Even with perfect code coverage there are potential dynamic paths that might never be taken unless some specific condition holds, which multi-path analysis is effectively better at capturing.
3. Achieving both of the above via build-time execution risks build system security and side effects, given the lack of a comprehensive virtualization enironment. In the end getting such an approach right is almost more of a service than a library...