Build artifacts depend node_modules, [email protected]

[0x0ece]

Bug Report

Build artifacts depend node_modules.

This is analogous to #1706, but for Nestjs 7.

The issue seems different as it's not a building issue, but an issue after building succeeds. I tried to follow the discussion in #1706 but I couldn't find a fix. Can anyone provide a resolution?

Current behavior

Repro:

1. nest new myrepro (choose yarn)
2. cd myrepro
3. yarn build
4. mv node_modules _node_modules
5. node dist/main.js

internal/modules/cjs/loader.js:969
  throw err;
  ^

Error: Cannot find module '@nestjs/core'
Require stack:
- /path/to/dist/main.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:966:15)
    at Function.Module._load (internal/modules/cjs/loader.js:842:27)
    at Module.require (internal/modules/cjs/loader.js:1026:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (/path/to/dist/main.js:3:16)
    at Module._compile (internal/modules/cjs/loader.js:1138:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)
    at Module.load (internal/modules/cjs/loader.js:986:32)
    at Function.Module._load (internal/modules/cjs/loader.js:879:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/path/to/dist/main.js' ]
}

Expected behavior

After build, the artifacts in dist should run without depending on node_modules.

Environment

Nest version: 7.4.2 (latest)

(I'm on node12/macosx, but the same happens if I build within Docker)

[jmcdo29]

This is common across all Typescript projects that use tsc (which nest does my default) instead of using a bundling tool (like webpack which nest is also compatible with). Even node projects by default depend on node_modules to properly run. Without a major change to how Node works, or without using a bundler, this is not going to behave any differently.

[0x0ece]

I tried with webpack but I get the same result (change build in --webpack). Meaning, the content of dist is different, but the error is the same.

In #1706 the problem seemed that they had to ignore some modules to make sure webpack could complete. Here webpack works, but then same exact Error: Cannot find module '@nestjs/core' as above.

[jmcdo29]

Right, Nest's webpack configuration uses webpack-node-externals, which means that everything your applications uses requires() for from the node_modules directory is still necessary and not webpacked. In the issue you linked, Kamil adds some insights on to why webpacking the backend doesn't usually work out too well due to needing things like C++ bindings for some lower level connections like database packages and the use of packages like bcrypt or argon2 for example.

You're always welcome to build your own webpack configuration that would bring everything into a single file, but Nest will not provide the config for that.

[0x0ece]

Oh ok, thank you for the insights. Closing this then.

[turkyden]

@0x0ece How did you solve this problem?

@jmcdo29 I want to integrate it into my electron application and bring everything into a single file. How do I do that ?

[0x0ece]

I didn't solve, I'm not packing into 1 single js file. Sorry...

[gentunian]

This is common across all Typescript projects that use tsc (which nest does my default) instead of using a bundling tool (like webpack which nest is also compatible with). Even node projects by default depend on node_modules to properly run. Without a major change to how Node works, or without using a bundler, this is not going to behave any differently.

Some contradictions arises, however. If nestjs was heavily designed with an "Angular" perspective, then this way of building is completely different. You may build an angular application (Typescript) and get a dist/ directory with all you need to serve it. Moreover, you don't even need node to serve it. Just put the dist/ directory in any webserver for it to be served.

To me, it should be an improvement for nestjs. Because as Angular, nestjs is heavily opinionated also. So having a bundler and produce an independent build will provide an extraordinary way of delivering nest applications. Adding to that, containers will get the benefits of simplifying and minimizing the deployment.

Today, you need to install dependencis, prune them for production, and moving node_modules around together with your dist. That's not a build, from my perspective. It's just a dev environment being served.

[jmcdo29]

@gentunian the reason Angular can create this production build option is because of how the browser environment works. There's no dependencies on C++ bindings or python libraries, there's no worries about database connections, there's just JavaScript, HTML, and CSS. With Node, you have these C++ bindings in packages like pg, bcrypt, argon2, sharp and others. These bindings pose huge problems when it comes to using webpack/snowpack/rollup/ and packing all the dependencies together because you end up needing a JS interpreter for those external tools. It's why webpack doesn't support this out of the box (to my knowledge). And as not everyone uses the same deployment strategy (docker, aws, gcloud, Heroku, Digital Ocean, whatever) it's hard to say "Here's the build script that will work for your case".

Heroku already manages this install, build, prune step for you. AWS is completely hands off and has multiple paths you could take (ECS, ECR, lambda, etc). Docker is another hands off approach and I have a sample tempalte here if you'd like to take a look using a multi-step build.

Nest's architecture is heavily inspired by Angular, that is and always will be true. But due to differences between the browser and node environment, providing the "right" build script to cover all cases (even all basic cases) would be difficult in its own right.

IF we were to attempt this (and that's a bit if) what do you say the build should do? Right now, build calls tsc or webpack and transpiles the code from un-runnable TS to runnable JS code. What else, in your opinion, should happen?

[gentunian]

@jmcdo29 good point. For one project I needed to make use of pkg that internally searches for every use of require() and bundles up a binary (not saying that nest should do that) and also tackles nodejs addons. I wonder if something similar for nest would make sense...

(edit to avoid getting people notified)
I saw your docker template and I'm amazed by practically the same approach we take, this is what I'm using to ship docker images from nest apps:

FROM node:12-alpine as deps
WORKDIR /app
COPY package*.json .
RUN npm ci

FROM node:12-alpine as build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build && npm prune --production

FROM node:12-alpine
WORKDIR /app
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
EXPOSE 3000
CMD [ "node", "dist/main" ]

[jmcdo29]

It might be interesting. It looks like pkg works on the compiled JS, so it could be a secondary step taken. It looks like pkg also bundles in a node runtime to the output file, which while it works, it also could end up causing bloat if you know you're going to run the code on a server with node already installed. Definitely an idea to think about, but I don't see us implementing this anytime soon. Feel free to create a POC if you really want though :)

[jordanst3wart]

Might be necroposting, but you can do this with ncc by vercel:

ncc build ./dist/main.js -m -C -o ./ncc
node ./ncc/index.js

example:
https://github.com/ishanuda/nestjs-ncc-bytenode/blob/main/server/package.json
repo:
https://github.com/vercel/ncc

I got an error with @mikro-orm/core though trying this. mikro-orm unreliably checks for versions being the same.

[github-contrib-avision]

Might be necroposting, but you can do this with ncc by vercel:

Our experience with this: Works like a charm unless you are using typeorm migration scripts. ncc does not compile these and thus typeorm is not able to find them at runtime. This is a known problem of ncc: vercel/ncc#578