Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NEXT-1143] Dev mode slow compilation #48748

Open
1 task done
jeengbe opened this issue Apr 23, 2023 · 493 comments
Open
1 task done

[NEXT-1143] Dev mode slow compilation #48748

jeengbe opened this issue Apr 23, 2023 · 493 comments
Assignees
Labels
linear: next Confirmed issue that is tracked by the Next.js team.

Comments

@jeengbe
Copy link

jeengbe commented Apr 23, 2023

⚠️ this original post has been edited by @timneutkens to reflect this comment ⚠️

Changes in the past week

I've been investigating this over the past week. Made a bunch of changes, some make a small impact, some make a large impact. Here's a list:

You can try them using npm install next@canary.

Help Investigate

In order to help me investigate this I'll ideally need an application that can be run, if you can't provide that (I understand if you can't) please provide the .next/trace file.

If possible follow these steps which would give me the best picture to investigate:

  • npm install next@canary (use the package manager you're using) -- We want to make sure you're using the very latest version of Next.js which includes the fixes mentioned earlier.
  • rm -rf .next
  • start development using the NEXT_CPU_PROF=1 and NEXT_TURBOPACK_TRACING=1 (regardless of if you're using Turbopack, it only affects when you do use Turbopack) environment variable. E.g.:
    • npm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 npm run dev
    • yarn: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 yarn dev
    • pnpm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 pnpm dev
  • Wait a few seconds
  • Open a page that you're working on
  • Wait till it's fully loaded
  • Wait a few seconds
  • Make an edit to a file that holds a component that is on the page
  • Wait for the edit to apply
  • Wait a few seconds
  • Make another edit to the same file
  • Wait a few seconds
  • Exit the dev command (ctrl c)
  • Upload the CPU traces put in the root of the application directory to https://gist.github.com
  • Upload the .next/trace file to https://gist.github.com -- Please don't run trace-to-tree yourself, as I use some other tools (e.g. Jaeger) that require the actual trace file.
  • If you're using Turbopack upload the .next/trace.log as well, if it's too large for GitHub gists you can upload it to Google Drive or Dropbox and share it through that.
  • Upload next.config.js (if you have one) to https://gist.github.com
  • Share it here

Known application-side slowdowns

To collect things I've seen before that cause slow compilation as this is often the root cause:

  • If you're on Windows, disable Windows Defender, it's a known cause of extreme slowdowns in filesystem access as it sends each file to an external endpoint before allowing to read/write
  • Filesystem slowness overall is what we've seen as the cause of problems, e.g. with Docker
  • react-icons, material icons, etc. Most of these libraries publish barrel files with a lot of re-exports. E.g. material-ui/icons ships 5500 module re-exports, which causes all of them to be compiled. You have to add modularizeImports to reduce it, here's an example: long compile times locally - along with "JavaScript heap out of memory" since upgrade to NextJS 13 #45529 (comment)
  • Custom postcss config, e.g. tailwindcss with a content setting that tries to read too many files (e.g. files not relevant for the application)

This and other slowdown reports are currently the top priority for our team. We'll continue optimizing Next.js with webpack where possible.
The Turbopack team is currently working on getting all Next.js integration tests passing when using Turbopack as we continue working towards stability of Turbopack.

Original post

Verify canary release

  • I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: linux
      Arch: x64
      Version: #1 SMP Fri Jan 27 02:56:13 UTC 2023
    Binaries:
      Node: 18.13.0
      npm: 8.19.3
      Yarn: 1.22.18
      pnpm: 7.30.5
    Relevant packages:
      next: 13.3.1
      eslint-config-next: 13.3.1
      react: 18.2.0
      react-dom: 18.2.0

Which area(s) of Next.js are affected? (leave empty if unsure)

No response

Link to the code that reproduces this issue

https://github.com/DigitalerSchulhof/digitaler-schulhof

To Reproduce

Note that I have been unable to replicate this issue in a demo repository.

Describe the Bug

The issue is that Next.js is generally slow in dev mode. Navigating to new pages takes several seconds:

[next] ready - started server on 0.0.0.0:3000, url: http://localhost:3000
[next] info  - Loaded env from /home/jeengbe/dsh/digitaler-schulhof/.env
[next] warn  - You have enabled experimental feature (appDir) in next.config.js.
[next] warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use at your own risk.
[next] info  - Thank you for testing `appDir` please leave your feedback at https://nextjs.link/app-feedback
[next] event - compiled client and server successfully in 1574 ms (267 modules)
[next] wait  - compiling...
[next] event - compiled client and server successfully in 219 ms (267 modules)
[next] wait  - compiling /(schulhof)/Schulhof/page (client and server)...
[next] event - compiled client and server successfully in 3.6s (1364 modules)
[next] wait  - compiling /(schulhof)/Schulhof/(login)/Anmeldung/page (client and server)...
[next] event - compiled client and server successfully in 1920 ms (1411 modules)
[next] wait  - compiling /api/schulhof/auth/login/route (client and server)...
[next] event - compiled client and server successfully in 625 ms (1473 modules)
[next] wait  - compiling /(schulhof)/Schulhof/Nutzerkonto/page (client and server)...
[next] event - compiled client and server successfully in 1062 ms (1482 modules)
[next] wait  - compiling /(schulhof)/Schulhof/Nutzerkonto/Profil/page (client and server)...
[next] event - compiled client and server successfully in 1476 ms (1546 modules)
[next] wait  - compiling /(schulhof)/Schulhof/Nutzerkonto/Profil/Einstellungen/page (client and server)...
[next] event - compiled client and server successfully in 2.1s (1559 modules)

The only somewhat reasonable time would be 600ms for the API route /api/schulhof/auth/login/route, even though that is still quite a lot slower than what it should be given its size.

It also doesn't look right to compile ~1500 modules for each page, as most of them should be cached. The pages are not very different.

Even an empty API route takes several hundreds of ms. The following example contains solely type exports:

[next] wait  - compiling /api/schulhof/administration/persons/persons/settings/route (client and server)...
[next] event - compiled successfully in 303 ms (107 modules)

I am not exactly sure how to read trace trees, but what stands out is that there are (over multiple runs) several entry next-app-loader that take 2 seconds to complete:

│  │  ├─ entry next-app-loader?name=app/(schulhof)/Schulhof/page&page=/(schulhof)/Schulhof/page&appPaths=/(schulhof)/Schulhof/page&pagePath=private-next-app-dir/(schulhof)/Schulhof/page.tsx&appDir=/home/jeengbe/dsh/digitaler-schulhof/app&pageExtensions=tsx&pageExtensions=ts&pageExtensions=jsx&pageExtensions=js&rootDir=/home/jeengbe/dsh/digitaler-schulhof&isDev=true&tsconfigPath=tsconfig.json&assetPrefix=&nextConfigOutput=! 1.9s

Find both dev and build traces here: https://gist.github.com/jeengbe/46220a09846de6535c188e78fb6da03e

Note that I have modified trace-to-tree.mjs to include event times for all events.

It also seems unusual that none of the modules have child traces.

Expected Behavior

Initial load and navigating should be substantially faster.

Which browser are you using? (if relevant)

No response

How are you deploying your application? (if relevant)

No response

From SyncLinear.com | NEXT-1143

@jeengbe jeengbe added the bug Issue was opened via the bug report template. label Apr 23, 2023
@joacub
Copy link

joacub commented Apr 24, 2023

same here, and in docker env is even worse, seems like is processing same files over and over without caching them.

@jinojacob15
Copy link

Same for me also dev env ,navigating to different pages via link component is pretty slow

@denu5
Copy link

denu5 commented Apr 25, 2023

1 its same here, hitting the page first time seems fine but routing via links gets stuck

@joacub
Copy link

joacub commented Apr 25, 2023

last canary version has a better cold build times improvements, still slow like 2-5 seconds (in docker env) waiting but much better

the version im talking about is 13.3.2-canary.6

@denu5
Copy link

denu5 commented May 2, 2023

Hey, @jeengbe there have been some patch updates (13.3.1 -> 13.3.4) did it improve for you?

@jeengbe
Copy link
Author

jeengbe commented May 2, 2023

Hi @denu5,

unfortunately, I can't report any real performance changes since I opened this issue.

You might want to check out the above linked issue in the TypeScript repo though - might be related.

@joacub
Copy link

joacub commented May 2, 2023

As @jeengbe mention there is no performance improvement, there is also a lot of I/O I don’t know why, one request gets pretty much like 1gb-2gb of io. And it is very slow.

@jeengbe
Copy link
Author

jeengbe commented May 2, 2023

As @jeengbe mention there is no performance improvement, there is also a lot of I/O I don’t know why, one request gets pretty much like 1gb-2gb of io. And it is very slow.

Unfortunately, I can't confirm this for my case

image

@joacub
Copy link

joacub commented May 2, 2023

As @jeengbe mention there is no performance improvement, there is also a lot of I/O I don’t know why, one request gets pretty much like 1gb-2gb of io. And it is very slow.

Unfortunately, I can't confirm this for my case

image

That’s pretty good, in my case there is a lot of I/O, maybe is because I’m using material-ui but I think is too much even though.

@jeengbe
Copy link
Author

jeengbe commented May 2, 2023

As @jeengbe mention there is no performance improvement, there is also a lot of I/O I don’t know why, one request gets pretty much like 1gb-2gb of io. And it is very slow.

Unfortunately, I can't confirm this for my case
image

That’s pretty good, in my case there is a lot of I/O, maybe is because I’m using material-ui but I think is too much even though.

Possibly, it would align with what your trace shows: #48407 (comment)

@langfordG
Copy link

langfordG commented May 4, 2023

I see that slow route changes in dev mode are showing a '[Fast Refresh] rebuilding' message in the browser console. Sometimes it performs a full page reload when changing routes even if no files have been edited.

@timneutkens timneutkens added the linear: next Confirmed issue that is tracked by the Next.js team. label May 10, 2023
@timneutkens timneutkens changed the title Dev mode very slow navigation (Slow entry next-app-loader spans?) [NEXT-1143] Dev mode very slow navigation (Slow entry next-app-loader spans?) May 10, 2023
@AsathalMannan
Copy link

its slowing down the development..!

@vajajak
Copy link

vajajak commented May 21, 2023

Having the same issue here, in the Docker environment it's come to a point where it's almost unusable, and sometimes I even have to do a hard reload, after waiting too long for navigation. This is the case both with component from next/navigation, as with the router.push (useRouter hook imported from next/navigation). We're using Next.js 13.4.2.

@joacub
Copy link

joacub commented May 22, 2023

Having the same issue here, in the Docker environment it's come to a point where it's almost unusable, and sometimes I even have to do a hard reload, after waiting too long for navigation. This is the case both with component from next/navigation, as with the router.push (useRouter hook imported from next/navigation). We're using Next.js 13.4.2.

same here, it is almost not usable in dokcer enviorements, but also outside docker is very slow, something is not working nice. this is painfully slow.

@JoshApp
Copy link

JoshApp commented May 27, 2023

Yeah same for me. I used to remote develop inside our k8s cluster but dev --turbo is super slow inside a container and causes my health check endpoint to sigkill it regularly.

The whole app router is super slow when containerized in Dev mode.

It works perfectly fine when I run both on my local machine and connect it via reverse proxy. This way it's faster than the old setup (which was not significantly faster before) and takes advantage of preloading pages via next/link. I see inconsistencies in caching too where it's a mix of instant navigation or long builds (around 3.5k modules for some things) around 2-10 sec.

Also there is this weird thing happening that a page compiles just fine and then later it grinds to a halt being stuck in waiting for compiling forever until the pod is crashed.

@Rykuno
Copy link

Rykuno commented May 27, 2023

I love next, but this is a complete show stopper. Sometimes it takes 10 seconds outside of docker for me on a Mac M2 to navigate one page.

This is insane.

@joacub
Copy link

joacub commented May 27, 2023

I love next, but this is a complete show stopper. Sometimes it takes 10 seconds outside of docker for me on a Mac M2 to navigate one page.

This is insane.

Yep even more I get sometimes 50 seconds in a simple page, that’s because is also building other things related to that in pralllel I guess.

not even outside docker, i just make a test to work outside docker and timing is exactly the same no difference…. Is getting slower and slower

@joacub
Copy link

joacub commented May 27, 2023

Btw webpack lazy building cold is faster than turbopack 🙂 by far

@Rykuno
Copy link

Rykuno commented May 27, 2023

Btw webpack lazy building cold is faster than turbopack 🙂 by far

Yes! I'm surprised this is not more prevalent as an issue atm; unless turbo will somehow fix all of this in 13.5 and they're waiting to address it.

What configs do you have for the faster webpack builds? I've tried quite a bit and can't lower my built time by much. I need a temporary fix for this ASAP :(

@oalexdoda
Copy link

A month later no updates on this? Makes development on appDir absolutely impossible. @timneutkens ?

Linked a bunch of related issues on this:
#50332

@JunkyDeLuxe
Copy link

I confirm that next app dir on dev mode and dynamic routing are very very slow on docker now

@timneutkens
Copy link
Member

timneutkens commented Jun 6, 2023

Changes in the past week

I've been investigating this over the past week. Made a bunch of changes, some make a small impact, some make a large impact. Here's a list:

You can try them using npm install next@canary.

Help Investigate

In order to help me investigate this I'll ideally need an application that can be run, if you can't provide that (I understand if you can't) please provide the .next/trace file.

If possible follow these steps which would give me the best picture to investigate:

  • npm install next@canary (use the package manager you're using) -- We want to make sure you're using the very latest version of Next.js which includes the fixes mentioned earlier.
  • rm -rf .next
  • start development using the NEXT_CPU_PROF=1 environment variable. E.g.:
    • npm: NEXT_CPU_PROF=1 npm run dev
    • yarn: NEXT_CPU_PROF=1 yarn dev
    • pnpm: NEXT_CPU_PROF=1 pnpm dev
  • Wait a few seconds
  • Open a page that you're working on
  • Wait till it's fully loaded
  • Wait a few seconds
  • Make an edit to a file that holds a component that is on the page
  • Wait for the edit to apply
  • Wait a few seconds
  • Make another edit to the same file
  • Wait a few seconds
  • Exit the dev command (ctrl c)
  • Upload the CPU traces put in the root of the application directory to https://gist.github.com
  • Upload the .next/trace file to https://gist.github.com -- Please don't run trace-to-tree yourself, as I use some other tools (e.g. Jaeger) that require the actual trace file.
  • Share it here

Known application-side slowdowns

To collect things I've seen before that cause slow compilation as this is often the root cause:

  • If you're on Windows, disable Windows Defender, it's a known cause of extreme slowdowns in filesystem access as it sends each file to an external endpoint before allowing to read/write
  • Filesystem slowness overall is what we've seen as the cause of problems, e.g. with Docker
  • react-icons, material icons, etc. Most of these libraries publish barrel files with a lot of re-exports. E.g. material-ui/icons ships 5500 module re-exports, which causes all of them to be compiled. You have to add modularizeImports to reduce it, here's an example: long compile times locally - along with "JavaScript heap out of memory" since upgrade to NextJS 13 #45529 (comment)
  • Custom postcss config, e.g. tailwindcss with a content setting that tries to read too many files (e.g. files not relevant for the application)

This and other slowdown reports are currently the top priority for our team. We'll continue optimizing Next.js with webpack where possible.
The Turbopack team is currently working on getting all Next.js integration tests passing when using Turbopack as we continue working towards stability of Turbopack.

@timneutkens timneutkens changed the title [NEXT-1143] Dev mode very slow navigation (Slow entry next-app-loader spans?) [NEXT-1143] Dev mode slow navigation Jun 6, 2023
@timneutkens timneutkens changed the title [NEXT-1143] Dev mode slow navigation [NEXT-1143] Dev mode slow compilation Jun 6, 2023
@timneutkens
Copy link
Member

Changed the initial post in this issue to reflect my reply above in order to ensure people see it as the first thing when opening the issue. I'm going to close the duplicate issues reporting similar slowdowns in favor of this one.

I'll need help from you all to ensure this thread doesn't spiral in "It is slow" comments that are not actionable when e.g. without traces / reproduction / further information. Thank you 🙏

@lluiscab
Copy link

Hi @timneutkens, sorry to hijack this issue but I'm trying to debug some slow compile times that our time has been experiencing and I'd like to know how do you process and load the trace file into Jaeger or a GUI representation (as seen in your last comment). I've been unable to find anything documenting or mentioning this process.

I've found the trace-to-tree.mjs script, which prints the trace in stdout. The trace-to-event-format.mjs which seems to convert the trace file to a different format type, which I assume is then loaded into some tool (?).

Also, I'm running a local jaeger deployment from https://github.com/vercel/opentelemetry-collector-dev-setup and if I run execute the send-trace-to-jaeger rust script, I get the following output: (traces-sites/vanilla.json is the .next/trace file)

$ cargo run ../../traces-sites/vanilla.json 
    Finished `dev` profile [unoptimized   debuginfo] target(s) in 0.33s
     Running `/<redacted>/next.js/target/debug/send-trace-to-jaeger ../../traces-sites/vanilla.json`
Jaeger trace will be available on http://127.0.0.1:16686/trace/a3fd2081f16b2ef3
body = Ok("No value at $[7].tags.layer reading List<Span> from json")
body = Ok("No value at $[0].tags.layer reading List<Span> from json")
body = Ok("No value at $[19].tags.layer reading List<Span> from json")

but when I open http://127.0.0.1:16686/trace/a3fd2081f16b2ef3, all I see is a HTTP Error: trace not found Jaeger UI error.

@timneutkens
Copy link
Member

@lluiscab I'd recommend using next internal turbo-trace-server /path/to/trace instead as that's a more specialized UI for both the Next.js trace file (which holds webpack instrumentation too) as well as Turbopack traces. That command is available on any Next.js canary install currently (will be on stable in the future)

@tiriana
Copy link

tiriana commented Jul 17, 2024

@timneutkens

I suspected my styles being responsible. And it is indeed much better then 2-3 weeks ago, when it really would take 20-30 seconds to recompile.

About dynamic imports - I implemented a dynamic loading of translation json files using useEffect and import().
It worked with regular dev but not with dev --turbo. Here's a gist with it: https://gist.github.com/tiriana/bc4d01f107b22e1b85eec13e1b7600ed

But on the other hand - when I use import() like a normal person, and not in useEffect - it works fine with Turbopack

Example:

// this works fine:
import dynamic from "next/dynamic";

export const DebugPanelDynamic = dynamic(() => import("@/components/layout/DebugPanel").then((m) => m.DebugPanel));

@cjmling
Copy link
Contributor

cjmling commented Jul 18, 2024

@dev-badace Do you have anti-virus enabled or something like that? That can cause such issues as it can block disk I/O on Windows

I can confirm this that it was because of anti-virus in my case. My code compile time were in xxx ms , but just when "Bitdefender" was installed it sky rocket to to 20 - 50 seconds!! . Found this post and https://stackoverflow.com/a/78236532/507203 which suggested me to exclude. Right when it get excluded it goes back to ms.

@GriffinSauce
Copy link

GriffinSauce commented Jul 18, 2024

@timneutkens Thanks for taking the time to help people here! I'm working on migrating our main web app from ejected CRA into Next and I'm running into some intense numbers for development builds.

So the main issues are: way slower restarts (no cache at all with app router?) and 18 MB client JS which takes significantly longer to load into the browser. The amount of modules is also concerning.

Switching to Turbo looks a lot better (even without cache) but sadly some quirks in my context are preventing me from jumping into Turbo now. In short:

  • unclarity about using Linaria - we're using Emotion now and looking to switch
  • no support for image loaders - to support the migration across multiple React apps we prefer to keep the current image import behaviour (outputting a string instead of a metadata object)

I've sent you the trace in a DM on Twitter/X, I've looked into it myself but I'm having trouble coming to any useful conclusions (besides our app just being huge). Edit: forgot to mention one very obvious thing: there's a lot of duplication of modules in the client bundle.

Thanks again for looking into these, I'd love to get us fully on board!

@timneutkens
Copy link
Member

@tiriana The reason that doesn't work is that you're combining path aliases @/ with highly dynamic imports, which is not supported yet. If you change it to a relative path it should work 👍 Related open issue: #63372 (comment)

@timneutkens
Copy link
Member

@Harbinder-web can you elaborate on what exactly is slow? Seeing initial compile take ~5 seconds, that's mostly because of the amount of barrel imports / large libraries you're importing like Mantine and react-icons, even with barrel optimization these can still be quite large

Recompiles when changing a file are taking 300-500ms.

@timneutkens
Copy link
Member

@GriffinSauce will follow up on twitter 👍

@PramodTKodag

This comment was marked as off-topic.

@PramodTKodag

This comment was marked as off-topic.

@timneutkens
Copy link
Member

@Harbinder-web can you make sure those steps are followed when collecting the trace? Then I can correctly inspect the trace 👍

@kl-thamm
Copy link

kl-thamm commented Jul 24, 2024

We are also experiencing this. For context: We are using Mac M1 & M2 in development and are on 14.2.5. We are using MUI so we set:

    optimizePackageImports: [
      "@mui/icons-material",
      "@mui/material",
      "mui-core",
      "date-fns",
      "lodash-es",
    ],
  },

We cannot use turbopack yet because of #68101.

On every page load I have absurd compile times:

POST /en/monitoring 404 in 2736ms
GET /en/app/actualPage 200 in 2207ms
 ○ Compiling /[lang]/icon.png ...
 ✓ Compiled /[lang]/icon.png in 965ms (6633 modules)

I am not sure if it matters, but we are roughly following https://nextjs.org/docs/app/building-your-application/routing/internationalization - so this is why you see the [lang] everywhere.

@timneutkens I know you want stack traces, but I got an error generating one:
Cannot generate CPU profiling: Error [ERR_INSPECTOR_COMMAND]: Inspector error -32000: No recording profiles found, there was still a cpufile as output, but opening it shows that its a corrupt json. I can share that with you (if there is a way to do so non-publicly).

If you give me any kind of instructions for debugging, I will follow along.

@timneutkens
Copy link
Member

@kl-thamm please at least share the trace file per instructions in the initial post.

@kl-thamm
Copy link

kl-thamm commented Jul 25, 2024

@timneutkens Did you see what i posted?

  • I have an error generating the trace per the instructions in the initial post. Should I share the corrupt file or is there anything to fix it (see the error message above)?
  • I would like to share the trace non publicly and the initial post suggests a public gist.

Edit: Create a private repo, uploaded trace from yesterday and shared access with you. If I can do more let me know how.

@timneutkens
Copy link
Member

Hey @kl-thamm, I'm still missing the .next/trace and .next/trace.log (if using Turbopack) files in that repository. The cpuprofiles don't matter much 🙂 Can you share the .next/trace and .next/trace.log.

@kl-thamm
Copy link

kl-thamm commented Jul 29, 2024

Hey, I am pretty sure that is related to the error I described above, because thats the only output. Let me know how to debug that and I will get back asap. I googled around a bit myself, but didnt find anything helpful.

Edit: Will try to just try again in the meantime.
Edit 2: Nope. Exact same error again: Cannot generate CPU profiling: Error [ERR_INSPECTOR_COMMAND]: Inspector error -32000: No recording profiles found
Edit 3: At least I now figured out where the error is coming from: Directly from the node runtime and since this is based on Chromes Javascript Rendering engine, it seems to be directly from chromes v8 (?), so it should be this profiler
(?) https://v8.dev/docs/profile.
I could even trace the line that throws the error: https://chromium.googlesource.com/v8/v8/ /7.4.77/src/inspector/v8-profiler-agent-impl.cc#279
But I have absolutely no clue how to fix it?

I added our next config to the repo you are in

@timneutkens
Copy link
Member

As said the cpuprofile files don't matter, the .next folder in your project will have the trace file as well as the trace.log file, those are the ones I need.

@kl-thamm
Copy link

As we dont use turbopack, I just have the trace file now and added that to the repo. Thanks for pointing out the location. That was the missing piece, as the profile was created in another directory.

@timneutkens
Copy link
Member

@kl-thamm Thanks, had a look, seems the majority of time spent is on reading files from disk.

Additional pointers:

  • Your instrumentation.ts (or .js) takes 190 seconds to run
  • services/graph_next/src/data-sources/graph/video/videosInRange/getVideosInRange.ts imports neo4j-driver which imports all of rxjs. Worth adding neo4j-driver to https://rc.nextjs.org/docs/app/api-reference/next-config-js/serverExternalPackages. Similarly there's a couple of places where all of Redis SDK is being imported.
  • services/graph_next/src/utils/cacheUtils/cacheHelper.ts imports all of Redis sdk, and the filereads of that are super slow in the trace
  • It seems you're on an old version of @supabase/ssr which ends up shipping additional code for ramda which has been removed in the latest version

Any specific reason you're not able to use Turbopack?

@kl-thamm
Copy link

kl-thamm commented Jul 31, 2024

Thanks a lot. I will make some changes based on that and report back with a new trace (if issues persist). However: We ran the same code in plain react at some point and it was insanely much faster at that point (that was before supabase however).

Is it expected that:
✓ Compiled /[lang]/icon.png in 965ms (6633 modules) we see things like this? That a single icon compiles 6k modules and takes almost a second?

Your instrumentation.ts (or .js) takes 190 seconds to run

Yes, thats ok!

Any specific reason you're not able to use Turbopack?

Yes:

We cannot use turbopack yet because of #68101.

Edit changes made:

  • Added neo4j-driver to serverComponentsExternalPackages
  • Upgrade supabase/ssr, supabase-js and supabase to the latest versions

Remarks:

  • Redis is just imported in cacheHelper.ts and cacheHelper.js, where the js file is just the transpiled version. That we use in a custom cache handler in the next.js config. In the transpiled file the import is var redis_1 = require("redis"); whereas in the ts file it is limited to: import { createClient } from "redis";
  • services/graph_next/src/utils/cacheUtils/cacheHelper.ts imports all of Redis sdk, and the filereads of that are super slow in the trace So this shouldnt be the case or is that an issue with modularizeImports again?

Edit 2:
No improvement. First compiles range from 6 seconds to 15 seconds and reloads still take up to 3 seconds. Adde trace2.

@wadamek65
Copy link

@timneutkens Hey there, we've also been struggling a little with compilation times. I have a feeling that in our case it's just due to the size of the application and the amount of dependencies and files we have but I figured it would be worth double-checking with someone much more knowledgeable.

Would you be able to take a look at the trace? Here is the gist: https://gist.github.com/wadamek65/735f9efa17cdc50d9dcc353b099d4ac8

For some reason no CPU profile was generated for me even though I followed the instructions and ran the app with NEXT_CPU_PROF=1, perhaps related to me being on Windows? Also, we are using an nx monorepo with [email protected] so the newest version.

@rzzapp
Copy link

rzzapp commented Aug 1, 2024

@timneutkens here is performance gist file, only issue you can face maybe is, we are using [email protected] due next@canary requires react@19 and not easy to bump it up. Our app is monorepo, load times were smaller but each page now takes around 15-50s and dev server restarts due memory usage few times each 30-40mins.

https://gist.github.com/rzzapp/ef8815c4162f4e2ff6fd479550740572

Edit: Additional infos:

  • My machine is M2 MBP/16GB Ram
  • Few times app compiled/loaded <1s but after route click few times compiles increased to 30s. Currently I'm averaging 30-90s for simple page.
  • Most long reloads logs Webpack warning:
[webpack.cache.PackFileCacheStrategy] Serializing big strings (112kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)

@timneutkens
Copy link
Member

@wadamek65 looking at your trace there is a lot of time unaccounted for, which would be coming from the plugins you added: https://gist.github.com/wadamek65/735f9efa17cdc50d9dcc353b099d4ac8#file-next-config-js-L1-L6. Would recommend removing these first. Also seeing slow file reads, given you're on windows make sure anti-virus which blocks file reads is disabled on your code.

@timneutkens
Copy link
Member

@rzzapp this stands out:

CleanShot 2024-08-02 at 14 13 20@2x

  • A lot of time spent on .css files, do you have customization (i.e. postcss)? If not it's filesystem access being slow... which would indicate antivirus as well given that you're on a Mac that should have fast filesystem access. Pretty much every .css file takes over 1 second, that's not normal.
  • There's a lot of these, likely filesystem access related too, but worth sharing the files so that we can run compilation on /Users/rufatzeynalli/Documents/config-app/packages/common/ui/src/index.ts
    CleanShot 2024-08-02 at 14 24 10@2x to check if there's a specific case in the transform that's slowing things down.

@rzzapp
Copy link

rzzapp commented Aug 2, 2024

@timneutkens thanks for checking! Yes, I'm on Mac also my team.

Will check with our hardware supplier team in case if we are having something locked in our system which can cause file reading unbearably slow. But generally laptops are running quite fast on smaller projects of CRA or vite.

common/ui is our local UI components which within monorepo and we are installing with pnpm workspace feature.

Tbh we don't have any custom postcss, we only have tailwindcss and everything runs based on that.

also worth to mentioned we do have custom nextjs proxy/middleware which run on each route change to detect it should load our next app pages or fallback legacy app (just redirects to that url which is external project) maybe this can cause slowness but I'm not sure. I remember once I tried to disable it, it was still slow. Probably nothing about this one is serious but worth to try to toggle it again on my side

I'll try few scenarios based on this then can provide few more details.

@wadamek65
Copy link

@timneutkens Thanks for taking a look! I think you're right; I think the slowness comes from the Vanilla Extract plugin, probably in combination with Mantine that we're using. I've seen quite a few issues regarding this in their repository. I tried adding optimizePackageImports: ['@mantine/core', '@mantine/dropzone', '@mantine/tiptap'], but it didn't seem to help a lot. But at this point, it seems more like a webpack issue to debug rather than a NextJS one, so I suppose I'll be taking a look into that next.

Thanks for taking the time to look into this. I really appreciate it! By the way, is there an instruction somewhere on how to load those traces into Jaeger and debug by ourselves?

@dstoyanoff
Copy link

@dstoyanoff I'll have a look at the trace soon, you should be able to try Turbopack with this configuration based on that being the only webpack configuration: https://x.com/timneutkens/status/1805180098703675882

Hello @timneutkens, were you able to look into this?

@rzzapp
Copy link

rzzapp commented Aug 7, 2024

@timneutkens thanks really for good inside, we did actually able to drop our compile/load time from 100-200s to around 1-2s. Main issues we missed:

  • content directory for tailwind.config.js which we just ignored node_modules check for one package (which we did it for 4 packages but 1 of them was missing it)
  • Barrel files within monorepo packages which were preventing tree-shaking, then added it to optimizePackageImports

Also we upgrade from v14.1.4 to v14.2.5.

So far cold start takes around 10-20s but later gets much much faster.

@kl-thamm
Copy link

kl-thamm commented Aug 7, 2024

@timneutkens Can we get a bit more systematic here and maybe also collect more general instructions? I know the detailed debugs are helpful for individual users and also for next to understand what is going on, but it will not allow for a good dx out of the box. Many recommendations you need to find by browsing through very large GitHub issues and lots of it is outdated.

In addition to that, there still seem to be a more fundamental issues, because I keep reading here about initial compile times of 10s and even reloads that exceed seconds after a very fair share of optimization. I can only compare that to other frameworks I worked with and given the very high hardware demands (e.g. 64GB RAM and M1 Max are not sufficient to be really fast), this is frustrating.

So what I gathered so far:

What When How Documentation Remark
Upgrade Next Always Depending on version https://nextjs.org/docs/pages/building-your-application/upgrading
Serve with turbopack Always Adapt package.json https://nextjs.org/docs/architecture/turbopack Might not work for all projects
Optimize Imports Importing from large Add setting in next config https://nextjs.org/docs/app/api-reference/next-config-js/optimizePackageImports Applies to some packages by default (only if setting is in file since its experimental?)
Modularize Imports Deprecated / https://nextjs.org/docs/architecture/nextjs-compiler#modularize-imports Use "Optimize Imports" instead, that does not need manual configuration
Disable Windows Defender Slow filesystem reads on Windows Computers Might not be possible on managed computers
Filesystem slowness overall is what we've seen as the cause of problems, e.g. with Docker ? ?

Maybe there is a chance you add something to the next documentation about this?

And then back to my problem:

  • We tried all optimizations we could find
  • We tried the optimizations you suggested
  • We cannot use turopack because of a bug with server relative imports
  • We are still having a very bad dx. Is there anything we can do to move forward from here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
linear: next Confirmed issue that is tracked by the Next.js team.
Projects
None yet
Development

No branches or pull requests