Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flattening feature creates lengthy folder structures #6830

Open
ishpagin opened this issue Jul 18, 2023 · 6 comments
Open

Flattening feature creates lengthy folder structures #6830

ishpagin opened this issue Jul 18, 2023 · 6 comments

Comments

@ishpagin
Copy link

ishpagin commented Jul 18, 2023

Hello pnpm team,

I'm having an issue on Windows because of the long path names that pnpm is creating. I believe this is due to the flattening feature that creates quite lengthy folder structures. Unfortunatly, this is causing problems when I'm trying to build my project, since Windows has a limit of 260 characters for the path.

For example, I have a package @nuxtjs [email protected]_@typescript-eslint [email protected][email protected][email protected]. As a result, I have very long path when I try build the project C:/Users/YourName/Documents/Projects/YourProject/node_modules/@nuxtjs [email protected]_@typescript-eslint [email protected][email protected][email protected]/some/other/directory/with/many/subdirectories/and/a/very/long/path.js, the build process can't find the files because the path is too long.

I've already tried several workarounds such as enabling long paths in Windows and moving my project closer to the root directory, but these haven't solved my problem completely.

This is not a bug of the @nuxtjs package. It occurs in our internal repositories that use the layers mechanism from Nuxt 3 and create very long paths when we want to import a file from a layer below. But this could be repeated for any deeply nested repository. For example, C:/Users/YourName/Documents/Projects/Some-Long-Path-To-Cause-An-Error-With-Pnpm-Resolution/YourProject/node_modules/@nuxtjs [email protected]_@typescript-eslint [email protected][email protected][email protected]/index.js. The only fix that helped me was setting the node-linker=hoisted variable to avoid creating long paths. Could you perhaps make the hashing shorter, for example, to avoid generating such long results when flattening?

Could you please look into this issue and perhaps consider a way to reduce the length of the path names created by pnpm? This would help a lot for those of us who are using Windows for development, but I read that Linux also has a problem that is not written anywhere how to solve.

Thank you for your time and for developing such a helpful tool!

Best regards,

pnpm version:

pnpm -v
8.5.1/latest

Code to reproduce the issue:

{
  "dependencies": {
    "nuxt": "^3.6.3",
    "@nuxt/types": "2.15.8",
    "@nuxt/typescript-build": "2.1.0",
    "@nuxtjs/eslint-config-typescript": "10.0.0",
    "eslint": "8.15.0"
  }
}

shamefully-hoist=true

import file from path with loooong name, or multiple pathes and from folder @nuxtjs eslint-config@10.0.0_@typescript-eslint [email protected][email protected][email protected]

Expected behavior:

No errors

Actual behavior:

ERROR Error: ENOENT: no such file or directory, realpath

Additional information:

  • node -v prints:
    node -v
    v18.16.1

  • Windows, macOS, or Linux?:
    Windows 10/11

@zkochan
Copy link
Member

zkochan commented Jul 20, 2023

We currently allow the directories inside node_modules/.pnpm have a length of not more than 120 chars. Here is the related code:

const MAX_LENGTH_WITHOUT_HASH = 120 - 26 - 1
export function depPathToFilename (depPath: string) {
let filename = depPathToFilenameUnescaped(depPath).replace(/[\\/:*?"<>|]/g, ' ')
if (filename.includes('(')) {
filename = filename
.replace(/(\)\()|\(/g, '_')
.replace(/\)$/, '')
}
if (filename.length > 120 || filename !== filename.toLowerCase() && !filename.startsWith('file ')) {
return `${filename.substring(0, MAX_LENGTH_WITHOUT_HASH)}_${createBase32Hash(filename)}`
}
return filename
}

We can try another approach (on Windows or on all systems). We can check what is the length of the path to the node_modules/.pnpm directory and if it is long, we can reduce the allowed max length of directories inside .pnpm.

@karlhorky
Copy link

I also ran into this just now, and reported an ESLint bug because it crashes on Windows with Cannot find package:

@MrZhang123
Copy link

MrZhang123 commented Apr 28, 2024

Same issues, pnpm v8.6.5.
I use pnpm patch creates too long hash, the path is too long on Windows, causing an error reporting that the file does not exist. So, how can this problem be solved? How can I make the hash shorter after pnpm patch?

@karlhorky
Copy link

karlhorky commented Apr 28, 2024

@MrZhang123 maybe you want to try upgrading to Node.js 22?

It looks like the Node.js v22 PR contains the fix "module,win: fix long path resolve (by @StefanStojanovic) nodejs/node#51097":

So maybe Node.js has support for long paths on Windows now!

(make sure that you have LongPathsEnabled set to true on the Windows machine you test on)

@Ice-mourne
Copy link

Ice-mourne commented Jun 3, 2024

@MrZhang123 maybe you want to try upgrading to Node.js 22?

It looks like the Node.js v22 PR contains the fix "module,win: fix long path resolve (by @StefanStojanovic) nodejs/node#51097":

So maybe Node.js has support for long paths on Windows now!

(make sure that you have LongPathsEnabled set to true on the Windows machine you test on)

Node 22 only partially fixes the problem it went from one issue to another and ultimately still can cause problems
Thats with the latest pnpm 9.1.4

@karlhorky
Copy link

karlhorky commented Jun 3, 2024

Node 22 only partially fixes the problem

Yes, sorry, my Node.js issue was reopened, Node.js v22 does not resolve the issue:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants