@tmeasday yep, I think we have critical perf bug here

Thanks @alexander-akait. Happy to help with a solution once the path forward is known. Naively it feels like perhaps there are opportunities to avoid scanning the require.context() when the event that triggers the update is a single file modification (and so is already known to match or not match the context).

@alexander-akait, with the help of @tmeasday, I found a workaround.

See storybookjs/storybook#14894 (comment)

@vidal7 Can you provide example?

@alexander-akait, sure instead of doing

module.exports = {
    stories: [
        './src/**/*.stories.ts',
    ]
};

I am doing something like this. I oversimplified my code just for this example.

const { sync } = require('glob');

module.exports = {
    stories: sync('./src/**/*.stories.ts')
};

I believe there's a developer experience caveat – In my tests, the technique described above has an undesirable side effect: the glob function generates an array of files at start time, and after that, it won't dynamically index newly added/renamed files.

This means developers need to restart Storybook whenever they add a new <filename>.stories.ts file, as the newly glob-generated array will then contain this new file.

Ideally, we'd have something fast, with the same "dynamic globbing" as before.

@kaelig, yes you are right. I have this caveat too. I didn't realized that by doing this, storybook on webpack rebuild does not scan new stories files. I don't have a better solution for now. Guess this issue will have to be resolved if you want new stories files to be indexed. I am sorry.

In theory you can create entry.js file and apply loader, inside loader run glob and create imports like:

import oneStory from "./one-story/index.js";
import twoStory from "./two-story/index.js";

inside loader, use glob-parent (i.e. extract src directory) and use this.addContextDependency https://webpack.js.org/api/loaders/#thisaddcontextdependency, it can be some
slow on initial run, but will be fast for incremental builds, also it will tract all changes inside src directory, i.e. we just generate entry file based on glob.

But why do not use https://webpack.js.org/configuration/experiments/#experimentslazycompilation with import()? All stories can be loaded using import(). If you need to add new story just add new lines to your source file with stories. This is actually very close to how it works in browser, even more it can work in browser without bundlers (in theory). WE don't have glob/require.context in browser and we should migrate from these hacky workarounds.

@alexander-akait - are you suggesting that they write a big file full of import() functions?

That is definitely possible, but for a development-time tool like Storybook which prioritizes DX over "correctness" in this way, we'd like to figure out a way to make it possible for users to provide a glob but still have good performance.

One idea I've been looking at is to have storybook generate that file of imports, in a similar way to your first suggestion, see this thread: https://webpack.slack.com/archives/C1LUX2DS9/p1633338458009500?thread_ts=1633265679.007900&cid=C1LUX2DS9

All this aside, I'm still not sure what webpack is doing that leads to the poor performance in the first place. When I use watchpack alone and make a change to a single file in a big directory full of stuff, it is fast. My guess is that webpack is re-scanning the glob every time watchpack detects a (set of?) change(s). I'm not sure why it would need to do that, but if there is a good reason I am very interested to know why.

Theoretically there is no reason to re-scan the glob after the initial scan and a stream of changes, especially if those changes are small.

Yes, problems in that we need scan whole directory (include nested) for each build, otherwise we can't catch new/deleted files...

Yes, problems in that we need scan whole directory (include nested) for each build, otherwise we can't catch new/deleted files...

This problem was already solved several years ago in the Jest, and jest-haste-map was made especially for this. It would be great to use it for such cases, but the problem is that it is not well documented.

Yes, hope somebody found time to help us here