Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: npm workspace and better Deno workspace support #24334

Merged
merged 101 commits into from
Jul 4, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
101 commits
Select commit Hold shift click to select a range
3660270
Start on workspace refactor.
dsherret Jun 17, 2024
2c591ef
Some work on tasks
dsherret Jun 18, 2024
81dc4a8
Switching more to workspace
dsherret Jun 18, 2024
271df70
More.
dsherret Jun 18, 2024
b155d9a
Remove cli/args/package_json.rs
dsherret Jun 18, 2024
c1d52f8
some work on deno compile
dsherret Jun 18, 2024
c8dbbb1
tasks
dsherret Jun 19, 2024
04fd3c6
update publishing
dsherret Jun 19, 2024
03b30f4
Adding fmting
dsherret Jun 19, 2024
1e8b196
Merge branch 'main' into workspace_start
dsherret Jun 19, 2024
cb6b545
some work on linting
dsherret Jun 19, 2024
5e9fd3f
finish lint
dsherret Jun 20, 2024
4fe5441
Some work on lsp and installing from all the packages
dsherret Jun 20, 2024
19ad5c4
separate out test options from workspace test options
dsherret Jun 20, 2024
6eed364
finish up test
dsherret Jun 20, 2024
c04afc7
Add bench
dsherret Jun 20, 2024
49b1a23
compiling
dsherret Jun 20, 2024
76dcc18
handle base properly
dsherret Jun 20, 2024
4205667
fix slow types linting
dsherret Jun 20, 2024
2033d89
Use SharedLocal and fix some vendor tests.
dsherret Jun 21, 2024
297d191
Start making import map optional
dsherret Jun 21, 2024
8583651
Start investigating @types issue
dsherret Jun 21, 2024
df4e20e
Fix. Wasn't thinking.
dsherret Jun 21, 2024
a7b5857
fix info
dsherret Jun 21, 2024
c5df354
fix watcher tests
dsherret Jun 21, 2024
77c0977
working npm tests
dsherret Jun 21, 2024
d38c216
some import specifier fixes
dsherret Jun 22, 2024
ffa9bb1
Fix some lsp issues.
dsherret Jun 24, 2024
e887d93
Fix test
dsherret Jun 24, 2024
64c20ac
We understand the npm workspace now
dsherret Jun 24, 2024
171bed6
Make deno compile work with relative specifiers
dsherret Jun 24, 2024
5c330c1
Fix in import_map/deno_emit
dsherret Jun 25, 2024
2b4247c
Compile fixes
dsherret Jun 25, 2024
3c9fec7
Fix some byonm stuff
dsherret Jun 25, 2024
e691125
Merge branch 'main' into workspace_start
dsherret Jun 25, 2024
4d3ac9e
Update to actual deps
dsherret Jun 26, 2024
e844ee1
refactor: move `PackageJson` to deno_config
dsherret Jun 26, 2024
5770c6b
Update
dsherret Jun 26, 2024
1ca7b82
fix failing tests
dsherret Jun 26, 2024
6b65431
Update deno_config
dsherret Jun 26, 2024
25439b0
fix failing test by clearing package json cache
dsherret Jun 26, 2024
c72410e
fix formatting
dsherret Jun 26, 2024
f41a348
Merge branch 'main' into workspace_start
dsherret Jun 26, 2024
b1e4645
Merge branch 'package_json_deno_config' into workspace_start
dsherret Jun 26, 2024
11eb6ff
Fixes for latest changes
dsherret Jun 26, 2024
1e28b4e
clippy
dsherret Jun 26, 2024
f626f10
Resolve todo
dsherret Jun 26, 2024
3fe1853
update from what will be main
dsherret Jun 26, 2024
c4b55bf
Resolve some todos
dsherret Jun 26, 2024
698d9aa
Merge branch 'main' into workspace_start
dsherret Jun 26, 2024
1281f76
Merge branch 'main' into workspace_start
dsherret Jun 26, 2024
217a217
fix failing tests
dsherret Jun 26, 2024
163bef4
npm workspace resolution
dsherret Jun 27, 2024
05ce93e
handle going to npm workspace deps for npm specifiers too
dsherret Jun 27, 2024
de1b298
Add test for workspace
dsherret Jun 27, 2024
2fd7ffa
add an npm specifier referencing a workspace package
dsherret Jun 27, 2024
0a9e1c3
move this down to deno_config
dsherret Jun 27, 2024
33ab587
updates for latest
dsherret Jun 27, 2024
2c35049
update to new Workspace::discover signature
dsherret Jun 28, 2024
6e9c118
Support `@workspace` tag in an npm specifier
dsherret Jun 28, 2024
a063f9a
Support pkg json "workspace:" specifiers
dsherret Jun 28, 2024
91c3c1b
Move test
dsherret Jun 28, 2024
8851529
Initialize node_modules dir with used workspace members
dsherret Jun 28, 2024
8dd40a9
Fix some tests and update var names
dsherret Jun 29, 2024
66c863b
Update workspace basic test
dsherret Jun 29, 2024
5df7730
Merge branch 'main' into workspace_start
dsherret Jun 29, 2024
44c189e
Merge branch 'main' into workspace_start
dsherret Jun 29, 2024
07a57b4
Update for failing compile
dsherret Jun 29, 2024
ebfe959
Merge branch 'main' of https://github.com/denoland/deno into workspac…
dsherret Jul 1, 2024
9a1e921
improve resolving root dir for deno compile
dsherret Jul 1, 2024
497c22d
Resolve more todos about errors in sub path resolution
dsherret Jul 1, 2024
b0fc65d
Add tests for and fix formatting
dsherret Jul 1, 2024
914be64
Fix some issues and add tests for linting
dsherret Jul 1, 2024
bb1988e
Add no slow types workspace lint test
dsherret Jul 1, 2024
36c1ca0
Merge branch 'main' into workspace_start
dsherret Jul 1, 2024
7fee5f6
Format and ignore some files from formatting
dsherret Jul 1, 2024
ea41334
Add tests for exports not found in npm workspace
dsherret Jul 1, 2024
59fb3db
Merge branch 'main' into workspace_start
dsherret Jul 1, 2024
f8cf42a
diagnostics
dsherret Jul 1, 2024
57fdbde
Add tests for lockfile going to the workspace root
dsherret Jul 1, 2024
8a97d8c
Add test for check
dsherret Jul 2, 2024
9ce4caa
Update deno_config
dsherret Jul 2, 2024
803b097
Add bench tests
dsherret Jul 2, 2024
754b308
add test
dsherret Jul 2, 2024
9d38ecf
Merge branch 'main' into workspace_start
dsherret Jul 2, 2024
6aaac91
fix specs
dsherret Jul 2, 2024
1448564
was using a backslash in a path
dsherret Jul 2, 2024
3f7569f
Bump ci cache
dsherret Jul 2, 2024
7e6644f
Switch back to resolving based on multiple args.
dsherret Jul 3, 2024
db95f58
update some tests
dsherret Jul 3, 2024
43f1f5f
Merge branch 'main' into workspace_start
dsherret Jul 3, 2024
372ceed
update tests
dsherret Jul 3, 2024
0c98628
fix copyright
dsherret Jul 3, 2024
51bed91
add tests for vendor
dsherret Jul 3, 2024
8ac81df
update test for pkg with no deps in lockfile in workspace
dsherret Jul 3, 2024
2a957e0
Merge branch 'main' into workspace_start
dsherret Jul 3, 2024
650a43d
- Fix private registries not working with auto-install in deno compile
dsherret Jul 3, 2024
ae7ebec
maybe fix windows tests
dsherret Jul 3, 2024
4d0d459
Updates from review
dsherret Jul 3, 2024
21673a3
Merge branch 'main' into workspace_start
dsherret Jul 3, 2024
b282f75
Upgrade deno_unsync. Remove fatal diagnostic test because it's in den…
dsherret Jul 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix some tests and update var names
  • Loading branch information
dsherret committed Jun 29, 2024
commit 8dd40a97282b92e32b62101a5b21d9c8d723c629
7 changes: 6 additions & 1 deletion cli/args/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 9,7 @@ mod package_json;

use deno_ast::SourceMapOption;
use deno_config::workspace::CreateResolverOptions;
use deno_config::workspace::PackageJsonDepResolution;
use deno_config::workspace::Workspace;
use deno_config::workspace::WorkspaceDiscoverOptions;
use deno_config::workspace::WorkspaceDiscoverStart;
Expand Down Expand Up @@ -1108,7 1109,11 @@ impl CliOptions {
.create_resolver(
CreateResolverOptions {
// todo(dsherret): this should be false for nodeModulesDir: true
pkg_json_dep_resolution: !self.use_byonm(),
pkg_json_dep_resolution: if self.use_byonm() {
PackageJsonDepResolution::Disabled
} else {
PackageJsonDepResolution::Enabled
},
specified_import_map: cli_arg_specified_import_map,
},
|specifier| {
Expand Down
4 changes: 2 additions & 2 deletions cli/args/package_json.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 46,7 @@ impl PackageJsonInstallDepsProvider {
{
workspace_pkgs.push(InstallNpmWorkspacePkg {
alias,
pkg_dir: pkg.package_json.dir_path().to_path_buf(),
pkg_dir: pkg.pkg_json.dir_path().to_path_buf(),
});
} else {
pkg_reqs.push(pkg_req)
Expand All @@ -58,7 58,7 @@ impl PackageJsonInstallDepsProvider {
}) {
workspace_pkgs.push(InstallNpmWorkspacePkg {
alias,
pkg_dir: pkg.package_json.dir_path().to_path_buf(),
pkg_dir: pkg.pkg_json.dir_path().to_path_buf(),
});
}
}
Expand Down
8 changes: 8 additions & 0 deletions cli/lsp/resolver.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 25,7 @@ use crate::util::progress_bar::ProgressBarStyle;
use dashmap::DashMap;
use deno_ast::MediaType;
use deno_cache_dir::HttpCache;
use deno_config::workspace::PackageJsonDepResolution;
use deno_config::workspace::WorkspaceResolver;
use deno_core::error::AnyError;
use deno_core::url::Url;
Expand Down Expand Up @@ -501,6 502,13 @@ fn create_graph_resolver(
.and_then(|d| d.package_json.clone())
.into_iter()
.collect(),
if config_data.map(|d| d.byonm).unwrap_or(false) {
PackageJsonDepResolution::Disabled
} else {
// todo(dsherret): this should also be disabled for when using
// auto-install with a node_modules directory
PackageJsonDepResolution::Enabled
},
)),
maybe_jsx_import_source_config: config_file
.and_then(|cf| cf.to_maybe_jsx_import_source_config().ok().flatten()),
Expand Down
3 changes: 3 additions & 0 deletions cli/standalone/binary.rs
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes deno compile to use relative instead of absolute specifiers. I started working on separating this out to a separate PR, but it's hard to untangle it from the workspaces changes.

Original file line number Diff line number Diff line change
Expand Up @@ -15,6 15,7 @@ use std::path::PathBuf;
use std::process::Command;

use deno_ast::ModuleSpecifier;
use deno_config::workspace::PackageJsonDepResolution;
use deno_config::workspace::Workspace;
use deno_core::anyhow::bail;
use deno_core::anyhow::Context;
Expand Down Expand Up @@ -76,6 77,7 @@ pub struct SerializedWorkspaceResolverImportMap {
pub struct SerializedWorkspaceResolver {
pub import_map: Option<SerializedWorkspaceResolverImportMap>,
pub package_jsons: BTreeMap<String, serde_json::Value>,
pub pkg_json_resolution: PackageJsonDepResolution,
}

#[derive(Deserialize, Serialize)]
Expand Down Expand Up @@ -615,6 617,7 @@ impl<'a> DenoCompileBinaryWriter<'a> {
)
})
.collect(),
pkg_json_resolution: workspace_resolver.pkg_json_dep_resolution(),
},
node_modules,
disable_deprecated_api_warning: cli_options
Expand Down
6 changes: 5 additions & 1 deletion cli/standalone/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -567,7 567,11 @@ pub async fn run(
Arc::new(pkg_json)
})
.collect();
WorkspaceResolver::new_raw(import_map, pkg_jsons)
WorkspaceResolver::new_raw(
import_map,
pkg_jsons,
metadata.workspace_resolver.pkg_json_resolution,
)
};
let cli_node_resolver = Arc::new(CliNodeResolver::new(
Some(cjs_resolutions.clone()),
Expand Down
18 changes: 6 additions & 12 deletions cli/tools/registry/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -252,10 252,8 @@ impl PublishPreparer {
let package = self
.prepare_publish(&member, graph, diagnostics_collector)
.await
.with_context(|| {
format!("Failed preparing '{}'.", member.package_name)
})?;
Ok::<_, AnyError>((member.package_name, package))
.with_context(|| format!("Failed preparing '{}'.", member.name))?;
Ok::<_, AnyError>((member.name, package))
}
.boxed()
})
Expand Down Expand Up @@ -412,19 410,19 @@ impl PublishPreparer {
"version": "{}",
"exports": "{}"
}}"#,
package.package_name,
package.name,
version,
suggested_entrypoint.unwrap_or("<path_to_entrypoint>")
);

bail!(
"You did not specify an entrypoint to \"{}\" package in {}. Add `exports` mapping in the configuration file, eg:\n{}",
package.package_name,
package.name,
deno_json.specifier,
exports_content
);
}
let Some(name_no_at) = package.package_name.strip_prefix('@') else {
let Some(name_no_at) = package.name.strip_prefix('@') else {
bail!("Invalid package name, use '@<scope_name>/<package_name> format");
};
let Some((scope, name_no_scope)) = name_no_at.split_once('/') else {
Expand Down Expand Up @@ -473,11 471,7 @@ impl PublishPreparer {
})
.await??;

log::debug!(
"Tarball size ({}): {}",
package.package_name,
tarball.bytes.len()
);
log::debug!("Tarball size ({}): {}", package.name, tarball.bytes.len());

Ok(Rc::new(PreparedPublishPackage {
scope: scope.to_string(),
Expand Down
4 changes: 2 additions & 2 deletions cli/tools/registry/publish_order.rs
Original file line number Diff line number Diff line change
Expand Up @@ -173,12 173,12 @@ fn build_pkg_deps(
specifier.as_str().starts_with(dir_url.as_str())
});
if let Some(root) = found_root {
deps.insert(root.1.package_name.clone());
deps.insert(root.1.name.clone());
}
}
}
}
members.insert(pkg_config.package_name.clone(), deps);
members.insert(pkg_config.name.clone(), deps);
}
Ok(members)
}
Expand Down
1 change: 1 addition & 0 deletions cli/tools/registry/unfurl.rs
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 389,7 @@ mod tests {
let workspace_resolver = WorkspaceResolver::new_raw(
Some(import_map),
vec![Arc::new(package_json)],
deno_config::workspace::PackageJsonDepResolution::Enabled,
);
let fs = Arc::new(RealFs);
let sloppy_imports_resolver = SloppyImportsResolver::new(fs);
Expand Down
1 change: 1 addition & 0 deletions cli/tools/vendor/test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 297,7 @@ fn build_resolver(
workspace_resolver: Arc::new(WorkspaceResolver::new_raw(
maybe_original_import_map,
Vec::new(),
deno_config::workspace::PackageJsonDepResolution::Enabled,
)),
maybe_jsx_import_source_config,
maybe_vendor_dir: None,
Expand Down
2 changes: 1 addition & 1 deletion tests/specs/install/future_install_global/__test__.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 5,7 @@
},
"steps": [
{
"args": "install --global --root ./bins --name deno-test-bin ./main.js",
"args": "install --global --root ./bins --name deno-test-bin ./pkg/main.js",
"output": "install.out"
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 2,7 @@
"tempDir": true,
"steps": [
{
"args": "install --root ./bins --name deno-test-bin ./main.js",
"args": "install --root ./bins --name deno-test-bin ./pkg/main.js",
"output": "install.out"
},
{
Expand Down
1 change: 1 addition & 0 deletions tests/specs/install/no_future_install_global/install.out
Original file line number Diff line number Diff line change
@@ -1,5 1,6 @@
⚠️ `deno install` behavior will change in Deno 2. To preserve the current behavior use the `-g` or `--global` flag.
Download http://localhost:4260/@denotest/esm-basic
Download http://localhost:4260/@denotest/esm-basic/1.0.0.tgz
[# there shouldn't be a line saying it initialized the node_modules folder here because this is a global install]
✅ Successfully installed deno-test-bin[WILDCARD]
[WILDCARD]
14 changes: 7 additions & 7 deletions tests/specs/run/workspaces/basic/main.out
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 2,19 @@
"imports": {
"chalk": "npm:chalk",
"chalk/": "npm:/chalk/",
"qwerqwer": "jsr:qwerqwer@^0.0.0",
"qwerqwer/": "jsr:/qwerqwer@^0.0.0/",
"asdfasdfasdf": "jsr:asdfasdfasdf@^0.0.0",
"asdfasdfasdf/": "jsr:/asdfasdfasdf@^0.0.0/"
"asdfasdfasdf/": "jsr:/asdfasdfasdf@^0.0.0/",
"qwerqwer": "jsr:qwerqwer@^0.0.0",
"qwerqwer/": "jsr:/qwerqwer@^0.0.0/"
},
"scopes": {
"./foo/": {
"~/": "./foo/",
"foo/": "./foo/bar/"
},
"./bar/": {
"@/": "./bar/",
"secret_mod/": "./bar/some_mod/"
},
"./foo/": {
"~/": "./foo/",
"foo/": "./foo/bar/"
}
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 1,5 @@
{
"args": "run -A main.ts",
"args": "run -A sub_dir/child/main.ts",
"output": "main.out",
"tempDir": true,
"exitCode": 1
Expand Down
4 changes: 3 additions & 1 deletion tests/specs/run/workspaces/member_outside_root_dir/main.out
Original file line number Diff line number Diff line change
@@ -1 1,3 @@
error: Workspace member '../other_folder' is outside root configuration directory[WILDCARD]
error: Workspace member must be nested in a directory under the workspace.
Member: file:///[WILDLINE]/sub_dir/other_folder/
Workspace: file:///[WILDLINE]/sub_dir/child/
2 changes: 1 addition & 1 deletion tests/specs/run/workspaces/nested_member/__test__.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 2,5 @@
"args": "run -A main.ts",
"output": "main.out",
"tempDir": true,
"exitCode": 1
"exitCode": 0
}
3 changes: 2 additions & 1 deletion tests/specs/run/workspaces/nested_member/bar/deno.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 4,6 @@
"imports": {
"@/": "./",
"secret_mod/": "./some_mod/"
}
},
"exports": "./mod.ts"
}
3 changes: 2 additions & 1 deletion tests/specs/run/workspaces/nested_member/foo/bar/deno.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 3,6 @@
"version": "0.0.0",
"imports": {
"chalk": "npm:chalk"
}
},
"exports": "./hello.ts"
}
3 changes: 2 additions & 1 deletion tests/specs/run/workspaces/nested_member/foo/deno.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 3,6 @@
"version": "0.0.0",
"imports": {
"~/": "./"
}
},
"exports": "./mod.ts"
}
5 changes: 4 additions & 1 deletion tests/specs/run/workspaces/nested_member/main.out
Original file line number Diff line number Diff line change
@@ -1 1,4 @@
error: Workspace member 'foo/bar' is nested within other workspace member 'foo'
Download http://localhost:4260/chalk
Download http://localhost:4260/chalk/chalk-5.0.1.tgz
buzz from foo
[Function: chalk] createChalk { level: 0 }