Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide in V2 a builtin rule to go from Digest -> Snapshot #7716

Closed
Eric-Arellano opened this issue May 14, 2019 · 1 comment · Fixed by #7725
Closed

Provide in V2 a builtin rule to go from Digest -> Snapshot #7716

Eric-Arellano opened this issue May 14, 2019 · 1 comment · Fixed by #7725

Comments

@Eric-Arellano
Copy link
Contributor

Eric-Arellano commented May 14, 2019

We often work with Digests in V2, because they are more lightweight than snapshots.

However, there are certain times that we need to be able to get the .files from a Digest so that we can for example inspect whether each folder has an __init__.py in it, as done in #7696.

Currently, the only way to go from a Digest to somehow getting the folder's paths is to use the builtin Rust rule Digest -> FilesContent, which unnecessarily materializes the file content.

--

Alternatively, we could provide a way to go from MergedDirectories->Snapshot, in addition to MergedDirectories->Digest.

@illicitonion
Copy link
Contributor

A quick guided tour of what I imagine this would look like:

  • Add a unit test in tests/python/pants_test/engine/test_fs.py
  • Teach the engine that it knows how to take an input of type digest and return a product of type snapshot, by adding an entry here:
    let intrinsics = vec![
    Intrinsic {
    product: types.snapshot,
    input: types.path_globs,
    },
    Intrinsic {
    product: types.snapshot,
    input: types.url_to_fetch,
    },
    Intrinsic {
    product: types.files_content,
    input: types.directory_digest,
    },
    Intrinsic {
    product: types.directory_digest,
    input: types.merged_directories,
    },
    Intrinsic {
    product: types.directory_digest,
    input: types.directory_with_prefix_to_strip,
    },
    Intrinsic {
    product: types.process_result,
    input: types.process_request,
    },
    ];
  • Teach the engine how to fulfil that edge, but adding a new match branch to this match:
    match &self.entry {
    which looks like:
&rule_graph::Rule::Intrinsic(Intrinsic { product, input }) if product == context.core.types.snapshot && input == context.core.types.directory_digest => {...}

where the ... will actually look like: Copy and paste the code from one of the other branches which uses a directory_digest (e.g. the one that produces files_content), and then call Snapshot::from_digest on the result. Probably with a map_err to a throw copied and pasted from one of the other branches too. Roughly:

self
  .select_product(&context, context.core.types.directory_digest, "intrinsic")
  .and_then(|directory_digest_val| lift_digest(&directory_digest_val).map_err(|str| throw(&str)))
  .and_then(|digest| Snapshot::from_digest(context.core.store(), digest).map_err(|str| throw(&str)))
  .map(move |snapshot| Snapshot::store_snapshot(&core, &snapshot))
  .to_boxed()

probably with some extra clones and map_errs around.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants