Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AsyncFn family of traits #119305

Merged
merged 3 commits into from
Jan 25, 2024
Merged

Conversation

compiler-errors
Copy link
Member

I'm proposing to add a new family of asynchronous Fn-like traits to the standard library for experimentation purposes.

Why do we need new traits?

On the user side, it is useful to be able to express AsyncFn trait bounds natively via the parenthesized sugar syntax, i.e. x: impl AsyncFn(&str) -> String when experimenting with async-closure code.

This also does not preclude AsyncFn becoming something else like a trait alias if a more fundamental desugaring (which can take many1 different2 forms) comes around. I think we should be able to play around with AsyncFn well before that, though.

I'm also not proposing stabilization of these trait names any time soon (we may even want to instead express them via new syntax, like async Fn() -> ..), but I also don't think we need to introduce an obtuse bikeshedding name, since AsyncFn just makes sense.

The lending problem: why not add a more fundamental primitive of LendingFn/LendingFnMut?

Firstly, for async closures to be as flexible as possible, they must be allowed to return futures which borrow from the async closure's captures. This can be done by introducing LendingFn/LendingFnMut traits, or (equivalently) by adding a new generic associated type to FnMut which allows the return type to capture lifetimes from the &mut self argument of the trait. This was proposed in one of Niko's blog posts.

Upon further experimentation, for the purposes of closure type- and borrow-checking, I've come to the conclusion that it's significantly harder to teach the compiler how to handle general lending closures which may borrow from their captures. This is, because unlike Fn/FnMut, the LendingFn/LendingFnMut traits don't form a simple "inheritance" hierarchy whose top trait is FnOnce.

flowchart LR
    Fn
    FnMut
    FnOnce
    LendingFn
    LendingFnMut
    
    Fn -- isa --> FnMut
    FnMut -- isa --> FnOnce
    
    LendingFn -- isa --> LendingFnMut
    
    Fn -- isa --> LendingFn
    FnMut -- isa --> LendingFnMut
Loading

For example:

fn main() {
  let s = String::from("hello, world");
  let f = move || &s;
  let x = f(); // This borrows `f` for some lifetime `'1` and returns `&'1 String`.

That trait hierarchy means that in general for "lending" closures, like f above, there's not really a meaningful return type for <typeof(f) as FnOnce>::Output -- it can't return &'static str, for example.

Special-casing this problem:

By splitting out these traits manually, and making sure that each trait has its own associated future type, we side-step the issue of having to answer the questions of a general LendingFn/LendingFnMut implementation, since the compiler knows how to generate built-in implementations for first-class constructs like async closures, including the required future types for the (by-move) AsyncFnOnce and (by-ref) AsyncFnMut/AsyncFn trait implementations.

Footnotes

  1. For example, with trait transformers, we may eventually be able to write: trait AsyncFn = async Fn;

  2. For example, via the introduction of a more fundamental "LendingFn" trait, plus a special desugaring with augmented trait aliases.

@rustbot
Copy link
Collaborator

rustbot commented Dec 25, 2023

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Dec 25, 2023
@compiler-errors
Copy link
Member Author

I'm nominating this for T-libs-api discussion -- not sure if what process is needed to introduce these traits as unstable :)

r? libs-api

@rustbot rustbot added the T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. label Dec 25, 2023
@rustbot rustbot assigned m-ou-se and unassigned Mark-Simulacrum Dec 25, 2023
@compiler-errors compiler-errors added I-libs-api-nominated Nominated for discussion during a libs-api team meeting. and removed T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Dec 25, 2023
@compiler-errors
Copy link
Member Author

also cc @rust-lang/wg-async

@traviscross traviscross added WG-async Working group: Async & await T-lang Relevant to the language team, which will review and decide on the PR/issue. labels Dec 27, 2023
@traviscross
Copy link
Contributor

traviscross commented Dec 27, 2023

Exciting work. Big thanks to @compiler-errors for pushing this forward.

@rustbot labels I-lang-nominated

Nominating this for T-lang. At the end of the day T-lang will need to be satisfied with the design here, so we should discuss it at this point at least for visibility.

And given that this is adding new lang items, we may want to charter this as a T-lang experiment.

@rustbot labels I-async-nominated

Nominating this for WG-async. Async closures have been a recent topic of discussion for WG-async and have been identified as a priority. The async WG will at the end of the day want to be happy with the design, so we too should discuss it.

@rustbot rustbot added I-lang-nominated Nominated for discussion during a lang team meeting. I-async-nominated Nominated for discussion during an async working group meeting. labels Dec 27, 2023
@traviscross traviscross added the A-async-await Area: Async & Await label Dec 27, 2023
@joshtriplett joshtriplett removed the I-lang-nominated Nominated for discussion during a lang team meeting. label Jan 3, 2024
@traviscross traviscross removed the I-libs-api-nominated Nominated for discussion during a libs-api team meeting. label Jan 3, 2024
@nikomatsakis
Copy link
Contributor

Discussed in the @rust-lang/lang meeting and we agreed to accept this as an experiment (I will server as the liaison). All things being equal the goal we would like as much interop and generalization between "async functions" and "functions returning traits" as we can get, but I clarified that @compiler-errors and I were already aligned on that but that there are some interesting complications that make that non-trivial, and that the intent is to make concrete progress towards an MVP, with an eye for future compatibility.

@nikomatsakis
Copy link
Contributor

@rustbot labels -I-lang-nominated

@nikomatsakis
Copy link
Contributor

Hey @rust-lang/libs, I did not remove the I-libs-api-nominated tag. My take is that this is more of a lang question than a libs one, even though it involves a trait, and I would expect lang to be primary decision maker with libs-api in a "consulting role" (i.e., free to raise objections, but not requiring active checkboxes). If you all would like a more active role, though, happy to discuss.

@traviscross
Copy link
Contributor

Regarding the I-libs-api-nominated tag, I removed that one because T-libs-api had not nominated it themselves; it had just been nominated for them while we worked out who would charter this work. As discussed above, T-lang is claiming this as an experiment, so there's probably not any remaining nominated question for T-libs-api. But as nikomatsakis suggested, of course, please renominate if there are open questions that T-libs-api should discuss.

@compiler-errors
Copy link
Member Author

I'm gonna pause this for a second -- still trying to decide whether this is better than just introducing LendingFn and LendingFnMut traits.

@rustbot author

@rustbot rustbot removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 3, 2024
@rustbot rustbot added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jan 3, 2024
@compiler-errors
Copy link
Member Author

After a lot of thought and experimentation, I've decided it's best to first introduce AsyncFn traits here, and then work on generalizing the behavior of async closures to work with the other "coroutine transform" flavors (gen and async gen) via a desugaring that decouples asyncness from the fact that these closures enable self-borrows.

A desugaring that is backed by LendingFn is more powerful, but it has some difficult interactions with the new trait solver I'm interested in writing up sometime soon. Given that @nikomatsakis has agreed to serve as liaison for this lang team experiment, and the interesting changes here are all in the type system, I'll r? oli-obk.

@rustbot ready

@rustbot rustbot assigned oli-obk and unassigned m-ou-se Jan 23, 2024
@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 23, 2024
@oli-obk
Copy link
Contributor

oli-obk commented Jan 24, 2024

@bors r rollup

@bors
Copy link
Contributor

bors commented Jan 24, 2024

📌 Commit 17b4333 has been approved by oli-obk

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 24, 2024
fmease added a commit to fmease/rust that referenced this pull request Jan 24, 2024
…=oli-obk

Add `AsyncFn` family of traits

I'm proposing to add a new family of `async`hronous `Fn`-like traits to the standard library for experimentation purposes.

## Why do we need new traits?

On the user side, it is useful to be able to express `AsyncFn` trait bounds natively via the parenthesized sugar syntax, i.e. `x: impl AsyncFn(&str) -> String` when experimenting with async-closure code.

This also does not preclude `AsyncFn` becoming something else like a trait alias if a more fundamental desugaring (which can take many[^1] different[^2] forms) comes around. I think we should be able to play around with `AsyncFn` well before that, though.

I'm also not proposing stabilization of these trait names any time soon (we may even want to instead express them via new syntax, like `async Fn() -> ..`), but I also don't think we need to introduce an obtuse bikeshedding name, since `AsyncFn` just makes sense.

## The lending problem: why not add a more fundamental primitive of `LendingFn`/`LendingFnMut`?

Firstly, for `async` closures to be as flexible as possible, they must be allowed to return futures which borrow from the async closure's captures. This can be done by introducing `LendingFn`/`LendingFnMut` traits, or (equivalently) by adding a new generic associated type to `FnMut` which allows the return type to capture lifetimes from the `&mut self` argument of the trait. This was proposed in one of [Niko's blog posts](https://smallcultfollowing.com/babysteps/blog/2023/05/09/giving-lending-and-async-closures/).

Upon further experimentation, for the purposes of closure type- and borrow-checking, I've come to the conclusion that it's significantly harder to teach the compiler how to handle *general* lending closures which may borrow from their captures. This is, because unlike `Fn`/`FnMut`, the `LendingFn`/`LendingFnMut` traits don't form a simple "inheritance" hierarchy whose top trait is `FnOnce`.

```mermaid
flowchart LR
    Fn
    FnMut
    FnOnce
    LendingFn
    LendingFnMut

    Fn -- isa --> FnMut
    FnMut -- isa --> FnOnce

    LendingFn -- isa --> LendingFnMut

    Fn -- isa --> LendingFn
    FnMut -- isa --> LendingFnMut
```

For example:

```
fn main() {
  let s = String::from("hello, world");
  let f = move || &s;
  let x = f(); // This borrows `f` for some lifetime `'1` and returns `&'1 String`.
```

That trait hierarchy means that in general for "lending" closures, like `f` above, there's not really a meaningful return type for `<typeof(f) as FnOnce>::Output` -- it can't return `&'static str`, for example.

### Special-casing this problem:

By splitting out these traits manually, and making sure that each trait has its own associated future type, we side-step the issue of having to answer the questions of a general `LendingFn`/`LendingFnMut` implementation, since the compiler knows how to generate built-in implementations for first-class constructs like async closures, including the required future types for the (by-move) `AsyncFnOnce` and (by-ref) `AsyncFnMut`/`AsyncFn` trait implementations.

[^1]: For example, with trait transformers, we may eventually be able to write: `trait AsyncFn = async Fn;`
[^2]: For example, via the introduction of a more fundamental "`LendingFn`" trait, plus a [special desugaring with augmented trait aliases](https://rust-lang.zulipchat.com/#narrow/stream/213817-t-lang/topic/Lending.20closures.20and.20Fn*.28.29.20-.3E.20impl.20Trait/near/408471480).
bors added a commit to rust-lang-ci/rust that referenced this pull request Jan 24, 2024
Rollup of 8 pull requests

Successful merges:

 - rust-lang#119305 (Add `AsyncFn` family of traits)
 - rust-lang#119389 (Provide more context on recursive `impl` evaluation overflow)
 - rust-lang#120062 (llvm: change data layout bug to an error and make it trigger more)
 - rust-lang#120099 (linker: Refactor library linking methods in `trait Linker`)
 - rust-lang#120201 (Bump some deps with syn 1.0 dependencies)
 - rust-lang#120230 (Assert that a single scope is passed to `for_scope`)
 - rust-lang#120278 (Remove --fatal-warnings on wasm targets)
 - rust-lang#120292 (coverage: Dismantle `Instrumentor` and flatten span refinement)

r? `@ghost`
`@rustbot` modify labels: rollup
bors added a commit to rust-lang-ci/rust that referenced this pull request Jan 25, 2024
…iaskrgr

Rollup of 10 pull requests

Successful merges:

 - rust-lang#119305 (Add `AsyncFn` family of traits)
 - rust-lang#119389 (Provide more context on recursive `impl` evaluation overflow)
 - rust-lang#119895 (Remove `track_errors` entirely)
 - rust-lang#120230 (Assert that a single scope is passed to `for_scope`)
 - rust-lang#120278 (Remove --fatal-warnings on wasm targets)
 - rust-lang#120292 (coverage: Dismantle `Instrumentor` and flatten span refinement)
 - rust-lang#120315 (On E0308 involving `dyn Trait`, mention trait objects)
 - rust-lang#120317 (pattern_analysis: Let `ctor_sub_tys` return any Iterator they want)
 - rust-lang#120318 (pattern_analysis: Reuse most of the `DeconstructedPat` `Debug` impl)
 - rust-lang#120325 (rustc_data_structures: use either instead of itertools)

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 8c6cf3c into rust-lang:master Jan 25, 2024
11 checks passed
@rustbot rustbot added this to the 1.77.0 milestone Jan 25, 2024
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Jan 25, 2024
Rollup merge of rust-lang#119305 - compiler-errors:async-fn-traits, r=oli-obk

Add `AsyncFn` family of traits

I'm proposing to add a new family of `async`hronous `Fn`-like traits to the standard library for experimentation purposes.

## Why do we need new traits?

On the user side, it is useful to be able to express `AsyncFn` trait bounds natively via the parenthesized sugar syntax, i.e. `x: impl AsyncFn(&str) -> String` when experimenting with async-closure code.

This also does not preclude `AsyncFn` becoming something else like a trait alias if a more fundamental desugaring (which can take many[^1] different[^2] forms) comes around. I think we should be able to play around with `AsyncFn` well before that, though.

I'm also not proposing stabilization of these trait names any time soon (we may even want to instead express them via new syntax, like `async Fn() -> ..`), but I also don't think we need to introduce an obtuse bikeshedding name, since `AsyncFn` just makes sense.

## The lending problem: why not add a more fundamental primitive of `LendingFn`/`LendingFnMut`?

Firstly, for `async` closures to be as flexible as possible, they must be allowed to return futures which borrow from the async closure's captures. This can be done by introducing `LendingFn`/`LendingFnMut` traits, or (equivalently) by adding a new generic associated type to `FnMut` which allows the return type to capture lifetimes from the `&mut self` argument of the trait. This was proposed in one of [Niko's blog posts](https://smallcultfollowing.com/babysteps/blog/2023/05/09/giving-lending-and-async-closures/).

Upon further experimentation, for the purposes of closure type- and borrow-checking, I've come to the conclusion that it's significantly harder to teach the compiler how to handle *general* lending closures which may borrow from their captures. This is, because unlike `Fn`/`FnMut`, the `LendingFn`/`LendingFnMut` traits don't form a simple "inheritance" hierarchy whose top trait is `FnOnce`.

```mermaid
flowchart LR
    Fn
    FnMut
    FnOnce
    LendingFn
    LendingFnMut

    Fn -- isa --> FnMut
    FnMut -- isa --> FnOnce

    LendingFn -- isa --> LendingFnMut

    Fn -- isa --> LendingFn
    FnMut -- isa --> LendingFnMut
```

For example:

```
fn main() {
  let s = String::from("hello, world");
  let f = move || &s;
  let x = f(); // This borrows `f` for some lifetime `'1` and returns `&'1 String`.
```

That trait hierarchy means that in general for "lending" closures, like `f` above, there's not really a meaningful return type for `<typeof(f) as FnOnce>::Output` -- it can't return `&'static str`, for example.

### Special-casing this problem:

By splitting out these traits manually, and making sure that each trait has its own associated future type, we side-step the issue of having to answer the questions of a general `LendingFn`/`LendingFnMut` implementation, since the compiler knows how to generate built-in implementations for first-class constructs like async closures, including the required future types for the (by-move) `AsyncFnOnce` and (by-ref) `AsyncFnMut`/`AsyncFn` trait implementations.

[^1]: For example, with trait transformers, we may eventually be able to write: `trait AsyncFn = async Fn;`
[^2]: For example, via the introduction of a more fundamental "`LendingFn`" trait, plus a [special desugaring with augmented trait aliases](https://rust-lang.zulipchat.com/#narrow/stream/213817-t-lang/topic/Lending.20closures.20and.20Fn*.28.29.20-.3E.20impl.20Trait/near/408471480).
@compiler-errors compiler-errors deleted the async-fn-traits branch January 26, 2024 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-async-await Area: Async & Await I-async-nominated Nominated for discussion during an async working group meeting. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-lang Relevant to the language team, which will review and decide on the PR/issue. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. WG-async Working group: Async & await
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants