r? @fmease

rustbot has assigned @fmease.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

@c410-f3r Has this been documented in the Reference already? The box is not ticked over at the tracking issue. If not, stabilization is blocked on adding sufficient documentation.

@rfcbot merge

@rfcbot concern document-in-reference

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @tmandry

Concerns:

• depth levels are confusing (Stabilize count, ignore, index, and length (macro_metavar_expr) #122808 (comment))
• document-in-reference resolved by Stabilize count, ignore, index, and length (macro_metavar_expr) #122808 (comment)
• hygiene-handled? resolved by Stabilize count, ignore, index, and length (macro_metavar_expr) #122808 (comment)

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

I brought this up in the doc PR but it belongs here – length should probably be renamed len before stabilization. The latter is de facto standard in the standard library, whereas the former is only used in a single unstable API. These metafunctions aren’t library items of course, but should presumably still be consistent with established names.

for everyone following along, @jdahlstrom's proposal was yote before the relevant teams, accepted, and merged in #124987.

After looking into this a bit deeper, I think some of the reason this API feels so confusing is that we do not have a way to refer to specific repetition groups directly. Instead, we try to refer to specific groups based on the variables they capture and an offset - this gets messy. E.g., this example from the reference PR:

macro_rules! innermost1 {
    ( $( $a:ident: $( $b:literal ),* );  ) => {
        [$( $( ${ignore($b)} ${index(1)}, )* ) ]
    };
}

Here you have to count how many repetition groups exist in the match expression, then count how many repetition groups the index(1) call is nested in, then backtrack by one to figure out what exactly is getting indexed. This is a lot to keep track of mentally.

I posted a pre-RFC (edit: now RFC) to discuss the idea of being able to name capture groups, which could then be referred to directly within metavar expressions. The above would become:

macro_rules! innermost1 {
    ( $outer_rep:( $a:ident: $inner_rep:( $b:literal ),* );  ) => {
        [$outer_rep( $inner_rep( ${index($outer_rep)}, )* ) ]
        // (side effect: `ignore($b)` isn't needed anymore since the repetition groups
        // are identified directly, rather than only by which metavars they contain)
    };
}

It is significantly easier to immediately see that index($outer_rep) refers to the outermost repetition group.

len($var) also tries to use $var as a proxy for the group it captures. len($outer_rep) (or possibly better count_repetitions($outer_rep)) would be more accurate here.

A nice side effect of the proposal is that our diagnostics could become more expressive, rather than gesturing at some code and trying to explain repetition errors in terms of what is captured.

One other thing is that the term depth seems backwards, since increasing the depth value actually decreases the level of depth/nesting. parent seems like a slightly better term for this argument, there might be something better yet. This can of course be fixed at any time without breaking anything.

In short: I feel great about stabilizing index() (no depth) and ignore($var) now. However, it seems like the API for len and count could be improved significantly if they were given a way to refer to groups themselves, rather than indirectly through captures.

(adding named repetition groups and using them for count/len would be backward compatible, it would just be nice to not support the offset/depth-based API at all.)

so then count would be something like the following?

macro_rules! m {
    ($outer:(a $middle:(b $inner:(c $v:ident)*)*)*) => {
        println!("{}", stringify!(
            ${count($outer, $inner)}
            [$outer(${count($middle, $inner)})*]
            ${count($outer, $middle)}
            ${count($outer)}
            [$outer(${count($middle)})*]
            [$outer([$middle(${count($inner)})*])*]
        ));
    }
}
m!(a b c v0 c v1 a a b b);
// prints (without the comments):
// 2 // two inners inside outer
// [ 2 0 0 ] // two inners inside the first middle, zero for the rest
// 3 // three middles inside outer
// 3 // three repetitions inside outer
// [ 2 0 2 ] // two repetitions inside first middle, zero for second, two for third
// // two repetitions inside inner in first middle, no inners in second middle, no repetitions inside both inners in third middle
// [ [ 2 ] [ ] [ 0 0 ] ]

Those all have the results I would expect, except I think [$outer(${count($middle)})*] would be [1 0 2] since there is only one $middle within the first $outer. And for the first three, I wasn't thinking multiple arguments would be needed since the variable is already scoped to a repetition group.

I am loosely using these rules:

• When querying more deeply nested variables/groups (downward), expressions only "see" those instances that are direct descendants
• When querying equally or less deeply nested variables/groups (upward), expressions "see" everything (i.e. neighbors, parents, everything at the same level)

m!(a b c v0 c v1 a a b b);

              /--------- (root) macro ------------\
             /                |                    \
        (outer0) a         (outer1) a           (outer2) a
           |                                   /        \
       (middle0) b                      (middle1) b    (middle2) b
      /        \
(inner0) c  (inner1) c
    |           |
 (v0) v0      (v1) v1

So, for your first three examples where I think I would change the matcher:

1. ${count($outer, $inner)} would just be ${count($inner)} -> 2. It gets called once at the root level and sees all instances of $inner that could possibly exist (2 here)
2. [$outer(${count($middle, $inner)})*] would just be [$outer(${count($inner)})*] -> [2 0 0]. It gets called once at outer0 (sees two $inners as descendants), once at outer1 (no $inners), and once at outer2 (no $inners)
3. ${count($outer, $middle)} would just be ${count($middle)} -> 3. It gets called once at the root level and sees all instances of $middle as descendants to count

Explanation of the remaining three:

4. ${count($outer)} -> 3: root scope, sees 3 total repetitions of $outer, returns 3
5. [$outer(${count($middle)})*] -> [1 0 2] sees one middle as a child at outer0, none at outer1, 2 at outer2.
6. [$outer([$middle(${count($inner)})*])*] -> [ [ 2 ] [ ] [ 0 0 ] ] just repeating the above

And then some others possibilities:

7. [$outer([$middle(${count($v})] -> [[ 2 ] [ ] [ 0 0 ]]: regular metavars can still be counted. In this case count($v) and count($inner) happen to give the same results
8. [$outer( ${count($outer)} )] -> [3 3 3]: Instantiated at outer1, outer2, outer3, can see everything at that level and above. ${len()} would basically be an alias of this.
9. [$outer([$middle([$inner( ${count($outer)} )*])*])*] -> [[[3 3] [ ] [ ]]: similar to above, just deeper nesting

(Examples 8 and 9 show cases where the current len() and count() produce identical results, with the only difference being restrictions on where they can be used. If we got named capture groups I think we could likely combine these, or at least make their differences more clear).

The use of a named placeholder to indicate the depth of a repetition seems orthogonal to the stabilization of the current approach accepted in the RFC. As such, it probably shouldn't be a blocking concern.

On the merits, the suggested approach looks overly verbose but nothing prevents a future inclusion if so desired by the responsible team.

I tend to agree with @tgross35 -- having to refer to capture groups by index is a really awkward and fragile API, I don't think we should stabilize that if better alternatives exist. After all we generally let the user name things they need to refer to (variables, functions, types, even loops to break out of) instead of using opaque numbers and counting.

Is there consensus for ${ignore($var)} and ${count($var)} at least? Those would make certain patterns much easier to handle.

ignore seems like a no-brainer to me. (Not sure this is the best possible name, but I can't come up with a better one either.)

count does seem to have very surprising interaction with repetitions. Judging from the reference PR:

macro_rules! count_value_nested {
    ( $( $name:ident: $( $value:literal ),* );  ) => {
        [ $(
            // using `count` within a repetition group will return the number
            // of times `$value` is matched _within that group_.
            ${count($value)},
        )  ]
    };
}

So, this is a repetition group that only contains $value. And somehow that becomes a once-per-$name repetition? That's not how repetition behaves anywhere else, is it? I would have expected that I must use $name somewhere in this repetition group to make this a once-per-$name group, and then I can count the $value in it -- the argument of $count would not be part of the automatic detection of the repetition group in my mental model, that has to already be clear without.

Which repetition group does this count?

macro_rules! count_test {
    ( $( $supername:ident { $( $name:ident: $( $value:ident )* ),  } )* ) => {
        [ $(
            ${count($value)},
        )  ]
    };
}

This could reasonably be for i in supername { i.total_numer_of_value() } but also for i in supername { for j in i.name { j.total_numer_of_value() } }.

Trying this out, it seems like like it is the former, counting the total number of value for each supername.

Usually a metavar "must appear in exactly the same number, kind, and nesting order of repetitions in the transcriber as it did in the matcher". With count that seems to be different. Is it something like "must appear in strictly fewer repetitions than it did in the matcher, and then it counts the number of inner repetitions in the current instance of the outer repetitions it occurs in"?

looks kinda like group.iter().flat_map(|expansion| expansion.get(metavar)).count()

As previously commented, the suggestion around named placeholders is orthogonal to the current behavior approved in the RFC and both can co-exist in the future.

Fate imposed yet another syntax-related blocking concern in a stabilization PR, as such, I ask the responsible party to decide the final outcome.

@rustbot labels I-lang-nominated

@rfcbot concern depth levels are confusing

Before reading this thread I had a strong reaction to the RFC exactly in line with @RalfJung's, and I had thoughts along the lines of what @tgross35 is proposing. So I think there's something worth paying attention to here. I see it as a major downside, quite possibly a failure, of the lang team process that we got this far without such syntax concerns being surfaced. With that said, even though the concern is a late breaking one, it feels significant enough to me that I don't think it should be minimized.

Rust's declarative macro syntax can already be quite intimidating to the uninitiated, and we should be careful in how we spend our "readability budget" here. This emphasis on readability is the reason I'm not that swayed by the argument of future compatibility: Even if we add named expansions later, the syntax for unnamed depth levels will always exist if we stabilize it now, and people who read Rust macro code will have to know about it.

To make my concern more concrete: I think adding recursion depths is a significant step up in both capability and complexity, but I would want that to be equally matched with stronger usability as well as motivation for the feature. The usability piece is quite lacking, in my opinion, in that the recursion depths are unlabeled integers without an obvious meaning. I certainly think they are non-obvious to the reader, and in some cases even to the writer who already knows about this feature (I speculate that many users might forget how count works with nested metavars and depths, even after using it a few times, because it doesn't seem to work like anything else in the language). I also don't see a motivation for these specific capabilities laid out in the RFC or the stabilization report.

In the interest of making forward progress, I'll put forward a proposal that I frame as a question. The implied question in each of these is how much the proposal limits the use cases enabled by these features. If the answer is "not that much" I think we should go forward with stabilizing this subset for now; otherwise, I hope it's a useful starting point for getting things moving.

• Can the stabilization proposal be subsetted not to support depth levels on any of these operators?
• Can count specifically be limited to supporting only metavars with a single recursion level?
  • If that's too limiting: What is the motivation for counting all nested expansions instead of only counting the outermost level (the one that would be expanded in the next use of $()*)? If hypothetically we switched it to instead only count the outermost level (without supporting explicit depths but holding open the possibility to support them in the future), would that enable significantly more use cases? edit: Nevermind, see Stabilize count, ignore, index, and length (macro_metavar_expr) #122808 (comment)

If that's too limiting: What is the motivation for counting all nested expansions instead of only counting the outermost level (the one that would be expanded in the next use of $()*)?

I don't exactly know what you mean here, could you give an example?

FWIW, I experimented with count a bit and here are the results. Without named repetitions, I don't think there is another interpretation that makes sense. Possibly there is a subset that makes sense, like only supporting count($x) when x is exactly one level inside the current nesting -- but it's not clear that that actually makes it more understandable.

The only point that gives me pause around count is -- if/when we have named repetition groups, how will they work with count, and is that compatible with today's behavior?

I don't exactly know what you mean here, could you give an example?

What I meant is, why not make count($x) equivalent to count($x, 1) instead of count($x, ∞)? We would then have to invent syntax for the latter if we wanted to preserve expressiveness, but this default aligns more with my expectations.

I'm not sure what count(x, 0) would mean, based on this from the RFC:

If repetitions are nested, then an optional depth parameter can be used to limit the number of nested repetitions that are counted. For example, a macro expansion like:

${count(x, 1)} ${count(x, 2)} ${count(x, 3)} $( a $( b $( $x )* )* )*

The three values this expands to are the number of outer-most repetitions (the number of times a would be generated), the sum of the number of middle repetitions (the number of times b would be generated), and the total number of repetitions of $x.

Reading this over again though, I no longer think it makes sense to limit to 1 by default. ${count(x)} should of course count the number of expansions for $x, not the number of expansions for the next-expandable metavariable that $x happens to be nested in. We could be conservative in only allowing it in the same contexts where $x can already be expanded, but I can see it being useful outside of those contexts.

I still think specifying a max depth level is non-obvious and of dubious value, since you could always rewrite the excerpt above as something like the following; are there cases where you can't do this?

 ${count(a)} ${count(b)} ${count(x)} $( a $( b $( $x )* )* )*

I'm not sure what count(x, 0) would mean, based on this from the RFC:

I have linked this already above, see here. That's what I reverse engineered based on my experiments, anyway. The default is definitely 0, according to that PR.

Maybe the semantics changed since the RFC was accepted?

Would be good to get an up=to-date description of the proposed semantics confirmed by the people involved in the implementation -- Cc @c410-f3r , not sure who else?

I still think specifying a max depth level is non-obvious and of dubious value, since you could always rewrite the excerpt above as something like the following; are there cases where you can't do this?

I think the issue is around matchers like $( [[ $( {{ $( $x, )* }} )* ]] )* where there is no variable in the outer repetition groups that one can refer to.

I will try to address all questions this weekend in my free time.

cc rust-lang/reference#1485 (comment)

I see it as a major downside, quite possibly a failure, of the lang team process that we got this far without such syntax concerns being surfaced.

Thank you. macro_metavar_expr has been around for 3 years and its implementation has been available for 2 years. Although part of the game, it is surprising and demotivating to see syntax concerns in stabilizing PRs (#95860).

The only point that gives me pause around count is -- if/when we have named repetition groups, how will they work with count, and is that compatible with today's behavior?

It is more complex but not impossible IMO. One way is to attach a corresponding matcher depth to each named repetition and then evaluate which ones can be associated based on the current transcriber position.

Syntax-wise, the distinction between numbers and strings can create appropriated implementation branches.

What I meant is, why not make count($x) equivalent to count($x, 1) instead of count($x, ∞)? We would then have to invent syntax for the latter if we wanted to preserve expressiveness, but this default aligns more with my expectations.

AFAIK it is currently equivalent to count($x, 0),

Before #117050 count($x) was equal to count($x, N) with N being the number of nested repetitions minus one. Now count($x) is equal to count($x, 0) which can also be seen as the innermost repetition.

rust-lang/reference#1485 (comment) will probably provide a better overview.

I still think specifying a max depth level is non-obvious and of dubious value, since you could always rewrite the excerpt above as something like the following; are there cases where you can't do this?

${count(a)} ${count(b)} ${count(x)} $( a $( b $( $x )* )* )*

Unfortunately it is not possible to count without a metavariable. AFAICT, a and b are just references used in the RFC to illustrative how the algorithm works.

But yeah, the specification of a depth is non-obvious as it depends on the number of nested matcher repetitions as well as the positioning.

the people involved in the implementation

@markbt, @petrochenkov and @mark-i-m

I see, I was confused by reading the RFC for more detail, but that is now outdated. @c410-f3r Can you please include a list of differences from the RFC in the stabilization report (with rationale; feel free to summarize and link to more context)?

I see, I was confused by reading the RFC for more detail, but that is now outdated. @c410-f3r Can you please include a list of differences from the RFC in the stabilization report (with rationale; feel free to summarize and link to more context)?

Sure. The report has been updated.