Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Macro naming and modularisation #1561

Merged
merged 3 commits into from
Aug 22, 2016
Merged

Conversation

nrc
Copy link
Member

@nrc nrc commented Mar 29, 2016

This RFC proposes making macros a first-class citizen in the Rust module system.
Both macros by example (macro_rules macros) and procedural macros (aka syntax
extensions) would use the same naming and modularisation scheme as other items
in Rust.

rendered

This RFC proposes making macros a first-class citizen in the Rust module system.
Both macros by example (`macro_rules` macros) and procedural macros (aka syntax
extensions) would use the same naming and modularisation scheme as other items
in Rust.
@nrc nrc self-assigned this Mar 29, 2016
@nrc nrc added the T-lang Relevant to the language team, which will review and decide on the RFC. label Mar 29, 2016
@brendanzab
Copy link
Member

This would be very useful for library developers, especially if we could reexport them as well! For example I would love this for my approx crate. Not sure about any subtle edge cases in this proposal though.

very different schemes for naming macros depending on whether a macro is defined
by example or procedurally. That would be inconsistent and annoying. However, I
hope we can make the new macro system appealing enough and close enough to the
existing system that migration is both desirable and easy.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could simply lint on #[macro_use] and #[macro_export]. This would never show up in dependencies, but only in the crate that's currently compiled

@ahicks92
Copy link

ahicks92 commented Apr 2, 2016

A question: does this mean I can put a macro in an impl block?
At the moment, we have MyType::new, the equivalent of a constructor. For data structures that wish to be initialized with lists of items, it might also be helpful to do MyType::new!, i.e. Vec::new![1, 2, 3, 4, 5] as opposed to the gloval vec! macro.
If not, it should be noted that this would solve the most common case for which I could see a C -style varargs or initializer list being useful while avoiding the pollution problem that we have currently.

@nrc
Copy link
Member Author

nrc commented Apr 2, 2016

@camlorn No. Name resolution happens before type information is collected, so that wouldn't necessarily follow from this RFC. This RFC does not preclude doing that in the future, but I don't see a nice way to implement it in the general case.


Some day, I hope that procedural macros may be defined in the same crate in
which they are used. I leave the details of this for later, however, I don't
think this affects the design of naming - it should all Just Work.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you provide some hints on how you plan to implement this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Procedural macros and any items marked with #[cfg(macro)] would be split off and compiled into one crate, everything else compiled into another. The first crate would be linked in to the compiler while it compiles the second (or equivalently, called using IPC).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So macros will only be able to call functions marked with #[cfg(macro)]?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems strange, #[cfg]-gating can only remove items with #[cfg(P)] where the cfg predicate P doesn't hold.

@solson
Copy link
Member

solson commented Apr 10, 2016

@nrc The Rendered link in the original post links to the original commit (https://github.com/nrc/rfcs/blob/573bd83aaac0896842f136262458c3b3bf671cd7/text/0000-macro-naming.md) rather than the most up-to-date version (https://github.com/nrc/rfcs/blob/macro-naming/text/0000-macro-naming.md).

@withoutboats
Copy link
Contributor

In the original blog series, it was suggested that macros imported using this system would be imported using a new symbol. I don't see that in this RFC, so I take it that this will change how macro_rules! macros can be imported. Are there any backwards compatibility issues connected with this change?

@nrc
Copy link
Member Author

nrc commented Apr 20, 2016

In the original blog series, it was suggested that macros imported using this system would be imported using a new symbol. I don't see that in this RFC

This is #1584.


macro! foo { ... }
```

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a good idea, because it won't play nice with macro-defining macros. For example, the following sort of thing should (and currently does) work:

  macro_rules! macro_defining { 
    ($name:ident) => { macro_rules! $name { () => (println!("Yay!");) } }
  }

  macro_defining!( say_yay );

  say_yay!();

...but it only works if the macros are expanded in the order they are written, and there's no reasonable static analysis that can tell us that fact.

@paulstansifer
Copy link

Regarding the issue of ! in use statements: since macro! isn't nailed down yet, we can achieve total consistency:

macro! {
  my_macro!(...) => {...}
}

and

use some_mod::my_macro!;

I think that this would be perfect if it weren't for attribute-style macros. Even so, I think that I prefer to use ! as if it were really part of the name for non-attribute-style macros:

use some_mod::{my_normal_macro!, my_attribute_macro, my_ordinary_function};

I feel like this is what I would try if I didn't know what the actual rule was.

@solson
Copy link
Member

solson commented May 25, 2016

@paulstansifer There has been a lot of discussion on that topic, but I guess I'll point to #1561 (comment) in particular.

It doesn't seem worthwhile to use ! here, and it's more consistent to keep use syntax uniformly unadorned.

@eddyb
Copy link
Member

eddyb commented Jul 17, 2016

I was under the impression definitions and uses would be orthogonal even if we had two systems for each, but @nrc pointed out on IRC that is not the case as currently proposed.

To which I make the counter-suggestion: allow the usage of #[macro_use] or use, regardless of how the macro was defined or even exported, at least across crates (as long as the macro is visible at the top-level of the crate from which the macro is being imported, in thr case of #[macro_use]).
At the same time, this provides 2 benefits:

  • allow crate authors to update without breaking their old users
  • allow crate users to upgrade without changing old dependencies

@nikomatsakis
Copy link
Contributor

nikomatsakis commented Aug 8, 2016

Hear ye, hear ye! This RFC is now entering final comment period. In the most recent @rust-lang/lang meeting, @nrc and I (the only two in attendance) felt inclined to accept. In particular, we're definitely in favor of the high-level aims of this RFC (controlling macros via ordinary use statements).

The major points of interest in the thread so far have been:

  • whether to place macros in their own namespace (the RFC does)
  • whether to import macros using a trailing ! (use foo::bar vs use foo::bar!). We two felt that use foo::bar was better, since the ! is more properly considered a "use macro operator" and not part of the macro name. As evidence for this position, consider that one can write higher-order macros that operate like macro_rules! foo { ($x:ident, $y:tt) => { $x!($y) } } (invoked like foo!(bar, baz), which is roughly equivalent to bar!(baz)). See this example on play. There is some good discussion here.
  • @paulstansifer's contention that not rely on macro-ordering will break macro-defining-macros; this is worth discussing more (I believe that not to be true, but there are some differences that emerge versus systems like Racket, which have a more mutable view of the universe).

(There may be some implementation details to be hashed out.)

@nikomatsakis
Copy link
Contributor

nikomatsakis commented Aug 8, 2016

@paulstansifer

I don't think this is a good idea, because it won't play nice with macro-defining macros

This is definitely an important point, thanks for raising it. In fact, I think the system as we've envisioned it plays great with macro-defining macros, but there are some patterns in Racket that will not work. Let's start with your example:

macro_rules! macro_defining { 
    ($name:ident) => { macro_rules! $name { () => (println!("Yay!");) } }
}
macro_defining!( say_yay );
say_yay!();

This interacts with the overall name resolution algorithm defined in #1560. The idea is roughly that expansion and name resolution is a process that occurs in rounds. In this case, in the first round, the path say_yay! will be "unresolved" -- because there is no definition for it. However, macro_defining! will resolve and expand. In the next round, then, say_yay will resolve. Errors are only reported if we have expanded all macros and still have paths that cannot be resolved at the end of the process.

One danger here is that expanding a macro may cause a path that resolved in the past to resolve differently. Mostly this isn't possible, because expanding a macro can only add items, and duplicate items cause errors. However, one case where it can cause trouble is around globs and shadowing -- if a path relies on item X that is later shadowed by another item X produced by a macro, then it resolves differently. To counter this, we make it an error (in #1560) for a macro to expand to an item X that shadows an item in an outer scope, at least if that item was used in a macro expansion path. (See @jseyfried's comment for more details.)

Personally I find the idea of item ordering being significant in Rust to be very unappealing. It is strikingly different from how most things in Rust work, and it particularly doesn't mix well with use statements in my mind. (Perhaps the ordering could be some sort of DAG derived from the use statements, but my efforts to find such a system that works out well mostly failed.)

But this does make it more difficult to have "comunicating" macros, which I understand in Racket at least often rely on expansion order. An example that I was given by @samth where this could break Racket patterns of macro interop was something about pattern matching. I think the idea was that a class! macro might generate some metadata that is later used by a match! macro (in Racket, I mean), and that this introduces an ordering dependency. However, he also added that in practice macros like match! would generally try to "reschedule" themselves to run late in the expansion process so that the relative ordering of a class declaration and a match wouldn't be important. (I don't remember more details than that.)

I think a potential example of who this could come into play in Rust might be that, e.g., a macro might want to access data from a type definition. (Perhaps one would be expected e.g. to annotate the type definition in one way, and this would enable uses elsewhere that "communicate" with that annotation to extract data about the types of the fields, or something.) This is not well supported in the current system, which is targeted at more "self-contained" macros.

I'm not sure what's a "general" fix here but I think a lot will depend on the details. For example, if the "uses" that wish to communicate with the definition are inside of fns or other code blocks, I expect we can do things to ensure that all module-levels macros and items are expanded first (given that names defined in fns can't be used from outside that fn).

I can also imagine patterns where a module is decorated (#[foo] mod { ... }) -- this would then allow code in the module to reference data generated by the #[foo] macro. This works just by default because the code that will reference what #[foo] generates is ultimately governed by foo and results from the expansion of foo itself.

Finally, we might develop somewhat more ad-hoc solutions, similar to how Racket macros might "reschedule" themselves to make themselves order independent, where a macro can ask to be deferred from executing if it finds that other macros it might depend on are not yet expanded.

In short, I'd rather tackle this in a declarative way than relying on the "AST ordering", which I don't think is a natural concept in Rust.

(As one last point, I'd sort of like to get rid of explicit mod declarations, at least for private modules, and instead infer their presence from the file-system -- but if we did so, it would interact very poorly with having a defined AST ordering.)

@ahicks92
Copy link

ahicks92 commented Aug 9, 2016

Can we get a macro_namespaced attribute added to current macros which would allow use of this feature now?

As I read it, this is applying to a hypothetical future in which we rewrite macros by example to use a new macro! keyword. But as far as I know, there isn't much of a timeline.

Specifically:

  • A macro with macro_namespaced requires explicit importing and reference per this RFC to be used.
  • All macros that choose not to use it can be used as they are now.
  • All macros that choose not to use it are treated as being in the root of their crate and may be used in the new style as well as the old.

I would like to see this feature before the macro redesign is finished, and this seems like a good compromise that allows for code transition. The disadvantage is that I believe the old macro_rules stuff is on its way out, and obviously this adds something else to it.

The name is off the top of my head. I'm not attached to it.

@withoutboats
Copy link
Contributor

withoutboats commented Aug 10, 2016

Maybe I missed it, but the RFC doesn't seem say anything about redefining / shadowing rules for macros in regard to other macros. Currently, the macro rules for this work very differently from the rules for functions. Currently, you can redefine macros whenever you want, and whichever definition is most recent in the source at the expansion site is the definition that is used.

If macros use the same import system as other symbols, I would expect their shadowing behavior to be the same. That is, you can't provide two definitions in the same module or import and define, but you can shadow prelude macros and you can shadow macros by providing a definition inside the body of a function.

(As one last point, I'd sort of like to get rid of explicit mod declarations, at least for private modules, and instead infer their presence from the file-system -- but if we did so, it would interact very poorly with having a defined AST ordering.)

(This is totally offtopic but this seems like a very good idea in the long term which would make Rust's module story more grokkable to users coming from other languages which work this way)

Also this PR needs the final-comment-period label. :-)

@nikomatsakis nikomatsakis added the final-comment-period Will be merged/postponed/closed in ~10 calendar days unless new substational objections are raised. label Aug 10, 2016
@strega-nil
Copy link

Agreed with @withoutboats about shadowing.

use foo::bar is much better than use foo::bar!

placing macros into their own namespace is also a good idea; it'd be weird to have macros be in the same namespace as types and values.

@arielb1
Copy link
Contributor

arielb1 commented Aug 11, 2016

(As one last point, I'd sort of like to get rid of explicit mod declarations, at least for private modules, and instead infer their presence from the file-system -- but if we did so, it would interact very poorly with having a defined AST ordering.)

But I like explicit mod declarations. The "every crate is 1 file" model is so liberating.

@nikomatsakis
Copy link
Contributor

Can we get a macro_namespaced attribute added to current macros which would allow use of this feature now?

That's a good question. I would hope we could have some way to use this with old-school macros, whether that require some sort of opt-in, or maybe we can make it work by default without breakage.

@nikomatsakis
Copy link
Contributor

nikomatsakis commented Aug 11, 2016

If macros use the same import system as other symbols, I would expect their shadowing behavior to be the same.

I'm a bit confused by this comment -- are you saying that this is why we can't easily use these new rules for existing macros? Or are you saying that you believe the newer rules in this RFC would cause macros to shadow in different ways than other names that are brought into scope via use? (Or are you referring to the errors that can result if you generate an item via a macro?)

EDIT: Re-reading, I see that you're just saying that the shadowing rules are not explicit enough in the RFC text. I think the answer is that they would follow the same rules as any other items, because we will literally be resolved macro paths using the same lookup rules we use for any other path.

@nrc nrc removed the I-nominated label Aug 11, 2016
@nikomatsakis
Copy link
Contributor

We discussed this in the @rust-lang/lang meeting and decided to leave it open for FCP a little longer. For one thing, we wanted to incorporate various bits of the discussion in to the RFC text. Nonetheless, I think we're all feeling generally positive about this direction.

@paulstansifer
Copy link

paulstansifer commented Aug 15, 2016 via email

@nikomatsakis
Copy link
Contributor

Huzzah! The @rust-lang/lang team has decided to accept this RFC.

@nikomatsakis
Copy link
Contributor

Tracking issue: rust-lang/rust#35896

If you'd like to keep following the development of this feature, please subscribe to that issue, thanks! :)

@kennytm kennytm mentioned this pull request Mar 12, 2017
@Centril Centril added A-resolve Proposals relating to name resolution. A-modules Proposals relating to modules. A-macros Macro related proposals and issues labels Nov 23, 2018
Techcable added a commit to Techcable/rust-cpython that referenced this pull request Aug 9, 2021
NOTE: We now get warnings because of the modularization of the macro system
specified in RFC rust-lang/rfcs#1561

However, fixing these warnings is out of scope for this PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-macros Macro related proposals and issues A-modules Proposals relating to modules. A-resolve Proposals relating to name resolution. final-comment-period Will be merged/postponed/closed in ~10 calendar days unless new substational objections are raised. T-lang Relevant to the language team, which will review and decide on the RFC.
Projects
None yet
Development

Successfully merging this pull request may close these issues.