Merge pull request #38872 from nikomatsakis/beta

Beta backports

Author: alexcrichton

src/bootstrap/dist.rs

@@ -391,6 +391,13 @@ pub fn rust_src(build: &Build) {
             }
         }
 
+        // If we're inside the vendor directory then we need to preserve
+        // everything as Cargo's vendoring support tracks all checksums and we
+        // want to be sure we don't accidentally leave out a file.
+        if spath.contains("vendor") {
+            return true
+        }
+
         let excludes = [
             "CVS", "RCS", "SCCS", ".git", ".gitignore", ".gitmodules",
             ".gitattributes", ".cvsignore", ".svn", ".arch-ids", "{arch}",

src/doc/book/SUMMARY.md

@@ -52,6 +52,7 @@
     * [Borrow and AsRef](borrow-and-asref.md)
     * [Release Channels](release-channels.md)
     * [Using Rust without the standard library](using-rust-without-the-standard-library.md)
+    * [Procedural Macros (and custom derive)](procedural-macros.md)
 * [Nightly Rust](nightly-rust.md)
     * [Compiler Plugins](compiler-plugins.md)
     * [Inline Assembly](inline-assembly.md)

src/doc/book/procedural-macros.md

@@ -0,0 +1,213 @@
+% Procedural Macros (and custom Derive)
+
+As you've seen throughout the rest of the book, Rust provides a mechanism
+called "derive" that lets you implement traits easily. For example,
+
+```rust
+#[derive(Debug)]
+struct Point {
+    x: i32,
+    y: i32,
+}
+```
+
+is a lot simpler than
+
+```rust
+struct Point {
+    x: i32,
+    y: i32,
+}
+
+use std::fmt;
+
+impl fmt::Debug for Point {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "Point {{ x: {}, y: {} }}", self.x, self.y)
+    }
+}
+```
+
+Rust includes several traits that you can derive, but it also lets you define
+your own. We can accomplish this task through a feature of Rust called
+"procedural macros." Eventually, procedural macros will allow for all sorts of
+advanced metaprogramming in Rust, but today, they're only for custom derive.
+
+Let's build a very simple trait, and derive it with custom derive.
+
+## Hello World
+
+So the first thing we need to do is start a new crate for our project.
+
+```bash
+$ cargo new --bin hello-world
+```
+
+All we want is to be able to call `hello_world()` on a derived type. Something
+like this:
+
+```rust,ignore
+#[derive(HelloWorld)]
+struct Pancakes;
+
+fn main() {
+    Pancakes::hello_world();
+}
+```
+
+With some kind of nice output, like `Hello, World! My name is Pancakes.`. 
+
+Let's go ahead and write up what we think our macro will look like from a user
+perspective. In `src/main.rs` we write:
+
+```rust,ignore
+#[macro_use]
+extern crate hello_world_derive;
+
+trait HelloWorld {
+    fn hello_world();
+}
+
+#[derive(HelloWorld)]
+struct FrenchToast;
+
+#[derive(HelloWorld)]
+struct Waffles;
+
+fn main() {
+    FrenchToast::hello_world();
+    Waffles::hello_world();
+}
+```
+
+Great. So now we just need to actually write the procedural macro. At the
+moment, procedural macros need to be in their own crate. Eventually, this
+restriction may be lifted, but for now, it's required. As such, there's a
+convention; for a crate named `foo`, a custom derive procedural macro is called
+`foo-derive`. Let's start a new crate called `hello-world-derive` inside our
+`hello-world` project.
+
+```bash
+$ cargo new hello-world-derive
+```
+
+To make sure that our `hello-world` crate is able to find this new crate we've
+created, we'll add it to our toml:
+
+```toml
+[dependencies]
+hello-world-derive = { path = "hello-world-derive" }
+```
+
+As for our the source of our `hello-world-derive` crate, here's an example:
+
+```rust,ignore
+extern crate proc_macro;
+extern crate syn;
+#[macro_use]
+extern crate quote;
+
+use proc_macro::TokenStream;
+
+#[proc_macro_derive(HelloWorld)]
+pub fn hello_world(input: TokenStream) -> TokenStream {
+    // Construct a string representation of the type definition
+    let s = input.to_string();
+    
+    // Parse the string representation
+    let ast = syn::parse_macro_input(&s).unwrap();
+
+    // Build the impl
+    let gen = impl_hello_world(&ast);
+    
+    // Return the generated impl
+    gen.parse().unwrap()
+}
+```
+
+So there is a lot going on here. We have introduced two new crates: [`syn`] and
+[`quote`]. As you may have noticed, `input: TokenSteam` is immediately converted
+to a `String`. This `String` is a string representation of the Rust code for which
+we are deriving `HelloWorld` for. At the moment, the only thing you can do with a
+`TokenStream` is convert it to a string. A richer API will exist in the future.
+
+So what we really need is to be able to _parse_ Rust code into something
+usable. This is where `syn` comes to play. `syn` is a crate for parsing Rust
+code. The other crate we've introduced is `quote`. It's essentially the dual of
+`syn` as it will make generating Rust code really easy. We could write this
+stuff on our own, but it's much simpler to use these libraries. Writing a full
+parser for Rust code is no simple task.
+
+[`syn`]: https://crates.io/crates/syn
+[`quote`]: https://crates.io/crates/quote
+
+The comments seem to give us a pretty good idea of our overall strategy. We
+are going to take a `String` of the Rust code for the type we are deriving, parse
+it using `syn`, construct the implementation of `hello_world` (using `quote`),
+then pass it back to Rust compiler.
+
+One last note: you'll see some `unwrap()`s there. If you want to provide an
+error for a procedural macro, then you should `panic!` with the error message.
+In this case, we're keeping it as simple as possible.
+
+Great, so let's write `impl_hello_world(&ast)`.
+
+```rust,ignore
+fn impl_hello_world(ast: &syn::MacroInput) -> quote::Tokens {
+    let name = &ast.ident;
+    quote! {
+        impl HelloWorld for #name {
+            fn hello_world() {
+                println!("Hello, World! My name is {}", stringify!(#name));
+            }
+        }
+    }
+}
+```
+
+So this is where quotes comes in. The `ast` argument is a struct that gives us
+a representation of our type (which can be either a `struct` or an `enum`).
+Check out the [docs](https://docs.rs/syn/0.10.5/syn/struct.MacroInput.html),
+there is some useful information there. We are able to get the name of the
+type using `ast.ident`. The `quote!` macro let's us write up the Rust code
+that we wish to return and convert it into `Tokens`. `quote!` let's us use some
+really cool templating mechanics; we simply write `#name` and `quote!` will
+replace it with the variable named `name`. You can even do some repetition
+similar to regular macros work. You should check out the
+[docs](https://docs.rs/quote) for a good introduction.
+
+So I think that's it. Oh, well, we do need to add dependencies for `syn` and
+`quote` in the `cargo.toml` for `hello-world-derive`.
+
+```toml
+[dependencies]
+syn = "0.10.5"
+quote = "0.3.10"
+```
+
+That should be it. Let's try to compile `hello-world`.
+
+```bash
+error: the `#[proc_macro_derive]` attribute is only usable with crates of the `proc-macro` crate type
+ --> hello-world-derive/src/lib.rs:8:3
+  |
+8 | #[proc_macro_derive(HelloWorld)]
+  |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+```
+
+Oh, so it appears that we need to declare that our `hello-world-derive` crate is
+a `proc-macro` crate type. How do we do this? Like this:
+
+```toml
+[lib]
+proc-macro = true
+```
+
+Ok so now, let's compile `hello-world`. Executing `cargo run` now yields:
+
+```bash
+Hello, World! My name is FrenchToast
+Hello, World! My name is Waffles
+```
+
+We've done it!

src/doc/reference.md

@@ -555,26 +555,24 @@ mod a {
 # fn main() {}
 ```
 
-# Syntax extensions
+# Macros
 
 A number of minor features of Rust are not central enough to have their own
 syntax, and yet are not implementable as functions. Instead, they are given
 names, and invoked through a consistent syntax: `some_extension!(...)`.
 
-Users of `rustc` can define new syntax extensions in two ways:
-
-* [Compiler plugins][plugin] can include arbitrary Rust code that
-  manipulates syntax trees at compile time. Note that the interface
-  for compiler plugins is considered highly unstable.
+Users of `rustc` can define new macros in two ways:
 
 * [Macros](book/macros.html) define new syntax in a higher-level,
   declarative way.
+* [Procedural Macros][procedural macros] can be used to implement custom derive.
+
+And one unstable way: [compiler plugins][plugin].
 
 ## Macros
 
 `macro_rules` allows users to define syntax extension in a declarative way.  We
-call such extensions "macros by example" or simply "macros" — to be distinguished
-from the "procedural macros" defined in [compiler plugins][plugin].
+call such extensions "macros by example" or simply "macros".
 
 Currently, macros can expand to expressions, statements, items, or patterns.
 
@@ -652,6 +650,28 @@ Rust syntax is restricted in two ways:
 
 [RFC 550]: https://github.com/rust-lang/rfcs/blob/master/text/0550-macro-future-proofing.md
 
+## Procedrual Macros
+
+"Procedrual macros" are the second way to implement a macro. For now, the only
+thing they can be used for is to implement derive on your own types. See
+[the book][procedural macros] for a tutorial.
+
+Procedural macros involve a few different parts of the language and its
+standard libraries. First is the `proc_macro` crate, included with Rust,
+that defines an interface for building a procedrual macro. The 
+`#[proc_macro_derive(Foo)]` attribute is used to mark the the deriving
+function. This function must have the type signature:
+
+```rust,ignore
+use proc_macro::TokenStream;
+
+#[proc_macro_derive(Hello)]
+pub fn hello_world(input: TokenStream) -> TokenStream
+```
+
+Finally, procedural macros must be in their own crate, with the `proc-macro`
+crate type.
+
 # Crates and source files
 
 Although Rust, like any other language, can be implemented by an interpreter as
@@ -2309,6 +2329,9 @@ impl<T: PartialEq> PartialEq for Foo<T> {
 }
 ```
 
+You can implement `derive` for your own type through [procedural
+macros](#procedural-macros).
+
 ### Compiler Features
 
 Certain aspects of Rust may be implemented in the compiler, but they're not
@@ -4112,6 +4135,16 @@ be ignored in favor of only building the artifacts specified by command line.
   in dynamic libraries. This form of output is used to produce statically linked
   executables as well as `staticlib` outputs.
 
+* `--crate-type=proc-macro`, `#[crate_type = "proc-macro"]` - The output
+  produced is not specified, but if a `-L` path is provided to it then the
+  compiler will recognize the output artifacts as a macro and it can be loaded
+  for a program. If a crate is compiled with the `proc-macro` crate type it
+  will forbid exporting any items in the crate other than those functions
+  tagged `#[proc_macro_derive]` and those functions must also be placed at the
+  crate root. Finally, the compiler will automatically set the
+  `cfg(proc_macro)` annotation whenever any crate type of a compilation is the
+  `proc-macro` crate type.
+
 Note that these outputs are stackable in the sense that if multiple are
 specified, then the compiler will produce each form of output at once without
 having to recompile. However, this only applies for outputs specified by the
@@ -4289,3 +4322,4 @@ that have since been removed):
 
 [ffi]: book/ffi.html
 [plugin]: book/compiler-plugins.html
+[procedural macros]: book/procedural-macros.html

src/libproc_macro/lib.rs

@@ -15,19 +15,16 @@
 //! Currently the primary use of this crate is to provide the ability to define
 //! new custom derive modes through `#[proc_macro_derive]`.
 //!
-//! Added recently as part of [RFC 1681] this crate is currently *unstable* and
-//! requires the `#![feature(proc_macro_lib)]` directive to use.
-//!
-//! [RFC 1681]: https://github.com/rust-lang/rfcs/blob/master/text/1681-macros-1.1.md
-//!
 //! Note that this crate is intentionally very bare-bones currently. The main
 //! type, `TokenStream`, only supports `fmt::Display` and `FromStr`
 //! implementations, indicating that it can only go to and come from a string.
 //! This functionality is intended to be expanded over time as more surface
 //! area for macro authors is stabilized.
+//!
+//! See [the book](../../book/procedural-macros.html) for more.
 
 #![crate_name = "proc_macro"]
-#![unstable(feature = "proc_macro_lib", issue = "27812")]
+#![stable(feature = "proc_macro_lib", since = "1.15.0")]
 #![crate_type = "rlib"]
 #![crate_type = "dylib"]
 #![cfg_attr(not(stage0), deny(warnings))]
@@ -55,12 +52,14 @@ use syntax::ptr::P;
 ///
 /// The API of this type is intentionally bare-bones, but it'll be expanded over
 /// time!
+#[stable(feature = "proc_macro_lib", since = "1.15.0")]
 pub struct TokenStream {
     inner: Vec<P<ast::Item>>,
 }
 
 /// Error returned from `TokenStream::from_str`.
 #[derive(Debug)]
+#[stable(feature = "proc_macro_lib", since = "1.15.0")]
 pub struct LexError {
     _inner: (),
 }
@@ -131,6 +130,7 @@ pub mod __internal {
     }
 }
 
+#[stable(feature = "proc_macro_lib", since = "1.15.0")]
 impl FromStr for TokenStream {
     type Err = LexError;
 
@@ -154,6 +154,7 @@ impl FromStr for TokenStream {
     }
 }
 
+#[stable(feature = "proc_macro_lib", since = "1.15.0")]
 impl fmt::Display for TokenStream {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         for item in self.inner.iter() {