Make compiletests (run-pass, compile-fail, etc) usable for testing arbitrary rust programs

[Manishearth]

Custom lints and syntax extensions would benefit from some in built support for testing.

[shepmaster]

If I understand this correctly, we would be able to write our own tests that ensure that a certain piece of code should not compile? If so, I'm in favor!.

[brson]

I'm a little unclear on what's desired here. Is support for testing that compilation fails the primary thing?

Cargo can test standalone executables already. Would adding a 'compile-fail' mode to Cargo be sufficient, or is there really a need to distribute a standalone compiletest exe?

compiletest is super-crufty (it's some of the oldest rust code in existence!) and currently tailored very specifically to rustc's in-tree needs.

[shepmaster]

For my cases, something built into Cargo that says "this line should fail to compile with this error" would be more than sufficient. I don't want to claim that's what @Manishearth wants though 😸

[Manishearth]

Well, also the matching comments with warnings would be nice.

We could also redesign the compile tests to be more similar to the #[test] system and provide most of the functionality it used to.

[huonw]

Is this a dupe of #12335?

[gnzlbg]

I think something like:

#[test]   // or #[compile_test]
#[should_not_compile(expected = "compiler error message/error type")]
fn compile_test() {}

would be very useful, if it is integrated with cargo.

[Manishearth]

dupe of #12335