Rewrite RFC in favour of mainCRTStartup alternative

Author: Diggsey

text/0000-windows-subsystem.md

@@ -22,7 +22,10 @@ The `WINDOWS` subsystem is commonly used on windows: desktop applications
 typically do not want to flash up a console window on startup.
 
 Currently, using the `WINDOWS` subsystem from rust is undocumented, and the
-process is non-trivial when targeting the MSVC toolchain:
+process is non-trivial when targeting the MSVC toolchain. There are a couple of
+approaches, each with their own downsides:
+
+## Define a WinMain symbol
 
 A new symbol `pub extern "system" WinMain(...)` with specific argument
 and return types must be declared, which will become the new entry point for
@@ -32,6 +35,13 @@ This is unsafe, and will skip the initialization code in `libstd`.
 
 The GNU toolchain will accept either entry point.
 
+## Override the entry point via linker options
+
+This uses the same method as will be described in this RFC. However, it will
+result in build scripts also being compiled for the windows subsystem, which
+can cause additional console windows to pop up during compilation, making the
+system unusable while a build is in progress.
+
 # Detailed design
 [design]: #detailed-design
 
@@ -44,52 +54,65 @@ In practice, only two subsystems are very commonly used: `CONSOLE` and
 `WINDOWS`, and from a user's perspective, they determine whether a console will
 be automatically created when the program is started.
 
-The solution this RFC proposes is to always export both `main` and `WinMain`
-symbols from rust executables compiled for windows. The `WinMain` function
-will simply delegate to the `main` function.
+## New crate attribute
+
+This RFC proposes two changes to solve this problem. The first is adding a
+top-level crate attribute to allow specifying which subsystem to use:
+
+`#![windows_subsystem = "windows"]`
+
+Initially, the set of possible values will be `{windows, console}`, but may be
+extended in future if desired.
 
-The exact signature is:
+The use of this attribute in a non-executable crate will result in a compiler
+warning. If compiling for a non-windows target, the attribute will be silently
+ignored.
+
+## Additional linker argument
+
+For the GNU toolchain, this will be sufficient. However, for the MSVC toolchain,
+the linker will be expecting a `WinMain` symbol, which will not exist.
+
+There is some complexity to the way in which a different entry point is expected
+when using the windows subsystem. Firstly, the C-runtime library exports two
+symbols designed to be used as an entry point:
 ```
-pub extern "system" WinMain(
-    hInstance: HINSTANCE,
-    hPrevInstance: HINSTANCE,
-    lpCmdLine: LPSTR,
-    nCmdShow: i32
-) -> i32;
+mainCRTStartup
+WinMainCRTStartup
 ```
 
-Where `HINSTANCE` is a pointer-sized opaque handle, and `LPSTR` is a C-style
-null terminated string.
+`LINK.exe` will use the subsystem to determine which of these symbols to use
+as the default entry point if not overridden.
 
-All four parameters are either irrelevant or can be obtained easily through
-other means:
-- `hInstance` - Can be obtained via `GetModuleHandle`.
-- `hPrevInstance` - Is always NULL.
-- `lpCmdLine` - `libstd` already provides a function to get command line
-  arguments.
-- `nCmdShow` - Can be obtained via `GetStartupInfo`, although it's not actually
-  needed any more (the OS will automatically hide/show the first window created).
+Each one performs some unspecified initialization of the CRT, before calling out
+to a symbol defined within the program (`main` or `WinMain` respectively).
 
-The end result is that rust programs will "just work" when the subsystem is
-overridden via custom linker arguments, and does not require `rustc` to
-parse those linker arguments.
+The second part of the solution is to pass an additional linker option when
+targeting the MSVC toolchain:
+`/ENTRY:mainCRTStartup`
 
-A possible future extension would be to add additional command-line options to
-`rustc` (and in turn, `Cargo.toml`) to specify the subsystem directly. `rustc`
-would automatically translate this into the correct linker arguments for
-whichever linker is actually being used.
+This will override the entry point to always be `mainCRTStartup`. For
+console-subsystem programs this will have no effect, since it was already the
+default, but for windows-subsystem programs, it will eliminate the need for
+a `WinMain` symbol to be defined.
+
+This command line option will always be passed to the linker, regardless of the
+presence or absence of the `windows_subsystem` crate attribute, except when
+the user specifies their own entry point in the linker arguments. This will
+require `rustc` to perform some basic parsing of the linker options.
 
 # Drawbacks
 [drawbacks]: #drawbacks
 
-- Additional platform-specific code.
+- A new platform-specific crate attribute.
 - The difficulty of manually calling the rust initialization code is potentially
   a more general problem, and this only solves a specific (if common) case.
-- This is a breaking change for any crates which already export a `WinMain`
-  symbol. It is likely that only executable crates would export this symbol,
-  so the knock-on effect on crate dependencies should be non-existent.
-
-  A possible work-around for this is described below.
+- The subsystem must be specified earlier than is strictly required: when
+  compiling C/C++ code only the linker, not the compiler, needs to actually be
+  aware of the subsystem.
+- It is assumed that the initialization performed by the two CRT entry points
+  is identical. This seems to currently be the case, and is unlikely to change
+  as this technique appears to be used fairly widely.
 
 # Alternatives
 [alternatives]: #alternatives
@@ -132,42 +155,6 @@ whichever linker is actually being used.
   support cross-compiling. If not compiling a binary crate, specifying the
   option is an error regardless of the target.
 
-- Have `rustc` override the entry point when calling `link.exe`, and tell it to
-  use `mainCRTStartup` instead of `winMainCRTStartup`. These are the "true"
-  entry points of windows programs, which first initialize the C runtime
-  library, and then call `main` or `WinMain` respectively.
-
-  This is the simplest solution, and it will not have any serious backwards
-  compatibility problems, since rust programs are already required to have a
-  `main` function, even if `WinMain` has been separately defined. However, it
-  relies on the two CRT functions to be interchangeable, although this does
-  *appear* to be the case currently.
-
-- Export both entry points as described in this RFC, but also add a `subsystem`
-  function to `libstd` determine which subsystem was used at runtime.
-
-  The `WinMain` function would first set an internal flag, and only then
-  delegate to the `main` function.
-
-  A function would be added to `std::os::windows`:
-
-  `fn subsystem() -> &'static str`
-
-  This would check the value of the internal flag, and return either `WINDOWS` or
-  `CONSOLE` depending on which entry point was actually used.
-
-  The `subsystem` function could be used to eg. redirect logging to a file if
-  the program is being run on the `WINDOWS` subsystem. However, it would return
-  an incorrect value if the initialization was skipped, such as if used as a
-  library from an executable written in another language.
-
-- Export both entry points as described in this RFC, but use the undocumented
-  MSVC equivalent to weak symbols to avoid breaking existing code.
-
-  The parameter `/alternatename:_WinMain@16=_RustWinMain@16` can be used to
-  export `WinMain` only if it is not also exported elsewhere. This is completely
-  undocumented, but is mentioned here: (http://stackoverflow.com/a/11529277).
-
 # Unresolved questions
 [unresolved]: #unresolved-questions