auto merge of #14638 : alexcrichton/rust/librustrt, r=brson

As part of the libstd facade efforts, this commit extracts the runtime interface
out of the standard library into a standalone crate, librustrt. This crate will
provide the following services:

* Definition of the rtio interface
* Definition of the Runtime interface
* Implementation of the Task structure
* Implementation of task-local-data
* Implementation of task failure via unwinding via libunwind
* Implementation of runtime initialization and shutdown
* Implementation of thread-local-storage for the local rust Task

Notably, this crate avoids the following services:

* Thread creation and destruction. The crate does not require the knowledge of
  an OS threading system, and as a result it seemed best to leave out the
  `rt::thread` module from librustrt. The librustrt module does depend on
  mutexes, however.
* Implementation of backtraces. There is no inherent requirement for the runtime
  to be able to generate backtraces. As will be discussed later, this
  functionality continues to live in libstd rather than librustrt.

As usual, a number of architectural changes were required to make this crate
possible. Users of "stable" functionality will not be impacted by this change,
but users of the `std::rt` module will likely note the changes. A list of
architectural changes made is:

* The stdout/stderr handles no longer live directly inside of the `Task`
  structure. This is a consequence of librustrt not knowing about `std::io`.
  These two handles are now stored inside of task-local-data.

  The handles were originally stored inside of the `Task` for perf reasons, and
  TLD is not currently as fast as it could be. For comparison, 100k prints goes
  from 59ms to 68ms (a 15% slowdown). This appeared to me to be an acceptable
  perf loss for the successful extraction of a librustrt crate.

* The `rtio` module was forced to duplicate more functionality of `std::io`. As
  the module no longer depends on `std::io`, `rtio` now defines structures such
  as socket addresses, addrinfo fiddly bits, etc. The primary change made was
  that `rtio` now defines its own `IoError` type. This type is distinct from
  `std::io::IoError` in that it does not have an enum for what error occurred,
  but rather a platform-specific error code.

  The native and green libraries will be updated in later commits for this
  change, and the bulk of this effort was put behind updating the two libraries
  for this change (with `rtio`).

* Printing a message on task failure (along with the backtrace) continues to
  live in libstd, not in librustrt. This is a consequence of the above decision
  to move the stdout/stderr handles to TLD rather than inside the `Task` itself.
  The unwinding API now supports registration of global callback functions which
  will be invoked when a task fails, allowing for libstd to register a function
  to print a message and a backtrace.

  The API for registering a callback is experimental and unsafe, as the
  ramifications of running code on unwinding is pretty hairy.

* The `std::unstable::mutex` module has moved to `std::rt::mutex`.

* The `std::unstable::sync` module has been moved to `std::rt::exclusive` and
  the type has been rewritten to not internally have an Arc and to have an RAII
  guard structure when locking. Old code should stop using `Exclusive` in favor
  of the primitives in `libsync`, but if necessary, old code should port to
  `Arc<Exclusive<T>>`.

* The local heap has been stripped down to have fewer debugging options. None of
  these were tested, and none of these have been used in a very long time.

Author: bors

mk/crates.mk

@@ -51,7 +51,7 @@
 
 TARGET_CRATES := libc std green rustuv native flate arena glob term semver \
                  uuid serialize sync getopts collections num test time rand \
-                 url log regex graphviz core rlibc alloc debug
+                 url log regex graphviz core rlibc alloc debug rustrt
 HOST_CRATES := syntax rustc rustdoc fourcc hexfloat regex_macros fmt_macros
 CRATES := $(TARGET_CRATES) $(HOST_CRATES)
 TOOLS := compiletest rustdoc rustc
@@ -60,7 +60,9 @@ DEPS_core :=
 DEPS_rlibc :=
 DEPS_alloc := core libc native:jemalloc
 DEPS_debug := std
-DEPS_std := core rand libc alloc collections native:rustrt native:backtrace
+DEPS_rustrt := alloc core libc collections native:rustrt_native
+DEPS_std := core libc rand alloc collections rustrt \
+	native:rust_builtin native:backtrace
 DEPS_graphviz := std
 DEPS_green := std native:context_switch
 DEPS_rustuv := std native:uv native:uv_support

mk/rt.mk

@@ -35,8 +35,8 @@
 # that's per-target so you're allowed to conditionally add files based on the
 # target.
 ################################################################################
-NATIVE_LIBS := rustrt hoedown uv_support morestack miniz context_switch \
-		rust_test_helpers
+NATIVE_LIBS := rust_builtin hoedown uv_support morestack miniz context_switch \
+		rustrt_native rust_test_helpers
 
 # $(1) is the target triple
 define NATIVE_LIBRARIES
@@ -52,8 +52,9 @@ NATIVE_DEPS_hoedown_$(1) := hoedown/src/autolink.c \
 			hoedown/src/version.c
 NATIVE_DEPS_uv_support_$(1) := rust_uv.c
 NATIVE_DEPS_miniz_$(1) = miniz.c
-NATIVE_DEPS_rustrt_$(1) := rust_builtin.c \
-			rust_android_dummy.c \
+NATIVE_DEPS_rust_builtin_$(1) := rust_builtin.c \
+			rust_android_dummy.c
+NATIVE_DEPS_rustrt_native_$(1) := \
 			rust_try.ll \
 			arch/$$(HOST_$(1))/record_sp.S
 NATIVE_DEPS_rust_test_helpers_$(1) := rust_test_helpers.c

src/liballoc/lib.rs

@@ -96,6 +96,21 @@ pub mod owned;
 pub mod arc;
 pub mod rc;
 
+// FIXME(#14344): When linking liballoc with libstd, this library will be linked
+//                as an rlib (it only exists as an rlib). It turns out that an
+//                optimized standard library doesn't actually use *any* symbols
+//                from this library. Everything is inlined and optimized away.
+//                This means that linkers will actually omit the object for this
+//                file, even though it may be needed in the future.
+//
+//                To get around this for now, we define a dummy symbol which
+//                will never get inlined so the stdlib can call it. The stdlib's
+//                reference to this symbol will cause this library's object file
+//                to get linked in to libstd successfully (the linker won't
+//                optimize it out).
+#[doc(hidden)]
+pub fn fixme_14344_be_sure_to_link_to_collections() {}
+
 #[cfg(not(test))]
 #[doc(hidden)]
 mod std {

src/liballoc/util.rs

@@ -28,20 +28,3 @@ fn align_to(size: uint, align: uint) -> uint {
     assert!(align != 0);
     (size + align - 1) & !(align - 1)
 }
-
-// FIXME(#14344): When linking liballoc with libstd, this library will be linked
-//                as an rlib (it only exists as an rlib). It turns out that an
-//                optimized standard library doesn't actually use *any* symbols
-//                from this library. Everything is inlined and optimized away.
-//                This means that linkers will actually omit the object for this
-//                file, even though it may be needed in the future.
-//
-//                To get around this for now, we define a dummy symbol which
-//                will never get inlined so the stdlib can call it. The stdlib's
-//                reference to this symbol will cause this library's object file
-//                to get linked in to libstd successfully (the linker won't
-//                optimize it out).
-#[deprecated]
-#[doc(hidden)]
-pub fn make_stdlib_link_work() {}
-

src/libcollections/lib.rs

@@ -72,6 +72,10 @@ fn expect<T>(a: core::option::Option<T>, b: &str) -> T {
     }
 }
 
+// FIXME(#14344) this shouldn't be necessary
+#[doc(hidden)]
+pub fn fixme_14344_be_sure_to_link_to_collections() {}
+
 #[cfg(not(test))]
 mod std {
     pub use core::fmt;      // necessary for fail!()

src/libcollections/slice.rs

@@ -862,7 +862,7 @@ mod tests {
     use std::prelude::*;
     use std::rand::{Rng, task_rng};
     use std::rc::Rc;
-    use std::unstable;
+    use std::rt;
     use slice::*;
 
     use vec::Vec;
@@ -1104,9 +1104,9 @@ mod tests {
     #[test]
     fn test_swap_remove_noncopyable() {
         // Tests that we don't accidentally run destructors twice.
-        let mut v = vec![unstable::sync::Exclusive::new(()),
-                         unstable::sync::Exclusive::new(()),
-                         unstable::sync::Exclusive::new(())];
+        let mut v = vec![rt::exclusive::Exclusive::new(()),
+                         rt::exclusive::Exclusive::new(()),
+                         rt::exclusive::Exclusive::new(())];
         let mut _e = v.swap_remove(0);
         assert_eq!(v.len(), 2);
         _e = v.swap_remove(1);

src/libcore/macros.rs

@@ -98,6 +98,20 @@ macro_rules! try(
     ($e:expr) => (match $e { Ok(e) => e, Err(e) => return Err(e) })
 )
 
+/// Writing a formatted string into a writer
+#[macro_export]
+macro_rules! write(
+    ($dst:expr, $($arg:tt)*) => (format_args_method!($dst, write_fmt, $($arg)*))
+)
+
+/// Writing a formatted string plus a newline into a writer
+#[macro_export]
+macro_rules! writeln(
+    ($dst:expr, $fmt:expr $($arg:tt)*) => (
+        write!($dst, concat!($fmt, "\n") $($arg)*)
+    )
+)
+
 #[cfg(test)]
 macro_rules! vec( ($($e:expr),*) => ({
     let mut _v = ::std::vec::Vec::new();

src/libgreen/basic.rs

@@ -20,7 +20,7 @@ use std::sync::atomics;
 use std::mem;
 use std::rt::rtio::{EventLoop, IoFactory, RemoteCallback};
 use std::rt::rtio::{PausableIdleCallback, Callback};
-use std::unstable::sync::Exclusive;
+use std::rt::exclusive::Exclusive;
 
 /// This is the only exported function from this module.
 pub fn event_loop() -> Box<EventLoop:Send> {
@@ -31,7 +31,7 @@ struct BasicLoop {
     work: Vec<proc():Send>,             // pending work
     remotes: Vec<(uint, Box<Callback:Send>)>,
     next_remote: uint,
-    messages: Exclusive<Vec<Message>>,
+    messages: Arc<Exclusive<Vec<Message>>>,
     idle: Option<Box<Callback:Send>>,
     idle_active: Option<Arc<atomics::AtomicBool>>,
 }
@@ -46,7 +46,7 @@ impl BasicLoop {
             idle_active: None,
             next_remote: 0,
             remotes: vec![],
-            messages: Exclusive::new(vec![]),
+            messages: Arc::new(Exclusive::new(Vec::new())),
         }
     }
 
@@ -61,19 +61,10 @@ impl BasicLoop {
 
     fn remote_work(&mut self) {
         let messages = unsafe {
-            self.messages.with(|messages| {
-                if messages.len() > 0 {
-                    Some(mem::replace(messages, vec![]))
-                } else {
-                    None
-                }
-            })
-        };
-        let messages = match messages {
-            Some(m) => m, None => return
+            mem::replace(&mut *self.messages.lock(), Vec::new())
         };
-        for message in messages.iter() {
-            self.message(*message);
+        for message in messages.move_iter() {
+            self.message(message);
         }
     }
 
@@ -125,13 +116,13 @@ impl EventLoop for BasicLoop {
             }
 
             unsafe {
+                let mut messages = self.messages.lock();
                 // We block here if we have no messages to process and we may
                 // receive a message at a later date
-                self.messages.hold_and_wait(|messages| {
-                    self.remotes.len() > 0 &&
-                        messages.len() == 0 &&
-                        self.work.len() == 0
-                })
+                if self.remotes.len() > 0 && messages.len() == 0 &&
+                   self.work.len() == 0 {
+                    messages.wait()
+                }
             }
         }
     }
@@ -165,33 +156,29 @@ impl EventLoop for BasicLoop {
 }
 
 struct BasicRemote {
-    queue: Exclusive<Vec<Message>>,
+    queue: Arc<Exclusive<Vec<Message>>>,
     id: uint,
 }
 
 impl BasicRemote {
-    fn new(queue: Exclusive<Vec<Message>>, id: uint) -> BasicRemote {
+    fn new(queue: Arc<Exclusive<Vec<Message>>>, id: uint) -> BasicRemote {
         BasicRemote { queue: queue, id: id }
     }
 }
 
 impl RemoteCallback for BasicRemote {
     fn fire(&mut self) {
-        unsafe {
-            self.queue.hold_and_signal(|queue| {
-                queue.push(RunRemote(self.id));
-            })
-        }
+        let mut queue = unsafe { self.queue.lock() };
+        queue.push(RunRemote(self.id));
+        queue.signal();
     }
 }
 
 impl Drop for BasicRemote {
     fn drop(&mut self) {
-        unsafe {
-            self.queue.hold_and_signal(|queue| {
-                queue.push(RemoveRemote(self.id));
-            })
-        }
+        let mut queue = unsafe { self.queue.lock() };
+        queue.push(RemoveRemote(self.id));
+        queue.signal();
     }
 }
 
@@ -216,7 +203,7 @@ impl Drop for BasicPausable {
 
 #[cfg(test)]
 mod test {
-    use std::task::TaskOpts;
+    use std::rt::task::TaskOpts;
 
     use basic;
     use PoolConfig;

src/libgreen/lib.rs

@@ -160,7 +160,7 @@
 //! # Using a scheduler pool
 //!
 //! ```rust
-//! use std::task::TaskOpts;
+//! use std::rt::task::TaskOpts;
 //! use green::{SchedPool, PoolConfig};
 //! use green::sched::{PinnedTask, TaskFromFriend};
 //!
@@ -221,10 +221,10 @@ use std::mem::replace;
 use std::os;
 use std::rt::rtio;
 use std::rt::thread::Thread;
+use std::rt::task::TaskOpts;
 use std::rt;
 use std::sync::atomics::{SeqCst, AtomicUint, INIT_ATOMIC_UINT};
 use std::sync::deque;
-use std::task::TaskOpts;
 
 use sched::{Shutdown, Scheduler, SchedHandle, TaskFromFriend, NewNeighbor};
 use sleeper_list::SleeperList;
@@ -319,7 +319,7 @@ pub fn run(event_loop_factory: fn() -> Box<rtio::EventLoop:Send>,
     let mut pool = SchedPool::new(cfg);
     let (tx, rx) = channel();
     let mut opts = TaskOpts::new();
-    opts.notify_chan = Some(tx);
+    opts.on_exit = Some(proc(r) tx.send(r));
     opts.name = Some("<main>".into_maybe_owned());
     pool.spawn(opts, main);
 

src/libgreen/sched.rs

@@ -10,11 +10,11 @@
 
 use std::mem;
 use std::rt::local::Local;
+use std::rt::mutex::NativeMutex;
 use std::rt::rtio::{RemoteCallback, PausableIdleCallback, Callback, EventLoop};
 use std::rt::task::BlockedTask;
 use std::rt::task::Task;
 use std::sync::deque;
-use std::unstable::mutex::NativeMutex;
 use std::raw;
 
 use std::rand::{XorShiftRng, Rng, Rand};
@@ -1022,7 +1022,7 @@ fn new_sched_rng() -> XorShiftRng {
 mod test {
     use rustuv;
 
-    use std::task::TaskOpts;
+    use std::rt::task::TaskOpts;
     use std::rt::task::Task;
     use std::rt::local::Local;
 
@@ -1475,7 +1475,7 @@ mod test {
 
     #[test]
     fn test_spawn_sched_blocking() {
-        use std::unstable::mutex::{StaticNativeMutex, NATIVE_MUTEX_INIT};
+        use std::rt::mutex::{StaticNativeMutex, NATIVE_MUTEX_INIT};
         static mut LOCK: StaticNativeMutex = NATIVE_MUTEX_INIT;
 
         // Testing that a task in one scheduler can block in foreign code

src/libgreen/simple.rs

@@ -15,10 +15,9 @@ use std::any::Any;
 use std::mem;
 use std::rt::Runtime;
 use std::rt::local::Local;
+use std::rt::mutex::NativeMutex;
 use std::rt::rtio;
-use std::rt::task::{Task, BlockedTask};
-use std::task::TaskOpts;
-use std::unstable::mutex::NativeMutex;
+use std::rt::task::{Task, BlockedTask, TaskOpts};
 
 struct SimpleTask {
     lock: NativeMutex,

src/libgreen/stack.rs

@@ -8,9 +8,9 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.
 
-use std::rt::env::max_cached_stacks;
+use std::sync::atomics;
 use std::os::{errno, page_size, MemoryMap, MapReadable, MapWritable,
-              MapNonStandardFlags, MapVirtual};
+              MapNonStandardFlags, MapVirtual, getenv};
 use libc;
 
 /// A task's stack. The name "Stack" is a vestige of segmented stacks.
@@ -151,6 +151,22 @@ impl StackPool {
     }
 }
 
+fn max_cached_stacks() -> uint {
+    static mut AMT: atomics::AtomicUint = atomics::INIT_ATOMIC_UINT;
+    match unsafe { AMT.load(atomics::SeqCst) } {
+        0 => {}
+        n => return n - 1,
+    }
+    let amt = getenv("RUST_MAX_CACHED_STACKS").and_then(|s| from_str(s.as_slice()));
+    // This default corresponds to 20M of cache per scheduler (at the
+    // default size).
+    let amt = amt.unwrap_or(10);
+    // 0 is our sentinel value, so ensure that we'll never see 0 after
+    // initialization has run
+    unsafe { AMT.store(amt + 1, atomics::SeqCst); }
+    return amt;
+}
+
 extern {
     fn rust_valgrind_stack_register(start: *libc::uintptr_t,
                                     end: *libc::uintptr_t) -> libc::c_uint;