tracing & tracing-subscriber have a design defect I believe #3223
Labels
Comments
|
Hmm, that doesn't seem right. The guard returned by Lines 1558 to 1570 in bfca546 The Lines 1051 to 1054 in bfca546 This is designed to prevent precisely the situation you're describing in this issue, so I'm not sure what's gone wrong. |
|
Looking at the panic in hyperium/hyper#3848, I see that and the stack indicates that we are in the span's It looks like the problem is this code in tracing/tracing-subscriber/src/registry/sharded.rs Lines 302 to 304 in bfca546 Note that here, in order to call the exit callback on every subscriber One solution, although it has unfortunate performance implications, would be for the |
hawkw
added
kind/bug
|
Can the performance be mitigated by checking if the thread being run on is still the same as the one the span was created on and only if there's a difference change the default thread subscriber? Not sure about the relative cost of the thread id check vs changing the TLS variable although maybe I'm missing the cost (I would think changing a single TLS variable is quite cheap on the order of checking the current thread but maybe setting the thread's default dispatcher is more involved) |
|
Hey, I am by no means a Rust expert and don't fully understand the complexity of this issue, but have there any steps toward a resolution for this? I would love to put in a PR rather than just complain, but this isn't necessarily my forte. I encountered this issue, and what I believe to be tangential issues:
Most issues I find under hyper-util and reqwest, where the maintainers take no interest in accommodating for this panic. They are using the I'm unsure if these are all 100% related, but I just wanted to make note of it that it seems recurring and prominent. How I encountered this (somewhat pseudocode) /// creates a default global logger, appending to file + stdout
fn global_logger() {
let subscriber: tracing_subscriber::fmt::Subscriber; /* constructed using tracing_subscriber::fmt() */
tracing::subscriber::set_global_default(subscriber);
}
/// creates a logger to be used in some span, appending to file + stdout
fn build_logger() -> (tracing_core::dispatcher::Dispatch, tracing_appender::non_blocking::WorkerGuard) {
let (.., guard) = tracing_appender::non_blocking(..);
let subscriber: tracing_subscriber::fmt::Subscriber; /* constructed using tracing_subscriber::fmt() */
let dispatch = Dispatch::new(subscriber);
(dispatch, guard)
}
fn main() {
// vvv uncommenting this, panics @ reqwest get, hyper_utils client future
// global_logger();
// ^^^ uncommenting this, panics @ reqwest get, hyper_utils client future
let thread_0 = std::thread::spawn(move || {
// get local spanned logger for this async runtime
let (dispatch, _guard) = build_logger();
let _local_default_guard = tracing::dispatcher::set_default(&dispatch);
let rt = tokio::runtime::Builder::new_current_thread()
.worker_threads(1)
.enable_all()
.build()
.unwrap();
// spawn async runtime on this thread
rt.block_on(async move {
// perform a reqwest::Client::get here
});
});
thread_0.join().unwrap();
}full trace: |
This was referenced Apr 2, 2025
|
https://github.com/arpadav/tracing-span-bug/ Made this, showcasing this issue |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version
tracing 0.1.41
tracing-subscriber 0.3.19
Platform
I'm pretty sure all of them, but definitely verified on Linux 6.12.9-arch1-1
Description
Using with_subscriber will trigger a panic if the async block creates a span & enters it on a different thread that will target the default subscriber. I believe both the with_subscriber AND the enter off-thread scenario are idiomatic. The reason for the crash is that the span.enter() try's recording the span exit into the default subscriber which isn't aware of it since it's registered by the scoped subscriber.
I encountered this with hyper which in GaiResolver has a block like:
This pattern seems pretty idiomatic: https://docs.rs/tracing/0.1.41/tracing/struct.Span.html#method.or_current
A simple repro on Linux (which uses GaiResolver) is:
Tried filing upstream but the author there confirmed it's likely an issue in this crate: hyperium/hyper#3848 (comment)
Either
with_subscriberhas a fatal flaw or the idiomatic pattern needs to be clarified on how to actually make it safe for the above snippet.The text was updated successfully, but these errors were encountered: