Fails to symbolicate when loaded as a .so on android

[thomcc]

If rust code using backtrace is loaded on android as a .so, symbolication always fails, even if symbols are available.

We've also seen it cause crashes due to OOM on arm android (32 and 64 bit) in the wild (mozilla/application-services#441, also other reports), which may or may not be related.

[thomcc]

Worth noting that on android shared object loading can be a bit weird, the object will often be loaded from inside the .apk (a zip file) directly, which their loader supports.

I'm not sure if this impacts how you symbolicate though.

[alexcrichton]

This does indeed sound bad! Unfortunately though it doens't sound familiar so it'd take some effort to investigate this. I don't have a ton of time right now to look into this, but if someone has questions that I can help with I can try to do that!

[thomcc]

Do you have any advice on where you'd start digging? E.g. Is this likely to be libbacktrace, the rust code, etc?

I don't mind looking into it, but I'm not exactly sure how all this fits together.

[alexcrichton]

My guess is that the Rust code probably isn't at fault here but probably something within libbacktrace. Unfotunately though libbacktrace is somewhat difficult to debug. You could also try tweaking the various features of this crate which controls which strategy is used for symbolication.

One possible candidate is libbacktrace finding and loading the ELF file image, which on Linux uses /proc/self/exe but there's platform-specific code for other platforms and it may be the case that Android isn't handled (which I think can probably also use /proc/self/exe)

[kjvalencik]

I don't think this was resolved by #228. I am getting backtraces when running a bin, but not when loading as a shared module.

[s1341]

I have a rust binary loading a c shared library via dlopen. I get symbols from the rust library, but no symbols from the shared library. This is not in an 'application'. Both files are debug versions.

I'd be happy to help debug this if you can point me in the right direction?

[alexcrichton]

Can a new issue be opened for this?

The reason this doesn't work is due to android not being listed here --

backtrace-rs/src/symbolize/gimli.rs

Lines 362 to 369 in c6e5bff

	} else if #[cfg(all(
	any(
	target_os = "linux",
	target_os = "fuchsia",
	target_os = "freebsd",
	),
	not(target_env = "uclibc"),
	))] {

-- which means by default this crate makes no attempt to symbolize anything

[kjvalencik]

I am using libbacktrace to symbolize (default-features = false, features = ["std", "libbacktrace"]). I was able to narrow down the issue a bit.

If I set android.defaultConfig.minSdkVersion to 21, it symbolizes correctly. If I set it higher, for example, to 23, it no longer works.

My hunch is that something in this section is only matching exactly 21 instead of >=21. I'll perform some more testing and open a PR if that's the case. Thanks!

backtrace-rs/crates/backtrace-sys/build.rs

Lines 145 to 168 in c6e5bff

	fn maybe_enable_dl_iterate_phdr_android(build: &mut cc::Build) {
	let expansion = cc::Build::new().file("src/android-api.c").expand();
	let expansion = match std::str::from_utf8(&expansion) {
	Ok(s) => s,
	Err(_) => return,
	};
	println!("expanded android version detection:\n{}", expansion);
	let marker = "APIVERSION";
	let i = match expansion.find(marker) {
	Some(i) => i,
	None => return,
	};
	let version = match expansion[i + marker.len() + 1..].split_whitespace().next() {
	Some(s) => s,
	None => return,
	};
	let version = match version.parse::<u32>() {
	Ok(n) => n,
	Err(_) => return,
	};
	if version >= 21 {
	build.define("HAVE_DL_ITERATE_PHDR", "1");
	}
	}

[kjvalencik]

I confirmed that HAVE_DL_ITERATE_PHDR is being set. I also confirmed that the exact same .so works with 21, but not 22. It must be a runtime issue and not a compile time issue.

[alexcrichton]

Note that the detection there is not part of the gimli implementation, it may need to get ported to the gimli implementation, however. Until dl_iterate_phdr is called with the gimli implementation this won't work by default.

I wouldn't recommend using the libbacktrace implementation because it's likely to get removed at some point and it's historically had at least a few bugs which could lead to segfaults or worse.

[s1341]

is gimli support for dl_iterate_phdr imminent?

[kjvalencik]

@alexcrichton Thanks! I'll take that under advisement. For what it's the cause is Android loading the shared library directly from the apk instead of extracting it to disk during install.

https://android.googlesource.com/platform/bionic/+/master/android-changes-for-ndk-developers.md#opening-shared-libraries-directly-from-an-apk

Setting android:extractNativeLibs="true" caused backtraces to start working again.

Cheers!

[alexcrichton]

@s1341 it needs someone to send a PR effectively.

@kjvalencik glad it works for you!

[s1341]

@alexcrichton I can take a hint! I'm not sure I'll have time for it soon though...

[kjvalencik]

@s1341 I got the ball rolling by opening a PR to libc to add dl_iterate_phdr to the Android targets (rust-lang/libc#2144).

I have a local branch of backtrace confirming that it is sufficient to get backtraces working with gimli on Android.