Cannot build [email protected] under Amazon Linux 2023 using cargo lambda. #1291
Comments
LegoWolf
added
bug
github-actions
bot
added
the
potential-regression
|
Can confirm this bug and the same errors when doing a Edit: this seems more like a For workarounds until this gets figured out - pin your deps like this Potentially relevant things |
|
For what it's worth, I ran into a similar failure on Friday while building in the cargo-lambda container. While digging I noticed that |
|
We are encountering issues here, probably related to this crate usage as well (still testing). Our software using 1.83 was fine. It gets killed with a SIGILL (Illegal instruction) when using version 1.85 of the sdk. We don't have yet a core-dump, but the kill happens within the routine that uploads a file to S3, which was working fine last week with 1.83. The crate is built on the same machine that runs the code, so it's unlikely that the issue arises from compiling on a different architecture, or on a CPU with different features than the one running the code. Additionally, while running This is on a r6idn.8xlarge machine (Xeon Platinum 8375C) |
|
The // no auto-optimize enabled, return and use the internal Rust implementation
#[cfg(feature = "optimize_crc32_auto")]
{
// for auto, default to the best available implementation based on CPU features
// in build scripts, the target architecture is only available via an environment variable
let target_arch = env::var("CARGO_CFG_TARGET_ARCH").unwrap();
if "x86" == target_arch {
// this is the only one supported on 32-bit x86 systems
crate::build_sse_v4s3x3()
}
#[cfg(any(target_arch = "x86_64", target_arch = "x86"))]
if is_x86_feature_detected!("vpclmulqdq")
&& is_x86_feature_detected!("avx512vl")
&& is_x86_feature_detected!("avx512f")
{
return build_avx512_vpclmulqdq_v3x2();
}
#[cfg(any(target_arch = "x86_64", target_arch = "x86"))]
if is_x86_feature_detected!("avx512vl")
&& is_x86_feature_detected!("avx512f")
&& is_x86_feature_detected!("pclmulqdq")
{
return crate::build_avx512_v4s3x3();
}
#[cfg(any(target_arch = "x86_64", target_arch = "x86"))]
if is_x86_feature_detected!("sse4.2") && is_x86_feature_detected!("pclmulqdq") {
crate::build_sse_v4s3x3()
}
}The build script does runtime detection of CPU features to do a build, rather than using the compiler target. You cannot trust that the binaries build on one machine will work on another. Given the numbers of download and users of |
|
I think that's exactly right: we are compiling on a AVX512 machine ( I can think of two ways to get out of this:
Also, I can confirm that upon reverting just the |
I would consider the If not in the aws-sdk itself, this should be addressed upstream in There is a nice guide to reproducing here: |
|
Indeed, I just added my comments to this issue because I believe breakage is related. But in our case, this has nothing to do with |
I would advocate for this path until it has been proven that |
|
Author of
I think this is the correct approach in the short-term. I've done extensive cross-compiling to/from various architectures, but apparently haven't yet covered all the edge cases (and/or don't fully understand the Rust mechanisms to improve this). Removing the feature flag (there's a reason it's a feature flag in the first place! 😄 ) will use the Rust implementation (which is still extremely fast). I'm traveling today, but will make a PR to that effect when I find some time if someone doesn't beat me to the punch. 👍 |
Yeah, that was my gut feeling too. I tried re-running my repro steps in Amazon Linux 2023 in a Docker container on my M1 Mac, and the
This error seems different from the one that I reported in the original bug description, because it is picking a valid architecture for the platform, but somehow it's picking one that either The takeaway for me is that I agree with the thread that one of the previously suggested fallbacks is best for now until these edge cases can be addressed. I do look forward to seeing the performance impact of |
|
I have a PR up removing the If anyone having cross compilation issues could test it via a git dependency like: [dependencies]
aws-smithy-checksums = { git = "https://github.com/smithy-lang/smithy-rs.git", branch = "landonxjames/crc-feature" }That would be helpful while I work to get my own cross compilation setup going. |
It looks good to me! I've compiled a test project with only this dependency in a few configurations: Amazon Linux 2023, Docker Container, M1 Mac: Clean builds with no errors. Any other tests that would be helpful to run? |
I've had a quick look at the source code, and I think the issue is that the crate uses two conflicting approaches for choosing the appropriate implementations. The pure Rust functions seem to do the right thing: pub(crate) unsafe fn update(state: u64, bytes: &[u8], params: CrcParams) -> u64 {
#[cfg(target_arch = "aarch64")]
{
// ...
}
#[cfg(all(target_arch = "x86_64", feature = "vpclmulqdq"))]
{
use std::arch::is_x86_feature_detected;
if bytes.len() >= 256 && is_x86_feature_detected!("vpclmulqdq") {
// ...
}
}
#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
{
// ...
}
#[cfg(not(any(target_arch = "x86", target_arch = "x86_64", target_arch = "aarch64")))]
return software::update(state, bytes, params);
}But the crate also uses C files for part of its implementation, and these can only be compiled for one targets, and one target only: extern "C" {
pub static ISCSI_TARGET: *const ::std::os::raw::c_char;
}
extern "C" {
#[doc = " Gets the target build properties (CPU architecture and fine-tuning parameters) for this implementation."]
pub fn get_iscsi_target() -> *const ::std::os::raw::c_char;
}
extern "C" {
#[doc = " Calculate CRC-32/ISCSI checksum using hardware acceleration\n\n @param crc0 Initial CRC value (typically 0)\n @param buf Pointer to input data buffer\n @param len Length of input data in bytes\n\n @return Calculated CRC-32/ISCSI checksum"]
pub fn crc32_iscsi_impl(crc0: u32, buf: *const ::std::os::raw::c_char, len: usize) -> u32;
}For example in CRC_EXPORT uint32_t crc32_iscsi_impl(uint32_t crc0, const char* buf, size_t len) {
crc0 = ~crc0;
for (; len && ((uintptr_t)buf & 7); --len) {
crc0 = _mm_crc32_u8(crc0, *buf++);
}
while (((uintptr_t)buf & 56) && len >= 8) {
crc0 = _mm_crc32_u64(crc0, *(const uint64_t*)buf);
buf += 8;
len -= 8;
}
if (len >= 384) {
__m128i z0;
/* First vector chunk. */
__m512i x0 = _mm512_loadu_si512((const void*)buf), y0;
__m512i x1 = _mm512_loadu_si512((const void*)(buf + 64)), y1;
__m512i x2 = _mm512_loadu_si512((const void*)(buf + 128)), y2;
/* ... */IMHO:
|
Nope that should be good. I was able to run the same tests in an AL 2023 Docker container as well and it all worked for me. Will get that PR merged and a release out. |
|
A new version of aws-smithy-checksums |
|
aws-sdk-s3 version 1.87.0 was just published a few minutes ago and it relies on |
|
Confirmed that it's fixed my original issue. Thank-you so much! |
|
Comments on closed issues are hard for our team to see. |
Describe the bug
I cannot use
cargo lambdato build any Rust project that includesaws-sdk-s3v1.86, whereas using v1.85 or earlier build fine. The build error only happens when usingcargo lambda buildnotcargo build.Regression Issue
Expected Behavior
No build errors.
Current Behavior
Following the reproduction steps produces this output:
Reproduction Steps
Install:
Create and build this project:
Possible Solution
The problem occurs when cargo tries to build
crc-fast, which is a new dependency ofaws-smithy-checksumsv0.63.2. I'm not certain whether the problem is in the crate itself or in the build environment. I'm not sure how to isolate the problem further than I have.Additional Information/Context
I am running Amazon Linux 2023 in Windows using WSL2.
uname -aproduces this output:Linux GLYN-PC 5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Version
Environment details (OS name and version, etc.)
Amazon Linux 2023, x86_64
Logs
No response
The text was updated successfully, but these errors were encountered: