Follow target ABI sign-/zero-extension rules for enum types

While attempting to port Rust to s390x, I ran into an ABI violation
(that caused rust_eh_personality to be miscompiled, breaking unwinding).
The problem is that this function returns an enum type, which is
supposed to be sign-extended according to the s390x ABI.  However,
common code would ignore target sign-/zero-extension rules for any
types that do not satisfy is_integral(), which includes enums.

For the general case of Rust enum types, which map to structure types
with a discriminant, that seems correct.  However, in the special case
of simple enums that map directly to C enum types (i.e. LLVM integers),
this is incorrect; we must follow the target extension rules for those.

Signed-off-by: Ulrich Weigand <[email protected]>

Author: uweigand

src/librustc_trans/abi.rs

@@ -26,6 +26,7 @@ use cabi_asmjs;
 use machine::{llalign_of_min, llsize_of, llsize_of_real, llsize_of_store};
 use type_::Type;
 use type_of;
+use adt;
 
 use rustc::hir;
 use rustc::ty::{self, Ty};
@@ -317,6 +318,14 @@ impl FnType {
                 if ty.is_integral() {
                     arg.signedness = Some(ty.is_signed());
                 }
+                // Rust enum types that map onto C enums (LLVM integers) also
+                // need to follow the target ABI zero-/sign-extension rules.
+                if let ty::TyEnum(..) = ty.sty {
+                    if arg.ty.kind() == llvm::Integer {
+                        let repr = adt::represent_type(ccx, ty);
+                        arg.signedness = Some(adt::is_discr_signed(&repr));
+                    }
+                }
                 if llsize_of_real(ccx, arg.ty) == 0 {
                     // For some forsaken reason, x86_64-pc-windows-gnu
                     // doesn't ignore zero-sized struct arguments.