Skip to content

Integer overflow in specialized _solve methods #681

Open
@c42f

Description

@c42f

As observed in #583 there's some weird errors with small-sized solves of integer matrices. For example:

julia> m = 2
       a = SMatrix{m,m}(rand(Int,m,m))
       b = SVector{m}(rand(Int,m))
2-element SArray{Tuple{2},Int64,1,2} with indices SOneTo(2):
  8315397486252833078
 -1243568792894157487

julia> Matrix(a) \ Vector(b)
2-element Array{Float64,1}:
 -1.0427738899729126
  0.5986498869255282

julia> a \ b
2-element SArray{Tuple{2},Float64,1,2} with indices SOneTo(2):
 4.043446204424333e-20
 8.620209721520739e-20

The underlying cause seems to be integer overflow in some specialized _solve functions. For example due using integer arithmetic here:

@inbounds return similar_type(b, T)((a[2,2]*b[1] - a[1,2]*b[2])/d,

CC @judober

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions