Skip to content

range end index 294912 out of range for slice of length 147456 #540

Closed
@andygrove

Description

@andygrove

Describe the bug

I just saw this when running benchmarks with latest from main and with xxhash64 disabled.

Caused by: org.apache.comet.CometNativeException: General execution error with reason org.apache.comet.CometNativeException: range end index 294912 out of range for slice of length 147456
        at comet::errors::init::{{closure}}(__internal__:0)
        at std::panicking::rust_panic_with_hook(__internal__:0)
        at std::panicking::begin_panic_handler::{{closure}}(__internal__:0)
        at std::sys_common::backtrace::__rust_end_short_backtrace(__internal__:0)
        at rust_begin_unwind(__internal__:0)
        at core::panicking::panic_fmt(__internal__:0)
        at core::slice::index::slice_end_index_len_fail(__internal__:0)
        at arrow_data::transform::variable_size::build_extend::{{closure}}(__internal__:0)
        at arrow_data::transform::MutableArrayData::extend(__internal__:0)
        at comet::execution::operators::copy_array(__internal__:0)
        at comet::execution::operators::copy_or_cast_array(__internal__:0)
        at <comet::execution::operators::copy::CopyStream as futures_core::stream::Stream>::poll_next(__internal__:0)
        at <datafusion_physical_plan::stream::RecordBatchStreamAdapter<S> as futures_core::stream::Stream>::poll_next(__internal__:0)
        at Java_org_apache_comet_Native_executePlan(__internal__:0)
        at <unknown>(__internal__:0).
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:102)
	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:125)
	at org.apache.spark.sql.comet.execution.shuffle.CometShuffleWriteProcessor.write(CometShuffleExchangeExec.scala:492)

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions