Add parquet filter and sort to bench.sh #6172
Conversation
| RESULTS_FILE="${RESULTS_DIR}/parquet.json" | ||
| echo "RESULTS_FILE: ${RESULTS_FILE}" | ||
| echo "Running parquet filter benchmark..." | ||
| $CARGO_COMMAND --bin parquet -- filter --path "${DATA_DIR}" --scale-factor 1.0 --iterations 5 -o ${RESULTS_FILE} |
There was a problem hiding this comment.
why we dont apply scale-factor for tpch? is it 1 by default?
There was a problem hiding this comment.
This is even more confusing -- the scale factor for tpch is actually a applied, but it is applied when we create the data. Specifically
The "parquet" benchmark actually doesn't use the tpch dataset at all, and instead generates its own data. Hwever, it overloads the "scale factor" terminology to describe the relative sizes
| RESULTS_FILE="${RESULTS_DIR}/parquet.json" | ||
| echo "RESULTS_FILE: ${RESULTS_FILE}" | ||
| echo "Running parquet filter benchmark..." | ||
| $CARGO_COMMAND --bin parquet -- filter --path "${DATA_DIR}" --scale-factor 1.0 --iterations 5 -o ${RESULTS_FILE} |
There was a problem hiding this comment.
It is somewhat ugly to have to run a second command(parquet) for different benchmarks -- I plan to combine them into a single benchmark runner over time
| RESULTS_FILE="${RESULTS_DIR}/parquet.json" | ||
| echo "RESULTS_FILE: ${RESULTS_FILE}" | ||
| echo "Running parquet filter benchmark..." | ||
| $CARGO_COMMAND --bin parquet -- filter --path "${DATA_DIR}" --scale-factor 1.0 --iterations 5 -o ${RESULTS_FILE} |
There was a problem hiding this comment.
This is even more confusing -- the scale factor for tpch is actually a applied, but it is applied when we create the data. Specifically
The "parquet" benchmark actually doesn't use the tpch dataset at all, and instead generates its own data. Hwever, it overloads the "scale factor" terminology to describe the relative sizes
Which issue does this PR close?
Follow on to #6131
Rationale for this change
Add support for running parquet filter and sort benchmarks to sort
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?
You can now run
and
to run benchmarks