[WIP] Remove COMET_SHUFFLE_FALLBACK_TO_COLUMNAR config

[andygrove]

Which issue does this PR close?

Closes #1254

Rationale for this change

The config COMET_SHUFFLE_FALLBACK_TO_COLUMNAR was added as a workaround for Spark SQL test failures and we need to remove it since users could run into the same bugs that is causing the Spark SQL failures.

What changes are included in this PR?

• Remove config
• Fix Spark SQL test failures:

core1

• test with low buffer spill threshold
• SPARK-32038: NormalizeFloatingNumbers should work on distinct aggregate
• reverse preceding/following range between with aggregation

core2

• subquery/exists-subquery/exists-orderby-limit.sql

core3

• SPARK-32649: Optimize BHJ/SHJ inner/semi join with empty hashed relation
• SPARK-38237: require all cluster keys for child required distribution for window query

How are these changes tested?

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.66%. Comparing base (f09f8af) to head (b3cf832).
Report is 195 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1736      +/-   ##
============================================
+ Coverage     56.12%   58.66%   +2.53%     
- Complexity      976     1138     +162     
============================================
  Files           119      129      +10     
  Lines         11743    12684     +941     
  Branches       2251     2372     +121     
============================================
+ Hits           6591     7441     +850     
- Misses         4012     4058      +46     
- Partials       1140     1185      +45     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.