Improved experience when remote object store URL does not end in /

[xiedeyantu]

Which issue does this PR close?

• Closes Improved experience when remote object store URL does not end in / #16302

Rationale for this change

It would be automatically add a / to the path if the first one was not found and try again.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[xiedeyantu]

@alamb Could you help reivew this PR?

[alamb]

@alamb Could you help reivew this PR?

Thanks @xiedeyantu ! We normally need to add tests as part of any code PR -- could you look into adding some tests and documentation about how this change works?

[xiedeyantu]

@alamb Could you help reivew this PR?

Thanks @xiedeyantu ! We normally need to add tests as part of any code PR -- could you look into adding some tests and documentation about how this change works?

@alamb Is there any example I can refer to for testing involving s3, because real access requires ak/sk and the real path?

[alamb]

@alamb Could you help reivew this PR?

Thanks @xiedeyantu ! We normally need to add tests as part of any code PR -- could you look into adding some tests and documentation about how this change works?

@alamb Is there any example I can refer to for testing involving s3, because real access requires ak/sk and the real path?

I think you can follow the one here: https://datafusion.apache.org/user-guide/cli/datasources.html#s3

To read from a public S3 bucket without signatures, use the aws.SKIP_SIGNATURE option:

CREATE EXTERNAL TABLE nyc_taxi_rides
STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/'
OPTIONS(aws.SKIP_SIGNATURE true);

[alamb]

@blaginin also did an end to end test with S3 in the CI tests. The instructions are here:

• https://github.com/apache/datafusion/blob/main/datafusion-cli/CONTRIBUTING.md#L30-L29