Arrow Flight SQL / ADBC datasource

[backkem]

Is your feature request related to a problem or challenge?

I want to fetch data from multiple remote databases.

Describe the solution you'd like

Since DataFusion is part of the Arrow ecosystem it seems to make sense to have an Arrow Flight SQL datasource.

Describe alternatives you've considered

splitgraph/seafowl's datafusion_remote_tables. However, it seems sensible to have a basic Arrow Flight SQL datasource out of the box.

Additional context

No response

[backkem]

Maybe adding an ADBC backed TableProviderFactory would be a good approach. That way the actual drivers can live under the apache/arrow-adbc project.

[alamb]

I think having an (optional) FlightSQL backed remote table sounds like a good idea to me.

[backkem]

Yes, I don't think it needs to be enabled by default.

I was looking into the adbc direction but the rust port seems less fleshed out than the C and Go ones so far. I did find this somewhat stale PR: apache/arrow-adbc#478

[alamb]

See discussion on #11938 -- we are going to try and put such table providers in other repos rather than the core datafusion one