Future support for Arrow Flight SQL

[jmcastagnetto]

Is your feature request related to a problem? Please describe.

A solution being worked in the Apache Arrow community is the use of an SQL layer over the Arrow Flight protocol, which will allow a client to communicate in a simpler way with a DB backend w/o the need for a multiplicity of drivers.

See for example: apache/arrow#10906 and https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/

Describe the solution you'd like

Use an Arrow Flight SQL driver to connect to multiple DBs:

(Image from: https://www.dremio.com/subsurface/arrow-flight-sql-a-universal-jdbc-driver/)

Describe alternatives you've considered

N/A

Additional context

N/A

[flambchap]

Hello @jmcastagnetto. Could you please tell me what benefits does it bring? What are the use cases? Thanks.

[jmcastagnetto]

The Arrow Flight SQL is a layer on top of the Arrow Flight protocol [1], and provides a known interface (SQL) to the "on-the-wire" representation of the data, making it easy to interact with datastores that support Arrow Flight.

Arrow Flight has better performance than JDBC or ODBC [2], and allows accessing huge amounts of data in batch mode, apart from supporting parallel and partitioned data access [1].

IMHO those are enough benefits (and not the only ones planned, AFAICT) to think about supporting (in the future) data access using either the lower level Apache Flight or a higher level Apache Flight SQL layer. Not sure what would be the amount of work needed or what changes will be needed in dBeaver. It seems like there is a reference implementation of a JDBC-style driver [3]

HTH

• [1] https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/
• [2] https://www.dremio.com/is-time-to-replace-odbc-jdbc/
• [3] https://www.dremio.com/subsurface/arrow-flight-sql-a-universal-jdbc-driver/

[jmcastagnetto]

Just an FYI, there seems to be a set of Java libs for Apache Arrow and related (Flight, Flight SQL, etc.), available at Maven: https://search.maven.org/search?q=g:org.apache.arrow%20AND%20v:10.0.0

Might also be useful for the issue #13033 here