Open Variant Type for semi-structured data

[wjones127]

I've been starting to experiment with implementing the Open Variant Type ¹ in Rust / DataFusion. There is a specification and Java library for this, and Spark will release this type in 4.0. There are also plans to integrate this into table formats such as Delta Lake ² and Iceberg ³. This would be a high-performance data type for semi-structured data, designed for better OLAP performance than JSON or BSON (discussed in #7845). I've discussed a little bit in the Arrow repo about it's potential as an Arrow extension type ⁴.

I'm working on creating an extension similar to datafusion-functions-json. If we could create a new repo datafusion-functions-variant, I'd be happy to develop that in the open.

Footnotes

1. https://github.com/apache/spark/tree/master/common/variant ↩

2. https://www.databricks.com/blog/introducing-open-variant-data-type-delta-lake-and-apache-spark ↩

3. https://lists.apache.org/thread/xnyo1k66dxh0ffpg7j9f04xgos0kwc34 ↩

4. https://github.com/apache/arrow/issues/42069 ↩

[alamb]

@samuelcolvin and @adriangb and I were speaking about datafusion-functions-json just today. They may be interested too

I have created https://github.com/datafusion-contrib/datafusion-functions-variant and added you as an admin 🚀

[samuelcolvin]

Interesting, this is exactly what we need.

[alamb]

FWIW the variant type has been added to the parquet spec, and is now much closer to standardization and we are beginning work on the support at the parquet layer. See epic

• [EPIC] [Parquet] Implement Variant type support in Parquet arrow-rs#6736

[ianmcook]

There's a discussion happening on the Arrow dev mailing list about adding Variant as a canonical extension type in the Arrow spec. Input from the DataFusion developer community would be much appreciated.

• The discussion thread in the mailing list archive: https://lists.apache.org/thread/w06cxdojjcmry4m9vb0bo7owd1jsbtz5
• How to subscribe to the Arrow dev mailing list: https://arrow.apache.org/community/#mailing-lists