Ideas: add directory support for FileIO

[dentiny]

Hi team, I notice

• FileIO is the only abstraction inside of iceberg-rust project to provide IO operation manipulation capabilities;
• Current implementation for FileIO focuses only on "files", or in object storage's word, "object".

I'm wondering if it makes sense to add "directory" support as well?
For example, the file hierarchy for iceberg table namespace and table is built upon directories.

The following is the output for rest catalog docker image, support our table has created namespace "default" and table "test_table".

[/workspaces/pg_moonlink] (hjiang/filesystem-catalog) 
vscode@4a9f5d144667$ ls /tmp/iceberg-test/default/test_table/
data  metadata

[dentiny]

More context why I want to have the directory support, I'm implementing a filesystem catalog feature for two purpose:

• Hermetic unit test
  • bringing rest docker image is a little painful, but I have ways to handle
  • MemoryCatalog doesn't support transaction features I need
• Local usage for customer initial try out (consider it production environment)
  • MemoryCatalog is not intended for production usage

[liurenjie1024]

Thanks @dentiny for raising this. FileIO is an abstraction over different storage systems, and we should only provide a minimum interface so that most storage systems can fit into it. As with directory operation, I don't think we should provide that for several reasons:

1. Apache iceberg spec has no requirement for file location, the file hierarchy you see is just default implementation, but not required in iceberg spec.
2. Most object stores's support for directory is limited and inefficient.

[dentiny]

Thank you for the reply!

I agree with your point, "directory" concepts, AFAIK, exists mostly on POSIX system and GCS.
Right now I'm implementing a filesystem based catalog, which definitely requires directory-related access, it's awkward to use file_io for file-centric operations, while std::fs for directory operation.

Take duckdb as an example, it has a filesystem interface, but not all the operations apply to all detailed filesystem instances.
For example, pipe only apply to local filesystem: https://github.com/duckdb/duckdb/blob/0069af20abea2660d67849257bc568570388d50c/src/include/duckdb/common/file_system.hpp#L173-L174, and set the default behavior to throw (for other filesystems).

I used it as a temporary workaround and leave the discussion thread here, and I completely understand and agree with your argument, would like to hear if you have other suggestions. :(

[liurenjie1024]

cc @Xuanwo is more familiar with storage systems, maybe he could help you here.

[Xuanwo]

Hi, thank you @dentiny for starting this.

The topic of a file-based catalog has come up multiple times throughout Iceberg's history and has been rejected for similar reasons:

• Iceberg is designed to avoid performing list object operations on storage services. It doesn't depend on storage services to provide listing support, which is one of the reasons it outperforms the Hive table format.
• The filesystem catalog itself offers only very limited features. For example, users cannot rename a table atomicly.

The latest discussion on this topic can be found at https://lists.apache.org/thread/v7x65kxrrozwlvsgstobm7685541lf5w.

Some people are exploring this direction (although not in exactly the same way) at https://olympiaformat.org. You might find it interesting.

---

Let’s return to our current discussion.

I personally agree that there are some interesting use cases for an FS-based catalog. For example, if the layout on S3 is organized in the way you mentioned, users can read from this catalog directly without needing to connect to other services.

(Note: This is an implementation detail regarding how other catalogs organize namespaces and tables. It’s possible for them to store the entire namespace in a Postgres database.)

I think the way forward for us is as follows:

We can expose a list operation in FileIO. This isn't being added solely for this discussion; our compaction and vacuum features will require this functionality as well.

You could then implement an FS catalog outside of iceberg-rust as a third-party catalog.

What do you think?

[dentiny]

Thank you for the thoughtful reply! I think all points you've made make sense to me.

As a newcomer, one thing I found filesystem catalog attractive is, it provides a way to demonstrate what metadata file and manifest files looks like in a straightforward way; rest catalog with "file" schema also generates a few files (manifest files and manifest list), but metadata files and version hint file seems to stored internally (i.e. in sqlite).

You could then implement an FS catalog outside of iceberg-rust as a third-party catalog.

Sure! I have an internal implementation for FS catalog, which only implements limited TableUpdates;
I will try to get more updates covered and put on my github for public access at least.

We can expose a list operation in FileIO. This isn't being added solely for this discussion; our compaction and vacuum features will require this functionality as well.

Nice! I'll definitely rely on the feature.

[dentiny]

We can expose a list operation in FileIO. This isn't being added solely for this discussion; our compaction and vacuum features will require this functionality as well.

Sorry if I mis-understand.

A second thought, iceberg is a location-unaware specification, which means data file could be placed anywhere, it's valid implementation as long as manifest files could track them.

For compaction, my understanding, is to
(1) get data files to compact from manifest files
(2) read data files and deletion files / deletion vector
(3) apply certain compaction strategies (i.e. sorting)
(4) write compacted data files into destination

So we don't actually require list operation involved here?

[Xuanwo]

So we don't actually require list operation involved here?

Yes, implementing compaction doesn't require list operations. I mentioned it here because most services that implement compaction often also need to implement vacuuming to ensure that data is garbage collected on the storage side, which does require list operations. For examples, some data files might been written in the storage but not committed for long.

[dentiny]

That makes sense to me.