Description
Describe the bug
Polaris uses some asynchronously executed tasks to run operations for table and manifest file cleanup. Those tasks are potentially executed in a separate thread in the same JVM. There is however no guarantee that those tasks will eventually run for multiple reasons:
- Tasks (e.g. via
org.apache.polaris.service.catalog.BasePolarisCatalog#dropTable
) are triggered after the fact. - Although tasks are persisted, there is no mechanism to pick up tasks that did not start or did not finish ("long lasting" failures, JVM terminates).
Overall this means that for example a "drop table with purge" returns a successful result to the user, the actual purge may never ever happen.
To Reproduce
No response
Actual Behavior
No response
Expected Behavior
No response
Additional context
No response
System information
No response