Skip to content

Commit b075ac4

Browse files
authored
Minor: consolidate doc roadmap pages (#11426)
1 parent dc21a6c commit b075ac4

File tree

3 files changed

+81
-97
lines changed

3 files changed

+81
-97
lines changed

docs/source/contributor-guide/quarterly_roadmap.md

Lines changed: 0 additions & 96 deletions
This file was deleted.

docs/source/contributor-guide/roadmap.md

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,3 +43,84 @@ start a conversation using a github issue or the
4343
make review efficient and avoid surprises.
4444

4545
[The current list of `EPIC`s can be found here](https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+epic).
46+
47+
# Quarterly Roadmap
48+
49+
A quarterly roadmap will be published to give the DataFusion community
50+
visibility into the priorities of the projects contributors. This roadmap is not
51+
binding and we would welcome any/all contributions to help keep this list up to
52+
date.
53+
54+
## 2023 Q4
55+
56+
- Improve data output (`COPY`, `INSERT` and DataFrame) output capability [#6569](https://github.com/apache/datafusion/issues/6569)
57+
- Implementation of `ARRAY` types and related functions [#6980](https://github.com/apache/datafusion/issues/6980)
58+
- Write an industrial paper about DataFusion for SIGMOD [#6782](https://github.com/apache/datafusion/issues/6782)
59+
60+
## 2022 Q2
61+
62+
### DataFusion Core
63+
64+
- IO Improvements
65+
- Reading, registering, and writing more file formats from both DataFrame API and SQL
66+
- Additional options for IO including partitioning and metadata support
67+
- Work Scheduling
68+
- Improve predictability, observability and performance of IO and CPU-bound work
69+
- Develop a more explicit story for managing parallelism during plan execution
70+
- Memory Management
71+
- Add more operators for memory limited execution
72+
- Performance
73+
- Incorporate row-format into operators such as aggregate
74+
- Add row-format benchmarks
75+
- Explore JIT-compiling complex expressions
76+
- Explore LLVM for JIT, with inline Rust functions as the primary goal
77+
- Improve performance of Sort and Merge using Row Format / JIT expressions
78+
- Documentation
79+
- General improvements to DataFusion website
80+
- Publish design documents
81+
- Streaming
82+
- Create `StreamProvider` trait
83+
84+
### Ballista
85+
86+
- Make production ready
87+
- Shuffle file cleanup
88+
- Fill functional gaps between DataFusion and Ballista
89+
- Improve task scheduling and data exchange efficiency
90+
- Better error handling
91+
- Task failure
92+
- Executor lost
93+
- Schedule restart
94+
- Improve monitoring and logging
95+
- Auto scaling support
96+
- Support for multi-scheduler deployments. Initially for resiliency and fault tolerance but ultimately to support sharding for scalability and more efficient caching.
97+
- Executor deployment grouping based on resource allocation
98+
99+
### Extensions ([datafusion-contrib](https://github.com/datafusion-contrib))
100+
101+
### [DataFusion-Python](https://github.com/datafusion-contrib/datafusion-python)
102+
103+
- Add missing functionality to DataFrame and SessionContext
104+
- Improve documentation
105+
106+
### [DataFusion-S3](https://github.com/datafusion-contrib/datafusion-objectstore-s3)
107+
108+
- Create Python bindings to use with datafusion-python
109+
110+
### [DataFusion-Tui](https://github.com/datafusion-contrib/datafusion-tui)
111+
112+
- Create multiple SQL editors
113+
- Expose more Context and query metadata
114+
- Support new data sources
115+
- BigTable, HDFS, HTTP APIs
116+
117+
### [DataFusion-BigTable](https://github.com/datafusion-contrib/datafusion-bigtable)
118+
119+
- Python binding to use with datafusion-python
120+
- Timestamp range predicate pushdown
121+
- Multi-threaded partition aware execution
122+
- Production ready Rust SDK
123+
124+
### [DataFusion-Streams](https://github.com/datafusion-contrib/datafusion-streams)
125+
126+
- Create experimental implementation of `StreamProvider` trait

docs/source/index.rst

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,6 @@ To get started, see
121121
contributor-guide/testing
122122
contributor-guide/howtos
123123
contributor-guide/roadmap
124-
contributor-guide/quarterly_roadmap
125124
contributor-guide/governance
126125
contributor-guide/inviting
127126
contributor-guide/specification/index

0 commit comments

Comments
 (0)