Skip to content

Commit 6ead16e

Browse files
jonmmeasealamb
jonmmease
and
alamb
authored
Add test crate to compile DataFusion with wasm-pack (#7633)
* Add test crate to compile DataFusion with wasm-pack * Add datafusion-wasm-app to exercise wasm build * prettier * Add license headers * tomlfmt * tomlfmt * Update datafusion/wasmtest/README.md Co-authored-by: Andrew Lamb <[email protected]> * --dev for faster build --------- Co-authored-by: Andrew Lamb <[email protected]>
1 parent 5f38135 commit 6ead16e

File tree

14 files changed

+7874
-1
lines changed

14 files changed

+7874
-1
lines changed

‎.github/workflows/rust.yml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,29 @@ jobs:
204204
cd datafusion-cli
205205
cargo doc --document-private-items --no-deps
206206
207+
linux-wasm-pack:
208+
name: build with wasm-pack
209+
runs-on: ubuntu-latest
210+
container:
211+
image: amd64/rust
212+
steps:
213+
- uses: actions/checkout@v4
214+
- name: Cache Cargo
215+
uses: actions/cache@v3
216+
with:
217+
path: /github/home/.cargo
218+
# this key equals the ones on `linux-build-lib` for re-use
219+
key: cargo-cache-
220+
- name: Setup Rust toolchain
221+
uses: ./.github/actions/setup-builder
222+
with:
223+
rust-version: stable
224+
- name: Install wasm-pack
225+
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
226+
- name: Build with wasm-pack
227+
working-directory: ./datafusion/wasmtest
228+
run: wasm-pack build --dev
229+
207230
# verify that the benchmark queries return the correct results
208231
verify-benchmark-results:
209232
name: verify benchmark results (amd64)

‎Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ members = [
3030
"datafusion/sql",
3131
"datafusion/sqllogictest",
3232
"datafusion/substrait",
33+
"datafusion/wasmtest",
3334
"datafusion-examples",
3435
"test-utils",
3536
"benchmarks",

‎datafusion/wasmtest/Cargo.toml

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
[package]
19+
name = "datafusion-wasmtest"
20+
description = "Test library to compile datafusion crates to wasm"
21+
version = { workspace = true }
22+
edition = { workspace = true }
23+
readme = { workspace = true }
24+
homepage = { workspace = true }
25+
repository = { workspace = true }
26+
license = { workspace = true }
27+
authors = { workspace = true }
28+
rust-version = "1.70"
29+
30+
[lib]
31+
crate-type = ["cdylib", "rlib",]
32+
33+
[dependencies]
34+
35+
# The `console_error_panic_hook` crate provides better debugging of panics by
36+
# logging them with `console.error`. This is great for development, but requires
37+
# all the `std::fmt` and `std::panicking` infrastructure, so isn't great for
38+
# code size when deploying.
39+
console_error_panic_hook = { version = "0.1.1", optional = true }
40+
41+
datafusion-common = { path = "../common", version = "31.0.0", default-features = false }
42+
datafusion-expr = { path = "../expr" }
43+
datafusion-optimizer = { path = "../optimizer" }
44+
datafusion-physical-expr = { path = "../physical-expr" }
45+
datafusion-sql = { path = "../sql" }
46+
47+
# getrandom must be compiled with js feature
48+
getrandom = { version = "0.2.8", features = ["js"] }
49+
parquet = { version = "47.0.0", default-features = false }
50+
wasm-bindgen = "0.2.87"

‎datafusion/wasmtest/README.md

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
## wasmtest
21+
22+
Library crate to verify that various DataFusion crates compile successfully to the `wasm32-unknown-unknown` target with wasm-pack.
23+
24+
Some of DataFusion's downstream projects compile to WASM to run in the browser. Doing so requires special care that certain library dependencies are not included in DataFusion.
25+
26+
## Setup
27+
28+
First, [install wasm-pack](https://rustwasm.github.io/wasm-pack/installer/)
29+
30+
Then use wasm-pack to compile the crate from within this directory
31+
32+
```
33+
wasm-pack build
34+
```
35+
36+
## Try it out
37+
38+
The `datafusion-wasm-app` directory contains a simple app (created with [`create-wasm-app`](https://github.com/rustwasm/create-wasm-app) and then manually updated to WebPack 5) that invokes DataFusion and writes results to the browser console.
39+
40+
From within the `datafusion/wasmtest/datafusion-wasm-app` directory:
41+
42+
```
43+
npm install
44+
npm run start
45+
```
46+
47+
Then open http://localhost:8080/ in a web browser and check the console to see the results of using various DataFusion crates.
48+
49+
**Note:** In GitHub Actions we test the compilation with `wasm-build`, but we don't currently invoke `datafusion-wasm-app`. In the future we may want to test the behavior of the WASM build using [`wasm-pack test`](https://rustwasm.github.io/wasm-pack/book/tutorials/npm-browser-packages/testing-your-project.html).
50+
51+
## Compatibility
52+
53+
The following DataFusion crates are verified to work in a wasm-pack environment using the default `wasm32-unknown-unknown` target:
54+
55+
- `datafusion-common` with default-features disabled to remove the `parquet` dependency (see below)
56+
- `datafusion-expr`
57+
- `datafusion-optimizer`
58+
- `datafusion-physical-expr`
59+
- `datafusion-sql`
60+
61+
The difficulty with getting the remaining DataFusion crates compiled to WASM is that they have non-optional dependencies on the [`parquet`](https://docs.rs/crate/parquet/) crate with its default features enabled. Several of the default parquet crate features require native dependencies that are not compatible with WASM, in particular the `lz4` and `zstd` features. If we can arrange our feature flags to make it possible to depend on parquet with these features disabled, then it should be possible to compile the core `datafusion` crate to WASM as well.

‎datafusion/wasmtest/datafusion-wasm-app/.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
node_modules
2+
dist

‎datafusion/wasmtest/datafusion-wasm-app/README.md

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
<div align="center">
2+
3+
<h1><code>create-wasm-app</code></h1>
4+
5+
<strong>An <code>npm init</code> template for kick starting a project that uses NPM packages containing Rust-generated WebAssembly and bundles them with Webpack.</strong>
6+
7+
<p>
8+
<a href="https://travis-ci.org/rustwasm/create-wasm-app"><img src="https://img.shields.io/travis/rustwasm/create-wasm-app.svg?style=flat-square" alt="Build Status" /></a>
9+
</p>
10+
11+
<h3>
12+
<a href="#usage">Usage</a>
13+
<span> | </span>
14+
<a href="https://discordapp.com/channels/442252698964721669/443151097398296587">Chat</a>
15+
</h3>
16+
17+
<sub>Built with 🦀🕸 by <a href="https://rustwasm.github.io/">The Rust and WebAssembly Working Group</a></sub>
18+
19+
</div>
20+
21+
## About
22+
23+
This template is designed for depending on NPM packages that contain
24+
Rust-generated WebAssembly and using them to create a Website.
25+
26+
- Want to create an NPM package with Rust and WebAssembly? [Check out
27+
`wasm-pack-template`.](https://github.com/rustwasm/wasm-pack-template)
28+
- Want to make a monorepo-style Website without publishing to NPM? Check out
29+
[`rust-webpack-template`](https://github.com/rustwasm/rust-webpack-template)
30+
and/or
31+
[`rust-parcel-template`](https://github.com/rustwasm/rust-parcel-template).
32+
33+
## 🚴 Usage
34+
35+
```
36+
npm init wasm-app
37+
```
38+
39+
## 🔋 Batteries Included
40+
41+
- `.gitignore`: ignores `node_modules`
42+
- `LICENSE-APACHE` and `LICENSE-MIT`: most Rust projects are licensed this way, so these are included for you
43+
- `README.md`: the file you are reading now!
44+
- `index.html`: a bare bones html document that includes the webpack bundle
45+
- `index.js`: example js file with a comment showing how to import and use a wasm pkg
46+
- `package.json` and `package-lock.json`:
47+
- pulls in devDependencies for using webpack:
48+
- [`webpack`](https://www.npmjs.com/package/webpack)
49+
- [`webpack-cli`](https://www.npmjs.com/package/webpack-cli)
50+
- [`webpack-dev-server`](https://www.npmjs.com/package/webpack-dev-server)
51+
- defines a `start` script to run `webpack-dev-server`
52+
- `webpack.config.js`: configuration file for bundling your js with webpack
53+
54+
## License
55+
56+
Licensed under either of
57+
58+
- Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
59+
- MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
60+
61+
at your option.
62+
63+
### Contribution
64+
65+
Unless you explicitly state otherwise, any contribution intentionally
66+
submitted for inclusion in the work by you, as defined in the Apache-2.0
67+
license, shall be dual licensed as above, without any additional terms or
68+
conditions.

‎datafusion/wasmtest/datafusion-wasm-app/bootstrap.js

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
// A dependency graph that contains any wasm must all be imported
19+
// asynchronously. This `bootstrap.js` file does the single async import, so
20+
// that no one else needs to worry about it again.
21+
import("./index.js")
22+
.catch(e => console.error("Error importing `index.js`:", e));

‎datafusion/wasmtest/datafusion-wasm-app/index.html

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
<!DOCTYPE html>
2+
<html>
3+
<head>
4+
<meta charset="utf-8">
5+
<title>Hello wasm-pack!</title>
6+
</head>
7+
<body>
8+
<h1>See console</h1>
9+
<noscript>This page contains webassembly and javascript content, please enable javascript in your browser.</noscript>
10+
<script src="bootstrap.js"></script>
11+
</body>
12+
</html>

‎datafusion/wasmtest/datafusion-wasm-app/index.js

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
import * as wasm from "datafusion-wasmtest";
19+
20+
wasm.try_datafusion();

0 commit comments

Comments
 (0)