Fix wasm32 build on version 46 #15102
Conversation
github-actions
bot
added
the
development-process
| - name: Install dependencies | ||
| run: | | ||
| apt-get update -qq | ||
| apt-get install -y -qq clang |
There was a problem hiding this comment.
Without this I got: https://github.com/XiangpengHao/datafusion/actions/runs/13744579987/job/38437951143#step:6:144
warning: [email protected]+zstd.1.5.6: ToolExecError: Command LC_ALL="C" "sccache" "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fno-exceptions" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "wasm-shim/" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-fvisibility=hidden" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-o" "/__w/datafusion/datafusion/target/wasm32-unknown-unknown/debug/build/zstd-sys-789e7626e12bcd14/out/44ff4c55aa9e5133-debug.o" "-c" "zstd/lib/common/debug.c" with args clang did not execute successfully (status code exit status: 2).cargo:warning=Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang` installed?
warning: [email protected]+zstd.1.5.6: Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang` installed?
warning: [email protected]+zstd.1.5.6: Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang` installed?
warning: [email protected]+zstd.1.5.6: Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang` installed?
warning: [email protected]+zstd.1.5.6: sccache: error: failed to execute compile
warning: [email protected]+zstd.1.5.6: sccache: caused by: cannot find binary path
So I added clang here
|
Thanks for cleaning up after me @XiangpengHao. |
There was a problem hiding this comment.
Thank you @XiangpengHao
It seems like this code is not covered and thus is likely to get broken again as part of refactoring.
We have a CI job that is supposed to check wasm like this:
datafusion/.github/workflows/rust.yml
Lines 251 to 265 in 618880e
It has a special crate that compiles some sort of test
https://github.com/apache/datafusion/blob/main/datafusion/wasmtest
Can you help me understand what is different about the parquet viewer that isn't covered by the existing test?
| @@ -45,7 +45,7 @@ chrono = { version = "0.4", features = ["wasmbind"] } | |||
| # all the `std::fmt` and `std::panicking` infrastructure, so isn't great for | |||
| # code size when deploying. | |||
| console_error_panic_hook = { version = "0.1.1", optional = true } | |||
| datafusion = { workspace = true } | |||
| datafusion = { workspace = true, features = ["parquet"] } | |||
There was a problem hiding this comment.
Can you help me understand what is different about the parquet viewer that isn't covered by the existing test?
Here's the secret @alamb . I think the workspace datafusion has all features disabled. If we enable parquet feature here, we should be able to see compile error.
There was a problem hiding this comment.
Indeed you are right -- I verified I got a compile error without the code changes in this PR
andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion/datafusion/wasmtest$ wasm-pack build --dev
Compiling datafusion-datasource-csv v46.0.0 (/Users/andrewlamb/Software/datafusion/datafusion/datasource-csv)
error[E0432]: unresolved import `crate::file_format::coerce_file_schema_to_view_type`
--> datafusion/datasource-parquet/src/opener.rs:23:40
|
23 | coerce_file_schema_to_string_type, coerce_file_schema_to_view_type,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| |
| no `coerce_file_schema_to_view_type` in `file_format`
| help: a similar name exists in the module: `coerce_file_schema_to_string_type`
|
note: found an item that was configured out
--> datafusion/datasource-parquet/src/file_format.rs:470:8
|
470 | pub fn coerce_file_schema_to_view_type(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: the item is gated here
--> datafusion/datasource-parquet/src/file_format.rs:469:1
|
469 | #[cfg(not(target_arch = "wasm32"))]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error[E0425]: cannot find function `coerce_file_schema_to_view_type` in this scope
--> datafusion/datasource-parquet/src/file_format.rs:726:27
|
519 | / pub fn coerce_file_schema_to_string_type(
520 | | table_schema: &Schema,
521 | | file_schema: &Schema,
522 | | ) -> Option<Schema> {
... |
571 | | }
| |_- similarly named function `coerce_file_schema_to_string_type` defined here
...
726 | if let Some(merged) = coerce_file_schema_to_view_type(&table_schema, &file...
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `coerce_file_schema_to_string_type`
|
I have been thinking about this issue I plan to:
|
Sounds great! I can also try to upgrade DataFusion in parquet viewer before every major release to see if I can help capture anything. |
|
BTW I think we should consider fixing this in a 46.0.0 release. |
|
Filed a ticket to track adding coverage |
|
Thanks again @XiangpengHao |
Which issue does this PR close?
Rationale for this change
I encountered a compile error when trying to upgrade DataFusion to V46 for paruqet viewer.
I think the bug was accidentally introduced in #14464 cc @logan-keede
What changes are included in this PR?
Reverted the wasm32 gate as it should work for wasm32. I also enabled the
parquetfeature for wasm-test so that we can capture the bug in ci.I also removed the fs feature on tokio, as I don't think parquet datasource relies on it, which breaks wasm build.
Are these changes tested?
Are there any user-facing changes?