[Windows] Is it possible to include Rust toolchain in Rtools?

[yutannihilation]

Currently libR-sys requires users to install standalone MSYS2 for libclang, but as extendr/extendr#13 suggests, it might be good to use Rtools' MSYS2 for simplicity.

Rtools' MSYS uses a dedicated repository, which has only the limited number of packages available. To add some packages there, we can send a pull request to r-windows/rtools-packages repo, according to this document: System Libraries in Rtools40.

[clauswilke]

Reading through the documents for Rtools, I wonder whether we may have to start a conversation with the maintainers first.

In this section it says that we can only contribute static libraries, but that probably won't work for libclang.

In this section it says that we can alternatively provide additional build tools but those should only be used to build more packman packages. But we need additional build tools to be used during the R package build.

@yutannihilation Do you want to initiate this conversation with the Rtools maintainers?

[yutannihilation]

In this section it says that we can only contribute static libraries, but that probably won't work for libclang.

Thanks, I didn't notice this... After some thinking, I now feel this is less important.

• For Windows user who just use CRAN packages, they can install CRAN precompiled binaries, so this won't affect them.
• For Windows user who try non-CRAN pakcages, they might be able to use a prebuilt static library (which seems undesirable according to Compiling on Windows extendr#10 (comment)). If we don't choose this approach, we might want to use Rtools' MSYS2 for the ease of installation.
• For Windows developers, probably they need to install Rust toolchain anyway for debugging etc.

Let's hold this issue on hold for a while.

[clauswilke]

Yes, I think we're good at this time. It might be more important to add Rust than libclang.

I've also been thinking that long-term, if this project gets widely used, we should try to convince the R core team to distribute the bindings as part of their official distribution. In this way we'd know that we always have the correct bindings for each R version and platform.

[Ilia-Kosenkov]

Rtools cannot be used for libclang. However, it can be used with precomputed bindings, see discussion in
extendr/rextendr#19.

[yutannihilation]

Not sure if this issue is worth keeping open, but I removed libclang from the issue title because we basically agreed it's not very important.

[Ilia-Kosenkov]

@yutannihilation, it looks like we have got the new Rtools version, like, today.
https://cran.r-project.org/bin/windows/Rtools/
Here is also a PR to bring the v2 to chocolatey
https://github.com/facorread/rtools-chocolatey/pull/3/files
I am going to check if it contains fixes we need and if it works as expected.
Perhaps the CI already pulls the latest Rtools version.

[yutannihilation]

Oh, cool. It might be just that they moved the artifacts from Bintray to somewhere because Bintray is retiring (I saw some commits related to this on r-windows/rtools-base recently), but it's worth checking!

[Ilia-Kosenkov]

@yutannihilation, I checked that it contains our fix, so now I am working to reproduce the errors we had before to see if this fix actually helps. I'll post updates here and in extendr/rextendr#91.

[Ilia-Kosenkov]

FYI, the CI pulls Rtools by hard-coded name, so unless they change it we won't get a new version:
https://github.com/r-lib/actions/blob/0d6c4b3efe82090bc03e99c585d9e89003117931/setup-r/src/installer.ts#L330-L336

[yutannihilation]

Can we specify rtools-version: 40v2?

https://github.com/r-lib/actions/blob/18d8b3cd131edfa357583fc071e138e82abdb0a9/setup-r/action.yml#L8-L10

[Ilia-Kosenkov]

@yutannihilation, I think we can because the version string is inserted as-is in the tools downloading URL.

[yutannihilation]

I created a pull request: r-lib/actions#287. Hope this works...

[Ilia-Kosenkov]

I am struggling to make the CI work. It not only does not seem to care about Rtools version, it also runs incorrect snapshot tests, taking the snapshot for use_extendr from yet unmerged PR (https://github.com/extendr/rextendr/blob/233eaf861d21ecb749ae3954cc89fe9f0369192a/tests/testthat/_snaps/use_extendr.md).
I think there is some sort of heavy caching that I cannot beat.

Locally though it seems to work fine.

[Ilia-Kosenkov]

With old Rtools:

r_path <- normalizePath(dirname(system2("where", "R.exe", stdout = TRUE)[1]))
cargo_path <- normalizePath(dirname(system2("where", "cargo.exe", stdout = TRUE)[1]))
rtools_home <- normalizePath("C:\\rtools40") # Point this to a correct Rtools
rtools_path <- normalizePath(paste(rtools_home, "usr\\bin", sep = "\\"))

old_path <- Sys.getenv("PATH")
new_path <- paste(r_path, cargo_path, rtools_path, sep = ";")
Sys.setenv(PATH = new_path)
Sys.setenv(RTOOLS40_HOME = rtools_home)
rextendr::rust_function("fn add(a:i32, b:i32) -> i32 {a + b}", toolchain = "stable-x86_64-pc-windows-msvc")
#> i build directory: C:\Users\[redacted]\AppData\Local\Temp\RtmpQZ4isS\file13c42b8519a5
#> Error: Rust code could not be compiled successfully. Aborting.
add(42L, 100500L)
#> Error in add(42L, 100500L): could not find function "add"

^{Created on 2021-05-04 by the reprex package (v2.0.0)}

With new Rtools:

r_path <- normalizePath(dirname(system2("where", "R.exe", stdout = TRUE)[1]))
cargo_path <- normalizePath(dirname(system2("where", "cargo.exe", stdout = TRUE)[1]))
rtools_home <- normalizePath("C:\\rtools40_test") # Point this to a correct Rtools
rtools_path <- normalizePath(paste(rtools_home, "usr\\bin", sep = "\\"))

old_path <- Sys.getenv("PATH")
new_path <- paste(r_path, cargo_path, rtools_path, sep = ";")
Sys.setenv(PATH = new_path)
Sys.setenv(RTOOLS40_HOME = rtools_home)
rextendr::rust_function("fn add(a:i32, b:i32) -> i32 {a + b}", toolchain = "stable-x86_64-pc-windows-msvc")
#> i build directory: C:\Users\[redacted]\AppData\Local\Temp\RtmpuCsL8P\file251c1567143f
#> v Writing file C:/Users/[redacted]/AppData/Local/Temp/RtmpuCsL8P/file251c1567143f/target/extendr_wrappers.R.
add(42L, 100500L)
#> [1] 100542

^{Created on 2021-05-04 by the reprex package (v2.0.0)}

[clauswilke]

One of the major annoyances of github actions that should be easy to fix but for some reason they don't is that you can't manually clear the cache. See here on how it can be done:
https://github.community/t/how-to-clear-cache-in-github-actions/129038

[Ilia-Kosenkov]

I have finally resolved all the issues. The faulty test was due to my local outdated version of {cli}, after updating it I got the new snapshot into the CI.
The tests are run using nearly-empty PATH, which is set to point to binaries of R, cargo, and rtools' /usr/bin (see extendr/rextendr#91 (comment)).
Here are the results:

• Failing R CMD check using old rtools40
  https://github.com/Ilia-Kosenkov/rextendr/runs/2501911452?check_suite_focus=true#step:9:314
• Successful R CMD check using new rtools40v2
  https://github.com/Ilia-Kosenkov/rextendr/runs/2501911453?check_suite_focus=true#step:10:182
  (the error is actually due to a warning issued in the absence of qpdf...)

There are some problems with pulling the latest rtools40v2, see discussion here:
r-lib/actions#287

[andy-thomason]

Pleased to hear this is going well.

…

On Tue, 4 May 2021, 16:55 Ilia, ***@***.***> wrote: I have finally resolved all the issues. The faulty test was due to my local outdated version of {cli}, after updating it I got the new snapshot into the CI. The tests are run using nearly-empty PATH, which is set to point to binaries of R, cargo, and rtools' /usr/bin (see extendr/rextendr#91 (comment) <extendr/rextendr#91 (comment)>). Here are the results: - Failing R CMD check using old rtools40 https://github.com/Ilia-Kosenkov/rextendr/runs/2501911452?check_suite_focus=true#step:9:314 - Successful R CMD check using new rtools40v2 https://github.com/Ilia-Kosenkov/rextendr/runs/2501911453?check_suite_focus=true#step:10:182 (the error is actually due to a warning issued in the absence of qpdf...) There are some problems with pulling the latest rtools40v2, see discussion here: r-lib/actions#287 <r-lib/actions#287> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAL36XCZNVKY5GLTD5GHZXLTMAKF5ANCNFSM4UVG2TPQ> .

[yutannihilation]

It seems it's not impossible to add Rust to Rtools. This recent email from Tomas Kalibera clarifies how to add a new library.

https://stat.ethz.ch/pipermail/r-devel/2021-December/081382.html

It's probably not very easy, though. There's an existing issue about adding Rust on the upstream (MXE) repository, but it seems there has been little discussion so far.

mxe/mxe#2499

[yutannihilation]

I'm closing this because

• the CRAN Windows machines now have Rust installed, so it seems this is not necessary to release packages on CRAN; to me, this means they decided not to include Rust in Rtools.
• non-CRAN packages can release the binary on R-universe, so the users can install it without Rust toolchain on their local

Feel free to reopen or file a new issue if necessary!