Some enhancements for using Julia in Quarto documents

[dmbates]

• I have a PR, Allow for .qmd extension and {julia} code blocks. domluna/JuliaFormatter.jl#570, in for JuliaFormatter to format the code in julia-based .qmd files. All the heavy work was already done, I just allowed a .qmd extension in addition to .jmd and allowed for braces around the julia specification for code blocks
• for some time the RCall package package has provided for R graphics to be displayed in a Jupyter notebook with a Julia kernel. It is almost ready for Quarto. At present the return value is interpreted by quarto as 2 sub-figures instead of 1 figure. There is an example in notebooks/mmt_sleepstudy.qmd in the https://github.com/RePsychLing/SMLP2022 repository. It can probably be fixed in RCall.jl If not, I will create a stand-alone example.
• ProgressMeter.jl also has special provision for Jupyter notebooks which almost works for Quarto documents. It seems to leave a few vestigial control codes in the output right now.

[jjallaire]

I'll try out the RCall example to see what might be going on with the sub-figures. In advance of this, if you add keep-ipynb: true and post the resulting ipynb here I can probably diagnose just as well that way.

We just made a change yesterday to strip ansi codes from Jupyter stream output -- perhaps that resolves the ProgressMeter.jl issue?

One other thing I'm hoping to see improved is high-resolution plots for printed output. You can see the current state of this here: JuliaLang/IJulia.jl#1035 I think either PDF figures should be made to work OR there should be an option to raise the DPI on PNG output without making the legend text smaller.

[jjallaire]

Attempted to repro the RCall issue but the render hangs for me on the RCall cell. My guess is though that if Jupyer is producing two plots that we can't easily overcome this (as we just literally replay the jupyter outputs as-is). If you post the ipynb I can confirm this -- in any case hopefully this is something that can be addressed in RCall.

[dmbates]

I hadn't meant to suggest that this was a problem with Quarto. I think the problem is the way that the R graphics image is being displayed and it may be a rather trivial fix in RCall.jl.

What is happening right now is that image is displayed and the return value, which is displayed as RObject{VecSxp}, is also returned. Unfortunately, if printing the return value is suppressed then the image is not displayed. I think the issue can be resolved in RCall.jl by explicitly returning nothing when a graphics file is displayed.

I added the .ipynb file to the repository in case you want to take a look at what happens now but don't feel that you should take action on it.

[dmbates]

@jjallaire I updated to quarto 0.9.326 and the problems with the control codes for the ProgressMeter output went away. Thanks for that - we use it to show the progress of the iterations when fitting a model and now the document contains the final summary of speed in a more-easily understood form.

[jjallaire]

Okay, great to hear!

The returning nothing behavior for RCall.jl sounds right to me.

[dmbates]

Unfortunately, returning nothing doesn't do what I wanted. I will need to delve into the post-eval hooks in the IJulia package.

[dmbates]

I have revived the discussion of displaying plots from R in a quarto/jupyter document with a Julia kernel using RCall.jl, JuliaInterop/RCall.jl#450. IIRC the ggplot2 and lattice packages both generate their plots when R's print method is invoked on the plot object. The way RCall is currently set up to display plots under Jupyter, the display of the "printed" value is already underway before we know if plots have been created through calls to the R instance. In other words, there is already a "print" of some results underway before we discover that we should be displaying plots instead.

I was successful in suppressing the printing of RObject{VecSxp} but Jupyter and the IJulia package still end up blank line of output, which seems to confuse quarto. I would create a gist with a Jupyter notebook but I have forgotten how to upload a file to a gist and after an hour of flailing around trying to do it, I have given up. (I know that I am supposed to be able to drag and drop files into the text block of a new gist but in my case it just pastes the file://... url and I really hate drag and drop interfaces.)

[jjallaire]

I think if you just send us a link to the ipynb within a regular Git repo we should be able to take a closer look.

Unrelated note: hoping to find links to some good Julia examples for my juliacon talk: #1261. @dmbates I went looking for your mixed-effects models book but didn't find it straight away. Is that published anywhere yet? Is the source code also available?

[dmbates]

The gist containing the .qmd file is https://gist.github.com/dmbates/87c56abd816045ceaad2564dbadeefcd

Two "in-progress" presentations using Quarto/Julia are https://crsl4.github.io/julia-workshop/ and https://repsychling.github.io/SMLP2022/

I am about to make the repository for the book public. The link will be https://juliamixedmodels.github.io/EmbraceUncertainty once I release it into the wide world. I said "end of June" so I best get a move on.

[dmbates]

EmbraceUncertainty is now live, but, like the other sites, still in-progress.

[jjallaire]

Great, thanks! I will definitely show that off in the talk.

[dmbates]

The display of R graphics in a julia-based Jupyter file through RCall is seen in the first figure of https://repsychling.github.io/SMLP2022/sleepstudy.html

I think what is happening is that the jupyter kernel passes an IOStream for the output from the code block. As a side-effect of the "printing" of the result in that stream RCall::ijulia_displayplots() is called which displays the "png" or "svg" files, if any have been created. However, the original IOStream is still around and, even if nothing it printed to it, it still gets returned. At least I think that is what is happening. @simonbyrne and I have been discussing this in JuliaInterop/RCall.jl#450 but neither of us remembers the details well. I guess what I should do is figure out what IJulia does to set up and take down that IOStream and, perhaps, what it does when the code block ends with ;, which suppresses output.

[dmbates]

I have been able to resolve the issue of having ggplot2 plots displayed in a Julia quarto document using RCall. See https://gist.github.com/dmbates/87c56abd816045ceaad2564dbadeefcd

The creation of the graphics output file (an SVG file in this case) is triggered when the ggplot2 object is printed. The fix is to do the call to print in R and then suppress the printed output from the julia cell by appending a ; after the @R_str macro call. The RCall.jl package has already registered an IJulia post-execution hook that looks for the graphics output files and displays them but the timing is off unless the call to print happens in the R code.

Resolves JuliaInterop/RCall.jl#450