mdbook-latex: Add mdBook backend script.

[flexibeast]

Addresses #407.

This script makes use of pandoc and pdfunite, the latter being provided by the poppler package. There's some temporary special-casing to deal with issues that are addressed in #414 and #415

The resulting PDF has some layout problems (e.g. on pp. 9 and 10).

i imagine we might want to incorporate this script into build.sh, but i've kept it separate for now, whilst it's in development. (We could, of course, keep it separate, and simply call it from build.sh if/when required.)

[flexibeast]

#414 and #415 merged; removed special-casing.

[ericonr]

Can pandoc itself generate PDFs? I thought it needed LaTeX available somewhere on the system. That's awesome!

Since it's a bigger script, I don't see too much of an issue with running it separately. Will try it in a bit!

[flexibeast]

Oh, sorry, no, i installed texlive-20200406_2 to provide pdflatex for pandoc.

[ericonr]

No problem! We should add these deps to the readme, then.

[flexibeast]

PR updated with changes to README.

[ericonr]

Hmm I wonder if we could somehow append the markdown documents together instead... Besides the formatting issues you pointed out, the page numbers are a bit weird, and we don't have clickable links :/

[flexibeast]

@ericonr: Sure, i'll rework it to concatenate the .md files first; i imagine that would mean that pdfunite would no longer be required. This might also fix the page numbering issue.

Yeah, i'm not sure why there aren't internal links; maybe they need to be actively enabled via hyperrefoptions?

[ericonr]

I'm afraid it might make weird things to the headers by concatenating everything. Let's see what happens!

Have you tested the alternative mdbook backend for ebooks? Maybe we could print a PDF from that? I know it looked weird when I tried it, but idk if it was caused by foliate or by the backend.

[ericonr]

Btw, when you're done remember to add the generated files to .gitignore :)

[flexibeast]

No, the headers seem fine when concatenating the Markdown sources (as does the page numbering). The problem is that new sections don't appear on new pages, and inserting FF or FS characters in the Markdown don't work. i'm about to try converting to LaTeX, inserting LaTeX page breaks where appropriate, then running pandoc. i'd prefer not to use the EPUB backend unless we really need to, since pandoc is still going to need texlive installed to convert the EPUB to PDF, so i'd like to avoid a possibly-unnecessary dependency.

[ericonr]

The reason I would consider the ebook backend is exactly that it would have already done the processing to split the pages where necessary as well as make the links work. But if you manage to make it work without that, all the better!

[flexibeast]

Trying to put together a LaTeX file suitable for direct use by pdflatex, involving adding \documentclass{book}, various \usepackages and the manual definition of a command defined by pandoc (tightlist), isn't yet producing anything satisfactory. (Not only is the layout utterly broken in terms of horizontal space, but the hyperlinks are looking for individual .md files.) But trying to use mdbook-epub, i get:

Error: Incompatible mdbook version, expected 0.3.7 but got 0.4.0

:-(

[flexibeast]

(If the EPUB stuff doesn't work out either, i can work on a Perl mdBook backend to generate the necessary LaTeX from the JSON representation - i've actually made an exploratory start on that.)

[ericonr]

Hmm I think mdbook-epub hasn't been updated in a while :/

rust-lang/mdBook#88 has some stuff, including software to collate the markdown files!

[ericonr]

https://github.com/cetra3/mdcollate this one seems quite interesting!

[flexibeast]

Actually, my Perl backend is coming along nicely: here's an excerpt of the output it's producing. i'm in the middle of working out how to do internal links; once that's done, i can work on producing LaTeX lists and tables.

If you have no objections, i'd like to focus on continuing to develop this, as it seems promising: this backend could give us total control over the output, with the ability to do things like create a nice ToC, use the listings package for sh codeblocks, create an index, etc. WDYT?

[ericonr]

I don't have any issues, and having full control is indeed very interesting! Good luck c:

[flexibeast]

Okay, thanks. i've updated the PR to include the mdbook-latex backend script, rather than pdf.sh, and changed the PR title accordingly.

[ericonr]

Oh, you're doing it the way we had discussed it a while ago, right? Using the JSON output of mdbook! Nice.

[flexibeast]

PR updated. Running pdflatex on the handbook.tex file in book/latex/ completes, once one removes the backslash from:

grep -m1 '(II) modeset([0-9]\+):' /var/log/Xorg.0.log

in xorg.md (which i believe shouldn't be there anyway).

There are very obviously many visual issues still to address, and although external links are (i believe) working, internal links are not, though the infrastructure for that is now basically in place.

[ericonr]

Would you like to make this into a separate packaged project or should it be kept in-tree? I can see the case for making it a package with all the dependencies and then we could talk to @the-maldridge about deploying the pdf in the mirror as well, for people to download.

[ericonr]

It would also help with PATH issues, since it's complaining about not finding the latex backend.

[flexibeast]

i was wondering about this also. My thinking has been that it should perhaps be kept in void-docs, as the code is (at least at this point) quite specific to the Void Handbook, and how we do things. Otoh, the arguments for a distinct repo under void-linux make sense as well.

Re. the PATH issue, yes, i've just been adding the path to my local void-docs repo to PATH. Obviously if it's a separate package, this is not an issue, but if we keep it in-tree, we'll need to work out how to best address this issue.

[ericonr]

Checklist time:

• enable code block wrap
• find out why the margins change every other page (might be because it's configured for a notebook style thing?)
• ideally fix tables as well
• code blocks have a weird backslash: \# f d i s k − l Disk / dev / sda
• local links

This is pretty cool already, though! Thanks :D

[flexibeast]

@ericonr: i believe i might have solved the codeblocks issue in general. :-D As a bonus, the style guide no longer requires special-casing overall (though i can spot at least one small issue). Could you please confirm? i've just pushed an update.

[ericonr]

Well, one can just pass the -output-directory= option to pdflatex.

That works, but I didn't know about it.

(Maybe the backend is supposed to handle excludes?)

Yes, sorry. I should have made that clear. I guess it's no longer needed, though :D

[ericonr]

(though i can spot at least one small issue)

I'm guessing the upside down interrogation mark?

i believe i might have solved the codeblocks issue in general

Indeed!

I've just noticed that the index is a bit off, e.g. the Wayland section should be under Graphical Session, not on the same level.

[flexibeast]

I'm guessing the upside down interrogation mark?

Yeah. i'll look into that.

Indeed!

Excellent! i'll remove the style-guide.tex file from this PR, and create a PR to revert #430.

I've just noticed that the index is a bit off, e.g. the Wayland section should be under Graphical Session, not on the same level.

i noticed something similar re. the "Hashboot" and "AppArmor" sections. i'll look into that too.

So, other things still left to do:

• The problem with the "If this is not run after installation ..." para in "Hashboot" seems to be partially addressed by hashboot: Remove unnecessary blockquoting. #433, but not completely - i don't know what's going on there yet, the LaTeX looks reasonable. Not sure if you want to make this a blocker.
• The Tor mirrors table is definitely indented too far.

Anything else? Have i forgotten anything?

[ericonr]

The problem with the "If this is not run after installation ..." para in "Hashboot" seems to be partially addressed by #433, but not completely - i don't know what's going on there yet, the LaTeX looks reasonable. Not sure if you want to make this a blocker.

Not a blocker, I believe.

For the build command:

diff --git a/build.sh b/build.sh
index 6a8ed10..9050e6a 100755
--- a/build.sh
+++ b/build.sh
@@ -26,5 +26,5 @@ pandoc \
     -o "void-docs.7" "void-docs.md"
 
 # Build PDF
-pdflatex -output-directory=book/latex/ book/latex/handbook.tex
-pdflatex -output-directory=book/latex/ book/latex/handbook.tex
+pdflatex -output-directory=book/latex/ book/latex/handbook.tex >/dev/null
+pdflatex -output-directory=book/latex/ book/latex/handbook.tex >/dev/null

In live-images/partition, < 2GB is being shown as ! 2 GB.

Oh, the default groups table is a bit screwed D:

The Intel video acceleration one is too.

[flexibeast]

Good point re. the >/dev/nulls - i've just added them locally.

Thanks for noting the other issues - i've added them to my TODO. :-)

[flexibeast]

Fixed:

• "< 2GB" being shown as "! 2 GB";
• the default groups table;
• the Intel video acceleration table;
• the style guide example.

Re. the Tor mirrors table, the LaTeX looks fine; i don't understand what's going on. Would it be okay to leave it for now, and come back to it later, as with that "Hashboot" para?

Re. the ToC. The issue is that the \tableofcontents looks for LaTeX \sections for the top level, and each Handbook section (i.e. Web page) is a 'section' in this context. In other words, unlike the HTML backend, it's not creating a ToC from the directory structure. i imagine we could manually replicate the Web site's ToC layout, though of course this would need to be manually changed every time the source structure is changed. Is this a blocker? If not, i can come back to this later. If so, i'll get to work. :-)

EDIT: Oh actually, i guess i could try to write code to create the ToC on the basis of the directory structure. So if this is indeed a blocker, that's the approach i'll try taking.

[ericonr]

I don't think any of these issues are blockers! So we can get to them in time.

[flexibeast]

Okay. :-) So is there anything else that needs to be done, or do you think this might be ready to merge?

[ericonr]

I don't think so. Might want to wait a bit to see if anyone has any comments. I guess we could list the to-do items in #407 ?

[ericonr]

handbook.pdf

If anyone wants to try it out :D

[ericonr]

I guess the only thing left to decide is regarding deployment, but this can be solved once we have deployment.

[flexibeast]

@ericonr: So do you think we should we wait longer for any feedback, or can we now merge?

[ericonr]

I asked @bobertlo to take a final look before merging, so it would be nice to wait for him.

Could you open an issue with the issues that still need to be solved in the generated PDF?

[flexibeast]

I asked @bobertlo to take a final look before merging, so it would be nice to wait for him.

Sure.

Could you open an issue with the issues that still need to be solved in the generated PDF?

i added them in a comment on #407, as you suggested here; do you think it would be better to have a distinct issue for them?

[ericonr]

i added them in a comment on #407, as you suggested here; do you think it would be better to have a distinct issue for them?

You can see I was making this comment at 4 in the morning :P Yes, you did list them there, and yes, it is a great place to put them. Sorry for the noise!

[bobertlo]

I don't see any blockers on this. I have opinions on improvements that I will put somewhere else but I'm really surprised at the quality of this after being away for so long. I want to print this and bind it!

[flexibeast]

@bobertlo: Thank you. :-) Merging.
@ericonr: Any thoughts on deployment? i'm guessing some changes need to be made to mdbook_build.sh.j2 in the mdbook Ansible role?