[reST] refactor

[jrappen]

Note

This PR is mainly meant to make future work easier, not necessarily
fix stuff now. If you have suggestions for stuff that should be fixed
now, leave a comment below.

---

current TODOs

• compare comments in syntax file
• add tests to prove linked issues can be closed

---

Additions:

• correctly hightlight directive options and their values as
  mapping.pair while checking value format
• default keymap
• symbol index lists implicit hyperlink targets (sections,
  footnotes, citations)
• highlight embedded python and raw-code directives (css, js, json,
  jsonc, html, python, toml, yaml), compare [reStructuredText] Code-blocks are not handled #3158
• explicit hyperlink targets and anonymous explicit hyperlink
  targets
• correctly highlight grid tables with optional inline markup
• correctly highlight inline interpreted text, optionally
  with roles
• correctly highlight substitution definitions
• completions with strict scope limits
• indentation settings based upon official docs
• neg. tests for comments
• highlight empty comments and section separators between empty
  lines

Fixes:

• line blocks (with optional inline markup)
• footnotes and citations
• pop explicit markup blocks before the next explicit markup block
• correctly highlight block comments
• include inline markup within block quotes
• allowed symbols for heading markers, compare [reStructuredText] Headers using some of the supported characters aren't highlighted #2894
• inline internal targets may contain symbol characters, compare
  [RestructuredText] single quotes break link targets #1204
• inline hyperlink references
• section headings with or without adornments
• literal blocks meta scope
• enumerated lists
• field lists
• punctuation of inline substitution references
• tests for embedded python

Changes:

• the syntax file now uses branching to correctly highlight
  section headings and their punctuation
• scope names have been adjusted with regard to the syntax
  scope naming guide
• match punctuation.definition.end rules for inline items
  before invalid.illegal.newline
• pop line blocks on empty line
• pop explicit markup blocks before next explicit markup block
• moved some rules from block-quote-block context to literal-block
• removed inline markup from literal blocks
• updated list of allowed characters for internal link labels,
  compare [reST] Internal link labels should allow more character types #793
• changed auto_complete_selector default
• re-ordered test file to match order of syntax file
• include linebreak in match for first line of directives
• differentiate known (constant.language) and unknown (constant.other)
  directive names

References:

• fixes [reST] Internal link labels should allow more character types #793
• fixes [RestructuredText] single quotes break link targets #1204
• fixes [reStructuredText] Headers using some of the supported characters aren't highlighted #2894
• fixes [reStructuredText] Code-blocks are not handled #3158
• duplicate the work @deathaxe had done in [Common] Add git conflict marker highlighting #4047, so his changes to the
  syntax don't get lost

Thanks to these contributors:

• deathaxe [email protected]
• Giampaolo Rodola [email protected]
• Victor Wheeler [email protected]

[giampaolo]

Thanks a lot for taking care of this.

• If possible, could you also add syntax highlighting for these kind of links?

  This is a paragraph `Process.exe()`_.
  
  .. _`Process.exe()`: https://psutil.readthedocs.io/en/latest/#psutil.Process.exe

Example file having lots of these: https://raw.githubusercontent.com/giampaolo/psutil/refs/heads/master/HISTORY.rst.

[vwheeler63]

Hello, @jrappen !

Good morning! And thank you for the invitation in sublimetext-io/docs.sublimetext.io#131.

My name is Vic.

I have been implementing these reST commands as I have found the need for them in my work and writing.

import sublime
import sublime_plugin


class VicsRstEncapsulateFieldCommand(sublime_plugin.TextCommand):
    """
    If (conditions applied in keymap):
    - we are in a reStructuredText source file, and
    - all selections have some text selected

    then wrap the selected text like this:

        :selected text:
    """
    def run(self, edit):
        sel_list = self.view.sel()

        # Replace each selection with edited string.
        for rgn in reversed(sel_list):
            selected_text = self.view.substr(rgn)
            new_text = ':' + selected_text + ':'
            self.view.replace(edit, rgn, new_text)

        # De-select each selection, leaving cursor on right edge of its
        # previous selection.  This avoids the danger of a stray keystroke
        # wiping out the new text.
        for i, rgn in enumerate(sel_list):
            del sel_list[i]
            sel_list.add(sublime.Region(rgn.end()))


class VicsRstEncapsulateLiteralCommand(sublime_plugin.TextCommand):
    """
    If (conditions applied in keymap):
    - we are in a reStructuredText source file, and
    - all selections have some text selected

    then wrap the selected text like this:

        ``selected text``
    """
    def run(self, edit):
        sel_list = self.view.sel()

        # Replace each selection with edited string.
        for rgn in reversed(sel_list):
            selected_text = self.view.substr(rgn)
            new_text = '``' + selected_text + '``'
            self.view.replace(edit, rgn, new_text)

        # De-select each selection, leaving cursor on right edge of its
        # previous selection.  This avoids the danger of a stray keystroke
        # wiping out the new text.
        for i, rgn in enumerate(sel_list):
            del sel_list[i]
            sel_list.add(sublime.Region(rgn.end()))


class VicsRstEncapsulateInterpretedTextRoleCommand(sublime_plugin.TextCommand):
    """
    If (conditions applied in keymap):
    - we are in a reStructuredText source file, and
    - all selections have some text selected

    then wrap the selected text like this:

        :role_name:`selected text`
    """
    def run(self, edit, role_name):
        sel_list = self.view.sel()

        # Replace each selection with edited string.
        for rgn in reversed(sel_list):
            selected_text = self.view.substr(rgn)
            new_text = f':{role_name}:`{selected_text}`'
            self.view.replace(edit, rgn, new_text)

        # De-select each selection, leaving cursor on right edge of its
        # previous selection.  This avoids the danger of a stray keystroke
        # wiping out the new text.
        for i, rgn in enumerate(sel_list):
            del sel_list[i]
            sel_list.add(sublime.Region(rgn.end()))

    // --------------------------------------------------------------------
    // reStructuredText Tools
    // --------------------------------------------------------------------
    {
        "keys": ["ctrl+alt+f"],
        "command": "vics_rst_encapsulate_field",
        "context":
        [
            { "key": "selector", "operand": "text.restructuredtext"},
            { "key": "selection_empty", "operand": false, "match_all": true },
        ]
    },
    {
        "keys": ["ctrl+alt+l"],  // Lower-case 'L'
        "command": "vics_rst_encapsulate_literal",
        "context":
        [
            { "key": "selector", "operand": "text.restructuredtext"},
            { "key": "selection_empty", "operand": false, "match_all": true },
        ]
    },
    {
        "keys": ["ctrl+alt+t"],
        "command": "vics_rst_encapsulate_interpreted_text_role",
        "args": {
            "role_name": "term"
        },
        "context":
        [
            { "key": "selector", "operand": "text.restructuredtext"},
            { "key": "selection_empty", "operand": false, "match_all": true },
        ]
    },

Use Case for vics_rst_encapsulate_field

This happens a lot working on other people's writings when they use (or over-use) unordered lists, or even worse, definition lists, that make what SHOULD be a nice, compact 8-item list of things into a pages-long thing that is more difficult to read. Then I convert these to field lists, which, in the RTD theme, makes a nice, compact, very readable list.

Use Case for vics_rst_encapsulate_literal

Occasionally I see something like FILE PATHS in a document are inconsistently marked—some as literals, some not—and I go through, say, 100 pages and make them all literals for consistency.

Use Case for vics_rst_encapsulate_interpreted_text_role

While building a glossary of a large work recently (over 100 pages), I lost count of how many times vics_rst_encapsulate_interpreted_text_role got used to create

:term:`word_or_phrase_in_glossary`

And that's just in 1 "document"!. It saved TONS of otherwise-error-prone keystrokes!

I expect more of these interpreted-text roles to come in the near future. I know vics_rst_encapsulate_interpreted_text_role is long, but I used it specifically so that people new to reST will be learning the correct terminology and so will know what to look up when they are learning reST

---

Helping Others Learn reST

I noticed you listed 3 useful reST learning and reference links in the Preferences > Package Settings > reStructuredText (reST) > Documentation portion of your reST menu.

I have accumulated a few more Docutils reference that I find I use regularly.

Additional Thoughts

reST alone, for any serious work, is kind of "bare-bones" without Sphinx. Sphinx should be very interesting to Sublime Text users because it is Python based, and like Sublime Text, is very customizable, and outputs in [html, dirhtml, singlehtml, json, htmlhelp, qthelp, devhelp, epub, latex, text, man, texinfo, xml, pseudoxml], and can even do external hyperlink checks, among other things.

- Documentation
    - Docutils (Basic reST)
        - [reST Introduction](https://docutils.sourceforge.io/docs/ref/rst/introduction.html)
        - [reST Primer](https://docutils.sourceforge.io/docs/user/rst/quickstart.html)
        - Reference
            - [Markup Specification](https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html)
            - [Tables](https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#tables)
            - [Substitution](https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#substitution-references)
        - [Directives](https://docutils.sourceforge.io/docs/ref/rst/directives.html)
        - [Interpreted Text Roles](https://docutils.sourceforge.io/docs/ref/rst/roles.html)
        - [Examples](https://docutils.sourceforge.io/docs/user/rst/demo.html)
        - [Quick Ref](https://docutils.sourceforge.io/docs/user/rst/quickref.html)
    - Sphinx (Sphinx-Extended reST; Extends Docutils Basics)
        - [Command Line Tools](https://www.sphinx-doc.org/en/master/man/index.html)
        - [sphinx-build Command Line](https://www.sphinx-doc.org/en/master/man/sphinx-build.html)
        - [Main Documentation Hub](https://www.sphinx-doc.org/en/master/)
        - [Configuration](https://www.sphinx-doc.org/en/master/usage/configuration.html)
        - [Directives](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html)
        - [Directives—Admonitions](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#admonitions-messages-and-warnings)
        - [Hyperlinks](https://www.sphinx-doc.org/en/master/usage/referencing.html)
        - [Interpreted Text Roles](https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html)
        - [Glossary](https://www.sphinx-doc.org/en/master/glossary.html#term-environment)
        - [Supported Programming Languages](https://pygments.org/docs/lexers/)
        - Themes
            - [Sphinx Themes Gallery](https://sphinx-themes.readthedocs.io/en/latest/)
            - [Customizing Output Using HTML Templates](https://www.sphinx-doc.org/en/master/development/html_themes/templating.html)

And between the Docutils and Sphinx documentation, the subject of hyperlinks is SO POORLY DOCUMENTED that I wrote my own reST + Sphinx Hyperlink Documentation here—which, IMO, is FAR easier to understand. (There are about 12 ways to create hyperlinks, and only about 8 of them are useful.)

I hope this helps.

Kind regards,
Vic

[vwheeler63]

@vwheeler63 I cleaned up our discussion a bit, as it makes it easier for me to read while going through TODOs here. I hope you don't mind.

Not at all.

@vwheeler63 Do you know of a page that lists Sphinx's additions to reST?

It's not in one place, but what the Sphinx doc guys did is they let the Docutils documentation handle the basics, and the Sphinx docs (e.g. Directives, Interpreted Text Roles, etc.) only documents things that Sphinx added (or significantly changed), which IMO, VERY much helps the reader understand what Sphinx contributed to the reST world, as well as where to look for the documentation, because in literally 99% of the cases, Sphinx didn't change any of the Docutils basics, but only ADDED to the functionality with additional directives. (Sphinx harnesses the Docutils engine internally, which was brilliantly designed for extension.)

[jrappen]

directive options are now highlighted

[jrappen]

⚠️ Current block quotes issues:

• attributions are not highlighted correctly
• can continue with same indentation past empty lines
• optional whitespace after attribution punctuation
• can have body elements

https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#block-quotes

[vwheeler63]

@jrappen May I make a suggestion?

Wherever you aren't already doing so, I suggest OR-ing in the following scope into the "context" conditions wherever they apply:

source.python comment.block.documentation.python

😄 I was just editing the header block comment of a significant Python, putting reST in it, and thought of this PR....

Thus...

{ "key": "selector", "operand": "(text.restructuredtext | source.python comment.block.documentation.python) - markup.raw" }

It is my understanding that grouping using parentheses works, but now I can't find the documentation on it....

[jrappen]

What is not working as expected? Can you give a code example? I don't understand what you want to do.

[deathaxe]

I guess he want's reST highlighting in python documentation comments.

This would require to remove various restrictions with regards to leading whitespace on the one hand and may cause confusion in python comments on the other hand.

Even though RST is still used for Sphinx, I've also already seen Markdown.

I wouldn't add highlighting in comments by default.

[vwheeler63]

For what it's worth, this maintainer of CPython confirms that the """ block comments in Python files are indeed reStructuredText.

[vwheeler63]

I wouldn't add highlighting in comments by default.

Confirmed, it's not the highlighting I am after, but the Key Bindings would be highly useful!

Edit:

...and completions and whatever else is applicable. Not highlighting.

[vwheeler63]

To illustrate, this is in the documentation section of one of my .py files:

Data Flow
---------

.. code-block:: text

    Inputs              Generated Source Files             Output
    -----------         ----------------------       ----------------------
    ./docs/src/   \
    ./src/         >===> ./docs/intermediate/  ===>  ./docs/build/<format>/
    ./examples/   /

    Once ./docs/intermediate/ is built, you can use all the Sphinx output
    formats, e.g.

    - make html
    - make latex
    - make man
    - make htmlhelp
    - etc.

[jrappen]

... I guess he wants reST highlighting in Python documentation comments ...

I won't be addressing this in this PR.

[deathaxe]

It may be useful/worth thinking about options for RST to not be too restrictive with regards to leading whitespace, which would enable it being embedded in other syntaxes seemlessly.

Markdown for instance ships an indented-markdown context, which is used wherever it is embedded into a syntax. HAML, Liquid, Astro and various other syntaxes make use of it to enable full fledged markdown highlighting even in indented code blocks.

If reST would provide something similar, it would probably rather easy for end users to create a Python (Sphinx) syntax adding reST highlighting in documentation strings.

...and completions and whatever else is applicable. Not highlighting.

Well, that's what you need to embed a syntax for, normally, as completions, syntax-specific key bindings etc. rely on certain scope, provided by syntax definitions.

[vwheeler63]

It may be useful/worth thinking about options for RST to not be too restrictive with regards to leading whitespace, which would enable it being embedded in other syntaxes seemlessly.

All reST markup works no matter how indented it is (though I'm not sure about section headings) -- all other stuff I work with regularly. This is by design in Docutils. Example: in a .. note:: directive, you can put a whole document, and syntax highlighting, links, nested lists, etc. etc.. All indented. (Note that footnotes and citations don't work that way though -- I sometimes wish they did. I think it's a bug.)

FYI, I don't need the highlighting in Python comments at all, but the other functionality (Completions/Snippets and other utilities) are very welcome. Example: the 3 I submitted above.

[vwheeler63]

Note: In testing, I am finding this key-binding selector very helpful. (The 3 commands I submitted above require selections to be non-empty.)

{ "key": "selector", "operand": "text.restructuredtext | source.python comment | source.python comment.block.documentation.python"},

[vwheeler63]

@jrappen I just found a bug with the existing reST highlighting in case you didn't already handle it:

5 back-ticks are actually legal in Docutils and simply represent a "literal backtick" character.

The current highlighting in ST build 4200 does "literal" highlighting on only the first 4 and leaves the 5th one "dangling". If you replace the middle (3rd) backtick with any other character, the highlighting is done correctly.

Hope this helps.