Proposal: RDoc extension system

[okuramasafumi]

Abstract

I'd like to propose an extension system for RDoc.

YARD has been popular for years and new comment usages such as rbs-inline has been introduced. We need to organize efforts around comment space so that we don't have to duplicate our effort.

The important thing is that RDoc is currently the only library bundled with Ruby that handles comments. RDoc is used to parse CRuby's C comment so it's not realistic to remove it in near future. Therefore, I think RDoc is a good place for collecting efforts around comments and documentation.

Then an extension system comes in. Because we have different needs. When each extension is implemented as a gem, it's possible for other gems to bundle their extension to extend RDoc behavior. For example, language servers can extend RDoc so that it can be used with editors.

Details

RDoc has three main components: parsers, store and generators. It would be beneficial to be able to extend all of them.

Parsers https://github.com/ruby/rdoc/tree/master/lib/rdoc/parser

One of my concerns is that RDoc format is not used anymore. As far as I know, most gems use YARD format. Currently, if a language server wants to implement documentation feature, it needs to adopt both RDoc format and YARD format.

So I'd like to add YARD parser to RDoc. While this could be implemented directly, it's possible to have other new formats with extensions.

This might be trivial with something like below:

class MyParser
end

RDoc::Parser.add MyParser, :my_parser
RDoc::Parser.find :my_parser

Store https://github.com/ruby/rdoc/blob/master/lib/rdoc/store.rb

It's extremely hard to implement perfect data store. Language servers are an example. We didn't expect them, but now they play an important role. It was difficult to design data store for language servers before they come.

It would be great to have a way to extend store. It's not clear to me how to achieve this, though.

Generators https://github.com/ruby/rdoc/tree/master/lib/rdoc/generator

There are some use cases for new generators. For example, some might want to use a sophisticated template. More importantly, we can generate cache for documentation for other tools with a new generator.

Current RDoc features

Current RDoc features could be implemented as extensions. ri could be a good example.

Summary

The purpose of my proposal is to maintain RDoc as a place for collecting all the efforts around documentation and comment space. RDoc extension system makes it possible to keep the core of RDoc small as a bundled gem and let developers customize it.

[st0012]

Parsers

I agree that it'd be great to support yard syntax too. But if that's the main goal, why don't we implement it first and then extract the API out of it later? From my experience working on Ruby LSP's addon system, it'll be extremely difficult to define good API interface with just one use case.

Store

What will be the benefit of extending store? Do you have any concrete examples in the language server's case?

Generators

It's already possible to register new generators, for example: https://github.com/BaseSecrete/rorvswild-theme-rdoc

Generally speaking, I think extensibility is just a means to achieve the goal. And I believe we can have more productive discussion if we can:

• Split each component extension into their own issues
• Have more detailed examples on each of their use case

[okuramasafumi]

@st0012 Sorry for the very late reply!

I agree that it'd be great to support yard syntax too. But if that's the main goal, why don't we implement it first and then extract the API out of it later? From my experience working on Ruby LSP's addon system, it'll be extremely difficult to define good API interface with just one use case.

I agree that it's better to have multiple examples for plugins. I have some ideas so I'll implement some of them if time allows.

What will be the benefit of extending store? Do you have any concrete examples in the language server's case?

Here, now I think extending Store object might not make sense in most cases. Instead, we can introduce env object that connects plugins and store data from them so that RDoc main components use that data later.

It's already possible to register new generators, for example: https://github.com/BaseSecrete/rorvswild-theme-rdoc

True, for adding generators, current implementation might be enough. However, if it turns out that we can integrate generators to our plugin system, we should be able to do so.

#1321 has YARD parser plugin and we are talking about the design of plugin system. For more productive discussion, I want to focus on Parser-related area in this post.

[st0012]

I want to focus on Parser-related area in this post.

Sure, can you update the proposal to:

• Remove the other 2 components
• Go deeper in parser plugin with more details & examples

[st0012]

After discussing this at RubyKaigi's coding party, we decided to change course and postpone the exploration of plugin system for now.
Instead, we'll experiment supporting yard parsing inside RDoc. As the first step, @okuramasafumi will add a new yard parser that supports these 3 tags:

• @yield
• @private
• @deprecated - this will also include (perhaps in a separate PR) adding a :deprecated: directive to RDoc in both rdoc and markdown marups.

cc @kou

[kou]

Thanks for the summary!

@okuramasafumi We don't need to do it as one large PR. Could you open an issue for it and break down it as small tasks in the issue? We can work on these small tasks step-by-step.

[okuramasafumi]

@st0012 Thank you for the summary, I'll implement those soon.
@kou Yes, I'll create issues to describe what to do to achieve effective YARD parsing with RDoc.