Configuration Provider API

[A2va]

The cpptools extension provide an API to consume a custom configuration provider.
For example the cmake-tools extension take advantage of that to have C++ completion without compile_commands file.

Could be possible to implement the same thing in vscode-clangd ?

[i-ky]

That API is specific to cpptools, so clangd extension will need its own API. It will also need to be supported by "configuration providers" including cmake-tools. Assuming that both cpptools and cmake-tools are now maintained by Microsoft, guess how likely it is to happen... VS Code docs don't even mention any alternative to cpptools.

[A2va]

Of course, I didn't expect clangd to have the same API, but to replicate the general idea.

I only created this issue to improve xmake-vscode, this extension already support cpptools and codelldb debugger, so I'm thinking to do the same for the C++ completion, with cpptools and clangd.

Some users only use VSCodium, and cpptools cannot be installed in this IDE, so a combo of clangd/code-lldb could work. This can help to reduce the minimal amount of configuration.

[HighCommander4]

At the protocol level, clangd does support the client specifying compile commands via workspace/didChangeConfiguration messages (see https://clangd.llvm.org/extensions#compilation-commands for details), as an alternative to using compile_commands.json.

The part I'm less sure about is whether another vscode extension can just get its hands on the LanguageClient instance created by vscode-clangd and use it to send such messages, or if vscode-clangd would need to expose an API to allow other extensions to do this. In the latter case, I imagine we'd be open to adding an such an API.

[A2va]

I made some test/research and actually this not possible to access to variables inside the extension. However I find how to expose an API:

// clangd extension
const api = new API();

export async function activate(context: vscode.ExtensionContext) {
return api;
}

// other extension
const clangd = vscode.extensions.getExtension("llvm-vs-code-extensions.vscode-clangd");

if (clangd) {
    let extension = undefined;
    if (!clangd.isActive) {
        extension = await cpptools.activate();
    } else {
        extension = clangd.exports;
    }

    const api = extension;
}

It's based on cpptools activation and cpptools npm package

[HighCommander4]

Thanks for checking.

Are you interested in writing a patch that implements the desired API?

[A2va]

Of course I can do it, but I have no idea when it will be ready.

[cameron-martin]

This would be useful for bazel integration. Currently we generate a large compile commands file for our whole repository, but requesting this information on demand for individual files would be nice for performance.

Generating the compile commands uses a separate command rather than as part of the build, so this has to be manually re-run when something that affects the compile commands changes. Possibly if the editor requested this it would be easier to keep up-to-date, although exactly when the editor re-requests this is presumably an open question.

[cameron-martin]

Another possible direction would be to use the build server protocol. If a BSP connection file exists and supports cpp, this extension could launch a BSP, get the target for the currently-open file, gather the compiler flags etc, and send this to clangd.

Or, clangd could support this natively: clangd/clangd#1903

[torshepherd]

@HighCommander4 another issue here I'm interested in tackling 😅

Context: at work we have a massive C++ bazel monorepo, which leads to two issues:

1. You have to manually extract and merge compile_commands.json with a bespoke extractor
2. As you merge more and more of the project, the json file grows extraordinarily large, leading to clangd having untenable RSS usage.

I suspect this is actually the indexer doing this, but regardless compile_commands.json is simply not a scalable approach to massive monorepos. We need to plug clangd directly into getting compile commands.

[torshepherd]

The reason I bring this up now is that buck2 and rust-analyzer have created a good integration:

buck2-side

r-a-side

[lmerynda]

We're running big repos too in my organization, including bazel based setups. With some scripting and talks with architects we were able to narrow down a few rules as to how the project is structured, which allows us to build our own heuristic based configuration setup which we lunch on startup and it delivers 95% accuracy which works very well for most of our users. The rest can be tweaked with repo persistent project configuration, or included in build system to access

One thing which we concluded in our jorney is that generating a series of .clangd files on a "module" basis yields much better experience than working with compile_commands.json. It's better for new files, it's better for test files, it's much smaller so we don't need clangd to spend the first 5-8s just reading the file for the first time. We basically went from 300k lines compile_commands.json to 300 lines "repo level" .clangd and 30-50 lines "module level" .clangd which gets generated dynamically.

In any case, by not relying on bazel build to deliver compile_commands.json we managed to get down to 10s setup instead of 5 minutes, definitely worth it in scale.

Chopping down compile_commands.json to see what's common across areas is an interesting excercise of itself, it could be used for security audits and activities around making sure we include (or not) certain compiler flags across the project, and/or we put good defaults in common locations

[torshepherd]

Wow, thats a great insight! I'll 100% be trying this.

I do still think long term it would be nice to create these integrations within vscode, bazel, and/or clangd.

[HighCommander4]

Following up on this earlier comment:

At the protocol level, clangd does support the client specifying compile commands via workspace/didChangeConfiguration messages (see https://clangd.llvm.org/extensions#compilation-commands for details), as an alternative to using compile_commands.json.

The part I'm less sure about is whether another vscode extension can just get its hands on the LanguageClient instance created by vscode-clangd and use it to send such messages, or if vscode-clangd would need to expose an API to allow other extensions to do this. In the latter case, I imagine we'd be open to adding an such an API.

As of version 0.1.29, vscode-clangd now has an API that exposes the LanguageClient object to other plugins: https://github.com/clangd/vscode-clangd/tree/master/api.

So, I think everything is now in place for other vscode extensions (e.g. a bazel one) to provide compile commands to clangd as an alternative to using compile_commands.json.

[torshepherd]

This is very cool! But this is more a push model right? An extension would have to listen for document opening events and then push compile commands to clangd?

[torshepherd]

Also, does clangd use compile_commands.json to do indexing? If so, then a push model rather than a discovery & pull model will suffer from incomplete indices, as the third-party extension will need to enumerate and push compile commands for all indexable files right?

[HighCommander4]

But this is more a push model right? An extension would have to listen for document opening events and then push compile commands to clangd?

I guess I imagined it as the extension having a notion of some sort of "project model", where it knows what files are part of the project, and pushing the compile commands for those files up front.

I think it could work "on demand" as you describe, too, you'd just need to be careful to send the command for a file before the textDocument/didOpen message for that file.

Also, does clangd use compile_commands.json to do indexing?

I've checked the code, and the workspace/didChangeConfiguration mechanism is hooked into the indexing process as well, so files for which a compile command is sent that way will get indexed.

If so, then a push model rather than a discovery & pull model will suffer from incomplete indices, as the third-party extension will need to enumerate and push compile commands for all indexable files right?

Not quite sure I follow this point, could you elaborate?

It's true that clangd will only index files it has a compile command for. In the scenario we're discussing, the extension is sending the commands, so it is in control of what clangd indexes and when. I'm not sure how else it could work.

[torshepherd]

Ok I buy what you're saying. Basically, a build system has the responsibility for understanding the structure of a program. For example, on changed BUILD files, it should push to clangd the updated compile commands (that might add in new -I flags or similar).

My issue is with how clangd scales to big repos. In the past, I've observed that clangd uses excessive memory or OOMs when you try to give it a compile commands that covers an entire massive codebase. So I guess, what's the best etiquette for a build system plugin? Eagerly push as many compile commands as possible to clangd, or be judicious with how much of the codebase we push commands for?

In any case I'll try to do this push model for now.

[HighCommander4]

So I guess, what's the best etiquette for a build system plugin? Eagerly push as many compile commands as possible to clangd, or be judicious with how much of the codebase we push commands for?

I don't think there's a "one size fits all" policy.

At my day job, we describe the repo containing our code as a "monorepo", but the total number of .cpp files it contains is ~10k; there's no issue with clangd indexing it up front (other than that it takes a while and we wish it was faster).

Google, as I understand it, has hundreds of millions of lines of C++ code, and they use clangd but with a custom indexing solution that's offloaded to distributed servers or some such.

There's probably various orders of magnitude in between and other strategies may be appropriate for them.