Add support for XTGETTCAP

[kchibisov]

How it works is available at https://invisible-island.net/xterm/ctlseqs/ctlseqs.html .

This is used to query for terminfo features and not relying on the actual files. It's supported by notcursors, most modern terms, etc. It could be sort of good when you ssh into system without TERM info you have and it could help terminals not having their terminfo spread to work with at least modern toolkits.

[chrisduerr]

Not a fan of this at all. Why do you want this?

[kchibisov]

mostly because it started to get used in toolkits.

I'd at least like to look into what it is a bit.

[chrisduerr]

mostly because it started to get used in toolkits.

Which one?

[kchibisov]

@chrisduerr I think I mentioned? it's notcurses, some recent go toolkits, etc. It's also present in basically any active terminal.

[roland-5]

From my experience, the currently released Neovim 0.10 uses XTGETTCAP as the escape sequence to detect OSC 52 support automatically and thus clipboard function within itself. This is very useful for me when I want to copy code from one remote machine and quickly paste it into another to check something.

[dcolascione]

Why wouldn't we want all terminal emulators to support this mechanism? TERM has serious long-term problems.

[chrisduerr]

Why wouldn't we want all terminal emulators to support this mechanism? TERM has serious long-term problems.

Because this also has serious problems, which is why it hasn't really been used by anything for like a century. But people got fed up with trying to make something new work so instead they're trying to switch from one set of problems to another. At the end of the day this just creates fragmentation with no benefits.

[dcolascione]

What are the serious problems?

[j4james]

99% of the features identified in a modern terminfo file are just standard ANSI or DEC sequences, or commonly supported Xterm extensions, that every modern terminal implements. This is why so many terminals can get away with just setting their TERM to xterm-256colors. Inventing a proprietary escape sequence for apps to look up a bunch of ANSI-standard escape sequences is patently ridiculous.

And if you're going to say that you're not using XTGETTCAP to query all those standard escape sequences, then why on earth would you expect terminals to report them? It's like asking someone their favorite color, but demanding that they provide the answer in the form of a 50,000 word novel, which you don't intend to read.

If there is a feature that isn't universally supported (like say OSC 52), there are better ways to report that than XTGETTCAP. You could propose a new DA1 feature, or a mode that can be queried with DECRQM, or even just extend the OSC 52 sequence itself with a feature reporting option. For example, if the first parameter is ?, then the terminal could report back the subset of OSC 52 that it can handle (not everyone supports paste, and even those that support copying don't all support the same clipboard targets).

[j4james]

Btw, notcurses is another example of unnecessary XTGETTCAP usage. The only capabilities it's querying are RGB and hpa. The former is used as a proxy for 24-bit SGR color support, which can be more accurately determined with a standard DECRQSS query. And hpa could be determined with a simple DSR-CPR test (i.e. attempt to move the cursor with HPA, and check whether it has actually moved with DSR-CPR).

Expecting terminals to implement all of the cruft associated with XTGETTCAP just so you can query two features that can already be determined via existing standards is absurd.

[kchibisov]

OSC 52 is a part of terminfo though already and was that for a while, I have no idea why neovim decided to not use terminfo as a fallback for it, sounds like just a push for XTGETTCAP just because they want to. And new flags are added to the terminfo, this XTGETTCAP is basically report a terminfo value, so those two are in sync.

The only real place where it kind of matters, is when you ssh into remote without your terminfo installed or with some dated version of it.

I'd state again that XTGETTCAP is just a terminfo reporting, so they use the same base file as a source of truth, it's not like one format is better, because they use the same database under the hood, it's just could be done via ssh and the other requires your hosts to be up to date/synced to some extent.

[j4james]

OSC 52 is a part of terminfo though

But it's such a convoluted definition considering what little information it provides. For Xterm it's defined as Ms=\E]52;%p1%s;%p2%s\007, which suggests other terminals could use a completely different escape sequence. Except that they'd also have to use p1 and p2 parameters that are exactly compatible with Xterm, which isn't even true of many of the existing OSC 52 implementations.

And in the case of nvim, they basically treat it like a boolean capability - if the definition starts with \e]52, it's just assumed to be compatible with Xterm's implementation. So what was the point of that whole parameterized string definition?

I'd state again that XTGETTCAP is just a terminfo reporting.

And that's the root of the problem, because terminfo was a great idea maybe 40 years ago, when you were trying to write an app that could work on many different hardware terminals with wildly different escape sequences. In this day and age, assuming you're writing a modern application targeting modern software terminals, terminfo causes more problems than it solves.

That said, if you (Alacritty) actually want to support XTGETTCAP, you're welcome to do so. I was just under the impression that you were against the idea, and I was attempting to explain to the person above why it wasn't worth the effort. But if you disagree, feel free to disregard everything I've said.

[chrisduerr]

That said, if you (Alacritty) actually want to support XTGETTCAP, you're welcome to do so. I was just under the impression that you were against the idea, and I was attempting to explain to the person above why it wasn't worth the effort. But if you disagree, feel free to disregard everything I've said.

There's also issues beyond what you mentioned, like the question of which terminfo to report and the fact that you need to query non-standard stuff like Smulx separately since processing is interrupted as soon as any invalid name is found in the requested list.

[kchibisov]

That said, if you (Alacritty) actually want to support XTGETTCAP, you're welcome to do so. I was just under the impression that you were against the idea, and I was attempting to explain to the person above why it wasn't worth the effort. But if you disagree, feel free to disregard everything I've said.

I personally don't mind if certain apps really force the rest by starting doing so, but I just don't like it myself either.

[kchibisov]

This is very useful for me when I want to copy code from one remote machine and quickly paste it into another to check something.

can you force neovim to enable it? Sounds like there should be a way to do so.

[dcolascione]

The mechanism still seems pretty useful to me. With XTGETTCAP, programs request only the capabilities they need, so why would it hurt anything that information on , say, ANSI capabilities would be harmful, even if most terminals will report the same strings for those capabilities? IMHO, there's a lot of value in having a single converged capability detection mechanism that works for both old, boring capabilities and new, exciting ones. Maybe it'd be simpler to support only Boolean flags instead of parameterized capabilities, but IMHO not simpler enough that it's worth making an alternative feature detection system. I plan to add support for XTGETTCAP to Emacs term, FWIW. I want to solve the ssh problem once and for all.

[kchibisov]

boring capabilities and new, exciting ones

Do you understand that terminfo will have new exciting ones as well, since they have the same source and one is based on another? Unless you're talking about a new protocol, nothing will change, except ssh miss-matches, as said.

[kchibisov]

I plan to add support for XTGETTCAP to Emacs term, FWIW. I want to solve the ssh problem once and for all.

Solved with one scp | tic command or running systems that are not 5-7 years old.

[chrisduerr]

IMHO, there's a lot of value in having a single converged capability detection mechanism that works for both old, boring capabilities and new, exciting ones.

Well this escape isn't that, so I don't see the point.

[Samasaur1]

This is very useful for me when I want to copy code from one remote machine and quickly paste it into another to check something.

can you force neovim to enable it? Sounds like there should be a way to do so.

Yes, you can:

vim.g.clipboard = {
  name = 'OSC 52',
  copy = {
    ['+'] = require('vim.ui.clipboard.osc52').copy('+'),
    ['*'] = require('vim.ui.clipboard.osc52').copy('*'),
  },
  paste = {
    ['+'] = require('vim.ui.clipboard.osc52').paste('+'),
    ['*'] = require('vim.ui.clipboard.osc52').paste('*'),
  },
}

(see :help clipboard-osc52)

[j4james]

the fact that you need to query non-standard stuff like Smulx separately since processing is interrupted as soon as any invalid name is found in the requested list.

Oh yeah, I forgot about that mess. And it's actually worse than you think. In practice you can never reliably query more than one capability at a time, regardless of whether they're standard or not, because every terminal handles multi-value queries differently.

[dcolascione]

Do you understand that terminfo will have new exciting ones as well, since they have the same source and one is based on another? Unless you're talking about a new protocol, nothing will change, except ssh miss-matches, as said.

Some systems don't have their terminfo databases updated for years. Many don't know about kitty at all, for example.

Solved with one scp | tic command or running systems that are not 5-7 years old.

I'm not sure why it's better to mutate state on other systems (on some of which, the system-global terminfo directories are immutable, meaning you need to use environment variable hacks to point at some custom directory you create) than to just have the terminal report its own capabilities. UA sniffing has been deprecated on the web for years. Why is it a good thing in terminals?

Do you understand that terminfo will have new exciting ones as well, since they have the same source and one is based on another? Unless you're talking about a new protocol, nothing will change, except ssh miss-matches, as said.

People often either connect to older systems or run old base images themselves. There are people still using Ubuntu 20.04 as a base. Backporting a new libterminfo just for one or two newer programs somebody might want to run doesn't make sense.

IMHO, there's a lot of value in having a single converged capability detection mechanism that works for both old, boring capabilities and new, exciting ones.
Well this escape isn't that, so I don't see the point.

It could be, if this bug were fixed:

Oh yeah, I forgot about that mess. And it's actually worse than you think. In practice you can never reliably query more than one capability at a time, regardless of whether they're standard or not, because every terminal handles multi-value queries differently.

An introspection mechanism solves multiple problems. There are no downsides AFAICT. The proposed workarounds are inconvenient, stateful, or incompatible with older systems. When everyone, or nearly everyone, supports introspection, users will benefit.

[kchibisov]

Some systems don't have their terminfo databases updated for years. Many don't know about kitty at all, for example.

And they would never support XTGETTCAP for sure then, since you never update those, so nothing is solved, it only matters for up to date systems, and systems we have now and that would be really old in the future, if you ssh into 5 year old host now, you have no luck with it unless you ssh with TERM=xterm or something, but XTGETTCAP won't solve anything for you now at all, or ploblems you have now, since up to date systems don't have problems, because terminfo (even alacritty one) is part of ncurses for a long time.

People often either connect to older systems or run old base images themselves. There are people still using Ubuntu 20.04 as a base. Backporting a new libterminfo just for one or two newer programs somebody might want to run doesn't make sense.

no one will backport XTGETTCAP patches to programs on system either, so what's the point?

I'm not sure why it's better to mutate state on other systems (on some of which, the system-global terminfo directories are immutable, meaning you need to use environment variable hacks to point at some custom directory you create) than to just have the terminal report its own capabilities. UA sniffing has been deprecated on the web for years. Why is it a good thing in terminals?

terminfo can be installed for a user, you don't need root, you mutate only your user and that's about it. User is the one responsible for copying it in any sane terminal.

An introspection mechanism solves multiple problems. There are no downsides AFAICT. The proposed workarounds are inconvenient, stateful, or incompatible with older systems. When everyone, or nearly everyone, supports introspection, users will benefit.

in a far future, once they update their systems to new programs running software supporting XTGETTCAP, so nothing will be solved for current users. Like the escape is old, and probably there's a reason why it was not that used...

[chrisduerr]

It could be, if this bug were fixed

That is no bug, it's intended to behave this way.

An introspection mechanism solves multiple problems. There are no downsides AFAICT. The proposed workarounds are inconvenient, stateful, or incompatible with older systems. When everyone, or nearly everyone, supports introspection, users will benefit.

There are no downsides, if we ignore the numerous downsides that have been laid out? Very convenient. You can't just hijack an old escape sequence and demand that it should suddenly behave differently because that would be convenient for you.

[dcolascione]

I'm confused. I've yet to see a mention of a downside to an introspection. I've seen vigorous insistence that existing workarounds are sufficient, but I haven't seen any downsides of introspection itself.

in a far future, once they update their systems to new programs running software supporting XTGETTCAP, so nothing will be solved for current users

Once programs generally support terminal introspection, the update problem will be solved for good from that moment onward even if the mechanism wouldn't help users of today's older systems. The world continues to evolve.

It's hard to grasp the opposition here to solving real user problems. I will be unable to recommend that people use or developers support terminals that are not interested in solving user problems.

[dcolascione]

Besides: what happens when an existing terminal emulator gains new capabilities and a program wanting to use those capabilities wants to know whether to enable them? With the static TERM approach, there's no way to describe version specific capabilities except by defining a new TERM name, and new TERM names cause users real inconvenience. Introspection solves the problem: it lets applications ask the specific terminal emulator binary they're using its capabilities instead of trying to infer them from the name of the program.

[chrisduerr]

It's hard to grasp the opposition here to solving real user problems. I will be unable to recommend that people use or developers support terminals that are not interested in solving user problems.

You're creating problems by further fragmenting the ecosystem, not solving anything.

[dcolascione]

How would you solve the old-host problem and the problem of multiple emulator versions under a single TERM name problem then?

[chrisduerr]

the problem of multiple emulator versions under a single TERM name

That's not a problem. Terminal doesn't matter, the only thing that matters is its capabilities. Nobody should be detecting things based on name.

How would you solve the old-host problem

Well I have no problem installing terminfos on my remotes. I don't disagree that there could be better solutions obviously, TERM itself is kinda crap, but you gotta be careful about just jumping on random existing escape sequences because most of them aren't used for good reasons.