Rich LLM logging #4866
Rich LLM logging #4866
Conversation
✅ Deploy Preview for continuedev ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
owtaylor
force-pushed
the
rich-llm-logging
branch
from
b00ffa2 to
ec83625
Compare
There was a problem hiding this comment.
@owtaylor just pair-reviewed with @Patrick-Erichsen, this is a sweet addition. Code is nicely inserted and agreed with most all decisions.
- regarding including llmLogger react app code in
gui, might move some files around as nitpicks later but overall we think you made the right decisions/compromises here, convenient to have same build output location for build script copying etc.
-As you mentioned not too bad to expose jetbrains log webview if we want, and not worth doing both for large experimental feature immediately, especially with opt-in behavior and the fact that improves default logging for jetbrains anyways
@Patrick-Erichsen and I will pull and smoke test a bit more, then should be good to merge
There was a problem hiding this comment.
@owtaylor some second thoughts, would agree with your concerns about memory, could you limit the logs array to maybe 10s of thousands to prevent memory from being a concern? Maybe start shifting at that limit
Also less important but could you do a quick review of if using HeroIcons is reasonable rather than @vscode/codicons icons? not a dealbreaker but should be cross platform and perhaps value of matching the sidebar icon set and not using vs code icons in jetbrains in the future >= value of not matching native vs code icons. Didn't look too much into it so if the hero icons are super lacking don't worry about it
There was a problem hiding this comment.
This is an excellent implementation! Playing with this locally was really fun. I feel like this is quickly going to become something we wonder how we lived without having for so long 😅
+1 to the feedback Dallin had above, went ahead and approved however since I think it's mergeable as is.
A few minor UI nitpicks:
- Chatted with @TyDunn on the name for the tab - can we call it "Continue Console" rather than "LLM Logs"?
- Could we use the phrase "Autocomplete" and "Edit" rather than "Fim" and "Completion"? Mostly to align with terminology in our docs.
Lastly - regarding the failing tests, we still have some issues with flake, so whenever you're ready to merge this just let me know and I'll re-run them til they pass.
|
Thanks @RomneyDa and @Patrick-Erichsen for the review!!
I'll come up with an additional commit to add a limit Probably limiting the number of interactions rather than the number of item, since that will be easier for communication with the view.
Will do.
Sure - "LLM Log" has some advantage of compactness and avoiding overflow of the panel tabs, but the leap from LLM => Continue is likely not obvious to all users. I suppose then "continue.enableModelLogging" changes "continue.enableConsole" and so forth. Will make a stab at it!
Hmm, @shermanhuman made a suggestion that what should appear in the list is the role rather than the "method" - I think that makes a lot of sense, and would align things with the terminology that you want, but does require the role to be passed through to the ILLM methods (probably in the options) - OK if I look at that as a follow-up? |
That was essentially the concern, e.g. a user seeing the tab and being unsure where it came from. Maybe a minor concern since it's opt-in, but we're about the same length as
Definitely, I'm okay with that being a follow up 👍 |
owtaylor
force-pushed
the
rich-llm-logging
branch
from
ec83625 to
f7e3470
Compare
Too late :-) - using Console in the code did fix some awkwardness of llmLog as a identifier/filename component. New version:
I think that addresses everything noted.
Let's see how this does. The failure in the last run had an odd screenshot - the sidebar ended up in the panel. I don't think it had anything to do with this because setting for the Log (console) will be off by default. |
|
Hmmm, re-running the tests and looks like the same issue Keyboard Shortcuts Fresh VS Code window → sidebar closed → cmd+L with no code highlighted → opens sidebar and focuses input → cmd+L closes sidebar In the e2e tests we do manually move the extension to the right sidebar: https://github.com/continuedev/continue/blob/main/extensions/vscode/e2e/actions/GUI.actions.ts#L15 I'm wondering if some of the iframe logic there has an assumption that there will only be a single iframe: https://github.com/continuedev/continue/blob/main/extensions/vscode/e2e/actions/GUI.actions.ts#L32 If you're able to run the e2e tests locally, I would just bump the timeout for that test to some arbitrarily long number (100000000) and that will allow you to poke around the IDE during the test to see if the feature is enabled or not. Feel free to timebox your troubleshooting to a reasonable length and then pass it over to us, the e2e tests are a bit tricky so I'm sure we'll be able to figure it out if it feels like you're going down a rabbit hole. |
The problem with moving the sidebar to the side is that the second quick pick for "View: Move View" lists all View Containers, including the empty "Continue Log" one, this upsets the hardcoded index of 14 (which is the Copilot Chat view container?) Changing |
When we catch/log/rethrow AbortError, throw it as itself, rather than throw Error(e.message) so that outer callers can detect it by the exception name.
We can't add citations to the rich log until we have them plumbed through with the correct data types. That will happen when they get added to the UI. Since the only appearance so far was in the old text log, just revert the addition for now. This reverts commit 6bc0bfc.
Instead of treating LLM logging as simply writing lines of text sequentially, emit a stream of typed data objects that include rich information about the interaction, such as timestamps, message types, tool calls, etc. These are then collected by a new custom webview in the vscode extension (the "Continue Console"), allowing a user or developer to browse through the different interactions and view them in detail. In progress interactions are now streamed live, instead of being held back to avoid concurrency issues. This webview could be also exposed for IntelliJ or file logging could be reimplemented on top of the new framework.
Add a streaming implementation of rich LLM logging to use if we don't have the webview connected; it converts the LLMInteractionItems into a human-readable stream with timestamps, handling of overlapping interactions, and so forth. Use this for the binary.
Only retain and display the 50 most recent interactions; this prevents the stored items from growing without bound if the user is using the vscode instance to edit for a long time.
Selecting the 14th view container ran into problems when an (emty) Continue Console view container was added. This also changes it to move to new secondary sidebar container rather than reusing one from Copilot.
owtaylor
force-pushed
the
rich-llm-logging
branch
from
f7e3470 to
25ee72e
Compare
|
The failure on this run: is actually a flake. |
|
@tomasz-stefaniak will hopefully be getting to the bottom of the test flake this week |
|
All tests passing! 25ee72e#diff-a6d8ec1e9272f89aa61d2e14627eef192f6b5ed76d7a6f1647991e56d9ff5ac4R21 Thanks for cleaning this up, that was bound to break at some point so apologies you were the first one to have to deal with it. -- I think we're good to merge here, if you're ready as well let's hit the button 👍 |
|
Handled a quick merge conflict that just arose |
Good to go from my perspective - I'll keep an eye out for any problems that are reported (or let me know) and will look at adding some enhancements like showing the role over the next few weeks. |
|
@owtaylor thanks for cleaning things up and working in the requested changes! Looks great. Merging. |
Description
Instead of treating LLM logging as simply writing lines of text sequentially, emit a stream of typed data objects that include rich information about the interaction, such as timestamps, message types, tool calls, etc.
These are then collected by a new custom webview in the vscode extension, allowing a user (developer) to browse through the different interactions and view them in detail. In progress interactions are now streamed live, instead of being held back to avoid concurrency issues.
There is a file logging implementation on top of this that includes the rich information and tries to be prettier and easier to read than the old logging; this is used for the binary and hence for Jetbrains for now, though exposing the rich webview in Jetbrains could be done too.
Checklist
Screenshots
Screencast.From.2025-03-27.16-03-32.webm
[ For visual changes, include screenshots. ]
Testing instructions
Review guide and notes
This is a rather large set of changes, some hints:
core/index.d.tsBaseLLMincore/llm/index.ts- this is where the actual logging is added - there is as lot of boilerplate that is replicated betweenstreamChat,streamComplete, etc. I have some ideas how this could be cleaned up, but it would make this patch less straightforward to review. One subtle thing in the changes here is the addingfinally {}blocks to catch cancellation via a call generator.return() - this should probably be documented, but would be easier if it wasn't duplicated in a bunch of places.logFormatter.tsandlogFormatter.test.tsare pretty big and somewhat complex, but don't have much relevance to the overall code-flow - they just adapt the new log interface into a stream.gui/src/components/llmLog/*is rather big, but it's "just" the react GUI. If you want to understand how it works,gui/src/hooks/useLLMLog.tsis a good place to start - it handles converting the linear stream of log items into state that can be efficiently rendered.Other notes
retainContextWhenHiddenis not set (to reduce overhead when hidden). Again, limiting the amount of history saved would help.Possible future extensions