Make Apply Streaming Cancelable #5694

RomneyDa · 2025-05-15T22:55:52Z

Made the "Apply" streaming process cancelable so users can stop an in-progress code apply operation in the UI or via the reject diffs OR cancel stream commands.

Approach:

Because of our apply architecture I decided to make a simple abort manager id -> abort controller singleton that can be used by the streamDiffLines function, which is the bottom level of all apply/edit streams.
Also makes the llm stream chat actually abort the controller rather than just stopping the generator when aborted message id found
Then, changes fetch package to catch abort errors right next to the fetch call and return a 499 client cancelled request error
This error is caught in the streamResponse function used by all streaming within Continue and just causes a premature return. This is for cases where fetch is aborted after streaming starts
Then I added specific error handling for 499 in the ~20 abortable non-streaming places - mostly odd APIs and complete requests throughout where response.json() is used
This requires publishing a new version of fetch (1.0.10), and openai-adapters (1.0.25) packages (DONE)

Also

Fixes VertexAI streams not being abortable
Shows Applying... and the cancel keyboard shortcuts if currently applying (same location as "Generating...)
Removes duplicate streaming logic - uses @continuedev/fetch package for streaming utils

To test:
Update package.jsons to use relative paths for fetch package and rebuild fetch package
Use a slower apply model
Cancel streaming mid-apply

netlify · 2025-05-15T22:55:57Z

✅ Deploy Preview for continuedev ready!

Name	Link
🔨 Latest commit	`dcb1615`
🔍 Latest deploy log	https://app.netlify.com/projects/continuedev/deploys/6833a3bfaa4a2200087791b4
😎 Deploy Preview	https://deploy-preview-5694--continuedev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

RomneyDa · 2025-05-15T22:56:39Z

core/core.ts

@@ -62,61 +61,7 @@ import { LLMLogger } from "./llm/logger";
 import { llmStreamChat } from "./llm/streamChat";
 import type { FromCoreProtocol, ToCoreProtocol } from "./protocol";
 import type { IMessenger, Message } from "./protocol/messenger";
-
-// This function is used for jetbrains inline edit and apply


Removed streamDiffLinesGenerator/moved logic inline because point of it was to account for aborted message ids which wasn't being used, made ~1/2 the lines duplicate, and is now defunct

RomneyDa · 2025-05-15T22:59:26Z

core/edit/streamDiffLines.ts

  input: string;
  language: string | undefined;
  onlyOneInsertion: boolean;
  overridePrompt: ChatMessage[] | undefined;
  rulesToInclude: RuleWithSource[] | undefined;
 }): AsyncGenerator<DiffLine> {
+  const abortManager = StreamAbortManager.getInstance();


Took the singleton approach so that functional stream diff lines function (also used in vscode-only code) can have persisted abort controllers that can be cancelled with core messaging

RomneyDa · 2025-05-15T23:00:30Z

core/llm/streamChat.ts

  } else {
    const gen = model.streamChat(
      messages,
-      new AbortController().signal,
+      abortController.signal,


This is significant because before, the connection would be hanging/running in the background until complete even if we've "aborted" using message ids at that point

RomneyDa · 2025-05-16T17:27:07Z

core/llm/llms/VertexAI.ts

@@ -323,14 +341,14 @@ class VertexAI extends BaseLLM {
      ? this.geminiInstance.removeSystemMessage(messages)
      : messages;
    if (this.vertexProvider === "gemini") {
-      yield* this.streamChatGemini(convertedMsgs, options);
+      yield* this.streamChatGemini(convertedMsgs, options, signal);


VertexAI fixes - weren't abortable

…top of them

continue-code-review · 2025-05-25T23:13:29Z

🚨 Code Review Error

GitHub API error (422): Unprocessable Entity

Please check the logs or contact the maintainers for assistance.

sestinj

Looks good. Nice that we were able to get rid of some of the uglier abort logic. Good tests for streamSse. I tested locally for a bit, but this is a PR to keep our eyes on while in pre-release

RomneyDa added 2 commits May 2, 2025 12:07

WIP: message abort controllers

b072c5f

cancelable apply

d93d665

github-project-automation bot added this to Issues and PRs May 15, 2025

github-project-automation bot moved this to Todo in Issues and PRs May 15, 2025

RomneyDa added 2 commits May 15, 2025 18:45

catch client abort errors

1a47f96

empty chat completion for custom non things

88d2bf2

RomneyDa marked this pull request as ready for review May 16, 2025 04:13

RomneyDa requested a review from a team as a code owner May 16, 2025 04:13

RomneyDa requested review from tomasz-stefaniak and removed request for a team May 16, 2025 04:13

dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label May 16, 2025

RomneyDa self-assigned this May 16, 2025

RomneyDa requested a review from sestinj May 16, 2025 04:13

RomneyDa added 2 commits May 16, 2025 10:15

bump fetch

7ad5587

bump openai-adapters

07313a0

RomneyDa commented May 16, 2025

View reviewed changes

RomneyDa added 2 commits May 16, 2025 10:29

cleanup

8f3ee86

use fetch package for streaming utils

35958b2

sestinj added a commit that referenced this pull request May 17, 2025

chore: ♻️ lift changes from #5694 to fetch package so I can build on …

564608d

…top of them

RomneyDa added 2 commits May 19, 2025 10:55

merge main

b8b893d

fetch updates - move tests to fetch package

9d2037c

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels May 19, 2025

fetch package updates from #5715

2d23065

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels May 19, 2025

RomneyDa added 3 commits May 19, 2025 16:42

bump fetch version

c496986

bump fetch for other sources

960990b

bump openai-adapters

05dafeb

RomneyDa and others added 3 commits May 19, 2025 16:52

lockfile updates

eea5e81

fix syntax error

10e3ae3

Merge branch 'main' into dallin/cancelable-apply

dcb1615

sestinj approved these changes May 25, 2025

View reviewed changes

github-project-automation bot moved this from Todo to In Progress in Issues and PRs May 25, 2025

dosubot bot added the lgtm This PR has been approved by a maintainer label May 25, 2025

sestinj merged commit 90d5aeb into main May 25, 2025
35 checks passed

sestinj deleted the dallin/cancelable-apply branch May 25, 2025 23:33

github-project-automation bot moved this from In Progress to Done in Issues and PRs May 25, 2025

github-actions bot locked and limited conversation to collaborators May 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make Apply Streaming Cancelable #5694

Make Apply Streaming Cancelable #5694

RomneyDa commented May 15, 2025 •

edited

Loading

Uh oh!

netlify bot commented May 15, 2025 •

edited

Loading

Uh oh!

RomneyDa May 15, 2025

Uh oh!

RomneyDa May 15, 2025

Uh oh!

RomneyDa May 15, 2025

Uh oh!

RomneyDa May 16, 2025

Uh oh!

continue-code-review bot commented May 25, 2025

Uh oh!

sestinj left a comment

Uh oh!

Uh oh!

Uh oh!

Make Apply Streaming Cancelable #5694

Make Apply Streaming Cancelable #5694

Conversation

RomneyDa commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for continuedev ready!

Uh oh!

RomneyDa May 15, 2025

Choose a reason for hiding this comment

Uh oh!

RomneyDa May 15, 2025

Choose a reason for hiding this comment

Uh oh!

RomneyDa May 15, 2025

Choose a reason for hiding this comment

Uh oh!

RomneyDa May 16, 2025

Choose a reason for hiding this comment

Uh oh!

continue-code-review bot commented May 25, 2025

Uh oh!

sestinj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

RomneyDa commented May 15, 2025 •

edited

Loading

netlify bot commented May 15, 2025 •

edited

Loading