feat: new query syntax #185

samwillis · 2025-06-18T19:45:17Z

Implements the new query syntax from #170

todo:

new intermedia representation
query builder that outputs the intermedia representation
compiler that compiles the ir to a D2 pipeline
test - and it works...

features:

The main entry point is a new createLiveQueryCollection function:

createLiveQueryCollection((q) =>
  q.from({ user: usersCollection }).select(({ user }) => ({
    id: user.id,
    name: user.name,
    age: user.age,
    email: user.email,
    active: user.active,
  }))
)

this wraps a new liveQueryCollectionOptions and createCollection call - the former of which can be used like the other xxxCollectionOptions function we have:

createCollection(liveQueryCollectionOptions({
  id: 'name-for-collection`,
  query: (q) =>
    q.from({ user: usersCollection }).select(({ user }) => ({
      id: user.id,
      name: user.name,
      age: user.age,
      email: user.email,
      active: user.active,
    })),
  // any other options
}))

createLiveQueryCollection can be called with either a query builder function, or a config object that takes the query param as a query builder function.

changeset-bot · 2025-06-18T19:45:20Z

⚠️ No Changeset found

Latest commit: c200ea8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

github-actions · 2025-06-18T19:46:38Z

Size Change: +669 B (+2.31%)

Total Size: 29.7 kB

Filename	Size	Change
`./packages/db/dist/esm/collection.js`	7.69 kB	-178 B (-2.26%)
`./packages/db/dist/esm/index.js`	569 B	+137 B (+31.71%)	🚨
`./packages/db/dist/esm/query/compiled-query.js`	0 B	-1.49 kB (removed)	🏆
`./packages/db/dist/esm/query/evaluators.js`	0 B	-1.06 kB (removed)	🏆
`./packages/db/dist/esm/query/extractors.js`	0 B	-870 B (removed)	🏆
`./packages/db/dist/esm/query/functions.js`	0 B	-1.28 kB (removed)	🏆
`./packages/db/dist/esm/query/group-by.js`	0 B	-976 B (removed)	🏆
`./packages/db/dist/esm/query/joins.js`	0 B	-1.14 kB (removed)	🏆
`./packages/db/dist/esm/query/order-by.js`	0 B	-1.42 kB (removed)	🏆
`./packages/db/dist/esm/query/pipeline-compiler.js`	0 B	-878 B (removed)	🏆
`./packages/db/dist/esm/query/query-builder.js`	0 B	-2.14 kB (removed)	🏆
`./packages/db/dist/esm/query/select.js`	0 B	-1.1 kB (removed)	🏆
`./packages/db/dist/esm/query/utils.js`	0 B	-1.13 kB (removed)	🏆
`./packages/db/dist/esm/utils.js`	0 B	-219 B (removed)	🏆
`./packages/db/dist/esm/query/builder/functions.js`	523 B	+523 B (new file)	🆕
`./packages/db/dist/esm/query/builder/index.js`	3.19 kB	+3.19 kB (new file)	🆕
`./packages/db/dist/esm/query/builder/ref-proxy.js`	851 B	+851 B (new file)	🆕
`./packages/db/dist/esm/query/compiler/evaluators.js`	1.34 kB	+1.34 kB (new file)	🆕
`./packages/db/dist/esm/query/compiler/group-by.js`	2.08 kB	+2.08 kB (new file)	🆕
`./packages/db/dist/esm/query/compiler/index.js`	1.4 kB	+1.4 kB (new file)	🆕
`./packages/db/dist/esm/query/compiler/joins.js`	1.16 kB	+1.16 kB (new file)	🆕
`./packages/db/dist/esm/query/compiler/order-by.js`	933 B	+933 B (new file)	🆕
`./packages/db/dist/esm/query/compiler/select.js`	657 B	+657 B (new file)	🆕
`./packages/db/dist/esm/query/ir.js`	310 B	+310 B (new file)	🆕
`./packages/db/dist/esm/query/live-query-collection.js`	1.98 kB	+1.98 kB (new file)	🆕

ℹ️ View Unchanged

Filename	Size
`./packages/db/dist/esm/deferred.js`	230 B
`./packages/db/dist/esm/errors.js`	150 B
`./packages/db/dist/esm/optimistic-action.js`	294 B
`./packages/db/dist/esm/proxy.js`	3.75 kB
`./packages/db/dist/esm/SortedMap.js`	1.24 kB
`./packages/db/dist/esm/transactions.js`	1.33 kB

_{compressed-size-action::db-package-size}

github-actions · 2025-06-18T19:47:18Z

Size Change: 0 B

Total Size: 561 B

ℹ️ View Unchanged

Filename	Size
`./packages/react-db/dist/esm/index.js`	152 B
`./packages/react-db/dist/esm/useLiveQuery.js`	409 B

_{compressed-size-action::react-db-package-size}

pkg-pr-new · 2025-06-18T19:47:26Z

@tanstack/db-example-react-todo

npm i https://pkg.pr.new/@tanstack/db@185

npm i https://pkg.pr.new/@tanstack/db-collections@185

npm i https://pkg.pr.new/@tanstack/react-db@185

npm i https://pkg.pr.new/@tanstack/vue-db@185

commit: c200ea8

samwillis · 2025-06-22T09:47:11Z

The one failing test in this at the moment is fixed by these d2mini PRs:

KyleAMathews · 2025-06-27T15:11:47Z

Installed this in the expo starter which has this for the query: const { data: todos } = useLiveQuery((q) => q.from({ todoCollection }));

There's no loader so the initial render has 0 items but it never re-renders after that. I called todoCollection.subscribeChanges and it does send in the inserts.

KyleAMathews · 2025-06-27T15:13:57Z

An optimistic/sync mutation does get it to re-render

KyleAMathews · 2025-06-27T15:26:17Z

I tried doing a select callback and got this js error for foo:

    q.from({ todoCollection }).select(({ todoCollection }) => {
      return {
        foo: `${todoCollection.id}`,
        id: todoCollection.id,
        text: todoCollection.text,
        completed: todoCollection.completed,
        created_at: todoCollection.created_at,
        updated_at: todoCollection.updated_at,
      };
    }),

Uncaught TypeError: Cannot convert object to primitive value

KyleAMathews · 2025-06-27T15:29:07Z

For the main operators — it'd be great to have an inline example or two to remind people of the syntax

KyleAMathews · 2025-06-27T16:13:39Z

I'm not sure I love the operators being individual exports (e.g. import { eq } from "@tanstack/react-db" as that makes them less discoverable e.g. right now I'm testing them & I need to look through the PR to get the full list. Maybe just operators so you can either do import { operators as $ } from "@tanstack/db" or import { operators: { eq }} from "@tanstack/db" ?

KyleAMathews · 2025-06-27T16:18:49Z

I tried doing a select callback and got this js error for foo:

Ohhhh that's because this isn't supported yet... just the operator selects... hmm that's going to confuse people

KyleAMathews · 2025-06-27T16:21:24Z

Another potentially annoying part about individual operators is that the autoimport tool suggests both drizzle & react-db 😅

KyleAMathews · 2025-06-27T16:25:43Z

isInFunc is a bit odd... inArray & notInArray is better perhaps like Drizzle https://orm.drizzle.team/docs/operators

KyleAMathews · 2025-06-27T16:26:27Z

I've changed my mind about putting operators under operator — there's not that many & w/ docs they'll be plenty discoverable 👍

KyleAMathews · 2025-06-27T16:36:12Z

concat doesn't seem to work?

I wrote this code: foo: concat([todoCollection.text, '-', todoCollection.completed]) in my select and got this output when rendering: [{},"-",{}] FOO3

Oh actually the test is misleading I guess? Removing the array made it work i.e. concat(todoCollection.text, '-', todoCollection.completed)

samwillis · 2025-06-27T19:56:22Z

I tried doing a select callback and got this js error for foo:

Ohhhh that's because this isn't supported yet... just the operator selects... hmm that's going to confuse people

It's planned, as I noted above and on Discord, it will come in another PR asap. We need to decide how to name them - they have to be distinct from the main method as they now take callbacks too.

samwillis · 2025-06-27T19:59:41Z

I'm not sure I love the operators being individual exports (e.g. import { eq } from "@tanstack/react-db" as that makes them less discoverable e.g. right now I'm testing them & I need to look through the PR to get the full list. Maybe just operators so you can either do import { operators as $ } from "@tanstack/db" or import { operators: { eq }} from "@tanstack/db" ?

There was a discussion about this over on the issue here: #170 (comment)

Happy to change this, we didn't really reach a conclusion. They could be an import, or provided for destructuring on the useLiveQuery callback.

samwillis · 2025-06-27T20:01:19Z

isInFunc

I thought I had it as isIn and aliased as in... clearly I messed something up.

samwillis · 2025-06-27T20:01:48Z

Oh actually the test is misleading I guess? Removing the array made it work i.e. concat(todoCollection.text, '-', todoCollection.completed)

Yes this is the correct syntax

samwillis · 2025-06-27T20:02:35Z

For the main operators — it'd be great to have an inline example or two to remind people of the syntax

I'de love to, I need to find a way to stop the type being so verbose though as I think it will push any jsdoc down in the prompt... 🤔

samwillis · 2025-06-27T20:03:52Z

There's no loader so the initial render has 0 items but it never re-renders after that. I called todoCollection.subscribeChanges and it does send in the inserts.

Is that with a shape that is now lazy? I think thats behaving as it should as there are no rows yet?

KyleAMathews · 2025-06-27T20:09:41Z

I need to find a way to stop the type being so verbose though

Explicit named types is the trick I think?

Is that with a shape that is now lazy? I think thats behaving as it should as there are no rows yet?

No the collection loads data just fine — the query just doesn't seem to detect the initial sync. Subsequent syncs & any optimistic changes do trigger a render.

… methods (#215)

samwillis · 2025-06-28T09:26:27Z

I need to find a way to stop the type being so verbose though

Explicit named types is the trick I think?

I've tried to make this work, but failed. No patter what I do it always shows the context in the generic in the prompt.

I have added jsdoc comments with examples, so thats an improvement. I think we can come back to trying to minimise the blob at the top later (and maybe a TS expert will come and work it out for us!)

samwillis · 2025-06-28T09:34:04Z

@KyleAMathews on isIn, it was mistakingly aliased in the test isInFunc, it is just isIn.

I'm open to renaming it, but not that it operates on both arrays and strings, so you can do isIn('string', 'isIn works on both arrays and strings')

Obv we can't call it in as thats a reserved word in JS. The another option is contains and flip the arguments around?

We could also split into two functions inArray and inString?

DawidWraga · 2025-06-28T12:46:21Z

I need to find a way to stop the type being so verbose though

Explicit named types is the trick I think?

I've tried to make this work, but failed. No patter what I do it always shows the context in the generic in the prompt.

I had a similar problem a while ago, this might be helpful

https://dev.to/dawidcodes/simplifying-complex-type-display-in-typescript-and-vs-code-4f1

KyleAMathews · 2025-06-28T13:32:29Z

Ah ok, maybe inArray and includes? It is a bit confusing to have both like and another string operator. It seems we could just have like and inArray?

samwillis added 6 commits June 18, 2025 18:29

starting point...

0450cff

checkpoint

7f6b55a

checkpoint

19b8749

tests

e28ec03

more

3c7ced9

tidy

7d03505

samwillis added 12 commits June 19, 2025 13:22

checkpoint

2211d48

more

5cc2b8a

basic test of full build -> compile -> run

d9cbd36

checkpoint

197e04a

more

85f6519

WIP groupby

006898b

groupby with having

c640cbf

fix lint errors

f5118c0

fix lint warnings

33f724e

fix tets

e7f9cd5

fix test type errors

119d193

fix lint

9171963

samwillis mentioned this pull request Jun 22, 2025

If a schema is passed, use that for the collection type #186

Merged

samwillis added 6 commits June 22, 2025 11:19

move tests

b4f0c2f

fix return type for select

d6dc39c

fix return type when no select

ffbc839

fix lint errors

4ae3cba

update d2mini

976bd07

Merge branch 'main' into query2

f04e87f

samwillis added 2 commits June 27, 2025 11:01

fix types

1a9f9ba

declutter type prompt in query builder

a315a5d

samwillis mentioned this pull request Jun 27, 2025

WIP functional variants of the select, where and having query builder methods #215

Merged

samwillis added 2 commits June 28, 2025 08:05

WIP functional variants of the select, where and having query builder…

8e1368d

… methods (#215)

jsdoc for the query builder

59b065c

samwillis force-pushed the query2 branch from 2653574 to 59b065c Compare June 28, 2025 09:25

samwillis force-pushed the query2 branch from 9d31396 to e44bef1 Compare June 28, 2025 09:31

remove alias in test for isIn

c200ea8

samwillis force-pushed the query2 branch from e44bef1 to c200ea8 Compare June 28, 2025 09:39

feat: new query syntax #185

Are you sure you want to change the base?

feat: new query syntax #185

Conversation

samwillis commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

github-actions bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samwillis commented Jun 22, 2025

Uh oh!

KyleAMathews commented Jun 27, 2025

Uh oh!

KyleAMathews commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KyleAMathews commented Jun 27, 2025

Uh oh!

KyleAMathews commented Jun 27, 2025

Uh oh!

KyleAMathews commented Jun 27, 2025

Uh oh!

KyleAMathews commented Jun 27, 2025

Uh oh!

KyleAMathews commented Jun 27, 2025

Uh oh!

KyleAMathews commented Jun 27, 2025

Uh oh!

KyleAMathews commented Jun 27, 2025

Uh oh!

KyleAMathews commented Jun 27, 2025

Uh oh!

samwillis commented Jun 27, 2025

Uh oh!

samwillis commented Jun 27, 2025

Uh oh!

samwillis commented Jun 27, 2025

Uh oh!

samwillis commented Jun 27, 2025

Uh oh!

samwillis commented Jun 27, 2025

Uh oh!

samwillis commented Jun 27, 2025

Uh oh!

KyleAMathews commented Jun 27, 2025

Uh oh!

samwillis commented Jun 28, 2025

Uh oh!

samwillis commented Jun 28, 2025

Uh oh!

DawidWraga commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KyleAMathews commented Jun 28, 2025

Uh oh!

Uh oh!

samwillis commented Jun 18, 2025 •

edited

Loading

changeset-bot bot commented Jun 18, 2025 •

edited

Loading

github-actions bot commented Jun 18, 2025 •

edited

Loading

github-actions bot commented Jun 18, 2025 •

edited

Loading

pkg-pr-new bot commented Jun 18, 2025 •

edited

Loading

KyleAMathews commented Jun 27, 2025 •

edited

Loading

DawidWraga commented Jun 28, 2025 •

edited

Loading