Skip to content

Conversation

@samwillis
Copy link
Collaborator

@samwillis samwillis commented Nov 27, 2025

stacked on #958

Enhanced multi-column orderBy support with lazy loading and composite cursor optimization.

Changes:

  • Create index on first orderBy column even for multi-column orderBy queries, enabling lazy loading with first-column ordering
  • Pass multi-column orderBy to loadSubset with precise composite cursors (e.g., or(gt(col1, v1), and(eq(col1, v1), gt(col2, v2)))) for backend optimization
  • Use wide bounds (first column only) for local index operations to ensure no rows are missed
  • Use precise composite cursor for sync layer loadSubset to minimize data transfer

Benefits:

  • Multi-column orderBy queries with limit now support lazy loading (previously disabled)
  • Sync implementations (like Electric) can optimize queries using composite indexes on the backend
  • Local collection uses first-column index efficiently while backend gets precise cursor

Prior to this change composite order by was broken as shown in the first commit CI run: https://github.com/TanStack/db/actions/runs/19741724803/job/56567001027

@changeset-bot
Copy link

changeset-bot bot commented Nov 27, 2025

🦋 Changeset detected

Latest commit: 9ecdc59

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages
Name Type
@tanstack/db Patch
@tanstack/angular-db Patch
@tanstack/db-collection-e2e Patch
@tanstack/electric-db-collection Patch
@tanstack/offline-transactions Patch
@tanstack/powersync-db-collection Patch
@tanstack/query-db-collection Patch
@tanstack/react-db Patch
@tanstack/rxdb-db-collection Patch
@tanstack/solid-db Patch
@tanstack/svelte-db Patch
@tanstack/trailbase-db-collection Patch
@tanstack/vue-db Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@pkg-pr-new
Copy link

pkg-pr-new bot commented Nov 27, 2025

More templates

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@926

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@926

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@926

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@926

@tanstack/offline-transactions

npm i https://pkg.pr.new/@tanstack/offline-transactions@926

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/@tanstack/powersync-db-collection@926

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@926

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@926

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@926

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@926

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@926

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@926

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@926

commit: 9ecdc59

@github-actions
Copy link
Contributor

github-actions bot commented Nov 27, 2025

Size Change: +829 B (+0.95%)

Total Size: 88.2 kB

Filename Size Change
./packages/db/dist/esm/collection/subscription.js 2.71 kB +166 B (+6.52%) 🔍
./packages/db/dist/esm/query/compiler/order-by.js 1.46 kB +206 B (+16.48%) ⚠️
./packages/db/dist/esm/utils/cursor.js 457 B +457 B (new file) 🆕
ℹ️ View Unchanged
Filename Size
./packages/db/dist/esm/collection/change-events.js 1.39 kB
./packages/db/dist/esm/collection/changes.js 977 B
./packages/db/dist/esm/collection/events.js 388 B
./packages/db/dist/esm/collection/index.js 3.24 kB
./packages/db/dist/esm/collection/indexes.js 1.1 kB
./packages/db/dist/esm/collection/lifecycle.js 1.67 kB
./packages/db/dist/esm/collection/mutations.js 2.31 kB
./packages/db/dist/esm/collection/state.js 3.42 kB
./packages/db/dist/esm/collection/sync.js 2.37 kB
./packages/db/dist/esm/deferred.js 207 B
./packages/db/dist/esm/errors.js 4.19 kB
./packages/db/dist/esm/event-emitter.js 748 B
./packages/db/dist/esm/index.js 2.66 kB
./packages/db/dist/esm/indexes/auto-index.js 742 B
./packages/db/dist/esm/indexes/base-index.js 766 B
./packages/db/dist/esm/indexes/btree-index.js 1.93 kB
./packages/db/dist/esm/indexes/lazy-index.js 1.1 kB
./packages/db/dist/esm/indexes/reverse-index.js 513 B
./packages/db/dist/esm/local-only.js 837 B
./packages/db/dist/esm/local-storage.js 2.1 kB
./packages/db/dist/esm/optimistic-action.js 359 B
./packages/db/dist/esm/paced-mutations.js 496 B
./packages/db/dist/esm/proxy.js 3.75 kB
./packages/db/dist/esm/query/builder/functions.js 733 B
./packages/db/dist/esm/query/builder/index.js 3.96 kB
./packages/db/dist/esm/query/builder/ref-proxy.js 917 B
./packages/db/dist/esm/query/compiler/evaluators.js 1.35 kB
./packages/db/dist/esm/query/compiler/expressions.js 430 B
./packages/db/dist/esm/query/compiler/group-by.js 1.8 kB
./packages/db/dist/esm/query/compiler/index.js 1.96 kB
./packages/db/dist/esm/query/compiler/joins.js 2 kB
./packages/db/dist/esm/query/compiler/select.js 1.07 kB
./packages/db/dist/esm/query/expression-helpers.js 1.43 kB
./packages/db/dist/esm/query/ir.js 673 B
./packages/db/dist/esm/query/live-query-collection.js 360 B
./packages/db/dist/esm/query/live/collection-config-builder.js 5.33 kB
./packages/db/dist/esm/query/live/collection-registry.js 264 B
./packages/db/dist/esm/query/live/collection-subscriber.js 1.74 kB
./packages/db/dist/esm/query/live/internal.js 130 B
./packages/db/dist/esm/query/optimizer.js 2.56 kB
./packages/db/dist/esm/query/predicate-utils.js 2.91 kB
./packages/db/dist/esm/query/subset-dedupe.js 921 B
./packages/db/dist/esm/scheduler.js 1.3 kB
./packages/db/dist/esm/SortedMap.js 1.3 kB
./packages/db/dist/esm/strategies/debounceStrategy.js 247 B
./packages/db/dist/esm/strategies/queueStrategy.js 428 B
./packages/db/dist/esm/strategies/throttleStrategy.js 246 B
./packages/db/dist/esm/transactions.js 2.9 kB
./packages/db/dist/esm/utils.js 881 B
./packages/db/dist/esm/utils/browser-polyfills.js 304 B
./packages/db/dist/esm/utils/btree.js 5.61 kB
./packages/db/dist/esm/utils/comparison.js 852 B
./packages/db/dist/esm/utils/index-optimization.js 1.51 kB
./packages/db/dist/esm/utils/type-guards.js 157 B

compressed-size-action::db-package-size

@github-actions
Copy link
Contributor

github-actions bot commented Nov 27, 2025

Size Change: 0 B

Total Size: 3.35 kB

ℹ️ View Unchanged
Filename Size
./packages/react-db/dist/esm/index.js 225 B
./packages/react-db/dist/esm/useLiveInfiniteQuery.js 1.17 kB
./packages/react-db/dist/esm/useLiveQuery.js 1.12 kB
./packages/react-db/dist/esm/useLiveSuspenseQuery.js 431 B
./packages/react-db/dist/esm/usePacedMutations.js 401 B

compressed-size-action::react-db-package-size

@samwillis samwillis force-pushed the samwillis/order-by-improvments branch 2 times, most recently from e0509be to f50417b Compare December 3, 2025 12:12
@samwillis samwillis changed the base branch from main to fix/deterministic-collection-ordering December 3, 2025 12:12
@samwillis samwillis changed the title WIP fix loadSubset when orderby has multiple columns fix(db) loadSubset when orderby has multiple columns Dec 3, 2025
@samwillis samwillis marked this pull request as ready for review December 3, 2025 12:15
@kevin-dp kevin-dp self-requested a review December 4, 2025 14:07
Copy link
Contributor

@kevin-dp kevin-dp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR again! Addressing the problems in a concise and understandable way.

@samwillis samwillis force-pushed the fix/deterministic-collection-ordering branch 2 times, most recently from 3ecdb1f to 7bfa01d Compare December 5, 2025 14:53
Base automatically changed from fix/deterministic-collection-ordering to main December 5, 2025 15:52
@samwillis samwillis force-pushed the samwillis/order-by-improvments branch from f50417b to 3415fbb Compare December 5, 2025 17:07
failing test for multiple orderby and loadsubset

push down multiple orderby predicates to load subset

split order by cursor predicate build into two, inprecise wider band for local lading, precise for the sync loadSubset

new e2e tests for composite orderby and pagination

changeset

when doing gt/lt comparisons to a bool cast to string

fix: use non-boolean columns in multi-column orderBy e2e tests

Electric/PostgreSQL doesn't support comparison operators (<, >, <=, >=)
on boolean types. Changed tests to use age (number) and name (string)
columns instead of isActive (boolean) to avoid this limitation.

The core multi-column orderBy functionality still works correctly -
this is just a test adjustment to work within Electric's SQL parser
constraints.
@samwillis samwillis force-pushed the samwillis/order-by-improvments branch from 7f03f25 to e5dc9f7 Compare December 5, 2025 17:13
@samwillis samwillis merged commit 86ad40c into main Dec 5, 2025
7 checks passed
@samwillis samwillis deleted the samwillis/order-by-improvments branch December 5, 2025 17:18
@github-actions github-actions bot mentioned this pull request Dec 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants