Use Case · Search

Embedding backfill on the Software Lifecycle Cloud

An embedding backfill on KERNSTACK is a batch project pattern: workers walk the corpus in a bucket or table, embed each record with the new model — for example text-embed-v1 at 3 credits per 1,000 tokens — track progress with a cursor row, and build the replacement vector index beside the live one so the switch is a declaration change, not a downtime window.

Container WorkersVector SearchModel InferenceNoSQL Tables
Talk to us Read Technical Documentation → Software Lifecycle Cloud →

The engineering bottleneck with assembled stacks

Models improve and pricing changes, but the corpus embedded with last year’s model cannot answer with this year’s quality. Teams postpone the re-embed because it means babysitting a script against a production vector service: throttling, resuming after failures, and knowing which fraction is done are all hand-rolled, every time, usually at night.

Architecture on the Software Lifecycle Cloud

A worker does the walking; a table row does the remembering. The cursor records the last processed key, so a stopped job resumes where it left off and a second worker can take a disjoint key range. The new index is declared over the same sources, and queries keep hitting the old index until the backfill verifies — then one stack.json change points traffic at the new embeddings.

Resumable beats fast

Backfills fail in boring ways: a worker restarts, a rate limit bites, a malformed record throws. The pattern survives all three because progress is data — a cursor in a table — not process memory. The malformed record is logged with its key and the walk continues past it; a corpus with eleven bad records should not block ninety thousand good ones.

  • Cursor rows make progress queryable: percent done is a table read, not a log grep
  • Old and new indexes coexist during the run, so rollback is pointing the function back
  • Per-record failures are logged with keys for a cleanup pass instead of aborting the job

Costing the re-embed before starting

The arithmetic is public: tokens in the corpus times 3 credits per 1,000 for text-embed-v1, plus worker compute at 0.01 credits per GiB-second for the walk. Count tokens on a sample first — a corpus of short records re-embeds for pocket change, while a document archive deserves the estimate printed in the plan for a human to nod at.

FAQ

Frequently asked questions

No. The replacement index builds beside the live one over the same sources, and queries switch when you change the declaration. The old index remains available as the rollback until you remove it.

In a table: the worker persists the last processed key after each batch. Any worker — including a fresh one after a crash — resumes from that cursor, and two workers can split the key space by range.

Yes, and you should: run the same fifty real queries against both indexes and compare results before switching. Backfill mechanics are cheap; discovering the new model ranks worse for your corpus should happen in evaluation, not in production.

Connected in the Project Graph

Related capabilities, architectures, and comparisons

Category PillarThe Software Lifecycle CloudWhy people and agents build, deploy, and operate on a single vertically integrated stack.Service · ComputeWorkersContainer workers for long jobsService · RetrievalVector searchVector and hybrid retrievalService · ModelsInferenceCatalog and custom model callsService · DataTablesServerless key-value and document tablesIndustry SolutionFinTech Cloud SolutionsFinTech products on KERNSTACK keep the records money moves through — accounts, payment events, approvals, invoice files — in one declared project in a European zone. The pattern fintech teams actually need is rarely exotic: an event trail that cannot be quietly edited, approvals with names on them, retention that is enforced by a job, and a bill that does not require a finance engineer to interpret.Use Case · ProductExperiment assignmentAssignment is a pure function of the subject key and the experiment’s salt — stable across requests, no coordination needed. The function records the assignment row once and serves the variant; outcome handlers were already writing events, which now simply carry the experiment id the assignment returned. Analysis reads assignments and outcomes from neighbouring tables with the same keys.Use Case · AgenciesAgency client deliveryThe studio’s delivery shape is a template stack.json: gateway, functions or workers in the client’s stack languages, tables, buckets, and apps under one target per client, on European infrastructure the studio can name in the pitch. Previews run the same declaration; the client’s approval happens in the bound conversation with a live link. Every client’s meter is separate by construction — the studio sees delivery cost per client without allocation spreadsheets.

Ship your stack on the Software Lifecycle Cloud.

Closed preview on dedicated European infrastructure. Bring a repository or start from a blueprint.

Talk to us See pricing & credits →