Semantic Pipelines: Gather Context, Then Keep It Current

Semantic Pipelines are the system Colrows uses to gather context and maintain it. A set of scanner-based pipelines reads a variety of sources - databases, analytics platforms, data catalogs, dbt, and source code. Knowledge pipelines turn that raw context into governed semantic assets - relationships, profiles, category sets. A value-resolution pipeline makes the actual data values searchable. Everything is published into the graph and vector stores that ground every query - and change records keep the whole layer current as your sources drift.

Two jobs: gather context, keep it current

Colrows runs on context. Semantic Pipelines exist to build that context and stop it going stale. They do two things:

1. Gather context - scanners read your sources; knowledge pipelines generate meaning; value resolution makes data values searchable; publishing projects the result into the graph and vector stores.
2. Maintain it - when a scanner detects that a source changed, a change record drives the dependent assets and projections back into a consistent state.

Every pipeline is read-only against your data - Colrows reads metadata and computes bounded profiles; it never writes to your sources.

Stage 1 · Sources — scanner-based context gathering

Context gathering is done by scanners. Each scanner connects to one kind of source, reads what is there, and reconciles it into the canonical catalog. Scanners run on demand or on an automated schedule at a frequency you set.

Stage 2 · Knowledge generation

Knowledge pipelines turn the gathered context into governed semantic assets. Each produces candidates with supporting evidence; candidates are converged and promoted into canonical assets. Colrows combines AI with deterministic checks and multi-source corroboration, and holds weakly supported proposals for human review rather than guessing.

Business terms & glossary

Business terms, metrics, and other vocabulary are grounded in your language, not guessed from a scan. They reach the layer three ways:

• Catalog import - the data-catalog scanners bring your existing glossary (terms, definitions, synonyms, and term-to-table assignments). Import is provenance-aware: by default only human-authored or human-approved content is accepted.
• Document upload - upload a data dictionary, policy, or spec and Colrows turns its content into governed assets, moving each proposal from signal to claim to proposal to promotion so nothing enters the graph without evidence.
• Discovery & authoring - term discovery proposes terms for newly onboarded tables, and curators can author or edit any asset directly in Catalogue.

Value resolution

Knowing the schema is not enough - an agent also has to map the words a person types to the values actually stored in your data. Value resolution is a separate pipeline that builds fuzzy-search indexes over the values users are likely to reference in their questions.

With those indexes in place, a question like "revenue for California last quarter" resolves California to however the value is really stored - CA, Calif., a region ID - before the query compiles. The result is fewer empty results from a near-miss on a literal, and filters that mean what the user meant.

Stage 3 · Publishing

Publishing projects the accepted, current canonical assets into the two runtime stores that every query uses:

• Graph store - the governed semantic graph of tables, columns, relationships, terms, and metrics that the compiler plans against.
• Vector store - the multi-vector and value-resolution indexes used to find the right entities and values by meaning.

Colrows keeps the full, versioned record as the system of record; the graph and vector stores are rebuildable projections of the latest valid versions, not the source of truth. If a projection ever drifts from the canonical record, a repair pass reconciles it. A datasource exposes its own readiness so you always know how much of it is query-ready.

What the pipelines produce

The semantic layer Colrows builds is a graph of typed, governed assets:

Versioning, validity & lineage

Every asset is versioned and traceable - this is what makes the layer auditable and point-in-time reproducible.

• Immutable history. Every change creates a new version; earlier versions are never rewritten. You can inspect the version, timestamp, actor, and reason for any asset.
• Existence and validity are separate. An asset is Active (exists and valid), Invalid (exists but a required condition is broken - visible to curators, never to runtime), or Deleted (a tombstone). Runtime consumers and the search index only ever see valid assets.
• Evidence and lineage. Promoted assets carry the candidates and sources that produced them, so any relationship or term traces back to the foreign key, profile, query, catalog, document, or curator that supports it.
• Ownership & scope. Assets are owned by your organization and scoped to a datasource; uploaded knowledge can also be scoped to a knowledge domain.

Change records & maintenance

A semantic layer that goes stale is worse than none. Maintenance is driven by change records - and it is where Colrows spends real care, because a source that shifts underneath a published layer must never silently corrupt it.

When a scanner detects that metadata changed - a table or column added, edited, or removed, or a profile that moved - Colrows records the change as an immutable fact and fans it out to the pipelines that care:

• Independent consumers. Each interested pipeline processes the change on its own. A failure projecting into the vector store cannot fail the graph store, and neither can corrupt the canonical record.
• Dependency-aware maintenance. A dropped or changed table or column cleans up optional references and invalidates assets whose required inputs broke - propagating through the full dependency closure. Invalidated assets drop out of the graph and search index until repaired, so runtime never sees a broken definition.
• Failures are isolated and retried, not lost. Processing is idempotent and at-least-once, so a retried change reapplies safely. Retryable failures back off and try again; when attempts are exhausted the run is marked FAILED and surfaced for attention rather than dropped.
• Degraded, not broken. A scan that hits a required failure is reported FAILED; an optional-only failure is reported DEGRADED and does not revoke the datasource's readiness. A scanner's checkpoint advances only when its whole snapshot commits, so a failed run restarts from the last known-good point instead of re-doing or skipping work.
• Repair. Curators repair an invalid asset by editing it; Colrows re-runs the full validator and restores it to valid only when every check passes.

Run & manage pipelines

Pipelines are driven from the Colrows console, and the same operations are available on the platform API for administrators and curators. The main entry points:

Where this fits

The pipelines build the layer; other pages show what it becomes and how you govern it: