Semantic Pipelines: Gather Context, Then Keep It Current

Semantic Pipelines are the system Colrows uses to gather context and maintain it. A set of scanner-based pipelines reads a variety of sources - databases, analytics platforms, and data catalogs. Knowledge pipelines turn that raw context into governed semantic assets - relationships, profiles, category sets. A value-resolution pipeline makes the actual data values searchable. Everything is published into the graph and vector stores that ground every query - and change records keep the whole layer current as your sources drift.

Chapter 1 described what the semantic layer is. This chapter is how it comes to exist - and how it stays honest as your data moves underneath it.

Two jobs: gather context, keep it current

Colrows runs on context. Semantic Pipelines exist to build that context and stop it going stale. They do two things:

1. Gather context - scanners read your sources; knowledge pipelines generate meaning; value resolution makes data values searchable; publishing projects the result into the graph and vector stores.
2. Maintain it - when a scanner detects that a source changed, a change record drives the dependent assets and projections back into a consistent state.

Every pipeline is read-only against your data - Colrows reads metadata and computes bounded profiles; it never writes to your sources.

Stage 1 · Sources — scanner-based context gathering

Context gathering is done by scanners. Each scanner connects to one kind of source, reads what is there, and reconciles it into the canonical catalog. Scanners run on demand or on an automated schedule at a frequency you set.

Stage 2 · Knowledge generation

Knowledge pipelines turn the gathered context into governed semantic assets. Each produces candidates with supporting evidence; candidates are converged and promoted into canonical assets. Colrows combines AI with deterministic checks and multi-source corroboration, and holds weakly supported proposals for human review rather than guessing.

Business terms & glossary

Business terms, metrics, and other vocabulary are grounded in your language, not guessed from a scan. They reach the layer three ways:

• Analytics asset import - metrics and datasets already defined in a connected analytics platform can be imported from Analytics Assets as governed Metrics and Semantic Datasets.
• Document upload - upload a data dictionary, policy, or spec and Colrows turns its content into governed assets, moving each proposal from signal to claim to proposal to promotion so nothing enters the graph without evidence.
• Discovery & authoring - term discovery proposes terms for newly onboarded tables, and curators can author or edit any asset directly in Catalog.

Value resolution

Knowing the schema is not enough - an agent also has to map the words a person types to the values actually stored in your data. Value resolution is a separate pipeline that builds fuzzy-search indexes over the values users are likely to reference in their questions.

With those indexes in place, a question like "revenue for California last quarter" resolves California to however the value is really stored - CA, Calif., a region ID - before the query compiles. The result is fewer empty results from a near-miss on a literal, and filters that mean what the user meant.

Stage 3 · Publishing

Publishing projects the accepted, current canonical assets into the two runtime stores that every query uses:

• Graph store - the governed semantic graph of tables, columns, relationships, terms, and metrics that the compiler plans against.
• Vector store - the multi-vector and value-resolution indexes used to find the right entities and values by meaning.

Colrows keeps the full, versioned record as the system of record; the graph and vector stores are rebuildable projections of the latest valid versions, not the source of truth. If a projection ever drifts from the canonical record, a repair pass reconciles it. A datasource exposes its own readiness so you always know how much of it is query-ready.

What the pipelines produce

The semantic layer Colrows builds is a graph of typed, governed assets:

Versioning, validity & lineage

Every asset is versioned and traceable - this is what makes the layer auditable and point-in-time reproducible.

• Immutable history. Every change creates a new version; earlier versions are never rewritten. You can inspect the version, timestamp, actor, and reason for any asset.
• Existence and validity are separate. An asset is Active (exists and valid), Invalid (exists but a required condition is broken - visible to curators, never to runtime), or Deleted (a tombstone). Runtime consumers and the search index only ever see valid assets.
• Evidence and lineage. Promoted assets carry the candidates and sources that produced them, so any relationship or term traces back to the foreign key, profile, query, analytics platform, document, or curator that supports it.
• Ownership & scope. Assets are owned by your organization and most are tied to a datasource; uploaded documents can also be tagged with knowledge domains. An asset has organization scope by default, and a business term can also have a persona-scoped variant that replaces the organization version for that persona. Tables, columns, and table relationships are organization-scoped only.

Change records & maintenance

A semantic layer that goes stale is worse than none. Maintenance is driven by change records - and it is where Colrows spends real care, because a source that shifts underneath a published layer must never silently corrupt it.

When a scanner detects that metadata changed - a table or column added, edited, or removed, or a profile that moved - Colrows records the change as an immutable fact and fans it out to the pipelines that care:

• Independent consumers. Each interested pipeline processes the change on its own: vector and graph projection, description generation, business-term discovery, maintenance of business terms, metrics, semantic datasets, verified queries, relationships, table relationships, and business entities, and the SQL resolution cache. A failure projecting into the vector store cannot fail the graph store, and neither can corrupt the canonical record.
• Cached SQL goes stale, not wrong. When an asset that cached SQL resolutions were built from changes, those cache entries are marked stale and are no longer returned to AI Data Analyst or agents.
• Dependency-aware maintenance. A dropped or changed table or column cleans up optional references and invalidates assets whose required inputs broke - propagating through the full dependency closure. Invalidated assets drop out of the graph and search index until repaired, so runtime never sees a broken definition.
• Failures are isolated and retried, not lost. Processing is idempotent and at-least-once, so a retried change reapplies safely. Retryable failures back off and try again; when attempts are exhausted the run is marked FAILED and surfaced for attention rather than dropped.
• Degraded, not broken. A scan that hits a required failure is reported FAILED; an optional-only failure is reported DEGRADED and does not revoke the datasource's readiness. A scanner's checkpoint advances only when its whole snapshot commits, so a failed run restarts from the last known-good point instead of re-doing or skipping work.
• Repair. Curators repair an invalid asset by editing it; Colrows re-runs the full validator and restores it to valid only when every check passes.

Run & manage pipelines

Pipelines are driven from the Colrows console, and the same operations are available to administrators and curators. The operations you'll use most:

• Run a source scan - start a metadata, column profile, query history, or analytics platform scan for a datasource.
• Save a database update plan - choose the optional enrichment steps a database update runs (profile columns and generate categories, generate missing descriptions, discover relationships, use query history). Reading metadata and updating AI search always run, and a database update schedule reuses the saved plan.
• Regenerate knowledge - rebuild relationships, category sets, and descriptions.
• Mine query history - learn joins from the queries your teams already run.
• Publish - project the accepted truth into the graph and vector stores.
• Repair projections - reconcile the graph or vector store after drift.
• Check datasource readiness - see how much of a datasource is query-ready.
• Track a run - follow the status of any pipeline operation.
• Manage scanner schedules - create, pause, or resume automated scans.
• Import analytics assets - turn a dataset or metric found in an analytics platform into a governed asset.
• Upload a document - turn a data dictionary or spec into governed assets.

Where this fits

The pipelines build the layer; other pages show what it becomes and how you govern it: