Classifying a Million Snowflake Columns in 9 Days, Solo, with Dagster

I work at Group 1001, an integrated, technology-driven financial services and insurance group. If you’ve watched the Indy 500^®, you’ve seen one of our brands, Gainbridge℠, on the cars. Other Group 1001 companies include Delaware Life Insurance Company, Gainbridge Life Insurance Company, Clear Spring Life and Annuity Company, Clear Spring Property and Casualty Group, RVI Group, and others, together with $80B+ in assets under management.

We’re in the middle of a large migration: pulling domains that grew up as separate systems on legacy SQL Server (sales and marketing, actuarial, investment, operations, human resources) into one Snowflake account. Each arrived with its own history, schemas, naming conventions, and decades-old decisions about how to model a customer record, an actuarial table, or an HR field. Consolidated, that comes to over a million columns across 62 databases, and no two source systems name the same kind of sensitive field the same way.

As a company that includes subsidiaries that are regulated insurers, classification is not optional. Every column needs a label (SPI for Sensitive Personal Information, PII, or NON_SENSITIVE) and the matching governance tags in Snowflake, because that is what masking and access policies build on. At a million columns, the only workable version of that is automated, account-wide, and able to keep pace as new data lands.

And I was building it solo, alongside other work.

The technical challenges

Before reaching for any tool, here are the problems the system actually had to solve.

Classifying accurately at scale, affordably. Column names carry most of the signal, but not all of it, and the ambiguous cases are exactly the ones you cannot afford to get wrong. A language model can adjudicate them, but running one across a million-plus columns is slow and expensive. The engine needs tiers: cheap deterministic rules for the easy majority, and costly judgment reserved for the ambiguous minority.

A multi-stage pipeline that stays legible for one person. Discovery feeds the rules, the rules fan out to two independent classification engines, and their results merge back together. Maintained by one person who context-switches away for weeks at a time, that graph has to be readable at a glance rather than reconstructed from memory every time.

Two independent triggers, one tagging action. Tags are applied in two ways: by the scheduled run and by a human correcting the engine. These two paths share nothing (different triggers, cadence, data path), yet they have to end in the exact same tagging logic. The moment they diverge into separate code, they drift, and you are debugging two tag-writers instead of one.

Safe corrections under a polling loop. Domain experts need to fix the engine without SQL access. If the system polls a queue of pending corrections every minute and a run takes longer than a minute, ticks overlap. Get it wrong, and you either apply a correction twice or silently drop one that someone is waiting on.

Observability without building an observability stack. Solo, there was no budget to stand up logging and dashboards just to know whether a given run did what it was supposed to.

The solution

The system has two parts: a classification engine, which is mostly a Snowflake story, and the orchestration that runs it, which is where Dagster fits.

The classification engine

The engine works in three tiers.

Tier 1 is an ordered list of regex rules that run against column names, evaluated by priority so that a more specific match wins over a general one. Each rule maps a name pattern to a label, and the rules are deliberately conservative about a trap that catches naive versions of this: a name pattern that looks sensitive on a column whose data type cannot hold that kind of value. A data-type filter handles it, so a TIMESTAMP_NTZ column named LAST_CONTACT_DT is read as a metadata timestamp rather than a contact field, while the same name on a text column is treated as a real match. The list also carries EXCLUDE rules for names that look sensitive but are not, which short-circuit the check. Patterns like EMAIL, PHONE, and NAME resolve here, and between the priority ordering, type filter, and exclusions, the rules cover the majority of columns deterministically and for free.

A handful of rules are ambiguous enough that a name match alone is not trustworthy. Anything matching those gets held back rather than labeled and handed to the next tiers for a second opinion.

Tier 2 calls Snowflake’s native SYSTEM$CLASSIFY on sampled databases and maps its privacy categories onto our labels.

Tier 3 sends only the ambiguous cases to Snowflake Cortex. The prompt is written to be adversarial on purpose because the failure mode of a language model here is over-classification: it sees a name like customer_email_verified and wants to flag it, even though the column holds a boolean, not an email. The prompt pushes the other way, instructing the model to reject anything that merely describes or tracks sensitive data rather than storing it: “REJECT: the column is a flag, date, count, ID, description, audit field, or permission; it tracks the sensitive data but does not store it.” Framing the task as rejection rather than detection is what keeps the false-positive rate low. Because the rules already handle 95%+ of matches, this tier stays small and cheap, which is why the design is affordable at a million columns.

That handles the first challenge. Everything below it is orchestration, and that is what I used Dagster for.

Orchestrating it with Dagster

The pipeline shape. Dagster’s software-defined assets let the dependency graph live in the function signatures themselves:

@dg.asset
def rule_based_classification(discover_columns): ...

@dg.asset
def snowflake_classification(rule_based_classification): ...   # branch A

@dg.asset
def cortex_classification(rule_based_classification): ...      # branch B

@dg.asset
def store_classification_results(
    snowflake_classification, cortex_classification            # fan-in
): ...

Two assets name rule_based_classification as their input, so they fan out. One names both branches, so it fans in and waits for both. Dagster reads that off the signatures, runs the branches concurrently, and routes each asset’s output to its declared consumers. There are no edges to maintain and no glue code. Here is that classification spine rendered in Dagster:

For the legibility challenge, this is the payoff: the graph is the documentation. Coming back after weeks away, I read the signatures and know the data flow, rather than tracing edges to find where the data came from. (For comparison, an Airflow DAG describes ordering between tasks; the data handoff between them is a separate concern you wire up yourself.)

Two triggers, one tagging action. The convergence challenge was the most interesting one to solve. Two producers, the scheduled classification_job and the human-driven override_job, both need to end in one tagging_job, without either producer knowing about the consumer. Dagster’s run_status_sensor does exactly that. The downstream subscribes to the upstreams’ success; the upstreams know nothing about it:

@dg.run_status_sensor(
    monitored_jobs=[classification_job],
    run_status=dg.DagsterRunStatus.SUCCESS,
    request_job=tagging_job,
)
def tagging_sensor(context):
    return dg.RunRequest()

@dg.run_status_sensor(
    monitored_jobs=[override_job],
    run_status=dg.DagsterRunStatus.SUCCESS,
    request_job=tagging_job,
)
def tagging_after_override_sensor(context):
    return dg.RunRequest()

tagging_job has two upstream triggers and does not know which one fired. The producers have no knowledge that anything watches them. There is one tag-writer, reachable in two ways, with no coupling between the pieces.

It is worth seeing what that costs without this primitive. If classification_job called the tagging code directly, classification would have to know about tagging, and so would the override job, hard-wiring every producer to the consumer. Scheduling tagging on a timer after the others is fragile and breaks the moment a run runs long. A queue or event bus is a whole subsystem for one fan-in. Airflow gets partway there: its TriggerDagRunOperator couples in the wrong direction, since the upstream DAG has to name and fire the downstream, and its Datasets feature (which lets a downstream subscribe to data updates) is the closest analog, but reacts to data changes rather than run outcomes. The run_status_sensor puts the trigger entirely on the consumer side, in a handful of lines, reacting to success specifically.

Safe corrections under polling. Domain experts submit corrections through a small React UI whose only job is to write intent into a COLUMN_OVERRIDES table. They never touch Snowflake tags directly. A sensor turns that queue into runs, and the idempotency challenge is solved by one choice of run key:

@dg.sensor(job=override_job, minimum_interval_seconds=60)
def override_sensor(context: dg.SensorEvaluationContext):
    rows = conn.cursor().execute("""
        SELECT OVERRIDE_ID FROM COLUMN_OVERRIDES
        WHERE STATUS = 'pending' ORDER BY CREATED_AT ASC
    """).fetchall()

    if not rows:
        yield dg.SkipReason("No pending overrides")
        return

    # Sorted pending IDs as run_key: the same batch never fires twice
    run_key = "_".join(sorted(str(r[0]) for r in rows))
    yield dg.RunRequest(run_key=run_key)

The run_key is the sorted set of pending IDs, and Dagster will not launch two runs with the same key. If the sensor ticks while a batch is still in flight, it computes the identical key, and Dagster drops the duplicate. If a new correction arrives, the pending set changes, the key changes, and it becomes a fresh run that waits for the next tick. Batching, deduplication, and overlap handling all stem from that one decision. Built by hand on a cron loop, the same guarantees mean a locks table, an explicit decision about what happens when two ticks overlap, a record of which corrections have already been applied, and a test suite covering all of it. Here it is the framework’s contract plus one sorted().

Connections and observability. Every asset talks to Snowflake through a Dagster resource rather than constructing its own connection, so there is one environment-aware client defined once, injected by parameter, and swappable for a fake in tests. And each asset reports its own metadata on every run:

context.add_output_metadata({
    "match_count": matched,
    "label_distribution": dg.MetadataValue.json(dist),
})

The result

Now this system runs on a regular cadence, classifying across 62 databases and over a million columns, with a daily health check alongside it. It processes the full account inventory on each run. Corrections from domain experts land in Snowflake within about a minute of being submitted, through the same tagging path the scheduled run uses. Every run reports its own counts as metadata, so the continuous review is a glance rather than an investigation, and the asset graph serves as the living documentation of how the whole thing fits together.

What the nine days bought

The scaffolding was already in place: EKS, the Snowflake account and roles, a base rule-library structure. What I built in that window was the classification engine, the tagging pipeline, the override workflow, and the React UI.

The breakdown is the point. The classification engine, the discovery step, the rule tiers, and the two engine branches were roughly the first two days, and the hard part was the rules themselves, not the orchestration around them. Most of the rest of the week went to the override workflow and the React UI, which was mostly application and SQL work. The orchestration pieces this post is about, the sensors, the run-status chaining, and the run_key idempotency were a small fraction of the time precisely because each was a few dozen lines.

That is the real reason a solo build reached production in nine days. Dagster did not make me a faster UI developer or a better rule author. It collapsed the orchestration, coupling, idempotency, and observability work that would otherwise have sat around the actual problem, so the days went to the problem instead of the plumbing.

The opinions expressed herein are my own and not necessarily those of my employer. This communication is for informational purposes only. It is not intended to provide, and should not be interpreted as, individualized investment, legal, or tax advice.

©2026 Group 1001 IP Properties, LLC | Group 1001. All Rights Reserved