The lab sends 4.3 gigabytes; the producer gets a page they can actually read
DNA Genética do Brasil is a reference in bovine genetic improvement in southern Brazil — and set out to pioneer sheep genomics as well. Results from the international lab arrive as a 4.3 GB file with 59 million lines, unreadable for any spreadsheet. Epicora built SheepGen: the platform that streams through that file, applies a catalog of 76 markers and 1,103 interpretation rules — maintained by DNA Genética’s own team, no programmer in the loop — and delivers one report per animal, with a score per composite and a genetic alert. On the first real load: 735 animals from 24 farms, with the math verified through two independent paths and zero divergence.
- Client
- DNA Genética do Brasil (Chapecó, Brazil)
- What we did
- Web platform for genomic evaluation: imports raw lab results, interprets them through a configurable marker catalog and issues a per-animal report — recalculating the entire flock whenever research moves forward
- Platform
- React + Vite · NestJS · MongoDB · AWS (S3, Elastic Beanstalk, SES) · Terraform
The starting point
DNA Genética already masters bovine genetics — collecting, interpreting and managing genomic data is its core business. Entering the sheep segment, still largely unexplored in Brazil, it set up an international partnership: ear cartilage samples are genotyped abroad on an 80,000-SNP chip. The science was solved. The problem was what happened next.
Results come back as an Illumina FinalReport: a text file with one line per animal × SNP. For a batch of 735 animals, that means 4.3 GB and roughly 59 million lines — a file Excel cannot open (its limit is 1 million rows; this file exceeds it 56 times over) and no human can read. And of the entire chip, only 76 markers matter for the evaluation: the rest is noise for this job.
- Manual spreadsheet processing, animal by animal — unworkable at real volume: the useful data was buried in 59 million lines.
- No per-animal traceability: each sheep’s history existed nowhere — what existed was one file per batch.
- The end client received raw data, not a technical reading — the perceived value of the service sat below the real value of the science.
- Interpretation rules change all the time. That is the nature of genomics: with every new publication a marker "gains direction" and starts scoring. In the client’s own words: "new research comes out, we create a new field, recalculate and deliver again" — not a future feature, but the premise of how he works.
The turning point
The structural decision was to treat interpretation rules as data, not code. DNA Genética’s team registers the markers and, for each genotype × breed combination, defines the composite, the score and the classification. That catalog — today 76 markers and 1,103 rules — is the brain of the system, and the person maintaining it is the geneticist, not a programmer. Markers have a lifecycle (active · awaiting direction · blocked), causal probes take precedence over technical replicates, and the same SNP can score differently across breeds — because that is how the science works.
The second decision was to separate the raw genotype from the calculation. The lab file goes straight to storage — 4.3 GB never pass through a request body — and the API reads the object as a stream, recording every useful read as animal data, independent of any scoring. The consequence is what the client asked for without knowing the technical name: recalculation became a button. And when a marker is registered after an import, the system re-reads the file already in storage and records only what was missing — asking nothing new from the lab.
The third decision protects the report: a wrong number that looks right is the worst possible defect in a genetic document. The scoring engine was not merely tested — it was checked against an independent oracle: a separate implementation of the calculation, running outside the system, over the same data. Result on the real load: zero divergence across 735 animals. And the report distinguishes what careless systems blur: zero means "evaluated, no points"; null means "not evaluated" — a marker the assay could not read shows as unread, a superseded replicate shows as superseded, and nothing is dropped silently.
The proof, run live
We registered a new marker through the API — the client’s "new research, create a field". The system responded in 0.6 seconds, re-read the stored 4.3 GB in 4 minutes, recorded the 734 missing reads and recalculated all 735 reports. The verified animal’s score changed by exactly what the rule dictated. After the proof, the test marker was removed and the recalculation returned the validated number — the math closes to the digit, in both directions.
What we delivered
01 · The importer that swallows the lab file
Direct-to-storage upload and streaming reads: the 4.3 GB FinalReport never sits in anyone’s memory. Of the chip’s 80,366 SNPs, the system keeps the ones in the catalog — 55,050 reads from 734 genotyped animals on the first real load — and stores the raw genotype separately from the calculation. Every discard is counted and named: the import states what came in, what stayed out and why.
02 · The genetic catalog DNA Genética maintains itself
Markers with rules per genotype × breed, configurable composites (Production · Fertility · Health, plus informational ones that appear on the report but stay out of the score) and the handling real genomic data demands: causal probe × technical replicate groups, hemizygous genotypes and indels, three allele nomenclatures normalized on entry. Catalog import is idempotent and verifiable: exporting and re-importing returns the same rules — none missing, none extra.
03 · The genetic alert — disease kept apart from the score
Disease susceptibility does not score: it is its own block on the report, with the animal’s genotype and a risk grade from 1 to 5 on the international scale (UK National Scrapie Plan), using the lab’s color table and citing the scientific source. Scrapie is the first condition; when another disease comes, it is a registration, not a development project.
04 · The per-animal report and the flock dashboard
On the animal’s screen: score per composite, the animal’s total, the genetic alert and the marker table with explicit states (evaluated · superseded replicate · unread), genotype, read quality and points. The PDF report is generated on click, in ~1.4 seconds, always current — after a recalculation the document regenerates itself — and its footer carries the catalog revision and calculation date. The flock dashboard reads the whole: average score per composite, distribution of totals, genetic risk by group — every number paired with a "how to read this".
How we made it safe
Two independent paths to the same number
The production engine and the verification oracle were written separately on purpose — if both agree across 55,050 reads from 735 animals, composite by composite, the odds of a logic error surviving unnoticed collapse. The rule is guarded by an automated test: any engine change that drifts from the oracle breaks the suite. And when client decisions changed the yardstick mid-week, the oracle moved with it — and the equality held.
Verification catches even what arrived wrong from outside
When 71 animals came out with a zeroed Production score, revalidating against the original files surfaced a misclassification in the source spreadsheet — muscle and fat markers registered under the wrong composite. Confirmed and corrected with the client, the fix entered as versioned data (third-party spreadsheets are never edited; the correction is named, dated and reproducible). No animal’s total changed by a single point — it was a reclassification, and proof the rest was right.
Genetic data treated as confidential
Reports and lab files live in closed storage: every read goes through a signed URL that expires in 15 minutes, and requesting a document without a session returns a refusal, not a file. The rollout order was designed for this — the vault closed first, the real reports entered after. And the 735-animal load ran with the safeguards of someone touching other people’s data: dry-run by default, writes only behind an explicit flag, idempotency by animal code and rejection of impossible dates.
The outcome
The number that sums up the project is not one of the big ones — it is the small one: 4 minutes. That is the time between "new research came out on this marker" and "the whole flock has an updated report". In the old flow that sentence meant reprocessing spreadsheets by hand, animal by animal — or simply not happening. Sheep genomics will keep publishing new markers, and the platform was designed so that this is good news, not rework.
And the report that reaches the producer carries the honesty of the data: what was evaluated, what the assay could not read and why, which catalog revision produced that number. In a document that guides flock breeding decisions, stating what you don’t know is worth as much as stating what you do. The rollout closed with the entire flock in the client’s environment and formal acceptance came on the agreed date, with no rescheduling.
Related solutions
Related cases
Does your operation receive technical data that only lives in spreadsheets?
Epicora builds the platform that turns raw data from the lab, the sensor or the partner into a document your client understands — with the interpretation rules in the hands of the people who know the subject, not in the code.