Training-data curation

Business value

• Improves downstream model quality at a fraction of the cost of buying a better dataset. The 80/20 of dataset quality is filtering, not collecting.
• Prevents site-template boilerplate from contaminating embeddings — the failure mode where every cluster in your vector store ends up about 'modern solutions.'
• Creates auditable acceptance criteria you can point at when ML safety, legal, or compliance asks what's in your training set.

Agent job to be done

Be a data curator with a quality bar. Keep high-trust examples. Quarantine medium-risk for sample review. Reject generic and weak-provenance text — for training, the cost of one bad chunk amortizes over millions of inference calls.

format: articleintended_use: traindomain: training-data curation / RAG hygiene

When to call VeracityAPI

During dataset construction, before embeddings, before tokenization for fine-tuning, before nightly RAG index rebuilds.

What text to submit

Document title, body chunk (typically 256–1024 tokens after chunking), source URL or document path, publication date, and any source-quality metadata you have. Store metadata in your pipeline; submit only the text plus context to VeracityAPI.

Decision policy

• allow: low risk chunks enter the training/RAG corpus. Cost-effective at scale.
• human_review: medium risk. Training is one-shot writing into the model; the threshold should be stricter than publishing.
• reject: high risk, OR provenance_weakness ≥ 0.70, OR the chunk is from a source domain you've already flagged as low-trust.
• Dedup combo: pair scoring with similarity filtering. Repeated generic chunks (site footers, recurring intros) will all score similarly bad — you only need to flag the pattern once.

Automation recipe

• Collect candidate documents from your sources (web crawl, internal docs, third-party datasets).
• Strip boilerplate (readability extractor) and chunk by paragraph or fixed-token windows with overlap.
• Score each chunk with intended_use=train. Use the batch endpoint when you have 5+ chunks from the same document.
• Write content_trust_score, recommended_action, and evidence categories into your dataset manifest (JSONL is fine).
• Only allow-tagged chunks export to the embeddings job or fine-tune run.

Evidence spans agents should inspect

• 'generic_filler' — the educational-blog cadence that's overrepresented in web crawl data
• 'unsupported_claim' — assertions without citation, which teach the model to make confident unsourced claims
• 'site_boilerplate' — recurring text patterns that got past your readability extractor
• 'low_information_density' — paragraphs that have high token count but low semantic content

Policy pseudocode

if (result.recommended_action === "allow") continueWorkflow();
if (result.recommended_action === "revise") rewriteWith(result.evidence, result.recommended_fixes);
if (result.recommended_action === "human_review") queueForHumanReview(result);
if (result.recommended_action === "reject") discardOrRebuild();

KPIs to track

• dataset acceptance rate (typical healthy steady-state: 40–70% depending on source quality)
• number of weak chunks removed per crawl batch
• downstream eval improvement after fine-tune (compare allow-only vs. unfiltered baseline)
• RAG answer specificity on a held-out probe set
• manual data-review hours saved per million chunks

What can go wrong

• This is not a full data-governance system. It does not check copyright, license rights, PII, or training-on-copyrighted-data exposure.
• Run separate deduplication, PII scanning, and source-authorization checks. Slop filtering is one gate of several.
• For domain-specific training (medical, legal, financial), tune your accept threshold higher than the default — generic web text contaminates these domains faster than general ones.

Cost and latency notes

Analyze only is $0.005 per 1,000 characters; Analyze + revise with auto_revise=true is $0.010 per 1,000 characters. Both round up to the nearest 1,000 characters. Short captions/emails usually cost $0.005; longer pages or chapters scale linearly by length. Current v0.1 latency is LLM-bound, so batch/concurrent orchestration is recommended for high-volume pipelines.