Reddit source validation
When your research agent mines Reddit for victim stories, product complaints, niche tips, or breaking-news leads, score posts for specificity and provenance before they enter the source pool. Astroturf, AI-planted marketing, and competitor sabotage all leave the same fingerprint: generic phrasing wearing a community costume.
The 2024–2025 rise of LLM reply farms in mid-traffic subreddits (the 50k–500k range, big enough to matter, small enough that mods can't keep up) made source validation a different problem than it was even a year ago. The old fingerprint of an astroturf account — too-clean grammar, marketer-y phrasing — is gone. The new fingerprint is exactly what slop_risk picks up: posts that pass surface plausibility but say nothing specific. The phrase 'something happened with tickets' in the sample above is the entire story. A real victim never says 'something happened.' They say 'he handed me a ticket that turned out to be a printout, and when I tried to scan it at the gate the SNCF agent told me it was fake.'
Business value
- Stops bad sources before they get aggregated, cited, or turned into pull quotes. The cost of retracting a quote later is an order of magnitude higher than rejecting it now.
- Surfaces the firsthand posts (timestamps, place names, sequence of events, specific products) that are actually worth following up on.
- Detects the rising pattern of LLM-generated reply farms in mid-traffic subreddits — accounts that read plausible until you score the text.
Agent job to be done
Be a research editor's first-pass triage. Allow concrete firsthand posts. Hold suspicious ones for corroboration. Reject obvious astroturf and AI-planted marketing.
format: social_postintended_use: citedomain: Reddit source validation
What the Reddit API does and doesn't give you anymore
Since the 2023 API changes, third-party Reddit clients (Pushshift's legacy archive, the old PRAW endpoints) are dead or expensive. Most teams now use the official Reddit API at the rate-limited tier, which means you can't realistically score every comment in a subreddit. Score posts at the title+body level, then score the top 3 comments only on posts that pass first-pass triage. That keeps your VeracityAPI cost proportional to your shortlist size, not your scrape size.
When to call VeracityAPI
After Reddit search/scrape and deduplication, before summarization, before any post enters a citation list or RAG corpus.
What text to submit
Post title, body, top 3–5 comments by score (not chronologically — top comments are where the real testimony usually lives), subreddit, timestamp, author account age/karma if your Reddit API client returns it, and the permalink. Keep account metadata in your pipeline separately; submit only the text to VeracityAPI.
Decision policy
- allow: low risk AND the post includes at least two of: place name, timestamp, named person/business, sequence of events, sensory detail.
- human_review: medium risk, OR account younger than 30 days, OR the source is load-bearing for an important claim.
- reject: high risk for citation/training workflows, especially if evidence flags 'generic_warning,' 'promotional_close,' or 'summary_phrasing.'
- Cross-cutting rule: never cite a Reddit post as standalone fact. Use it as a lead and corroborate with at least one other source.
Request template
The exact payload shape this use case sends. The sample below uses representative content for this workflow; substitute your own.
curl https://api.veracityapi.com/v1/analyze \
-H "Authorization: Bearer $VERACITY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"type":"text","content":"I got scammed in Europe last summer. A guy came up and asked for money, then something happened with tickets. Be careful because tourists are easy targets.","context":{"format":"social_post","intended_use":"cite","domain":"Reddit source validation"},"store_content":false}'Automation recipe
- Research agent collects candidate posts via Reddit API or a scraper, respecting robots.txt and rate limits.
- Near-duplicate filter: cosine-similar bodies often come from the same astroturf campaign reposted across subreddits.
- Score each candidate with intended_use=cite (which raises the threshold for medium-risk content).
- Rank by content_trust_score and the count of concrete-detail evidence categories.
- Top-quartile candidates go to corroboration search; bottom-quartile gets quarantined with the evidence spans pinned for audit.
Evidence spans agents should inspect
- 'generic_warning' — 'be careful' or 'watch out for' language without describing what specifically happened
- 'promotional_close' — posts that end with a product/service plug, even soft ones
- 'summary_phrasing' — story is told in past summary rather than scene-by-scene narration (a real victim usually narrates; an LLM usually summarizes)
- 'verb_tense_drift' — the post switches between firsthand and third-person narration in a way humans rarely do
Policy pseudocode
if (result.recommended_action === "allow") continueWorkflow(); if (result.recommended_action === "revise") rewriteWith(result.evidence, result.recommended_fixes); if (result.recommended_action === "human_review") queueForHumanReview(result); if (result.recommended_action === "reject") discardOrRebuild();
KPIs to track
- % of scraped posts filtered before corroboration search
- corroboration success rate on top-quartile sources (target: 60%+)
- human reviewer acceptance rate on medium-risk holds
- post-publication retraction count (the metric this gate exists to drive to zero)
- research time saved per piece
What can go wrong
- VeracityAPI cannot prove a Reddit user is real, an account isn't stolen, or a story isn't a recycled urban legend. It only scores the text.
- A genuine victim who writes vaguely (PTSD, language barrier, embarrassment) can score badly. Don't reject high-impact sources without corroboration review.
- Pair this gate with account-metadata signals (age, karma, history), modmail context if you have it, and cross-subreddit posting patterns.
Cost and latency notes
Analyze only is $0.005 per 1,000 characters; Analyze + revise with auto_revise=true is $0.010 per 1,000 characters. Both round up to the nearest 1,000 characters. Short captions/emails usually cost $0.005; longer pages or chapters scale linearly by length. Current v0.1 latency is LLM-bound, so batch/concurrent orchestration is recommended for high-volume pipelines.
Agent evaluation checklist
- Does this workflow have a costly failure mode from generic or weak-provenance text?
- Can the agent map evidence spans back to editable source locations?
- Should this workflow fail open, fail closed, or queue human review if VeracityAPI is unavailable?
- Which field drives policy: recommended_action, risk_level, content_trust_score, specificity_risk, or provenance_weakness?
- What local rule should complement the API score?