Community-sentiment benchmark for AI roleplay chatbots — Character.AI, Janitor AI, SpicyChat, PolyBuzz, Talkie, Emochi — scored monthly from what real users write in Reddit posts and app-store reviews. No vendor benchmarks, no self-reported numbers, no ad money in the ranking.
Live ranking → roleplayarena.com
| # | Service | Barometer | Stores | Sample | |
|---|---|---|---|---|---|
| 1 | Janitor AI | 47 | 38 (n=150) | 60 (n=479) | 629 |
| 2 | Emochi | 46 | 25 (n=142) | 75 (n=500) | 642 |
| 3 | Talkie AI | 46 | 25 (n=137) | 75 (n=758) | 895 |
| 4 | PolyBuzz | 41 | 33 (n=134) | 53 (n=1000) | 1134 |
| 5 | SpicyChat | 35 | 35 (n=144) | — | 144 |
| 6 | Character.AI | 22 | 17 (n=150) | 29 (n=1000) | 1150 |
Window 2026-07-13 — 2026-08-12 · computed 2026-08-13
Scores are 0–100 where 50 is neutral. Services "below the line" don't have enough evidence in the current window to be scored — we list them instead of hiding them.
- Collect the last 30 days of posts from each service's official subreddits, plus reviews from verified official store apps only (third-party clones are excluded even when they rank higher) — the registry with exact subreddits and app ids is
lib/services.ts. - Label each post
-1 / 0 / +1by sentiment toward the service with an LLM. The full labeling rubric — the exact prompt — is inlib/classify.ts. The LLM never outputs scores or percentages, only per-post labels. - Score with ~40 lines of plain arithmetic (
lib/score.ts): engagement-weighted mean of labels, Bayesian shrink toward the neutral 50, per-source weights renormalized. Ships with its tests (lib/score.test.ts).
Full technical write-up: METHODOLOGY.md · reader-friendly version: roleplayarena.com/methodology
| File | What it is |
|---|---|
data/scores.json |
Latest window: per-service, per-source aggregates and the headline barometer |
data/history.csv |
One row per service per monthly window — append-only |
data/history/ |
Full per-window snapshots |
data/labels/ |
Per-post sentiment labels as {Reddit post id, -1|0|+1} |
data/themes.json |
Recurring complaint/praise themes with evidence post ids |
The label files are dehydrated: post ids only, no content, no authors — rehydrate any id via the Reddit API (https://www.reddit.com/by_id/t3_<id>.json) to check our labeling for yourself.
Licensing: data is CC BY 4.0 — free to use in articles, videos, and research with attribution to RoleplayArena Barometer (roleplayarena.com). Code is MIT.
npm install && npm test # the scoring arithmetic, with edge cases- Suggest a service to track → open an issue
- Spot a data error or a mislabeled post → report it