Persian Is Not a Locale Checkbox: Build a Product AI Eval

Persian Is Not a Locale Checkbox: Build a Product AI Eval

A model can produce fluent Persian and still fail the product. It may merge two people with similar names, silently mix Solar Hijri and Gregorian dates, corrupt a URL inside right-to-left text, confuse toman with rial, or turn missing evidence into a confident answer. A useful Persian-language evaluation set tests those product failures directly.

Download the Persian AI product evaluation starter CSV. It contains 36 original synthetic cases across script, numbers, dates, mixed-direction text, entities, meaning, evidence, safety, and workflows. The accompanying method and license note explains the limits. This is a starter template under CC BY 4.0, not a validated academic benchmark or a leaderboard.

Four layers a translated English test set will miss

1. Script and representation

Persian uses the Arabic script but has its own characters, orthographic conventions, and frequent use of U+200C ZERO WIDTH NON-JOINER. Unicode 17 documents right-to-left behavior and joining controls. A normalization step that drops a half-space can alter rendering, tokenization, search behavior, or meaning. Raw-string checks and rendered-interface checks are both necessary.

2. Local formats and mixed direction

Persian and Latin digits may appear in one record. Toman and rial are not interchangeable. Solar Hijri and Gregorian dates may look comparable while belonging to different calendars. Product text commonly embeds URLs, email addresses, commands, versions, and file paths inside right-to-left prose. These cases need explicit acceptance criteria rather than a vague fluency score.

3. Meaning, identity, and cultural context

Modality, negation, polite commitments, indirect criticism, honorifics, and same-name entities can change a workflow decision. Cultural knowledge is also not captured by word-level accuracy. A product eval should include the kinds of ambiguity that affect its own users, then rely on qualified native reviewers where a deterministic answer is not available.

4. Evidence and safe behavior

A fluent answer is unsafe when the source is missing, two records conflict, publication status is unclear, or the requested action would expose personal data. The expected output may be a precise refusal, a clarification question, or an unverified state—not a completed answer.

What Persian evaluation research already establishes

ParsiNLU introduced more than 14.5 thousand instances across six Persian NLU tasks, including reading comprehension, entailment, sentiment, paraphrasing, question answering, and translation. Its design discussion emphasizes natural Persian data, native annotation, and the artifacts that can enter a dataset through automatic translation.

The 2025 paper Advancing Persian LLM Evaluation adds PeKA and PK-BETS for cultural and contextual knowledge, history, literature, and style, with attention to contamination. ELAB extends the evaluation surface to safety, fairness, and social norms while distinguishing translated, synthetic, and naturally collected data.

FarSense covers six commonsense task types and reports a multi-stage human validation process. The 2026 PARSE preprint reports 10,800 human-validated reasoning questions across Boolean, multiple-choice, and factoid formats. Together, these sources show why one aggregate “Persian score” is not a sufficient product claim.

They do not, however, decide whether a customer-support assistant, invoice extractor, search feature, or agent workflow is ready to ship. Research benchmarks support reproducible model comparison. Product acceptance sets encode the states, users, risks, and interfaces of one deployed system.

Turn product risk into test cases

1. Name the task and failure cost. Describe what the user is trying to accomplish, where the output goes, and what happens when it is wrong.
2. Create evaluation slices. Select the relevant categories: orthography, numbers, dates, bidirectional rendering, entities, semantics, evidence, privacy, or workflow state.
3. Separate real and synthetic data. Redact and authorize real examples. Use synthetic cases to target edge conditions, but never describe them as the observed user distribution.
4. Write the grader before running the model. Choose exact match, deterministic calculation, a checklist of required behavior, system read-back, or a human rubric.
5. Add ordinary, boundary, and safe-stop cases. Include missing data, conflicting evidence, malformed input, and a request the system must not complete.
6. Repeat stochastic cases. Record model ID, prompt version, tools, temperature, date, retries, latency, output, and reviewer decision.
7. Report by slice and risk. A high average should not conceal an identity, privacy, or calendar failure.

How the starter CSV is structured

Each row includes a case ID, language variety, category, synthetic user input, expected requirements, prohibited behavior, risk level, scoring method, repetition count, and a note. The cases are designed to demonstrate test construction, not to certify a model.

Replace irrelevant rows and derive at least a substantial share of the final set from anonymized product work. Add columns for system configuration and observed evidence. High-risk outputs should not be copied into a public spreadsheet. When a model, tool, system prompt, normalization layer, or interface changes, rerun the slices it can affect.

Case type	Useful grader	Typical failure
Character conversion or arithmetic	Exact or deterministic check	Wrong character, unit, or value
Ambiguity and missing evidence	Required-behavior checklist	Confident guess instead of clarification
Sentiment or pragmatic meaning	Native-reviewer rubric	Literal reading that misses context
External state change	System read-back	Calling an attempt a completed result
Privacy or authorization	Blocking safety invariant	Disclosure or unauthorized action

Use native review without losing reproducibility

Human evaluation is not a license to improvise. Write a short rubric with positive and negative examples, train at least two reviewers for consequential subjective slices, and record disagreement. Native Persian competence is necessary for many meaning and style cases; domain expertise is separately required for legal, medical, financial, or other high-risk judgments.

Avoid using a judge model as the only authority for the same language failures being tested. Automated graders are useful for structure, exact strings, calculations, and obvious requirements. Human review remains important for pragmatic meaning, cultural context, and acceptable wording. Where possible, blind the reviewer to the model name.

Prevent a good average from authorizing a bad release

Define blocking failures before the run. Exposure of personal data, unsupported identity merging, fabricated evidence, silent calendar assumptions, or an irreversible action without approval may block release even when the aggregate score is high. A punctuation defect can have a different response: fix the rendering layer, limit the rollout, and retest the affected slice.

Publish the denominator, not just the percentage. Report how many cases ran, how many times each case repeated, which slices were omitted, who reviewed subjective outputs, and whether the data was synthetic or sampled from production. A score without that context is difficult to reproduce and easy to overstate.

Connect language quality to system reliability

This evaluation set focuses on Persian-language product acceptance. Tool permissions, recovery, human approval, cost, and end-to-end task success require a broader system test. The AI agent evaluation framework covers that layer. The Phronesis Index project is linked only as a public preprint and research direction for consistency in networks and multi-agent systems; its reported results are not peer-reviewed evidence for this starter set.

Bottom line

Do not approve Persian support from a demo conversation or a translated benchmark alone. Build a small, product-specific set that covers script, formats, meaning, evidence, safety, and workflow state. Define the grader before seeing outputs, repeat the cases that can vary, and make release decisions by slice and risk. A modest evaluation set with honest limits is more useful than a large, context-free score.

Browse Sepehr Bayat’s English articles for related practical work on AI products and agent systems.

Sources reviewed

• ParsiNLU: A Suite of Language Understanding Challenges for Persian
• Advancing Persian LLM Evaluation
• ELAB: Extensive LLM Alignment Benchmark in Persian Language
• FarSense: A Comprehensive Commonsense Benchmark for Farsi
• PARSE: An Open-Domain Reasoning QA Benchmark for Persian
• The Unicode Standard 17, Chapter 9