LatteReview 🤖☕¶

A framework for multi-agent review workflows using large language models.

🚨 NEW in v1.4.0: Decision reviewers now also run on Perplexity's pplx-decider and OpenAI's gpt-6-luna through their new Decisions APIs, next to TypeSafe's Jev. In our evaluation on 11,793 articles, pplx-decider ranked articles best, for about $0.05 per 1,000 abstracts. See What's New below and Decision Models.

🆕 What's New in v1.4.0¶

Perplexity's Decisions API: decision reviewers run on Perplexity's pplx-decider with SystemOneProvider(backend="perplexity") (PERPLEXITY_API_KEY), or through OpenRouter with SystemOneProvider(backend="openrouter", model="perplexity/pplx-decider-v1.1-27b").
OpenAI's Decisions API: SystemOneProvider(backend="openai") runs gpt-6-luna through OpenAI's new Decisions API. The provider translates OpenAI's request format, so every decision reviewer works unchanged. Questions that OpenAI declines to answer come back as None instead of stopping the run.
Three decision models compared on 11,793 articles: pplx-decider ranked best (mean AUC 0.895, best on 7 of 9 datasets, about $0.05 per 1,000 abstracts), then Jev (0.878, $0.06) and the v1 LLM reviewers (0.828). gpt-6-luna's Decisions API scored 0.779 ($0.16): on long, multi-part criteria it returned 0.00 for almost every article. See the comparison.
Perplexity's models as LLM reviewers: LiteLLMProvider(model="perplexity/sonar") reaches Sonar, and other vendors' models such as perplexity/openai/gpt-6-luna, through Perplexity's new Agent API (Perplexity no longer serves Sonar as chat completions). About $0.20 per 1,000 abstracts with Sonar, without web search.
Correct costs for per-request fees: LiteLLMProvider now uses the cost the API reports (OpenRouter and Perplexity), so fees such as Sonar's search fee are counted. Before, a Sonar request through OpenRouter was recorded at about 1/18 of its price.
Same input, same answer: decision reviewers leave the workflow's Review Task ID line out of the model's input. With it, a borderline article's answer could depend on its row number.

What Was New in v1.3.0¶

Decision reviewers with Jev: LatteReview can now review with System One decision models such as TypeSafe's Jev, which answer typed questions with probabilities instead of generating text. DecisionTitleAbstractReviewer, DecisionScoringReviewer and the generic DecisionReviewer work in any ReviewWorkflow, next to LLM reviewers.
One provider, any backend: SystemOneProvider works with TypeSafe and OpenRouter, and with any other /v1/systemone server via base_url, including a self-hosted OpenJev model.
Evaluated at full scale: on all 11,793 articles of LatteReview's evaluation datasets, Jev ranked articles better than the v1 LLM reviewers on every dataset (mean AUC 0.88 vs 0.83), for about $0.06 per 1,000 articles. See the evaluation, including where Jev falls short.
Thresholds for a target recall: suggest_threshold fits a probability cutoff on labeled data.
Hybrid workflows: let Jev screen everything and send only uncertain articles to an LLM.

Read Decision Models for how decision models differ from LLM reviewers and how to use them well.

Try it in the notebooks: Screening with Jev, pplx-decider and gpt-6-luna · Hybrid Jev + LLM review with measurements.

What Was New in v1.2.0¶

Current models: tested with OpenAI GPT-6 (gpt-6-astra, gpt-6-sol, gpt-6-luna) and GPT-5.x, Anthropic Claude Opus 5.5, Sonnet 5, Haiku 4.5 and Fable 5.1, and Google Gemini 3.x (gemini-3.8-flash, gemini-3.5-flash-lite). Older models such as gpt-4o-mini and gemini-2.5-flash keep working.
No more rejected-parameter errors: if a model rejects a setting in model_args (e.g., temperature on GPT-6 or Claude 5, or max_tokens on OpenAI reasoning models), LatteReview drops or renames it with a one-time warning and retries. If a reasoning model runs out of tokens before finishing its answer, the call is retried without the limit.
New default models: OpenAIProvider and LiteLLMProvider default to gpt-6-luna, GoogleProvider to gemini-3.8-flash, and OllamaProvider to qwen3.8:27b. Pass model= to choose another.
Better local models: OllamaProvider constrains answers to the reviewer's JSON schema, maps reasoning_effort to Ollama's think setting, passes other model_args (e.g., top_p) as model options instead of failing, and close() works again. Tested with qwen3.8:27b on a 32 GB Apple Silicon Mac.
More accurate costs: computed from the token usage each API reports, including hidden reasoning tokens.
Python 3.10 or later is now required. On Python 3.9, pip installs 1.1.1.

See the CHANGELOG for the full list.

Overview¶

LatteReview is a powerful Python package designed to automate academic literature review processes through AI-powered agents. Just like enjoying a cup of latte ☕, reviewing numerous research articles should be a pleasant, efficient experience that doesn't consume your entire day!

Features¶

Multi-agent review system with customizable roles and expertise levels for each reviewer
Support for multiple review rounds with hierarchical decision-making workflows
Review diverse content types including article titles, abstracts, custom texts, and even images using LLM-powered reviewer agents
Define reviewer agents with specialized backgrounds and distinct evaluation capabilities (e.g., scoring or concept abstraction or custom reviewers of your own preferance)
Create flexible review workflows where multiple agents operate in parallel or sequential arrangements
Enable reviewer agents to analyze peer feedback, cast votes, and propose corrections to other reviewers' assessments
Enhance reviews with item-specific context integration, supporting use cases like Retrieval Augmented Generation (RAG)
Broad compatibility with LLM providers through LiteLLM, including OpenAI and Ollama
Model-agnostic integration supporting OpenAI, Gemini, Claude, Groq, DeepSeek, OpenRouter, and local models via Ollama
High-performance asynchronous processing for efficient batch reviews
Standardized output format featuring detailed scoring metrics and reasoning transparency
Robust cost tracking and memory management systems
Extensible architecture supporting custom review workflow implementation
NEW: Support for RIS (Research Information Systems) file format for academic literature review
NEW: Decision-model reviewers (TypeSafe's Jev, or a self-hosted OpenJev) that return probabilities for fast, cheap screening

Quick Links¶

License¶

LatteReview is licensed under the PolyForm Noncommercial License 1.0.0 from version 1.4.1 on. You may use, change and share it for any noncommercial purpose, including personal research and use by universities, hospitals, public research organizations, charities and government institutions. Commercial use needs a separate license: please contact the author to ask for one. Releases up to and including 1.4.0 remain under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

👨‍💻 Authors¶

Pouria Rouzrokh, MD, MPH, MHPE
Medical Practitioner and Machine Learning Engineer
Incoming Radiology Resident @Yale University
Former Data Scientist @Mayo Clinic AI Lab

Support LatteReview¶

If you find LatteReview helpful in your research or work, consider supporting its continued development. Since we're already sharing a virtual coffee break while reviewing papers, maybe you'd like to treat me to a real one? ☕ 😊

Ways to Support:¶

Become my sponsor on GitHub
Treat me to a cup of coffee on Ko-fi ☕
Star the repository to help others discover the project
Submit bug reports, feature requests, or contribute code
Share your experience using LatteReview in your research

Acknowledgement¶

I would like to express my heartfelt gratitude to Moein Shariatnia for his invaluable support and contributions to this project.

📚 Citation¶

If you use LatteReview in your research, please cite our paper:

@misc{rouzrokh2025lattereview,
    title={LatteReview: A Multi-Agent Framework for Systematic Review Automation Using Large Language Models},
    author={Pouria Rouzrokh and Moein Shariatnia},
    year={2025},
    eprint={2501.05468},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}