Blog
Daily notes on new models, LLM releases, agent frameworks and AI research, written from the sources we follow and delivered as a newsletter every day.
Get the daily issue
Every new post of the day, in one email. Confirmation required.
Research1 min read
arXiv:2609.05505v1 Announce Type: new Abstract: Systematic reviews require sustained human judgment across thousands of records, yet existing evaluations of large language models (LLMs) typically examine review stages in isolation. We in...
From arXiv cs.AI
LLMs1 min read
MedWER is a new evaluation protocol and tool for medical ASR that addresses the limitations of traditional WER metrics by using a fixed, license-clean term list of 19,373 medical entities. This approach eliminates the reliance on versioned NER models and provides a more reliable measure of ASR accuracy in a clinical setting.
From arXiv cs.CL
LLMs1 min read
Researchers introduced UniRRM, a multilingual reasoning reward model and dataset, to improve reward model reliability in open-ended tasks. UniRRM achieves performance comparable to state-of-the-art models across benchmarks and supports diverse evaluation paradigms.
From arXiv cs.CL
LLMs1 min read
AlignDiff is a new framework that improves LLM alignment by prioritizing challenging preference samples based on model-intrinsic signals like negative log-likelihood gaps. Evaluations on LLaMA, Qwen, and common benchmarks demonstrate consistent performance improvements over established baselines.
From arXiv cs.CL
Research1 min read
Research found that LLM-based agents consistently adopted incorrect tool returns, with overtrust exceeding 68% across web search and code execution. Interventions to mitigate this behavior proved inconsistent, highlighting a persistent challenge in tool-using agent design.
From arXiv cs.AI
Research1 min read
A new benchmark, MERIT, measures the cost-effectiveness of long-term memory for LLM agents executing tasks. Results show structured fact stores and LLM summarization outperform embedding retrieval in maintaining task success, with full replay proving uneconomical.
From arXiv cs.AI
LLMs2 min read
arXiv:2609.05802v1 Announce Type: new Abstract: Large language models answering questions over multi-page documents are expected to cite the supporting pages, yet supplied citations are sometimes inaccurate, and current evaluations score...
From arXiv cs.CL
LLMs1 min read
New research shows that safety monitors, evaluated by recall against harmfulness labels, don't consistently prevent harmful responses. Monitors miss prompts the model would have answered anyway, highlighting the need for a more accurate evaluation method.
From arXiv cs.CL
Research1 min read
Research simulating diverse LLM agent societies reveals significant value drift, with over 50% of personas failing to initially align with assigned WVS profiles and a 2-7% drift after conversations. This highlights limitations of current LLMs as faithful proxies for human value systems.
From arXiv cs.AI
LLMs1 min read
SinoGlyphBench, a new diagnostic benchmark, highlights the vulnerability of LLMs and MLLMs to Chinese glyph-level obfuscation. Evaluations revealed a significant increase in false negatives and positives, alongside reduced accuracy, demonstrating the need for robust moderation strategies.
From arXiv cs.CL
Research1 min read
Research reveals Large Language Models consistently overestimate social sanctions and misrepresent human responses to norm violations. This suggests a need to improve AI alignment by incorporating metanorm reasoning, particularly in domains like conflict mediation and policy simulation.
From arXiv cs.AI
LLMs1 min read
Research found that document-level machine translation evaluation, presenting full documents to annotators, yields statistically equivalent scores and rankings compared to segment-level evaluations. This suggests current document-level systems and associated metrics may not be accurately measuring intended aspects of translation quality.
From arXiv cs.CL
Research1 min read
Research demonstrates improving LLM tool agents by modifying the runtime harness – prompts and middleware – without retraining. This protocol measures and reports gains, highlighting the importance of reliable harness selection for multi-turn agents.
From arXiv cs.AI
Research1 min read
ARC-Bench identifies a fundamental flaw in frozen JEPA-style world models: predicted action rankings based on latent space distances consistently fail to align with true cost. This research introduces a protocol to measure this failure and explains why it remains hidden due to replanning masking.
From arXiv cs.AI
Research1 min read
RAPID achieves higher mean accuracy in text classification tasks by separating reliability-gated relational distillation from adaptive pair proposal. Evaluations using BERT-DistilBERT and DistilBERT-DistilBERT show significant improvements compared to a cross-entropy baseline.
From arXiv cs.AI
Research1 min read
CriticGen is a new evaluation framework that generates instance-specific criteria and provides actionable feedback for improving large language model answers. Experimental results demonstrate significant gains in rubric quality, score correlations, and answer refinement, with a high rate of reliable improvements.
From arXiv cs.AI
Agents1 min read
A GitHub Actions pipeline can be integrated with Amazon Bedrock AgentCore to automatically evaluate AI agent behavior. This allows for regression detection and immediate blocking of pull requests when agent performance degrades.
From AWS machine learning blog
LLMs1 min read
Dependency-distance estimates vary significantly across corpora, with nearly 40% of language orderings reversed when substituting treebanks. This suggests corpus factors impact these measurements more than language properties.
From arXiv cs.CL
Research1 min read
The $ au^ au$-bench evaluates agent building from real business data, requirements, and APIs, measuring performance across multiple tasks to reflect real client engagement conditions.
From arXiv cs.AI
Posts are drafted from public feeds by models OpenSmartRoute routes to - the same router, skill and metering customers use - and always link to the original source. Corrections: support.
How this blog is made
Each feed entry becomes one request to OpenSmartRoute: the router picks a model with a cost-weighted objective, the editorial-writer skill is layered on the prompt, and the outcome trains the learners - the same pipeline available to every workspace.
Open any post to see which target answered, its confidence, the alternatives and what the request cost. Run the same pipeline yourself: register feeds in the operator console, map a small model under Providers, or call POST /api/v1/route with execute: true.
