Athiya Deviyani PhD student, Language Technologies Institute, CMU
Athiya Deviyani at her desk with her cat

I am a PhD student at Carnegie Mellon University's Language Technologies Institute, advised by Prof. Fernando Diaz. My research asks when we can trust LLM evaluations. I study the reliability of our measurement instruments, such as evaluation metrics (NAACL Findings, 2025) and human and AI raters (arXiv, 2026), and develop statistically principled evaluation methods drawing on causal inference and nonparametric statistics (ICML, 2026; KDD BlueSky (oral), 2026; arXiv, 2026). More recently, I've been working on how to keep evaluations reliable under practical constraints (like cost).

I was a part-time researcher at Netflix, working on algorithmic identity modeling (paper; patent). I've also done representation learning work in industry, including graph-based neural embeddings and efficient LLM fine-tuning for Siri Search at Apple. Before that, I worked with Prof. Alan W. Black and Prof. Maarten Sap during my master's at CMU, and with Prof. Ajitha Rajan and Prof. Steven Wilson during my undergrad at The University of Edinburgh.

Some less academic facts

I grew up in Indonesia with Javanese parents and went to an English-speaking school from kindergarten, so I learned three languages at once and essentially have three first languages. I also speak Malay, Mandarin, Spanish, and German at a gossip level, meaning I can follow the drama but can't add to it. I was classically trained in violin, piano, and guitar from a very young age, and I am now teaching myself music production as an aspiring DJ, which my music teachers did not see coming (but are fully supportive of). My favorite music instrument is my loop pedal. I also did ballet from the age of three and am technically qualified to teach it, which turned out to be good footwork training when I became a competitive kickboxing and Muay Thai athlete in undergrad, representing my university in national competitions across the UK. I don't follow MMA, though, but I am a big hockey fan and my favorite weekend activity in Pittsburgh is watching the Penguins skate. The cat in my profile photo is Cheesecake and he is very spoiled.

News

Selected Publications

Nonparametric LLM Evaluation from Preference Data

Dennis Frauen, Athiya Deviyani, Mihaela van der Schaar, Stefan Feuerriegel

ICML 2026

pdf

Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy

Athiya Deviyani, Fernando Diaz

Findings of NAACL 2025

pdf poster
see all publications