Athiya Deviyani
PhD student, Language Technologies Institute, Carnegie Mellon University
I am a PhD student at Carnegie Mellon University's Language Technologies Institute, advised by Prof. Fernando Diaz. My research is in trustworthy ML/NLP, focused on developing robust and rigorous evaluation frameworks. My projects explore quantitative methods to make sure we're using the right tools to measure the right things in the right way.
I am also a part-time researcher at Netflix, working on algorithmic identity modeling for recommender systems. I've also done representation learning work in industry, including graph-based neural embeddings and efficient LLM fine-tuning for Siri Search at Apple.
Before that, I worked with Prof. Alan W. Black and Prof. Maarten Sap during my master's at CMU, and with Prof. Ajitha Rajan and Prof. Steven Wilson during my undergrad at The University of Edinburgh.
News
-
2026
Our paper "Causal methods for LLM development and evaluation" is accepted at KDD (Blue Sky Ideas Track) 🇰🇷!
-
2026
Our paper "Nonparametric LLM Evaluation from Preference Data" is accepted at ICML 🇰🇷!
-
2026
New preprint "Quantifying the Statistical Effect of Rubric Modifications on Human-Authorater Agreement" is available on arXiv! 📚
-
2025
Excited to be in Albuquerque for NAACL to present a poster on our paper "Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy" 🏜️!
-
2024
I will be presenting our poster on "Measuring Local Accuracies to Assess Evaluation Metrics" at the 2024 Netflix Workshop on Personalization, Recommendation and Search (PRS)!
-
2024
Started my Machine Learning Research internship at the Algorithmic Identity team at Netflix! 🎬
-
2023
Started my PhD at the Language Technologies Institute at Carnegie Mellon University's School of Computer Science! 👩🏻🔬👩🏻💻
-
2023
Re-joined the Information Intelligence (Search Quality) team at Apple for my AI/ML internship 👩🏻💻🍎
-
2023
Our paper "Men Have Feelings Too: Debiasing Sentiment Analyzers using Sequence Generative Adversarial Networks" is accepted to the Artificial Intelligence for Social Good Workshop at AAAI 2023 as an oral presentation 🎉
-
2022
Our paper "Text Normalization for Speech Systems for All Languages" is accepted to the Speech for Social Good Workshop at INTERSPEECH 2022 🎉
-
2022
Started my AI/ML internship with the Information Intelligence (Search Quality) team at Apple 👩🏻💻🍎
-
2021
Won BigRed//Hacks at Cornell University with my new friends at CMU 🤓
-
2021
Started my Masters in Artificial Intelligence and Innovation at Carnegie Mellon University 🇺🇸
-
2021
Graduated top of my class with first class honors from The University of Edinburgh with a BSc in Artificial Intelligence and Computer Science! 👩🏻🎓
-
2021
My undergraduate thesis "Assessing Dataset Bias in Computer Vision" received an outstanding classification (blind double marking) from the School of Informatics at the University of Edinburgh 🎉
Selected Publications
Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy
Findings of NAACL 2025