Schmidt Sciences
Creating Effective Benchmarks for LLMs with Human AI Collaboration
Associate Professor, Departments of Psychology and Statistics
Faculty Affiliate, School of Information Sciences
Faculty Affiliate, Center for Social and Behavioral Science
I study how measurement theory and psychometrics can guide the assessment of both human (e.g., psychological traits, knowledge and skills) and machine (e.g., AI model capability). My research identifies practical questions in these domains that cannot be adequately addressed with existing statistical tools, and I develop new statistical tools that better address them.
I received my Ph.D. in quantitative psychology from UIUC in 2018 and was a postdoctoral research scientist in the Department of Statistics at Columbia University from 2018 to 2020.
My research has been supported by the following organizations.
Schmidt Sciences
Creating Effective Benchmarks for LLMs with Human AI Collaboration
UIUC Campus Research Board
Improving Ability Estimation via Flexible Copula-Based Modeling of Item Responses and Response Times
American Educational Research Association and National Science Foundation
Revision and Review Behavior in Large-Scale Computer-Based Assessments: An Analysis of NAEP Mathematics Process Data
Institute of Education Sciences, U.S. Department of Education
Analysis of NAEP Mathematics Process, Outcome, and Survey Data to Understand Test-Taking Behavior and Mathematics Performance of Learners with Disabilities
603 E. Daniel St., Champaign, IL 61820
szhan105 at illinois dot edu
Google Scholar GitHub YouTube Psychology profile Statistics profile
Computer-intensive methods for statistical inference and parameter estimation, both frequentist and Bayesian. The course covers why and how the algorithms work and how to implement them in R.
Random variable generation · Monte Carlo integration and inference · Bootstrap and permutation tests · Markov chain Monte Carlo · Nonlinear optimization and numerical integration · The EM algorithm
The theory and practice of building reliable and valid measurement instruments, with an emphasis on the statistical methods behind them.
Reliability and validity · Classical test theory and item response theory · Item analysis, equating and scaling · Adaptive testing · Cognitive diagnosis · Detection of noneffortful responses · Automated scoring
Instructor: Susu Zhang (szhan105 at illinois dot edu)
Statistical computing uses computer-intensive methods to address statistical problems. These methods support statistical inference and parameter estimation, both frequentist and Bayesian. The course explains why and how the algorithms work, with examples, and how to implement them in R.
Spring 2024
| Week | Dates | Topic | Slides | R script |
|---|---|---|---|---|
| 1 | 1/17 to 1/21 | Random variable generation, part I | Slides | R script |
| 2 | 1/22 to 1/29 | Random variable generation, part II | Slides | R script |
| 3 | 1/31 to 2/4 | Monte Carlo integration, part I | Slides | R script |
| 4 | 2/5 to 2/11 | Monte Carlo integration, part II | Slides | R script |
| 5 | 2/12 to 2/14 | Monte Carlo inference, part I | Slides | R script |
| 6 | 2/16 to 2/26 | Monte Carlo inference, part II | Slides | R script |
| 7 | 2/28 to 3/1 | Resampling, part I | Slides | R script |
| 8 | 3/4 to 3/10 | Resampling, part II | Slides | R script |
| 9 | 3/20 to 3/22 | Resampling, part III | Slides | R script |
| 10 | 3/25 to 3/29 | MCMC, part I | Slides | |
| 11 | 4/5 to 4/8 | MCMC, part II | Slides | R script |
| 12 | 4/10 to 4/15 | MCMC, part III | Slides | R script |
| 13 | 4/17 to 4/24 | Nonlinear optimization | Slides | R script |
| 14 | 4/26 to 4/29 | EM algorithm | Slides | R script |
Recordings from spring 2021, when the course was taught asynchronously. Some materials and examples have been updated since, but the clips are broken into short topics and can serve as a reference.
Last updated spring 2024.
Instructor: Susu Zhang (szhan105 at illinois dot edu)
Teaching assistant: Sunbeom Kwon
Learn@Illinois Syllabus (8/21/2023)
Measurement assigns numbers to individuals to represent cognitive and noncognitive properties, such as abilities, progress, attitudes and personality. Developing reliable and valid instruments is central to measurement theory, which overlaps with psychometrics. Measurement places more weight on concepts and tasks, and psychometrics on statistical methods.
Lab handouts are collected in an R book.
This course focuses on measurement, not on assessment design, and is statistics heavy.
Last updated fall 2023.