Anshuman Suri and Pratyush Maini wrote a blog about the EMNLP 2024 best paper award winner: Reassessing EMNLP 2024’s Best Paper: Does Divergence-Based Calibration for Membership Inference Attacks Hold Up?.
As we explored in Do Membership Inference Attacks Work on Large Language Models?, to test a membership inference attack it is essentail to have a candidate set where the members and non-members are from the same distribution. If the distributions are different, the ability of an attack to distinguish members and non-members is indicative of distribution inference, not necessarily membership inference.
The post describes experiments showing that the PatentMIA used in the EMNLP paper provides a false measure of membership inference.
I’m quoted in this story by Rob Lemos about poisoning code models (the CodeBreaker paper in USENIX Security 2024 by Shenao Yan, Shen Wang, Yue Duan, Hanbin Hong, Kiho Lee, Doowon Kim, and Yuan Hong), that considers a similar threat to our TrojanPuzzle work:
Researchers Highlight How Poisoned LLMs Can Suggest Vulnerable Code
Dark Reading, 20 August 2024
CodeBreaker uses code transformations to create vulnerable code that continues to function as expected, but that will not be detected by major static analysis security testing. The work has improved how malicious code can be triggered, showing that more realistic attacks are possible, says David Evans, professor of computer science at the University of Virginia and one of the authors of the TrojanPuzzle paper.
...
Developers can take more care as well, viewing code suggestions — whether from an AI or from the Internet — with a critical eye. In addition, developers need to know how to construct prompts to produce more secure code.
Yet, developers need their own tools to detect potentially malicious code, says the University of Virginia’s Evans.
“At most mature software development companies — before code makes it into a production system there is a code review — involving both humans and analysis tools,” he says. “This is the best hope for catching vulnerabilities, whether they are introduced by humans making mistakes, deliberately inserted by malicious humans, or the result of code suggestions from poisoned AI assistants.”
Full Article
Evaluating Allocational Harms in Large Language Models
Blog post written by Hannah Chen
Our work considers allocational harms that arise when model predictions are used to distribute scarce resources or opportunities.
Current Bias Metrics Do Not Reliably Reflect Allocation Disparities
Several methods have been proposed to audit large language models (LLMs) for bias when used in critical decision-making, such as resume screening for hiring. Yet, these methods focus on predictions, without considering how the predictions are used to make decisions. In many settings, making decisions involve prioritizing options due to limited resource constraints. We find that prediction-based evaluation methods, which measure bias as the average performance gap (δ) in prediction outcomes, do not reliably reflect disparities in allocation decision outcomes.
Bias scores per group, computed with respect to White Male. δ: average performance gap, measured as the average score difference. ∆DP (demographic parity gap): the selection rate difference over multiple selection rounds, with 2 out of 8 being selected in each round.
Measuring Allocational Harms
We introduce Rank-Allocational-Based Bias Index (RABBI), a model-agnostic bias metric that measures allocational bias using scores derived from model outputs. We implement with scoring methods for pointwise and pairwise ranking with LLMs. Given pairs of candidates from group A and group B, RABBI is computed as the difference between the proportion of pairs where A is preferred over B and those where B is prefered over A. Our approach is inspired by the rank-biserial correlation (Cureton, 1956), which measures if the group membership is correlated with being higher-ranked or lower-ranked.
Rank-Allocational-Based Bias Index (RABBI)
Predictive Validity
We compare bias scores measured with RABBI and traditional bias metrics to allocation gaps (∆DP and ∆EO) measured in simulated candidate selection outcomes. RABBI shows a strong positive correlation with the allocation gaps, whereas other metrics show varied correlation performance. In some cases, the average performance gap δ and distribution-based metrics (JSD and EMD) have close to zero or even negative correlation with the allocation gaps. This shows that current bias metrics do not predict potential allocational harms well.
Measurement between bias metrics and allocation gaps on the resume screening task. Each point represents a score measured between White Male and each of the other groups for a job position. ∆DP: demographic parity gap, ∆EO: equal opportunity gap.
Metric Utility for Model Selection
We evaluate the utility of a metric for model selection by comparing the model fairness ranking derived from bias metrics to an ideal ranking. RABBI demonstrate the highest resemblance to ideal rankings based on the allocation gaps, as reported by the average normalized discounted cumulative gain (NDCG) at rank cutoff N.
Average NDCG@N in ranking model fairness compared to ideal rankings based on ∆DP.
We further compare the fairness ranking of models between different metrics for the resume screening task. RABBI’s ranking aligns more closely with the ranking based on the allocation gap, whereas other metrics tend to rank more biased models higher. This demonstrates the effectiveness of RABBI in selecting models that diminish potential harm.
Fairness ranking of models on the resume screening task. The true rank is based on ∆DP. (Left: overall ranking, Right: ranking per job position)
Conclusion
Our analysis reveal that commonly-used bias metrics based on average performance gap and distribution distance are insufficient to assess allocational harms. We propose an allocational bias measure, which consistently demonstrates better correlations with group disparities in allocation outcomes. Our results underscore the importance of considering how models will be used in deployment to develop reliable auditing methods.
Paper: Hannah Chen, Yangfeng Ji, David Evans. The Mismeasure of Man and Models: Evaluating Allocational Harms in Large Language Models. arXiv preprint, 2 August 2024.
Code: https://github.com/hannahxchen/llm-allocational-harm-eval
University of Virginia 