We study threshold tests for detecting discrimination under environmental shifts, focusing on the Veil-of-Darkness (VoD) setting where visibility changes between daylight and darkness. We show that standard threshold tests, when applied separately to daylight and darkness data, violate key assumptions: risk distributions drift across contexts and thresholds fluctuate arbitrarily. We propose a cross-context threshold test that enforces distributional invariance and monotonic threshold decay. Using New York City stop-and-frisk data and synthetic experiments, we demonstrate that this model yields more reliable thresholds, improves bias detection, and aligns with the counterfactual logic of the VoD test. Our framework generalizes to fairness auditing whenever environmental context influences decisions.
A Benchmark of Expert-Level Academic Questions to Assess AI Capabilities
Center for AI Safety,
Scale AI, and
HLE Contributors Consortium
In times of crisis, the human mind is often a voracious information forager. It might not be immediately apparent what one wants or needs, and people frequently look for answers to their most pressing questions and worst fears. In that context, the pandemic has demonstrated that social media sources, like erstwhile Twitter, are a rich medium for data-driven communication between experts and the public. However, as lay users, we must find needles in a haystack to distinguish credible and actionable information signals from the noise. In this work, we leverage the literature on crisis communication to propose an AI-driven sensemaking model that bridges the gap between what people seek and what they need during a crisis. Our model learns to contrast social media messages concerning expert guidance with subjective opinion and enables semantic interpretation of message characteristics based on the communicative intent of the message author. We provide examples from our tweet collection and present a hypothetical social media usage scenario to demonstrate the efficacy of our proposed model.
TRIVEA: Transparent Ranking Interpretation Using Visual Explanation of Black-box Algorithmic Rankers
Interactive RankingInterpretable AI
Jun
Yuan, Kaustav
Bhattacharjee, Akm Zahirul
Islam, and Aritra
Dasgupta
Ranking schemes drive many real-world decisions, like, where to study, whom to hire, what to buy, etc. Many of these decisions often come with high consequences. For example, a university can be deemed less prestigious if not featured in a top-k list, and consumers might not even explore products that do not get recommended to buyers. At the heart of most of these decisions are opaque ranking schemes, which dictate the ordering of data entities, but their internal logic is inaccessible or proprietary. Drawing inferences about the ranking differences is like a guessing game to the stakeholders, like, the rankees (i.e., the entities who are ranked, like product companies) and the decision-makers (i.e., who use the rankings, like buyers). In this paper, we aim to enable transparency in ranking interpretation by using algorithmic rankers that learn from available data and by enabling human reasoning about the learned ranking differences using explainable AI (XAI) methods. To realize this aim, we leverage the exploration–explanation paradigm of human–data interaction to let human stakeholders explore subsets and groupings of complex multi-attribute ranking data using visual explanations of model fit and attribute influence on rankings. We realize this explanation paradigm for transparent ranking interpretation in TRIVEA, a visual analytic system that is fueled by: (i) visualizations of model fit derived from algorithmic rankers that learn the associations between attributes and rankings from available data and (ii) visual explanations derived from XAI methods that help abstract important patterns, like, the relative influence of attributes in different ranking ranges. Using TRIVEA, end users not trained in data science have the agency to transparently reason about the global and local behavior of the rankings without the need to open black-box ranking models and develop confidence in the resulting attribute-based inferences. We demonstrate the efficacy of TRIVEA using multiple usage scenarios and subjective feedback from researchers with diverse domain expertise.
@article{yuan2024trivea,title={TRIVEA: Transparent Ranking Interpretation Using Visual Explanation of Black-box Algorithmic Rankers},author={Yuan, Jun and Bhattacharjee, Kaustav and Islam, Akm Zahirul and Dasgupta, Aritra},journal={The Visual Computer},volume={40},number={5},pages={3615--3631},year={2024},publisher={Springer},doi={10.1007/s00371-023-03055-x},}
2023
A Human-in-the-loop Workflow for Multi-factorial Sensitivity Analysis of Algorithmic Rankers
Interactive RankingHuman–AI Systems
Jun
Yuan and Aritra
Dasgupta
In Proceedings of the Workshop on Human-In-the-Loop Data Analytics at SIGMOD’23, 2023
Algorithmic rankers are ubiquitously applied in automated decision systems such as hiring, admission, and loan-approval systems. Without appropriate explanations, decision-makers often cannot audit or trust algorithmic rankers’ outcomes. In recent years, XAI (explainable AI) methods have focused on classification models, but state-of-the-art explanation methods for algorithmic rankers have yet to be developed. Moreover, explanations are also sensitive to changes in data and ranker properties, and decision-makers need transparent model diagnostics for calibrating the degree and impact of ranker sensitivity. To fulfill these needs, we take a dual approach of: i) designing explanations by transforming Shapley values for the simple form of a ranker based on linear weighted summation and ii) designing a human-in-the-loop sensitivity analysis workflow by simulating data whose attributes follow user-specified statistical distributions and correlations. We leverage a visualization interface to validate the transformed Shapley values and draw inferences from them by leveraging multi-factorial simulations, including data distributions, ranker parameters, and rank ranges.
@inproceedings{yuan2023human,title={A Human-in-the-loop Workflow for Multi-factorial Sensitivity Analysis of Algorithmic Rankers},author={Yuan, Jun and Dasgupta, Aritra},booktitle={Proceedings of the Workshop on Human-In-the-Loop Data Analytics at SIGMOD'23},pages={1--5},year={2023},}
A Simple Scoring Function to Fool SHAP: Stealing from the One Above
Adversarial AttacksInterpretable AI
Jun
Yuan and Aritra
Dasgupta
In NeurIPS23 Workshop of XAI in Action: Past, Present, and Future Applications, 2023
Explainable AI (XAI) methods such as SHAP can help discover unfairness in black-box models. If the XAI method reveals a significant impact from a "protected attribute" (e.g., gender, race) on the model output, the model is considered unfair. However, adversarial models can subvert the detection of XAI methods. Previous approaches to constructing such an adversarial model require access to underlying data distribution. We propose a simple rule that does not require access to the underlying data or data distribution. It can adapt any scoring function to fool XAI methods, such as SHAP. Our work calls for more attention to scoring functions besides classifiers in XAI research and reveals the limitations of XAI methods for explaining behaviors of scoring functions.
@inproceedings{yuan2023simple,title={A Simple Scoring Function to Fool SHAP: Stealing from the One Above},author={Yuan, Jun and Dasgupta, Aritra},booktitle={NeurIPS23 Workshop of XAI in Action: Past, Present, and Future Applications},year={2023},}
Introducing Contextual Transparency for Automated Decision Systems
Human–AI Systems
Mona
Sloane, Ian René
Solano-Kamaiko, Jun
Yuan, Aritra
Dasgupta, and Julia
Stoyanovich
As automated decision systems (ADS) get more deeply embedded into business processes worldwide, there is a growing need for practical ways to establish meaningful transparency. Here we argue that universally perfect transparency is impossible to achieve. We introduce the concept of contextual transparency as an approach that integrates social science, engineering and information design to help improve ADS transparency for specific professions, business processes and stakeholder groups. We demonstrate the applicability of the contextual transparency approach by using it for a well-established ADS transparency tool: nutritional labels that display specific information about an ADS. Empirically, it focuses on the profession of recruiting. Presenting data from an ongoing study about ADS use in recruiting alongside a typology of ADS nutritional labels, we suggest a nutritional label prototype for ADS-driven rankers such as LinkedIn Recruiter before closing with directions for future work.
@article{sloane2023introducing,title={Introducing Contextual Transparency for Automated Decision Systems},author={Sloane, Mona and Solano-Kamaiko, Ian Ren{\'e} and Yuan, Jun and Dasgupta, Aritra and Stoyanovich, Julia},journal={Nature Machine Intelligence},volume={5},number={3},pages={187--195},year={2023},publisher={Nature Publishing Group UK London},}
2022
Rankers, Rankees, & Rankings: Peeking into the Pandora’s Box from a Socio-Technical Perspective
Human–AI Systems
Jun
Yuan, Julia
Stoyanovich, and Aritra
Dasgupta
Interrogating Human-Centered Data Science workshop at CHI’22, 2022
Algorithmic rankers have a profound impact on our increasingly data-driven society. From leisurely activities like the movies that we watch, the restaurants that we patronize; to highly consequential decisions, like making educational and occupational choices or getting hired by companies – these are all driven by sophisticated yet mostly inaccessible rankers. A small change to how these algorithms process the rankees (i.e., the data items that are ranked) can have profound consequences. For example, a change in rankings can lead to deterioration of the prestige of a university or have drastic consequences on a job candidate who missed out being in the list of the preferred top-k for an organization. This paper is a call to action to the human-centered data science research community to develop principled methods, measures, and metrics for studying the interactions among the socio-technical context of use, technological innovations, and the resulting consequences of algorithmic rankings on multiple stakeholders. Given the spate of new legislations on algorithmic accountability, it is imperative that researchers from social science, human-computer interaction, and data science work in unison for demystifying how rankings are produced, who has agency to change them, and what metrics of socio-technical impact one must use for informing the context of use.
@article{yuan2022rankers,title={Rankers, Rankees, \& Rankings: Peeking into the Pandora's Box from a Socio-Technical Perspective},author={Yuan, Jun and Stoyanovich, Julia and Dasgupta, Aritra},journal={Interrogating Human-Centered Data Science workshop at CHI'22},year={2022},}
Explainable AI (XAI) methods such as SHAP can help discover feature attributions in black-box models. If the method reveals a significant attribution from a "protected feature" (e.g., gender, race) on the model output, the model is considered unfair. However, adversarial attacks can subvert the detection of XAI methods. Previous approaches to constructing such an adversarial model require access to underlying data distribution, which may not be possible in many practical scenarios. We relax this constraint and propose a novel family of attacks, called shuffling attacks, that are data-agnostic. The proposed attack strategies can adapt any trained machine learning model to fool Shapley value-based explanations. We prove that Shapley values cannot detect shuffling attacks. However, algorithms that estimate Shapley values, such as linear SHAP and SHAP, can detect these attacks with varying degrees of effectiveness. We demonstrate the efficacy of the attack strategies by comparing the performance of linear SHAP and SHAP using real-world datasets.
@article{yuan2024fooling,title={Fooling SHAP with Output Shuffling Attacks},author={Yuan, Jun and Dasgupta, Aritra},note={In preparation},year={2027},}
ConsensusLens: A Design Study for Interpreting Course Evaluation Student Surveys
Information VisualizationHuman–AI Systems
Vrushali
Koli, Bayan
Divaaniaazar, Jun
Yuan, Yi
Meng, Eugene
Deess, and Aritra
Dasgupta
2027
In preparation
A Scalable Human–AI Annotation Pipeline for Low-Resource Spoken Languages