Health AI: “The Next Health Equity Frontier”

Lisa Bowleg, PhD, MA, Founder and CEO, The Intersectionality Training Institute

September 27, 2026

TLDR: Conversations about intersectionality, equity and community governance are among the most important ones that we should be having about health AI but aren’t.  This was a key insight from the riveting conversation with Dr. Oni Blackstock and Dr. Elle Lett, our salon guests at the Intersectionality Training Institute’s August 2026 Intersectionality Salon.  Oni Blackstock, MD, MPH, the Founder and Executive Director of Health Justice, and Elle Lett, MD, PhD, MA, MBiostat, Emergency Medicine Resident Physician at the Hospital of the University of Pennsylvania (ISI 2022), are Black women physician-researchers, health equity and anti-racist activists, and intersectionality visionaries in the vanguard of those sounding the alarm about the equity and ethical consequences of generative and predictive health AI.   Dr. Blackstock’s work advocates for greater community governance over generative AI use in health care systems.  Dr. Lett’s research shows that intersectional debiasing prediction models for emergency admissions are significantly more effective at reducing bias than marginal debiasing with no trade-off between fairness and overall accuracy.

Key Takeaways:

    • Equity is rarely a consideration when health care systems decide to adopt health AI.
    • Communities are rarely included in key decisions about whether AI should be used, for which purposes, and which data should be used for training AI systems.
    • Intersectionality is an important framework for health AI because the same historically oppressed communities (e.g., those who are Black, Latino/a/x and Indigenous; poor, have disabilities, and/or are sexual and gender minorities) who are disproportionately affected by health inequities, are the same communities at greatest risk for algorithmic harm.
    • Governance —community and multi-national — is vital to reducing algorithmic harm, establishing equity metrics, regulating environmental impact and harm, and determining the level of acceptable bias in health AI.

Equity and Governance: Two Conversations Health Care Systems Should Be Having about  Health AI (But Aren’t)

Health care systems are adopting AI with lightning speed, and often backwardly as Blackstock has written, “to address a problem they think AI can solve.”   For example, “What if, “Blackstock asked, “the solution is not AI, but a community health navigator?”

Moreover, there are two important conversations that health care systems should be having about health AI, but for the most part, according to Blackstock and Lett, aren’t: equity and governance.  Lett opined that “interest in implementing AI has far outpaced our abilities to be good stewards of the technology making AI and health the next health equity frontier.”

The need for greater community governance over health AI is the focus of Blackstock’s work.  Blackstock lamented the lack of conversations about community governance. She identified numerous questions that remain unanswered in the health AI space such as: what shapes health, which technologies are used (or should be used), and what are the terms and conditions surrounding the use of AI in health care.

To address the critical gaps about community governance and health AI, Blackstock co-leads the Grounded Innovation Lab @ Health Justice, which  developed the ORCHID (Organizational Readiness to Engage in Community-Led Health AI Partnership) framework.  Blackstock also co-leads LIFHE (Locally Informed Futures for Health Equity), a community-based research project with the Harlem Health Initiative at the CUNY School of Public Health. The goal of LIFHE is to facilitate collaboration between community members to develop their own terms and conditions for the use of health AI.

Multilevel governance was also essential, Lett argued, to prospectively decide what degree of intersectional bias is acceptable in health AI.  Without such governance, Lett explained, the implicit assumption is that the intersectional bias that is already being redoubled, is acceptable bias.” She warned that “the people who are already harmed will continue being harmed. That’s the implicit assumption that we’re making by using these models without thinking through how this bias will be propagated.”

The Motivations Behind Blackstock and Lett’s Community Governance and Intersectional Debiasing Health AI Work

Despite being a self-described “healthy AI skeptic” in the case Blackstock, and an “unhappy bedfellow to AI,” in the case of Lett, both noted their strong commitments to this work.  Community was at the center of Blackstock’s work.  She wanted to be better informed about AI to “be able to help figure out what’s best for us and for our communities.”

Lett, in line with her advocacy for better stewardship of intersectionality, said she felt a moral imperative to be a good steward of the technology or to force others to be stewards of the technology to ensure that we were not widening existing societal inequities.

Is Intersectional Bias Inevitable in Health AI?

Health AI reflects the intersectional biases of the society in which they are developed, making intersectional bias inevitable, according to Lett.   AsLett noted, the data generating inputs for AI are racist, sexist, misogynistic, heterosexist, cisgenderist, ableist, xenophobic and classist (to name just a few).

As one examples of intersectional bias, Blackstock described Seyyed-Kalantari et al.’s 2021 study that found that AI algorithms applied to chest x-rays consistently and selectively underdiagnosed “intersectional under-served subpopulations” such as Black and Hispanic women. Black girls and women (0-20 years) with Medicaid insurance had the largest rates of underdiagnosis.

Because the chest x-ray prediction models relied on the natural language in the clinical reports for training data, Blackstock explained, the data reflected the “[intersectional] bias or racism and sexism and misogynoir that the clinical providers have and so it just ended up getting embedded in the data that the algorithms flagged.”

Intersections of racism and sexism in technology are well-documented.  Safiya Umoja Noble’s award-winning 2018 book, Algorithms of Oppression: How Search Engines Reinforce Racism documented that Google keyword searches for “Black, Latina or Asian girls” invariably generated links to pornography or that queries of “why are Black women so…” frequently autocompleted a litany of racist and sexist responses.

Blackstock provided as another example, journalist Karen Hao’s award-winning 2025 book, The Empire of AI: The Reckless Race for Total Domination (also published as Empire of AI: Dreams and Nightmares in Sam Altman’s Open AI).  Hao argues that AI is the new colonialism, and documents how AI companies’ rush to advance generative AI has encoded historic racism, sexism, and systemic inequality.

Why Intersectional Debiasing Should Be the “Gold Standard” for Predictive Health AI

If, as Lett asserted, intersectional bias is inevitable, a logical next question is: is it possible to eliminate intersectional bias in health AI?

No, said Lett, but her intersectional debiasing research with predictive models for which emergency department patients need to be admitted to the hospital, shows that it is possible to mitigate intersectional bias.

Lett et al.’s 2025 study  compared marginal debiasing (i.e., single-axis; gender or race) in fair clinical prediction models such as a model for “women” and a model for “Black patients,” to intersectional debiasing, fair prediction models that accounted for intersectional positions (e.g., gender and race) for intersectional groups such as Black women.

The study found that intersectional debiasing was significantly more effective at reducing bias in calibration errors (accurate risk scores) and false negatives (missed admissions) than marginal debiasing models, with no trade-off between fairness and overall accuracy.

The results prompted Lett and colleagues to recommend that intersectional debiasing become the new “gold standard” for more equitable prediction models for emergency admissions.

Call to Action: Some Strategies to Promote More Equitable and Intersectional Health AI

    • If you’re a physician other health care professional or work in a health department or health care system, get up to speed on Blackstock and colleagues’ ORCHID framework so that you can advocate for community governance at your workplace.
      • Follow the lead of Leah Marcotte, MD, MPH, a primary care physician at the University of Washington School of Medicine and member of the ISI 2023 cohort, who emailed to say that after the salon, she had emailed her organization’s AI committee about community governance. She shared that now has a meeting “in a few weeks to discuss what [ community governance in health AI] might look like at UW.”  She signed off: “Salons inspiring action!” Of course, I LOVED learning this.  Brava Leah!
    • If you work with communities in any capacity in which AI is being implemented, as Blackstock advocates in this video about community governance, “start asking not just what AI can do, but who gets to decide how it’s built and used.”
    • Follow and support the work of “the mostly Black women, femmes and trans folks” that Blackstock described as “the moral compass for AI.” Of course, Blackstock and Lett are in this number.  Despite enduring death threats and other hostile backlash for their AI ethics and equity work, Blackstock saluted the research and advocacy of leaders such as:
      • Joy Buolalmwini, founder of the Algorithmic Justice League.
      • Sasha Constanza -Chock, a scholar, writer and designer advocating for more equitable community-led processes;
      • Zainab Garba-Sani, the founder of ACCESS AI, a framework for multidisciplinary stakeholders to engage communities to advance AI equity
      • Timnit Gebru, the founder and CEO of DAIR (Distributed AI Research Institute), whom Google fired for raising awareness about workplace discrimination; and
      • Alex Hanna, a sociologist and Director of Research at DAIR.
    • Want to get more involved in equity and intersectional bias reducing health AI work and advocacy? The Intersectionality Collective’s chat space is a great place to continue the conversation from this salon, share resources, and find potential collaborators to brainstorm and share strategies on this topic.

Missed the Salon on Community Governance, Intersectional Debiasing and Health AI?

Browse our YouTube channel to see more clips of Blackstock and Lett’s conversation about community governance, intersectional debiasing ,and equity in health AI .You can also apply to join the Intersectionality Collective , our online community to see the list of recommended readings for the chat and other resources discussed or shared in the Zoom chat. You’ll also find information and resources from past salons.

Want to Be a Part of More Great Conversations About Intersectionality?

You can!  ITI’s free monthly Intersectionality Salons are the place to usually be on the second Wednesday of each month (5-6:30 pm ET) for informative, educational, and engaging conversations about all things intersectionality.

FAQ

What does ISI mean?

ISI is the acronym for the Intersectionality Summer Intensive, the Intersectionality Training Institute’s engaging 5-day in-person “intensive” focused on the application of intersectionality to qualitative, quantitative, and mixed methods health equity and social research, practice and community engagement.  Their other stellar credentials notwithstanding, we proudly claim our ISI cohort members.  Dr. Lett is a member of the inaugural ISI 2022 cohort and Dr. Marcotte is a member of the ISI2023 cohort.

What’s the difference between generative and predictive AI?

Generative AI, the focus of Dr. Blackstock’s community governance work, is a type of AI that Creates new text, images, audio, or code based on prompts (e.g., ChatGPT, Claude, Siri, Alexa).  Predictive AI, the focus of Dr. Lett’s work, is a type of AI that predicts future outcomes and trends using historical data.

What’s intersectional bias? 

Intersectional bias refers to the differential treatment, stigma or discrimination based on the intersection of two or more historically oppressed social or demographic categories (e.g., racialized or ethnic minoritized status, sexual or gender minority status, being a woman, having a disability, poor or working class, undocumented, and/or religious minority.  Intersectional bias is qualitatively different from the single-axis bias (e.g., bias based on racial group or gender minority status).  You’ll find the term intersectional bias in the health equity and bias research literature, as well as that on algorithmic fairness in generative and predictive AI.

What is community governance in the context of health and AI Look like? 

Watch Dr. Blackstock describe community governance using the example of a public health department or health care organization developing a generative AI chatbot to answer questions about vaccines.  Here, Blackstock defines community governance as a process by which:

“… instead of deciding everything internally, [the organization] invites community members as true partners from the start, and compensates them for their time, their expertise, their contributions and their insights.  And together, they help determine which data should and shouldn’t be used, and they flag language that could cause harm. So, community has a real say in what moves forward and what doesn’t, and that shapes how the chatbot is ultimately built and used.”

What is marginal debiasing?

Typically, “fair AI” technologies seek to correct bias via a single-axis focus (e.g., a fair model for “women” or Black patients.”).

What is intersectional debiasing?

In contrast to marginal debiasing, intersectionality debiasing attends to fairness for intersectional groups (such as Black women) to ensure that the models acknowledge the intersection of “race and gender” rather than rely only on single-axis categories (e.g., women or Black people).  Failing to account for specific intersection positions functions to erase the experiences, needs and concerns of intersectional groups such as Black women.  For more information about intersectional debiasing, see Lett et al.’s (2025) research.