From non-consensual intimate images and cyberbullying to white supremacist speech and misinformation, online harms are a daily reality. With support from MIDAS, U-M researchers are
using AI and data science to detect abuse, design interventions and inform policy, helping reshape how platforms, organizations and governments protect people in digital spaces.
The message arrived on a Tuesday afternoon. Someone sent her a screenshot: an intimate photo she had shared months earlier with a partner she trusted. Now it was circulating in a private Discord server with hundreds of strangers.
The college student – we’ll call her Maya – felt her stomach drop. She searched frantically across platforms. The image had already been reposted. She reported it everywhere she could, but each site had different rules and processes. Some takedowns happened quickly. Others took days. New copies kept appearing.
Her phone became a source of dread. She stopped going to campus events, afraid someone would recognize her. Her grades slipped. She withdrew from friends because explaining what happened meant reliving it.
Maya’s experience is far from rare. With support from the Michigan Institute for Data and AI in Society, University of Michigan researchers are using AI and data science to understand online harms, design better interventions, and inform policy, in order to reshape how platforms, schools, and governments protect people in digital spaces.
The scale of online harm
Online harassment has become a pervasive part of digital life. A 2020 Pew Research Center survey found that 41 percent of U.S. adults had experienced online harassment; 25 percent reported more severe forms such as threats or sustained abuse. Nearly half of U.S. teens report being bullied online.
These experiences are not merely unpleasant. Sustained harassment is linked to anxiety, depression, and withdrawal from public life. Journalists, activists, and researchers, especially women, often self-censor or leave platforms entirely.
Non-consensual intimate images are among the most damaging forms of abuse. Survivors describe violations that echo physical sexual assault. The images can resurface years later, affecting careers, relationships, and mental health.
“For survivors, the harm is deeply personal,” said Sarita Schoenebeck, a professor at the School of Information and director of the Living Online Lab. “But it’s also structural. People cannot participate safely online or offline when their image or likeness can be modified and shared online as nude or sexual.”
Beyond detection
For years, platform responses followed a familiar pattern: users reported abuse, human moderators reviewed it, and content was removed – or not. Billions of posts circulate daily; no human workforce can review them all. Machine learning offered a way to scale detection. AI systems can flag hate speech, threats, and harassment, performing reasonably well for overt cases. But they struggle with context and evolving tactics.
Listening to survivors
Schoenebeck and collaborators surveyed nearly 4,000 people across 14 regions worldwide about their experiences with online harassment and the platform responses they found most helpful. The study, Women’s Perspectives on Harm and Justice after Online Harassment, examined reactions to insults, stalking, threats, and non-consensual image sharing.
The findings were consistent. Women perceived greater harm across nearly all scenarios and strongly favored responses such as rapid content removal and permanent bans for repeat offenders. Financial compensation or temporary suspensions were seen as far less effective. The research also showed that context matters. Trust in platforms, cultural norms, and legal recourse varied across regions, meaning no single response works everywhere.
“But the principles are clear,” Schoenebeck said. “Victims want agency, accountability, and responsiveness. Delayed or dismissive responses overlook and enable the problem.”
These insights challenge moderation approaches that focus solely on punishment or takedowns without addressing safety, dignity, and repair.
Designing deterrence
Another strand of work in Schoenebeck’s group focuses on preventing harm before it happens through design. One project, led by PhD student Qiwei Li, developed a prototype app called Hands-Off, which requires users to make a specific hand gesture above their phone before viewing an image. The gesture makes simultaneous screenshotting nearly impossible, adding friction that deters non- consensual sharing.
Performing interlace and binoculars gestures in Hands-Off to deter non-consensual mobile screenshots
“Technology isn’t neutral,” Schoenebeck said. “Design choices shape what’s easy and what’s hard for a user to do. Thoughtful friction can deter harmful behaviors without compromising usability.”
The approach doesn’t eliminate abuse, but it raises the barrier for casual violations and shifts responsibility from victims to system design.
What platforms miss
While Schoenebeck centers survivor experience, Libby Hemphill, associate professor at the School of Information and director of the Social Media Archive at ICPSR, studies how platforms detect (and fail to detect) harmful content.
In the project What Social Media Platforms Miss About White Supremacist Speech, Hemphill analyzed thousands of posts from forums such as Stormfront and extremist subreddits. The research showed that harmful ideology often spreads through euphemisms, humor, historical references, and claims of victimhood, strategies that evade keyword-based moderation.
“Platforms are getting better at catching the most obvious content,” Hemphill said. “But that pushes extremists to become more sophisticated.”
The work produced carefully curated datasets that support better detection models and are shared through the Social Media Archive, allowing other researchers to build on validated, ethically managed data.
De-escalation, not just deletion
Hemphill’s research also asks whether platforms can intervene before conversations spiral into abuse.
One project tested bots that post gentle reminders when conversations show signs of escalation. The messages encourage users to pause, reflect, or reconsider tone. The approach reframes AI as a tool for de escalation rather than punishment. Early results suggest modest but measurable reductions in hostile follow-ups. “We focus so much on deletion,” Hemphill said. “But what if platforms helped people step back.
Generative AI raises the stakes
Generative AI has intensified these challenges. Deepfake tools now allow anyone to create realistic non-consensual sexual images using a single photo. What once required technical expertise now takes seconds.
“Generative AI has changed the scale and accessibility of harm,” Schoenebeck said. “It forces us to rethink consent, privacy, and platform responsibility.”
In 2025, MIDAS provided another pilot grant to Schoenebeck and Li, along with co-advisor Eric Gilbert, to build web-based AI agents that act on behalf of victim-survivors to locate, report, and monitor non-consensual content across the web.
“The law is catching up slowly,” Schoenebeck said. “In the meantime, platforms need better tools; and victims need faster, clearer pathways to help.”
Technology and misinformation
Harassment is not the only major threat in the online world. Another threat is misinformation, and the broader swirl of political information that shapes what people believe and how they act.
With MIDAS support, Michael Traugott and colleagues have studied how political information and misinformation move through online ecosystems, and how exposure to that content connects to people’s political attitudes and perceptions. Their work treats “what spreads” as a measurable phenomenon, tracking how campaign messages travel across news and social media, who encounters them, and what that means for public opinion.
That research thread is captured in Words That Matter: How the News and Social Media Shaped the 2016 Presidential Campaign, a book that synthesizes evidence on how campaign information flowed through journalism, social media, and public attention, and the analysis of misinformation and “fake news” as part of that information environment.
What’s next
The challenges continue to evolve. New
platforms emerge. Moderation policies shift.
Generative tools accelerate the creation and
spread of misinformation.
“The landscape shifts rapidly,” Schoenebeck said.
“But the core principles are durable. People
deserve safety, agency, and accountability
online.”
Maya eventually found support. The images
never disappeared entirely, but the most widely
shared copies were removed. She now advocates
for clearer policies and resources for others
facing similar harm.
Research such as that supported by MIDAS gives
substance to that advocacy. It shows that online
harms and misinformation are not inevitable,
and that AI, when guided by care and evidence,
can be part of the solution instead of the
problem.