BBL Speaker Series: Dr. Elijah Bouma-Sims: Evaluating SMS Scam Warnings in the Field
Talk Title: Evaluating SMS Scam Warnings in the Field
Speaker: Dr. Elijah Bouma-Sims, Assistant Professor, School of Computing, Jarvis College of Computing and Digital Media, DePaul University.
Location: IRB 4105 and Zoom
Abstract: SMS and other instant messaging protocols are a growing vector for scams. There is interest from both researchers and industry in using generative AI to detect these scams and explain them to users. But do tailored explanations actually protect people better than a simple generic warning? To our knowledge, no study has evaluated SMS warnings, AI-generated or otherwise, in the field. In this talk, I’ll present a series of studies in which we designed, refined, and deployed an LLM-based warning system.
We first conducted a task-based interview study (n = 20) with US-based Android users in which participants assessed SMS messages using an inbox-style interface with various warning designs. These initial warning prototypes were refined across three online experiments (n = 903), focusing especially on how the length of the AI-generated text may affect people’s behavior. We tested a final prototype in a field experiment in which 91 Android users installed an instrumented SMS app and received simulated scam messages over two weeks, with no warnings in one week and either generic or context-specific, LLM-generated warnings in the other.
We found that SMS warnings significantly reduce click-through on scam messages. However, context-specific warnings offered no measurable advantage over generic ones: they did not further reduce clicking, increase reporting, or improve performance on a later phishing identification task, though participants rated some aspects more favorably. I’ll discuss what these findings mean for designing better SMS warnings, as well as lessons learned from the challenges of a real-world deployment study. These insights are valuable to both the usable security community and mobile HCI.
Bio: Dr. Elijah Bouma-Sims is an Assistant Professor at DePaul University specializing in usable privacy and security. His research investigates why individuals fall victim to scams and how to design better protections. His recent work examines how people seek support on social media after encountering scams, as well as the design of anti-scam warnings for SMS. He earned his PhD from Carnegie Mellon University in 2026, alongside a B.S. in Computer Engineering and a B.A. in History from North Carolina State University. His research has been published in top venues including ACM CCS, NDSS, ACM CHI, and SOUPS. In his free time, he enjoys training for marathons along the Lakefront Trail in Chicago.
