Mind the Third Eye! Benchmarking Privacy Awareness in MLLM‑powered Smartphone Agents

Zhixin Lin1, Jungang Li2,3, Shidong Pan4, Yibo Shi5, Yue Yao1,†, Dongliang Xu1,†
1 Shandong University
 2 Hong Kong University of Science and Technology (Guangzhou)
 3 Hong Kong University of Science and Technology
 4 Columbia University
 5 Xi'an Jiaotong University
† Corresponding Authors

If our project helps you, please give us a star ⭐ on GitHub to support us 🥸

Code Paper 🤗 Dataset

Abstract

We present SAPA‑Bench, a large‑scale benchmark of 7,138 scenarios to measure privacy awareness in MLLM‑powered smartphone agents across recognition, localization, category, sensitivity level, and risk response. We define five metrics (PRR, PLR, PLAR, PCAR, RA) and evaluate seven representative agents.

Figures

Introduction

Figure A — Introduction

Benchmark

Figure B — Benchmark

Data Construction

Figure C — Data Construction

Risk Awareness (RA) per Model

Figure D — Risk Awareness by Model

BibTeX

@article{lin2025sapa,
  title   = {Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agents},
  author  = {Lin, Zhixin and Li, Jungang and Pan, Shidong and Shi, Yibo and Yao, Yue and Xu, Dongliang},
  journal = {arXiv preprint arXiv:2508.19493},
  year    = {2025}
}