Life of Research
Appearance
Preprint Open access CC BY 4.0

A Benchmark Hit Is Not an Exposure: What Published Vision-Language Geolocation Accuracy Says About Location Privacy, Why One Model Locates 61 Percent of One Photo Set to a Kilometre and Fewer Than 12 Percent of Another to a Hundred, and the Exposure Estimand a Privacy Claim Needs

  1. SONYTECH

Preprint

DOI10.5281/zenodo.24382475 LOR-00254

Download PDF (375 KB) Cite

Cite

A Benchmark Hit Is Not an Exposure: What Published Vision-Language Geolocation Accuracy Says About Location Privacy, Why One Model Locates…

Read

Abstract

(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0.

Life of Research Since 2024 a growing literature has argued that general-purpose vision-language models can work out where a photograph was taken well enough to make image geolocation an immediate privacy risk, and a second literature has begun building defences against it. In this analysis the author reviews and synthesises published results to show that the direction of that claim survives the evidence but its usual support does not: the accuracies quoted for it are properties of particular photo sets, adversaries and distance thresholds, and none of them is the quantity a privacy claim is about. The same model places 61 percent of a curated, text-rich stock-photo set within one kilometre, about 23 percent of a public Flickr benchmark, and fewer than 12 percent of sampled street-level images within one hundred kilometres. One video-frame benchmark reports 65.6 percent within a kilometre for clips from one platform and 6.4 percent for clips from another. The obvious deflation, that benchmark images were seen in training, does not explain the capability: on photographs collected by the authors and not drawn from any public source, frontier models still name the city about 60 percent of the time, and freshly taken photographs of Seoul are located about as well as Street View images of the same city. The author separates five variables that published results leave unfixed (image population, harm radius, adversary configuration, counterfactual baseline, and the unit of exposure), shows that defences are mostly scored against a fixed adversary although a published photograph must resist every later one, and proposes an exposure protocol that states each variable before a number is reported.

Keywords

Files (1)

Cite this record

Mahendrakar, P. (2026). A Benchmark Hit Is Not an Exposure: What Published Vision-Language Geolocation Accuracy Says About Location Privacy, Why One Model Locates 61 Percent of One Photo Set to a Kilometre and Fewer Than 12 Percent of Another to a Hundred, and the Exposure Estimand a Privacy Claim Needs [Preprint]. Life of Research. https://doi.org/10.5281/zenodo.24382475

APA 7th edition
Export to a reference manager
BibTeX
@misc{mahendrakar2026benchmark,
  author       = {Mahendrakar, Pranay},
  title        = {{A Benchmark Hit Is Not an Exposure: What Published Vision-Language Geolocation Accuracy Says About Location Privacy, Why One Model Locates 61 Percent of One Photo Set to a Kilometre and Fewer Than 12 Percent of Another to a Hundred, and the Exposure Estimand a Privacy Claim Needs}},
  year         = {2026},
  month        = nov,
  date         = {2026-11-20},
  publisher    = {Life of Research},
  howpublished = {Preprint},
  doi          = {10.5281/zenodo.24382475},
  url          = {https://lifeofresearch.com/preprints/254},
  abstract     = {(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research Since 2024 a growing literature has argued that general-purpose vision-language models can work out where a photograph was taken well enough to make image geolocation an immediate privacy risk, and a second literature has begun building defences against it. In this analysis the author reviews and synthesises published results to show that the direction of that claim survives the evidence but its usual support does not: the accuracies quoted for it are properties of particular photo sets, adversaries and distance thresholds, and none of them is the quantity a privacy claim is about. The same model places 61 percent of a curated, text-rich stock-photo set within one kilometre, about 23 percent of a public Flickr benchmark, and fewer than 12 percent of sampled street-level images within one hundred kilometres. One video-frame benchmark reports 65.6 percent within a kilometre for clips from one platform and 6.4 percent for clips from another. The obvious deflation, that benchmark images were seen in training, does not explain the capability: on photographs collected by the authors and not drawn from any public source, frontier models still name the city about 60 percent of the time, and freshly taken photographs of Seoul are located about as well as Street View images of the same city. The author separates five variables that published results leave unfixed (image population, harm radius, adversary configuration, counterfactual baseline, and the unit of exposure), shows that defences are mostly scored against a fixed adversary although a published photograph must resist every later one, and proposes an exposure protocol that states each variable before a number is reported.},
  keywords     = {image geolocation, vision-language models, location privacy, threat models, benchmark validity, contextual integrity, adversarial perturbation, privacy evaluation},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}

Using Zotero or Mendeley? Their browser connector saves this record from this page, with its PDF when there is one. On search and browse pages it lists every result so you can save many at once.

Guides How to cite a preprint Preprints vs journal articles What CC BY 4.0 allows