Geospatial Sparse Attention Helps TabPFN Tackle Spatial Data Challenges

Our group has published a new paper in the International Journal of Geographical Information Science (IJGIS) titled, “Do foundation models work for geospatial tabular data? An investigation of TabPFN and a proposed enhancement based on geospatial sparse attention”. The paper was first-authored by PhD researcher Rui Deng, in collaboration with Dr Mingshu Wang and Dr Ziqi Li (Florida State University).

The study investigates the zero-shot capabilities of the tabular foundation model, TabPFN, on spatial regression tasks. As a powerful, general-purpose model, TabPFN is designed to solve generic, domain-agnostic tabular tasks. Consequently, when applied directly to geospatial datasets, it lacks an explicit mechanism to account for spatial dependence. This limits its predictive performance in spatial contexts and also leads to memory/computational overheads on large datasets.

To bridge this gap, the team developed Geospatial Sparse Attention (GSA), a lightweight, inference-time framework. By applying a grid-based spatial partitioning mechanism, TabPFN-GSA guides the model’s attention mechanism to prioritise geographically proximate observations. This effectively provides the model with spatial context without the need for retraining.

Benchmark evaluations across over 40 synthetic spatial datasets and 4 real-world datasets demonstrated that TabPFN-GSA significantly improves predictive accuracy and robustness. Furthermore, the sparse attention mechanism resolved the original model’s memory/computational overheads, successfully scaling to datasets of up to 70,000 spatial sample points.

This research was supported by the NVIDIA Academic Grant Programme and Google Cloud Research Credits.

The breakthrough has recently been highlighted by a University of Glasgow official press release and is receiving wider coverage across several scientific media outlets, including FutureScot, Spatial Source, and myScience.

The open-source code is available on GitHub, with supporting data hosted on figshare.