The research investigates whether language model embeddings encode structured real-world information, specifically temporal and geographic signals. A projection-based method is employed, operating directly on output embeddings. The method uses a small set of seed examples to define an axis in embedding space and ranks texts or entities based on their projection onto that axis. This approach is model-agnostic, requiring only embeddings and without access to model weights or internal activations. The study applied this method to temporal and geographic datasets, revealing that embedding projections recover meaningful chronological and spatial structure. The results suggest that output embeddings encode signals relevant to time and space. This offers a practical tool for interpretability and for downstream temporal and geographic information retrieval tasks, such as temporal ordering, geographic ranking, and tagging.
Source: https://arxiv.org/abs/2609.05721