The research addresses the challenge of systematically identifying references to datasets within unstructured text across various domains, including research papers and humanitarian reports. Existing methods require extensive manual labeling, which is impractical due to the volume of relevant documents. The presented framework offers a weakly supervised approach, reducing the dependency on large, labeled datasets. It employs a lightweight model initially trained on general research literature to generate candidate dataset mentions from unlabeled domain documents. A frontier large language model then reviews these candidates within the context of the document, validating or rejecting them and adjusting extraction boundaries. The resulting annotations are supplemented with synthetic and contrastive examples, enabling fine-tuning of the lightweight model for large-scale extraction.
The system was evaluated on an independent benchmark of 1,706 text passages. At the mention level, the model achieved 74.1% precision and 70.5% recall. Notably, precision reached 89.5% when considering passages containing dataset references. At the passage level, the model achieved 88.2% accuracy and 88.6% specificity in distinguishing passages with dataset references from those without. These results demonstrate a practical method for constructing domain-specific supervision when labeled data are limited. The framework provides a technical foundation for analyzing data use and identifying potential gaps in the displacement data landscape.
The approach leverages the capabilities of large language models to automate the labeling process, reducing the need for manual annotation. This is particularly valuable in domains where labeled data are scarce and expensive to obtain. The framework’s modular design allows for adaptation to different datasets and domains, and the use of synthetic data further enhances its scalability. The benchmark performance indicates the potential for this approach to be applied to a wide range of data analysis tasks.
Source: https://arxiv.org/abs/2609.12107
