Research Data Leeds Repository

Citation

Li, Fangjun and Hogg, David C. and Cohn, Anthony G. (2024) RoomSpace. University of Leeds. [Dataset] https://doi.org/10.5518/1518

Dataset description

This dataset, associated with the IJCAI-24 paper "Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning", aims to enhance spatial reasoning evaluations in language models. Our benchmark encompasses a broad spectrum of qualitative spatial relationships, including topological, directional, and distance relations. These are presented with different viewing points, varied granularities, and density of relation constraints to mimic real-world complexities, promoting more accurate evaluation of language models' capabilities in spatial reasoning tasks.

Keywords:	qualitative spatial reasoning, evaluating language models
Divisions:	Faculty of Engineering and Physical Sciences > School of Computing
Related resources:	Location Type https://doi.org/10.48550/arXiv.2405.15064 Publication
License:	Creative Commons Attribution 4.0 International (CC BY 4.0)
Date deposited:	05 Jun 2024 20:05
URI:	https://archive.researchdata.leeds.ac.uk/id/eprint/1293
Additional details Additional details Creators or authors: Creators ORCID Other ID Email Primary affiliation Primary affiliation ID Primary affiliation ID type Secondary affiliation Secondary affiliation ID Secondary affiliation ID type Li, Fangjun https://orcid.org/0000-0002-1109-6285 University of Leeds https://ror.org/024mrxd33 ror Hogg, David C. https://orcid.org/0000-0002-6125-9564 University of Leeds https://ror.org/024mrxd33 ror Cohn, Anthony G. https://orcid.org/0000-0002-7652-8907 University of Leeds https://ror.org/024mrxd33 ror Type of data: Dataset Research funders: (1) Microsoft Research - Accelerating Foundation Models Research program, with the provision of Azure resources to access GPT; (2) the Turing’s Defence and Security programme through a partnership with the UK government in accordance with the framework ag, Economic and Social Research Council Grant numbers: ES/W003473/1 Publication date: 2024 Resource language: English Metadata language: English Publisher: University of Leeds Contact email: scfli@leeds.ac.uk Last Modified: 20 Jun 2024 13:23

Files

Documentation

README_Li-etal_2024.txt [4kB] README_Li-etal_2024.txt [4kB] README_Li-etal_2024.txt [4kB] Security: Anyone File name: README_Li-etal_2024.txt Content type: Documentation Mime-Type: text/plain License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 4kB

Data

RoomSpace-Data-SD-1K.zip [7GB] RoomSpace-Data-SD-1K.zip [7GB] RoomSpace-Data-SD-1K.zip [7GB] Security: Anyone File name: RoomSpace-Data-SD-1K.zip Content type: Data Description: Contains data for 1,000 room samples, with the first 100 rooms identical to those in `SD-100` Mime-Type: application/zip License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 7GB

RoomSpace-Data-SD-10K-robot_south.zip [16GB] RoomSpace-Data-SD-10K-robot_south.zip [16GB] RoomSpace-Data-SD-10K-robot_south.zip [16GB] Security: Anyone File name: RoomSpace-Data-SD-10K-robot_south.zip Content type: Data Description: Contains data for 10,000 room samples, with the first 1,000 rooms identical to those in `SD-1K` Mime-Type: application/zip License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 16GB

RoomSpace-Data-SD-10K-robot_south_rec.zip.001 [16GB] RoomSpace-Data-SD-10K-robot_south_rec.zip.001 [16GB] RoomSpace-Data-SD-10K-robot_south_rec.zip.001 [16GB] Security: Anyone File name: RoomSpace-Data-SD-10K-robot_south_rec.zip.001 Content type: Data Description: Contains data for 10,000 room samples, with the first 1,000 rooms identical to those in `SD-1K` License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 16GB

RoomSpace-Data-SD-10K-robot_south_rec.zip.002 [15GB] RoomSpace-Data-SD-10K-robot_south_rec.zip.002 [15GB] RoomSpace-Data-SD-10K-robot_south_rec.zip.002 [15GB] Security: Anyone File name: RoomSpace-Data-SD-10K-robot_south_rec.zip.002 Content type: Data Description: Contains data for 10,000 room samples, with the first 1,000 rooms identical to those in `SD-1K` License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 15GB

RoomSpace-Data-SD-10K-top-down.zip [17GB] RoomSpace-Data-SD-10K-top-down.zip [17GB] RoomSpace-Data-SD-10K-top-down.zip [17GB] Security: Anyone File name: RoomSpace-Data-SD-10K-top-down.zip Content type: Data Description: Contains data for 10,000 room samples, with the first 1,000 rooms identical to those in `SD-1K` Mime-Type: application/zip License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 17GB

RoomSpace-Data-SD-100.zip [692MB] RoomSpace-Data-SD-100.zip [692MB] RoomSpace-Data-SD-100.zip [692MB] Security: Anyone File name: RoomSpace-Data-SD-100.zip Content type: Data Description: Contains data for 100 room samples Mime-Type: application/zip License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 692MB