Research Data Leeds Repository

RoomSpace

Citation

Li, Fangjun and Hogg, David C. and Cohn, Anthony G. (2024) RoomSpace. University of Leeds. [Dataset] https://doi.org/10.5518/1518¹

Dataset description

This dataset, associated with the IJCAI-24 paper "Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning", aims to enhance spatial reasoning evaluations in language models. Our benchmark encompasses a broad spectrum of qualitative spatial relationships, including topological, directional, and distance relations. These are presented with different viewing points, varied granularities, and density of relation constraints to mimic real-world complexities, promoting more accurate evaluation of language models' capabilities in spatial reasoning tasks.

Keywords:	qualitative spatial reasoning, evaluating language models
Divisions:	Faculty of Engineering and Physical Sciences > School of Computing2
Related resources:	Location Type https://doi.org/10.48550/arXiv.2405.150643 Publication
License:	Creative Commons Attribution 4.0 International (CC BY 4.0)
Date deposited:	05 Jun 2024 20:05
URI:	https://archive.researchdata.leeds.ac.uk/id/eprint/12934
Additional details Additional details Creators or authors: Creators ORCID Other ID Email Primary affiliation Primary affiliation ID Primary affiliation ID type Secondary affiliation Secondary affiliation ID Secondary affiliation ID type Li, Fangjun https://orcid.org/0000-0002-1109-62855 University of Leeds https://ror.org/024mrxd33 ror Hogg, David C. https://orcid.org/0000-0002-6125-95646 University of Leeds https://ror.org/024mrxd33 ror Cohn, Anthony G. https://orcid.org/0000-0002-7652-89077 University of Leeds https://ror.org/024mrxd33 ror Type of data: Dataset Research funders: (1) Microsoft Research - Accelerating Foundation Models Research program, with the provision of Azure resources to access GPT; (2) the Turing’s Defence and Security programme through a partnership with the UK government in accordance with the framework ag, Economic and Social Research Council Grant numbers: ES/W003473/1 Publication date: 2024 Resource language: English Metadata language: English Publisher: University of Leeds Contact email: scfli@leeds.ac.uk8 Last Modified: 20 Jun 2024 13:23

Files

Documentation

README_Li-etal_2024.txt [4kB] 9 README_Li-etal_2024.txt [4kB] 10 README_Li-etal_2024.txt [4kB] 11 Security: Anyone File name: README_Li-etal_2024.txt12 Content type: Documentation Mime-Type: text/plain License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 4kB

Data

RoomSpace-Data-SD-1K.zip [7GB] 13 RoomSpace-Data-SD-1K.zip [7GB] 14 RoomSpace-Data-SD-1K.zip [7GB] 15 Security: Anyone File name: RoomSpace-Data-SD-1K.zip16 Content type: Data Description: Contains data for 1,000 room samples, with the first 100 rooms identical to those in `SD-100` Mime-Type: application/zip License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 7GB

RoomSpace-Data-SD-10K-robot_south.zip [16GB] 17 RoomSpace-Data-SD-10K-robot_south.zip [16GB] 18 RoomSpace-Data-SD-10K-robot_south.zip [16GB] 19 Security: Anyone File name: RoomSpace-Data-SD-10K-robot_south.zip20 Content type: Data Description: Contains data for 10,000 room samples, with the first 1,000 rooms identical to those in `SD-1K` Mime-Type: application/zip License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 16GB

RoomSpace-Data-SD-10K-robot_south_rec.zip.001 [16GB] 21 RoomSpace-Data-SD-10K-robot_south_rec.zip.001 [16GB] 22 RoomSpace-Data-SD-10K-robot_south_rec.zip.001 [16GB] 23 Security: Anyone File name: RoomSpace-Data-SD-10K-robot_south_rec.zip.00124 Content type: Data Description: Contains data for 10,000 room samples, with the first 1,000 rooms identical to those in `SD-1K` License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 16GB

RoomSpace-Data-SD-10K-robot_south_rec.zip.002 [15GB] 25 RoomSpace-Data-SD-10K-robot_south_rec.zip.002 [15GB] 26 RoomSpace-Data-SD-10K-robot_south_rec.zip.002 [15GB] 27 Security: Anyone File name: RoomSpace-Data-SD-10K-robot_south_rec.zip.00228 Content type: Data Description: Contains data for 10,000 room samples, with the first 1,000 rooms identical to those in `SD-1K` License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 15GB

RoomSpace-Data-SD-10K-top-down.zip [17GB] 29 RoomSpace-Data-SD-10K-top-down.zip [17GB] 30 RoomSpace-Data-SD-10K-top-down.zip [17GB] 31 Security: Anyone File name: RoomSpace-Data-SD-10K-top-down.zip32 Content type: Data Description: Contains data for 10,000 room samples, with the first 1,000 rooms identical to those in `SD-1K` Mime-Type: application/zip License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 17GB

RoomSpace-Data-SD-100.zip [692MB] 33 RoomSpace-Data-SD-100.zip [692MB] 34 RoomSpace-Data-SD-100.zip [692MB] 35 Security: Anyone File name: RoomSpace-Data-SD-100.zip36 Content type: Data Description: Contains data for 100 room samples Mime-Type: application/zip License: Creative Commons Attribution 4.0 International (CC BY 4.0) File size: 692MB

• 1. https://doi.org/10.5518/1518
• 2. https://archive.researchdata.leeds.ac.uk/view/divisions/SC/
• 3. https://doi.org/10.48550/arXiv.2405.15064
• 4. https://archive.researchdata.leeds.ac.uk/id/eprint/1293
• 5. https://orcid.org/0000-0002-1109-6285
• 6. https://orcid.org/0000-0002-6125-9564
• 7. https://orcid.org/0000-0002-7652-8907
• 8. mailto:scfli@leeds.ac.uk
• 9. https://archive.researchdata.leeds.ac.uk/1293/7/README_Li-etal_2024.txt
• 10. https://archive.researchdata.leeds.ac.uk/1293/7/README_Li-etal_2024.txt
• 11. https://archive.researchdata.leeds.ac.uk/1293/7/README_Li-etal_2024.txt
• 12. https://archive.researchdata.leeds.ac.uk/1293/7/README_Li-etal_2024.txt
• 13. https://archive.researchdata.leeds.ac.uk/1293/1/RoomSpace-Data-SD-1K.zip
• 14. https://archive.researchdata.leeds.ac.uk/1293/1/RoomSpace-Data-SD-1K.zip
• 15. https://archive.researchdata.leeds.ac.uk/1293/1/RoomSpace-Data-SD-1K.zip
• 16. https://archive.researchdata.leeds.ac.uk/1293/1/RoomSpace-Data-SD-1K.zip
• 17. https://archive.researchdata.leeds.ac.uk/1293/2/RoomSpace-Data-SD-10K-robot_south.zip
• 18. https://archive.researchdata.leeds.ac.uk/1293/2/RoomSpace-Data-SD-10K-robot_south.zip
• 19. https://archive.researchdata.leeds.ac.uk/1293/2/RoomSpace-Data-SD-10K-robot_south.zip
• 20. https://archive.researchdata.leeds.ac.uk/1293/2/RoomSpace-Data-SD-10K-robot_south.zip
• 21. https://archive.researchdata.leeds.ac.uk/1293/3/RoomSpace-Data-SD-10K-robot_south_rec.zip.001
• 22. https://archive.researchdata.leeds.ac.uk/1293/3/RoomSpace-Data-SD-10K-robot_south_rec.zip.001
• 23. https://archive.researchdata.leeds.ac.uk/1293/3/RoomSpace-Data-SD-10K-robot_south_rec.zip.001
• 24. https://archive.researchdata.leeds.ac.uk/1293/3/RoomSpace-Data-SD-10K-robot_south_rec.zip.001
• 25. https://archive.researchdata.leeds.ac.uk/1293/4/RoomSpace-Data-SD-10K-robot_south_rec.zip.002
• 26. https://archive.researchdata.leeds.ac.uk/1293/4/RoomSpace-Data-SD-10K-robot_south_rec.zip.002
• 27. https://archive.researchdata.leeds.ac.uk/1293/4/RoomSpace-Data-SD-10K-robot_south_rec.zip.002
• 28. https://archive.researchdata.leeds.ac.uk/1293/4/RoomSpace-Data-SD-10K-robot_south_rec.zip.002
• 29. https://archive.researchdata.leeds.ac.uk/1293/5/RoomSpace-Data-SD-10K-top-down.zip
• 30. https://archive.researchdata.leeds.ac.uk/1293/5/RoomSpace-Data-SD-10K-top-down.zip
• 31. https://archive.researchdata.leeds.ac.uk/1293/5/RoomSpace-Data-SD-10K-top-down.zip
• 32. https://archive.researchdata.leeds.ac.uk/1293/5/RoomSpace-Data-SD-10K-top-down.zip
• 33. https://archive.researchdata.leeds.ac.uk/1293/6/RoomSpace-Data-SD-100.zip
• 34. https://archive.researchdata.leeds.ac.uk/1293/6/RoomSpace-Data-SD-100.zip
• 35. https://archive.researchdata.leeds.ac.uk/1293/6/RoomSpace-Data-SD-100.zip
• 36. https://archive.researchdata.leeds.ac.uk/1293/6/RoomSpace-Data-SD-100.zip