This project contains the source code for the paper titled "Overcoming the Long-tail Problems for Safety-Critical Autonomy with Ego Spatial Sense". It includes three primary experiments: the 2D Escaping Ball Game, the F1Tenth Racing Simulation, and the Vision-Language Spatial Sense Model.
-
2D Escaping Ball Game: The 2D escaping ball game simulates an ego ball navigating within a square window to avoid collisions with independently moving background balls. The game ends upon the first collision or after a fixed number of time steps, with survival duration serving as the final score.
-
F1Tenth Racing Simulation: The F1Tenth Racing experiment uses the F1Tenth Gym simulator to evaluate an ego car navigating a 2D track with static obstacles using LiDAR-based perception. The game ends upon completing a lap or encountering a collision, boundary violation, or stalling, with distance traveled recorded as the score.
-
Vision-Language Spatial Sense Model (VLSS): The Vision-Language Spatial Sense Model combines the ego spatial sense model with a pre-trained VLM to generate context-aware, spatially informed trajectories from front-view images in previously unseen driving scenarios. The VLM extracts commonsense knowledge, while the spatial sense module performs structured spatial reasoning, with training and testing based on nuScenes data.
Ego-Spatial-Sense
|- 2D-Escaping-Ball: source code and instructions for the 2D Escaping Ball Game.
|- F1Tenth-Racing: source code and instructions for the F1Tenth Racing Simulation.
|- VLSS-Model: source code and instructions for the Vision-Language Spatial Sense Model.
|- README.md
|- LICENSE.txt
The hardware and software requirements for this repository are flexible and broadly compatible across various platforms. However, we recommend not using macOS, as certain dependencies and functionalities may not perform reliably. We also advise against running the code on remote servers, as visualization features may not function properly. The following environments have been thoroughly validated:
-
Hardware:
- CPU with x86 architecture
- NVIDIA GPU with CUDA (optional but recommended for accelerated model training and testing)
-
Operating System:
- Ubuntu 22.04
- Windows 11
-
Python Environment:
- Python 3.10 (via native installation or within an Anaconda virtual environment).
Detailed installation instructions are available in each experiment's respective directory. In general, the installation process is straightforward and typically requires approximately 5 to 10 minutes on a standard computing system. Each experiment is self-contained and can be executed independently.
Detailed usage instructions are provided within each experiment's respective directory, covering expert demonstration, data collection, model training, and model evaluation. Additionally, pretrained models are available for immediate testing purposes. Each experiment is self-contained and can be executed independently.
-
Zhijie Qiao: [email protected]
-
Zhong Cao: [email protected]
This project is licensed under the PolyForm Noncommercial License 1.0.0.
- Henry Liu: [email protected]