This is the official repository for the NeurIPS2024 poster paper "Real-time Stereo-based 3D Object Detection for Streaming Perception". [Paper]
(1) Download the KITTI 3D tracking dataset including velodyne, stereo images, and calibration matrices. Then move the extra_data/planes.zip to data/kitti/training/ directory and unzip it.
The folders are organized as follows:
ROOT_PATH
├── data
│ ├── kitti_tacking
│ │ │── training
│ │ │ ├──calib & velodyne & label_2 & image_2 & image_3 & planes
├── pcdet
├── mmdetection-v2.22.0
(2) Generate KITTI tracking split file.
python tools/gen_split_kitti_tracking.py --save_split --sample_mode 3(3) Generate KITTI data list and joint Stereo-Lidar Copy-Paste database for training.
python -m pcdet.datasets.kitti_streaming.lidar_kitti_streaming create_kitti_infos
python -m pcdet.datasets.kitti_streaming.lidar_kitti_streaming create_gt_database_only --image_cropsKeep in mind that put the pre-computed road plane to ./kitti_tracking/training/planes for precise copy-paste augmentation.
(1) Install mmcv-1.4.0 library.
# pycocotools==2.0.2 requires a lower version of cython
pip install cython==0.29.36
pip install pycocotools==2.0.2
pip install torch==1.7.1 torchvision==0.8.2
mim install mmcv-full==1.4.0(2) Install the spconv library.
sudo apt install libboost-dev
git clone https://github.com/traveller59/spconv --recursive
cd spconv
git reset --hard f22dd9
git submodule update --recursive
python setup.py bdist_wheel
pip install ./dist/spconv-1.2.1-xxx.whl(3) Install the included mmdetection-v2.22.0.
cd mmdetection-v2.22.0
pip install -e .(4) Install OpenPCDet library.
pip install tqdm timm scikit-image==0.21.0 scikit-learn==1.3.0
pip install -e .(1) Get model inference time for streaming simulations.
python tools/gen_infer_time.py --cfg_file configs/stream/kitti_models/stream_dsgn_r18-token_prev_next-feature_align_avg_fusion-lka_7-mcl.yamlPlease ensure that the inference time is faster than the frame interval, as this is the core of end-to-end streaming perception.
(2) Then train the model by
python tools/train.py --cfg_file configs/stream/kitti_models/stream_dsgn_r18-token_prev_next-feature_align_avg_fusion-lka_7-mcl.yaml --fix_random_seed(3) We provide checkpoints in the extra_data/checkpoint_epoch_20.pth directory. You can evaluate the model directly by
python tools/test.py --cfg_file configs/stream/kitti_models/stream_dsgn_r18-token_prev_next-feature_align_avg_fusion-lka_7-mcl.yaml --ckpt extra_data/checkpoint_epoch_20.pthThe evaluation results can be found in the outputing model folder.
Our code is based on several released code repositories. We thank the great code from DSGN++, OpenPCDet, and ASAP.