| Author | GitHub profile |
|---|---|
| Pawel Dymek | pdymek |
| Nikolai Pchelin | Niko1707 |
| Nil Oller | NilOller |
| Francesc Fons | ffons9 |
Hands on project consisting in the implementation of deeep learning (DL) techniques applied to the 3D objects detection and segmentation based on LiDAR technology in real autonomous driving scenarios.
Deployment of DL application targeting automotive autonomous driving / advanced driving assistance systems (AD/ADAS) by combining the following technologies and artifacts:
- Neural Network Model: PointNet
- Point Cloud Sensing: Velodyne HDL-64E LiDAR
- Dataset: KITTI / SemanticKITTI
- Point Cloud Viewer Tool: semantic-kitti-api
- Computation Platform: Desktop PC equipped with CPU and GPU
The goal of this project was to put in practice the knowledge acquired along the postgraduate course by understanding and digesting the full development cycle of DL applications.
The dataset selected in this project has been KITTI / SemanticKITTI.
https://www.cvlibs.net/datasets/kitti/
http://www.semantic-kitti.org/
The 3D point cloud data is acquired through the Velodyne HDL-64E LiDAR Sensor.
The point cloud data format is depicted next.
The Neural Network Model in use is PointNet.
The main challenge of this project has been the fact of adapting and connecting many different pieces together: autonomous driving dataset, 3D LiDAR point clouds, segmentation NN model, visualization tools, etc. and programming all the application in python and pytorch programming language.
The software project has been architected to make our solution modular, flexible and scalable. To this aim, the full application is decomposed in building blocks that are easily interconnected giving place to a simple processing flow, as illustrated below.
The short period of time assigned to this project has forced the authors to teamwork following agile methodologies in order to iterate the final product in short sprints. The fact of having a modular architecture enabled each developer to focus on one specific building block with reasonable freedom of interference.
The NN processing is split in two phases: training and inference.
The outcome of the training (and validation) phase is a model stored in the way of a .pth file format.
This .pth file is used later in the second phase related to inference (test) to perform the predictions.
The project has been executed in a desktop PC consisting of one CPU and one GPU. DEVELOP
The training is performed in the GPU whereas the test can run in the CPU.
Two different variants of the dataset have been used for training: (i) the original KITTI/SemanticKITTI dataset consisting of 22 different sequences of around 4500 time steps each and (ii) a lightweight version of KITTI/SemanticKITTI with less objects and a very reduced size of points.
The full training process can be monitored through TensorBoard.
The best way to assess our results is by having a look at the point cloud, as shown below.
Ground truth:
Prediction results:
Despite the short period of time devoted to the development of this project, authors could complete the full design cycle to reach some results that we consider are good enough according the objectives of this one-semester course.
All in all, this hands on exercise has been a good learning session in order for the authors to better understand and digest all the concepts and knowledge on Deep Learning presented along the course.
Great team and good job!
Link for dataset: http://www.semantic-kitti.org/dataset.html
Data for each of the scences should be organised with following structure. The mnimial rquired version is having one directory with velodyne data and one with labels.
.../<directory_name>/sequences/
/<scence_number>/velodyne
/labels
/<scence_number>/velodyne
/labels
...
To build the virtual environment from the terminal. For instance, you can do it manually from the powershell in Visual Studio Code by running the following instructions:
python -m venv virtualenv
virtualenv\Scripts\activate.bat
pip install -r requirements.txt
Run the program through the following command from the terminal.
For training there are two required paramters only: --hparamDatasetPath and --hparamActionType (=train). The others would be taken as default from hyperparam.py file. However they also could be customized.
Minimal example (with usage of default hyper parameters):
python main.py --hparamActionType train --hparamDatasetPath G:\Project\_kitti_test\data_odometry_velodyne\dataset\sequences\
More evaluated example
python main.py --hparamActionType train --hparamDatasetPath G:\Project\_kitti_test\data_odometry_velodyne\dataset\sequences\ --hparamDatasetSequence 00 --hparamDatasetSequence 04
The other parameters that could be used in train:
--hparamDatasetSequencenumber of sequence used for training, default '00'--hparamValDatasetSequencenumber of sequence used for validation, default '00'--hparamNumPointsdefault 4000, number of points for each of the scenes used in training--hparamTrainBatchSizetraining batch size--hparamValBatchSizevalidation batch size--hparamYamlConfigPathin case of use external .yaml config file--hparamPointDimension3 or 4 for Kitty dataset (4 for the inclusion of reflectance)--hparamNumberOfClassesnumber of classes--hparamTrainNumEpochs--hparamValNumEpochs
During evaluation for the selected scene would be created folder with predicted labels (in <seqence_number>/predictions catalog).
.../<directory_name>/sequences/
/<scence_number>/velodyne
/labels
/predictions
...
For test there are four required paramters only: --hparamDatasetPath,--hparamActionType (=test) and --hparamModelPthPath and --hparamTestDatasetSequence. The others would be taken as default from hyperparam.py file. However they also could be customized.
Minimal example (with usage of default hyper parameters):
python main.py --hparamActionType test --hparamDatasetPath G:\Project\_kitti_test\data_odometry_velodyne\dataset\sequences\ --hparamModelPthPath G:\Project\_kitti_test\seg_model_bus_99.pth --hparamTestDatasetSequence 11
Execute the following commands from the project root in the terminal to visualize both point cloud and voxels:
python main.py --hparamDatasetPath E:\Project\Lidar_KITI\kitti\dataset --hparamDatasetSequence 11 --hparamPredictionsPath E:\Project\Lidar_KITI\kitti\dataset --hparamActionType visualize
Used parameters
--hparamActionType-"visualize"--hparamDatasetPath- path for root dataset catalog--hparamDatasetSequence- number of visualized sequence--hparamPredictionsPath- path for a directory with predictions
Along the different stages of our project, we have been inspired by previous related works available in other repositories and websites. They have been useful material that provided us many insights about theoretical approaches, problem-solving, and also for some code.
Neural Network:
- https://github.com/Yvanali/KITTISeg
- https://github.com/fxia22/pointnet.pytorch
- https://github.com/marionacaros/3D-object-segmentation-light-PointNet
Point Cloud Visualization Tool: