Updated 07 Apr 2019
For full instructions please refer to the README.md file.
This is the MATLAB code for training a point cloud classification network using 3D modified Fisher Vectors.
This work was presented in IROS 2018 in Madrid, Spain and was also published in
Robotics and Automation Letters.
Modern robotic systems are often equipped with a direct 3D data acquisition device, e.g. LiDAR, which provides a rich 3D point cloud representation of the surroundings. This representation is commonly used for obstacle avoidance and mapping. Here, we propose a new approach for using point clouds for another critical robotic capability, semantic understanding of the environment (i.e. object classification). Convolutional neural networks (CNN), that perform extremely well for object classification in 2D images, are not easily extendible to 3D point clouds analysis. It is not straightforward due to point clouds' irregular format and a varying number of points. The common solution of transforming the point cloud data into a 3D voxel grid needs to address severe accuracy vs memory size tradeoffs. In this paper we propose a novel, intuitively interpretable, 3D point cloud representation called 3D Modified Fisher Vectors (3DmFV). Our representation is hybrid as it combines a coarse discrete grid structure with continuous generalized Fisher vectors. Using the grid enables us to design a new CNN architecture for real-time point cloud classification. In a series of performance analysis experiments, we demonstrate competitive results or even better than state-of-the-art on challenging benchmark datasets while maintaining robustness to various data corruptions.
The code requires at least MATLAB 2019a (it is the first to support 3D CNNs). Additionally a supported GPU is required with a ComputeCapability of at least 3.0. and CUDA 10.0
1. Download The data directory from the onedrive folder in the link below.
2. Run `train.m` ( with desired parameters - number of Gaussians, number of points, augmentations, etc).
This will also create a log directory and sub-directories based on the number of points and Gaussians.
1.a. Download the log directory form the onedrive link below.
1.b. Alternatively, test your own trained model by training first.
2. Run `test.m` (make sure to set the path to the desired trained model).
link to data and log:
It is trained on the ModelNet40 dataset by Princeton, so please cite their work as well.
To train on your own data the directory structure should be:
1. Subdirectories named test and train.
2. Categories sub-sub-directories ( e.g. dataset_name/test/category_name/filename.txt)/
3. Each point cloud has to be saved as a separate `.txt` file.
Itzik Ben Shabat (2020). 3DmFV-Net : Point Cloud classification using 3D CNNs (https://www.github.com/sitzikbs/3DmFV-Net-MATLAB), GitHub. Retrieved .
Ben-Shabat, Yizhak, et al. “3DmFV: Three-Dimensional Point Cloud Classification in Real-Time Using Convolutional Neural Networks.” IEEE Robotics and Automation Letters, vol. 3, no. 4, Institute of Electrical and Electronics Engineers (IEEE), Oct. 2018, pp. 3145–52, doi:10.1109/lra.2018.2850061.
Sorry for the late reply - for some reason I am not getting email notifications for comments here so feel free to email me.
regarding the question - the 2D visualization "flattens" the 4D 3DmFV tensor into an image where the columns are Gaussians and the rows are the derivatives with respect to the different parameters and different symmetric functions.
Having problems visualising 3D Gaussian using the function visualize_2d_3dmfv
Is there a visualize_3d_3dmfv function that you haven't uploaded or am i missing something?
Thanks in advance.
Minor changes to the readme file