Summary:<\/em><\/strong> \u00a0Autonomous vehicles (AUVs) and many other systems that need to accurately perceive the world around them will be much better off when image classification moves from 2D to 3D.\u00a0 Here we examine the two leading approaches to 3D classification, Point Clouds and Voxel Grids.<\/em><\/p>\n \u00a0<\/p>\n In the AUV world, simultaneous localization and mapping (SLAM) is the technical term for how the vehicle maintains awareness of its surroundings, both static (traffic lights) and moving (other cars and pedestrians).\u00a0 If the car could visualize itself in 3D space instead of a series of 2D snapshots then performance would be significantly better.<\/p>\n \u00a0<\/p>\n Sensors Already See in 3D<\/strong><\/span><\/p>\n The two primary sensors on AUVs for this dynamic awareness are LIDARs and RGB-D cameras.\u00a0 RGB-D, also known a depth cameras capture not only the 2D RGB data but also depth based on \u2018time of flight\u2019, literally the time it takes the photons to reach the sensor.\u00a0 Not too many years ago this was enormously expensive and complex but sensor technology has made it reasonable to put these on lots of devices.\u00a0 Think Microsoft Kinect from 2010.<\/p>\n So by putting two RGB-D cameras on a car you have introduced stereoscopic vision and the ability to capture a full 3D dataset of all the objects surrounding the car.<\/p>\n But curiously, the 3D data from both LIDAR and RGB-D cameras has so far been analyzed only using our existing 2D CNN algorithms.\u00a0 Basically we\u2019ve been throwing all that valuable 3D data away.<\/p>\n \u00a0<\/p>\n Storing the Data for Analysis<\/strong><\/span><\/p>\n Deep learning is finally catching up with techniques for 3D CNNs.\u00a0 The techniques are relatively new but are rapidly approaching commercialization.<\/p>\n There are fundamentally two ways to store the 3D image data for 3D CNN image classification.<\/p>\n Voxel grids<\/strong> (b) are 3D versions of pixels (a mash up of \u2018volume\u2019 and \u2018pixel\u2019).\u00a0 In our 2D CNN world we analyze only one \u2018slice\u2019 of pixels.\u00a0 In the Voxel world there are many different 2D slices that add up to the full 3D object.\u00a0 Resolution depends on the size of the pixel as well as the depth of the \u2018slice\u2019.<\/p>\n Point clouds can literally be an infinite number of points in space with coordinates that \u2018float\u2019 versus Voxel grids where each voxel has a discrete coordinate within a predefined space.<\/p>\n Point clouds are by definition unordered while Voxel grids are ordered data.<\/p>\n Both versions appear to have the same computational obstacle.\u00a0 Given the common 256 x 256 image size for 2D CNN classification, it would seem that taking a 3D layer would require 256^3 pixels and that would impose very high computational and memory costs.\u00a0 In fact, through experiment, 3D voxels of 32^3 or 64^3 yields accuracy similar to the larger 2D image.\u00a0 A 64^3 image has essentially the same requirement as a 512^2 image.<\/p>\n \u00a0<\/p>\n Performance Comparisons<\/strong><\/span><\/p>\n Research is increasingly aided by a growing number of open source data bases for training.\u00a0 Notably:<\/p>\n The leading model packages so far are:<\/p>\n For Point-based: PointNet and PointNet++<\/p>\n For Voxel-base: Voxel ResNet and Voxel CNN<\/p>\n Although research continues along both fronts, a recent study published by Nvdia comparing traditional, point-based, and voxel-based systems shows the Voxel approach in the lead.\u00a0 The Point cloud data could be used but the accuracy was lower and the computational cost higher than the Voxel approaches.<\/p>\n Voxelization converged faster making it preferable for real time object classification and in addition was more accurate.<\/p>\n For more background try these papers here<\/a><\/u><\/em> and here<\/a><\/u><\/em> or the original NVDIA study here<\/a><\/u><\/em>.<\/p>\n \u00a0\u00a0<\/p>\n Other articles by Bill Vorhies<\/u><\/em><\/a><\/p>\n \u00a0<\/p>\n About the author:\u00a0 Bill is Contributing Editor for Data Science Central.\u00a0 Bill is also President & Chief Data Scientist at Data-Magnum and has practiced as a data scientist since 2001.\u00a0 His articles have been read more than 2 million times.<\/p>\n He can be reached at:<\/p>\n Bill@DataScienceCentral.com<\/a> or<\/span> Bill@Data-Magnum.com<\/a><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<\/a>One of the well-known problems in CNN image classification is that because the CNN classifier sees only a 2D image of the object it won\u2019t recognize that same object if it\u2019s rotated.\u00a0 The solution thus far has been to train on many different orthogonal views of the same object and that vastly expands the problem of training data and training time.<\/p>\n
<\/a>Point Clouds<\/strong> (a) are simply collections of 3D points in space as might be collected from the rapidly rotating LIDAR beam on our AUV.\u00a0 They have an \u2018xyz\u2019 address in space and may also capture the \u2018rgb\u2019 data to better differentiate the object.\u00a0 This data is \u2018per pixel\u2019 converted to point clouds for processing.<\/p>\n
\n
<\/a><\/p>\n