Authors

Abstract

Autonomous harvesting and evaluation of apples reduce the labour cost. Segmentation of apple point clouds from consumer-grade RGB-D camera is the most important and challenging step in the harvesting process due to the complex structure of apple trees. This paper put forward a segmentation method of apple point clouds based on regions of interest (ROI) in RGB images. Firstly, an annotated RGB dataset of apple trees was built and applied to train the optimized Faster R-CNN to locate ROI containing apples in RGB images. Secondly, the relationship between RGB images and depth images was built to roughly segment the apple point clouds by ROI. Finally, the quality control procedure (QCP) was proposed to improve the quality of segmented apple point clouds. Images for training mainly included two lighting condition, two colours and three apple varieties in orchard, making this method more suitable for practical applications. QCP performed well in filtering noise points and achieved Purity as 96.7% and 96.2% for red and green apples, respectively. Through the comparison method, experimental results indicated that the segmentation method based on ROI is more effective and accurate for red and green apples in orchard. The segmentation method of point clouds based on ROI has great potential for segmentation of point clouds in unstructured scenes.

Abstract in Chinese

苹果的自动采摘和评估降低了劳动成本。在采摘过程中由于苹果树的复杂结构，分割消费级RGB-D相机获得的苹果点云是最重要且具有挑战性的一步。本文提出基于彩色图像中感兴趣区域的点云分割方法。首先制作了一个用来训练优化过的Faster-RCNN的苹果树数据集，定位出彩色图像中包含有苹果的感兴趣区域。然后，构建了彩色图与深度图之间的关系量，依照此关系量使用感兴趣区域对点云进行快速的粗分割。最后，针对粗分割的点云特征提出质量控制程序，提升苹果点云的分割质量。用来训练的彩色图像主要包含了果园中的两种光照条件以及三个品种、两种颜色的苹果，提升了此方法在实际场景中的适用性。质量控制程序的处理效果理想，对于红绿两色苹果分别得到了96.7%和96.2%的纯净度。通过对比实验，结果证明了基于彩色图像中感兴趣区域的苹果点云分割方法对于果园中红绿两色的苹果均有更高的效率和准确率, 在非结构化场景中具有非常大的应用前景