To Örebro University

Fusing color (RGB) images and range or depth (D) data in the form of RGB-D or multi-sensory setups is a relatively new but rapidly growing modality for many agricultural tasks. RGB-D data have potential to provide valuable information for many agricultural tasks that rely on perception, but collection of appropriate data and suitable ground truth information can be challenging and labor-intensive, and high-quality publicly available datasets are rare. This paper presents a survey of the existing RGB-D datasets available for agricultural robotics, and summarizes key trends and challenges in this research field. It evaluates the relative advantages of the commonly used sensors, and how the hardware can affect the characteristics of the data collected. It also analyzes the role of RGB-D data in the most common vision-based machine learning tasks applied to agricultural robotic operations: visual recognition, object detection, and semantic segmentation, and compares and contrasts methods that utilize 2-D and 3-D perceptual data.

There is an increasing interest in the use of automation in plant production settings. Here, we employed a robotic platform to induce controlled mechanical stimuli (CMS) aiming to improve basil quality. Semi-targeted UHPLC-qToF-MS analysis of organic acids, amino acids, phenolic acids, and phenylpropanoids revealed changes in basil secondary metabolism under CMS, which appear to be associated with changes in taste, as revealed by different means of sensory evaluation (overall liking, check-all-that-apply, and just-about-right analysis). Further network analysis combining metabolomics and sensory data revealed novel links between plant metabolism and sensory quality. Amino acids and organic acids including maleic acid were negatively associated with basil quality, while increased levels of secondary metabolites, particularly linalool glucoside, were associated with improved basil taste. In summary, by combining metabolomics and sensory analysis we reveal the potential of automated CMS on crop production, while also providing new associations between plant metabolism and sensory quality.

In this paper we present a multidisciplinary approach combining technical practices with sensory data to optimize cultivation practices for production of plants using sensory evaluation and further the how it affects nutritional content. We apply sensory evaluation of plants under mechanical stress, in this case robot cultivated basil. Plant stress is a research field studying plants' reactions to suboptimal conditions leading to effects on growth, crop yield, and resilience to harsh environmental conditions. Some of the effects induced by mechanical stress have been shown to be beneficial, both in futuristic commercial growing paradigms (e.g., vertical farming), as well as in altering the plant's nutritional content. This pilot study uses established sensory methods such as Liking, Just-About-Right (JAR) and Check-All-That-Apply (CATA) to study the sensory effect of mechanical stress on cropped basil induced by a specially developed robotic platform. Three different kinds of cropped basil were evaluated: (a) mechanically stressed-robot cultivated, (b) non-stressed -robot cultivated from the same cropping bed (reference); and (c) a commercially organic produced basil. We investigated liking, critical attributes, sensory profile, and the use of a semi-trained culinary panel to make any presumptions on consumer acceptance. The semi-trained panel consisted of 24 culinary students with experience of daily judging sensory aspects of specific food products and cultivated crops. The underlying goal is to assess potential market aspects related to novel mechanical cultivation systems. Results shows that basil cropped in a controlled robot cultivated platform resulted in significantly better liking compared to commercially organic produced basil. Results also showed that mechanical stress had not negatively affected the sensory aspects, suggesting that eventual health benefits eating stressed plants do not come at the expense of the sensory experience.

This research evaluates the effect on herbal crops of mechanical stress induced by two specially developed robotic platforms. The changes in plant morphology, metabolite profiles, and element content are evaluated in a series of three empirical experiments, conducted in greenhouse and CNC growing bed conditions, for the case of basil plant growth. Results show significant changes in morphological features, including shortening of overall stem length by up to 40% and inter-node distances by up to 80%, for plants treated with a robotic mechanical stress-induction protocol, compared to control groups. Treated plants showed a significant increase in element absorption, by 20-250% compared to controls, and changes in the metabolite profiles suggested an improvement in plants' nutritional profiles. These results suggest that repetitive, robotic, mechanical stimuli could be potentially beneficial for plants' nutritional and taste properties, and could be performed with no human intervention (and therefore labor cost). The changes in morphological aspects of the plant could potentially replace practices involving chemical treatment of the plants, leading to more sustainable crop production.

Current harvesting robots have limited performance, due to the unstructured and dynamic nature of both the target crops and their environment. Efforts to date focus on improving sensing and robotic systems. This paper presents a parallel approach, to "design" the crop and its environment to best fit the robot, similar to robotic integration in industrial robot deployments.

A systematic methodology to select and modify the crop "design" (crop and environment) to improve robotic harvesting is presented. We define crop-dependent robotic features for successful harvesting (e.g., visibility, reachability), from which associated crop features are identified (e.g., crop density, internode length). Methods to influence the crop features are derived (e.g., cultivation practices, climate control) along with a methodological approach to evaluate the proposed designs. A case study of crop "design" for robotic sweet pepper harvesting is presented, with statistical analyses of influential parameters. Since comparison of the multitude of existing crops and possible modifications is impossible due to complexity and time limitations, a sequential field experimental setup is planned. Experiments over three years, 10 cultivars, two climate control conditions, two cultivation techniques and two artificial illumination types were performed. Results showed how modifying the crop effects the crops characteristics influencing robotic harvesting by increased visibility and reachability. The systematic crop "design" approach also led to robot design recommendations. The presented "engineering" the crop "design" framework highlights the importance of close synergy between crop and robot design achieved by strong collaboration between robotic and agronomy experts resulting in improved robotic harvesting performance.

This paper presents the development, testing and validation of SWEEPER, a robot for harvesting sweet pepper fruit in greenhouses. The robotic system includes a six degrees of freedom industrial arm equipped with a specially designed end effector, RGB-D camera, high-end computer with graphics processing unit, programmable logic controllers, other electronic equipment, and a small container to store harvested fruit. All is mounted on a cart that autonomously drives on pipe rails and concrete floor in the end-user environment. The overall operation of the harvesting robot is described along with details of the algorithms for fruit detection and localization, grasp pose estimation, and motion control. The main contributions of this paper are the integrated system design and its validation and extensive field testing in a commercial greenhouse for different varieties and growing conditions. A total of 262 fruits were involved in a 4-week long testing period. The average cycle time to harvest a fruit was 24 s. Logistics took approximately 50% of this time (7.8 s for discharge of fruit and 4.7 s for platform movements). Laboratory experiments have proven that the cycle time can be reduced to 15 s by running the robot manipulator at a higher speed. The harvest success rates were 61% for the best fit crop conditions and 18% in current crop conditions. This reveals the importance of finding the best fit crop conditions and crop varieties for successful robotic harvesting. The SWEEPER robot is the first sweet pepper harvesting robot to demonstrate this kind of performance in a commercial greenhouse.

Current practice for vine yield estimation is based on RGB cameras and has limited performance. In this paper we present a method for outdoor vine yield estimation using a consumer grade RGB-D camera mounted on a mobile robotic platform. An algorithm for automatic grape cluster size estimation using depth information is evaluated both in controlled outdoor conditions and in commercial vineyard conditions. Ten video scans (3 camera viewpoints with 2 different backgrounds and 2 natural light conditions), acquired from a controlled outdoor experiment and a commercial vineyard setup, are used for analyses. The collected dataset (GRAPES3D) is released to the public. A total of 4542 regions of 49 grape clusters were manually labeled by a human annotator for comparison. Eight variations of the algorithm are assessed, both for manually labeled and auto-detected regions. The effect of viewpoint, presence of an artificial background, and the human annotator are analyzed using statistical tools. Results show 2.8-3.5 cm average error for all acquired data and reveal the potential of using lowcost commercial RGB-D cameras for improved robotic yield estimation.

A methodology for planning the sequence of tasks for a harvesting robot is presented. The fruit targets are situated at unknown locations and must be detected by the robot through a sequence of sensing tasks. Once the targets are detected, the robot must execute a harvest action at each target location. The traveling salesman paradigm (TSP) is used to plan the sequence of sensing and harvesting tasks taking into account the costs of the sensing and harvesting actions and the traveling times. Sensing is planned online. The methodology is validated and evaluated in both laboratory and greenhouse conditions for a case study of a sweet pepper harvesting robot. The results indicate that planning the sequence of tasks for a sweet pepper harvesting robot results in 12% cost reduction. Incorporating the sensing operation in the planning sequence for fruit harvesting is a new approach in fruit harvesting robots and is important for cycle time reduction. Furthermore, the sequence is re-planned as sensory information becomes available and the costs of these new sensing operations are also considered in the planning.

In this paper we present the usage of PointNet, a deep neural network that consumes raw un-ordered point clouds, for detection of grape vine clusters in outdoor conditions. We investigate the added value of feeding the detection network with both RGB and depth, contradictory to common practice in agricultural robotics of relying on RGB only. A total of 5057 pointclouds (1033 manually annotated and 4024 annotated using geometric reasoning) were collected in a field experiment conducted in outdoor conditions on 9 grape vines and 5 plants. The detection results show overall accuracy of 91% (average class accuracy of 74%, precision 53% recall 48%) for RGBXYZ data and a significant drop in recall for RGB or XYZ data only. These results suggest the usage of depth cameras for vision in agricultural robotics is crucial for crops where the color contrast between the crop and the background is complex. The results also suggest geometric reasoning can be used for increased training set size, a major bottleneck in the development of agricultural vision systems.