To Örebro University

This paper presents an approach for tracking multiple persons using a combination of colour and thermal vision sensors on a mobile robot. First, an adaptive colour model is incorporated into the measurement model of the tracker. Second, a new approach for detecting occlusions is introduced, using a machine learning classifier for pairwise comparison of persons (classifying which one is in front of the other). Third, explicit occlusion handling is then incorporated into the tracker.

This thesis addresses the problem of people detection and tracking by mobile robots in indoor environments. A system that can detect and recognise people is an essential part of any mobile robot that is designed to operate in populated environments. Information about the presence and location of persons in the robot’s surroundings is necessary to enable interaction with the human operator, and also for ensuring the safety of people near the robot.

The presented people tracking system uses a combination of thermal and colour information to robustly track persons. The use of a thermal camera simplifies the detection problem, which is especially difficult on a mobile platform. The system is based on a fast and efficient samplebased tracking method that enables tracking of people in real-time. The elliptic measurement model is fast to calculate and allows detection and tracking of persons under different views. An explicit model of the human silhouette effectively distinguishes persons from other objects in the scene. Moreover the process of detection and localisation is performed simultaneously so that measurements are incorporated directly into the tracking framework without thresholding of observations. With this approach persons can be detected independently from current light conditions and in situations where other popular detection methods based on skin colour would fail.

A very challenging situation for a tracking system occurs when multiple persons are present on the scene. The tracking system has to estimate the number and position of all persons in the vicinity of the robot. Tracking of multiple persons in the presented system is realised by an efficient algorithm that mitigates the problems of combinatorial explosion common to other known algorithms. A sequential detector initialises an independent tracking filter for each new person appearing in the image. A single filter is automatically deleted when it stops tracking a person. While thermal vision is good for detecting people, it can be very difficult to maintain the correct association between different observations and persons, especially where they occlude one another, due to the unpredictable appearance and social behaviour of humans. To address these problems the presented tracking system uses additional information from the colour camera. An adaptive colour model is incorporated into the measurement model of the tracker to improve data association. For this purpose an efficient integral image based method is used to maintain the real-time performance of the tracker. To deal with occlusions the system uses an explicit method that first detects situations where people occlude each other. This is realised by a new approach based on a machine learning classifier for pairwise comparison of persons that uses both thermal and colour features provided by the tracker. This information is then incorporated into the tracker for occlusion handling and to resolve situations where persons reappear in a scene.

Finally the thesis presents a comprehensive, quantitative evaluation of the whole system and its different components using a set of well defined performance measures. The behaviour of the system was investigated on different data sets including different real office environments and different appearances and behaviours of persons. Moreover the influence of all important system parameters on the performance of the system was checked and their values optimised based on these results.

This paper presents a vision-based approach for tracking people on a mobile robot using thermal images. The approach combines a particle filter with two alternative measurement models that are suitable for real-time tracking. With this approach a person can be detected independently from current light conditions and in situations where no skin colour is visible. In addition, the paper presents a comprehensive, quantitative evaluation of the different methods on a mobile robot in an office environment, for both single and multiple persons. The results show that the measurement model that was learned from local greyscale features could improve on the performance of the elliptic contour model, and that both models could be combined to further improve performance with minimal extra computational cost.

In this paper we present a vision-based approach to detect, track and identify people on a mobile robot in real time. While most vision systems for tracking people on mobile robots use skin color information, we present an approach using thermal images and a fast contour model together with a Particle Filter. With this method a person can be detected independently from current light conditions and in situations were no skin color is visible (the person is not close or does not face the robot). Tracking in thermal images is used as an attention system to get an estimate of the position of a person. Based on this estimate we use a pan-tilt camera to zoom to the expected face region and apply a fast face tracker in combination with face recognition to identify the person.

While most vision systems for tracking people on mobile robots use skin color information, we present an approach using thermal images and two different measurement models together with a Particle Filter. With this method a person can be detected independently from current light conditions and in situations were no skin color is visible (the person is not close or does not face the robot). The results show that a measurement model that was learned from local greyscale features improved on the performance of an elliptic contour model, and that both models could be used in combination to further improve performance with minimal extra computational cost

Whenever people move through their environments they do not move randomly. Instead, they usually follow specific trajectories or motion patterns corresponding to their intentions. Knowledge about such patterns enables a mobile robot to robustly keep track of persons in its environment and to improve its behavior. This paper proposes a technique for learning collections of trajectories that characterize typical motion patterns of persons. Data recorded with laser-range finders is clustered using the expectation maximization algorithm. Based on the result of the clustering process we derive a Hidden Markov Model (HMM) that is applied to estimate the current and future positions of persons based on sensory input. We also describe how to incorporate the probabilistic belief about the potential trajectories of persons into the path planning process. We present several experiments carried out in different environments with a mobile robot equipped with a laser range scanner and a camera system. The results demonstrate that our approach can reliably learn motion patterns of persons, can robustly estimate and predict positions of persons, and can be used to improve the navigation behavior of a mobile robot.

Future service robots will need to keep track of the persons in their environment. A number of people tracking systems have been developed for mobile robots, but it is currently impossible to make objective comparisons of their performance. This paper presents a comprehensive, quantitative evaluation of a state-of-the-art people tracking system for a mobile robot in an office environment, for both single and multiple persons.

This paper addresses the problem of detecting and identifying persons with a mobile robot, by sensory fusion of thermal and colour vision information. In the proposed system, people are first detected with a thermal camera, using image analysis techniques to segment the persons in the thermal images. This information is then used to segment the corresponding regions of the colour images, using an affine transformation to solve the image correspondence between the two cameras. After segmentation, the region of the image containing a person is further divided into regions corresponding to the person's head, torso and legs. Temperature and colour features are then extracted from each region for input to a pattern recognition system. Three alternative classfication methods were investigated in experiments with a moving mobile robot and moving persons in an office environment. The best identification performance was obtained with a dynamic recognition method based on a Bayes classifier, which takes into account evidence accumulated in a sequence of images.

This paper addresses the problem of tracking a moving person with a single, omnidirectional camera. An appearance-based tracking system is described which uses a self-acquired appearance model and a Kalman filter to estimate the position of the person. Features corresponding to ``depth cues'' are first extracted from the panoramic images, then an artificial neural network is trained to estimate the distance of the person from the camera. The estimates are combined using a discrete Kalman filter to track the position of the person over time. The ground truth information required for training the neural network and the experimental analysis was obtained from another vision system, which uses multiple webcams and triangulation to calculate the true position of the person. Experimental results show that the tracking system is accurate and reliable, and that its performance can be further improved by learning multiple, person-specific appearance models