Deformable linear objects (DLOs), such as cables, wires, ropes, and sutures, are important components in various applications in robotics. Although automating DLO manipulation tasks through robot deployment can offer benefits in terms of cost reduction and increased efficiency, it presents difficult challenges. Unlike rigid objects, DLOs can deform and possess high-dimensionalstate space, significantly amplifying the complexity of their dynamics. These inherent characteristics, combined with the absence of distinctive features and the occurrence of occlusion, contribute to the difficulties involved in DLO manipulation tasks.
This dissertation focuses on developing novel approaches for two aspects: modeling and tracking DLOs. Both aspects are important in DLO manipulation, yet they remain open research questions. Current analytical physics-based methods for modeling DLO dynamics are either time-consuming or inaccurate and often undifferentiable, which hampers their applications in robot planning and control. Although deep learning methods have shown promise in modeling object dynamics, there is still a gap in learning DLO dynamics in a 3D environment. As for the tracking, many current methods rely on assumptions such as knowing the DLO initial state and segmented DLO point sets, which are rarely fulfilled in real-world scenarios, significantly limiting their practical applicability.
This dissertation aims to answer three research questions: How can data-driven models be used for learning DLO dynamics? How can the data-driven models be efficiently trained for real-world DLO manipulation tasks? How can images be used to track the state of DLOs during manipulation in uncontrolled real-world settings?
The first contribution of this dissertation is a data-driven model that effectively simulates DLO state transitions. To bridge the current gap in learning full 3D DLO dynamics, a new DLO representation and a recurrent network module are introduced to facilitate better effect propagation between different segments along the DLO. Meanwhile, the model is differentiable, enabling efficient model predictive control for real-world DLO shape control tasks. However, data-driven approaches demand a large amount of training data, which can be time-consuming and laborious to collect in practice. Thus, the second and third contributions propose two frameworks for minimizing the burden incurred by the data collection process. Specifically, a framework is proposed for learning the data-driven model on synthetic data from simulation. Parameters of the simulation model are identified by solving an optimization problem using the differential evolution algorithm with only a few trajectories of a real DLO required. This dissertation also proposes a trial-and-error interaction approach inspired by model-based reinforcement learning, which significantly reduces the need for training data and automates the data collection process.
The above contributions rely on artificial markers for tracking the DLO state during data collection and closed-loop control, which is acknowledged as a limitation. To address this, the fourth contribution proposes a novel approach that utilizes a particle filter within a low-dimensional state embedding learned by an autoencoder. This approach achieves robust tracking under occlusion and eliminates the need for high-fidelity physics simulations or manually designed constraints. Furthermore, the particle-filter-based method is employed and extended to track the state of a branched deformable linear object (BDLO), which is more challenging because of its complex branched structure. The proposed approach learns a likelihood prediction function directly from depth images in simulation, without requiring segmented point sets of the BDLO.
In conclusion, with the proposed methods for modeling and tracking DLOs, this dissertation contributes to advancing a broad range of applications, including DLO simulation, tracking, and manipulation. The development of these approaches lays the foundations for various directions of future research, which are further discussed in the dissertation.