To Örebro University

The first WARA Robotics Mobile Manipulation Challenge, held in December 2024 at ABB Corporate Research in Västerås, Sweden, addressed the automation of task-intensive and repetitive manual labor in laboratory environments -- specifically the transport and cleaning of glassware. Designed in collaboration with AstraZeneca, the challenge invited academic teams to develop autonomous robotic systems capable of navigating human-populated lab spaces and performing complex manipulation tasks, such as loading items into industrial dishwashers. This paper presents an overview of the challenge setup, its industrial motivation, and the four distinct approaches proposed by the participating teams. We summarize lessons learned from this edition and propose improvements in design to enable a more effective second iteration to take place in 2025. The initiative bridges an important gap in effective academia-industry collaboration within the domain of autonomous mobile manipulation systems by promoting the development and deployment of

Branched deformable linear objects (BDLOs), such as wire harnesses, are important connecting components in manufacturing industries. However, due to deformability, a lack of distinct visual features, and complex branched structure, automating tasks involving these BDLOs remains a challenge. In this paper, we propose a particle-filter-based method to track the state of a BDLO. To circumvent the high cost of tracking the complex high-dimensional BDLO state, we instead track each branch as an individual B-spline. Our method learns a data-driven model to predict the likelihood of each particle conditioned on depth image observation. In contrast to current state-of-the-art approaches based on non-rigid registration, we do not require pre-segmenting the BDLO, thus alleviating a strong and limiting assumption. We train our approach on domain-randomized depth data from simulation and achieve zero-shot transfer to real-world BDLOs, achieving state-of-the- art tracking performance when the pre-segmentation fails.

Deformable linear objects (DLOs), such as cables, wires, ropes, and sutures, are important components in various applications in robotics. Although automating DLO manipulation tasks through robot deployment can offer benefits in terms of cost reduction and increased efficiency, it presents difficult challenges. Unlike rigid objects, DLOs can deform and possess high-dimensionalstate space, significantly amplifying the complexity of their dynamics. These inherent characteristics, combined with the absence of distinctive features and the occurrence of occlusion, contribute to the difficulties involved in DLO manipulation tasks.

This dissertation focuses on developing novel approaches for two aspects: modeling and tracking DLOs. Both aspects are important in DLO manipulation, yet they remain open research questions. Current analytical physics-based methods for modeling DLO dynamics are either time-consuming or inaccurate and often undifferentiable, which hampers their applications in robot planning and control. Although deep learning methods have shown promise in modeling object dynamics, there is still a gap in learning DLO dynamics in a 3D environment. As for the tracking, many current methods rely on assumptions such as knowing the DLO initial state and segmented DLO point sets, which are rarely fulfilled in real-world scenarios, significantly limiting their practical applicability.

This dissertation aims to answer three research questions: How can data-driven models be used for learning DLO dynamics? How can the data-driven models be efficiently trained for real-world DLO manipulation tasks? How can images be used to track the state of DLOs during manipulation in uncontrolled real-world settings?

The first contribution of this dissertation is a data-driven model that effectively simulates DLO state transitions. To bridge the current gap in learning full 3D DLO dynamics, a new DLO representation and a recurrent network module are introduced to facilitate better effect propagation between different segments along the DLO. Meanwhile, the model is differentiable, enabling efficient model predictive control for real-world DLO shape control tasks. However, data-driven approaches demand a large amount of training data, which can be time-consuming and laborious to collect in practice. Thus, the second and third contributions propose two frameworks for minimizing the burden incurred by the data collection process. Specifically, a framework is proposed for learning the data-driven model on synthetic data from simulation. Parameters of the simulation model are identified by solving an optimization problem using the differential evolution algorithm with only a few trajectories of a real DLO required. This dissertation also proposes a trial-and-error interaction approach inspired by model-based reinforcement learning, which significantly reduces the need for training data and automates the data collection process.

The above contributions rely on artificial markers for tracking the DLO state during data collection and closed-loop control, which is acknowledged as a limitation. To address this, the fourth contribution proposes a novel approach that utilizes a particle filter within a low-dimensional state embedding learned by an autoencoder. This approach achieves robust tracking under occlusion and eliminates the need for high-fidelity physics simulations or manually designed constraints. Furthermore, the particle-filter-based method is employed and extended to track the state of a branched deformable linear object (BDLO), which is more challenging because of its complex branched structure. The proposed approach learns a likelihood prediction function directly from depth images in simulation, without requiring segmented point sets of the BDLO.

In conclusion, with the proposed methods for modeling and tracking DLOs, this dissertation contributes to advancing a broad range of applications, including DLO simulation, tracking, and manipulation. The development of these approaches lays the foundations for various directions of future research, which are further discussed in the dissertation.

Tracking deformable linear objects (DLOs) is a key element for applications where robots manipulate DLOs. However, the lack of distinctive features or appearance on the DLO and the object’s high-dimensional state space make tracking challenging and still an open question in robotics. In this paper, we propose a method for tracking the state of a DLO by applying a particle filter approach, where the posterior probability of each sample is estimated by a learned predictor. Our method can achieve accurate tracking even with no prerequisite segmentation which many related works require. Due to the differentiability of the posterior probability predictor, our method can leverage the gradients of posterior probabilities with respect to the latent states to improve the motion model in the particle filter. The preliminary experiments suggest that the proposed method can provide robust tracking results and the estimated DLO state converges quickly to the true state if the initial state is unknown.

Robots manipulating deformable linear objects (DLOs) – such as surgical sutures in medical robotics, or cables and hoses in industrial assembly – can benefit substantially from accurate and fast differentiable predictive models. However, the off-the-shelf analytic physics models fall short of differentiability. Recently, neural-network-based data-driven models have shown promising results in learning DLO dynamics. These models have additional advantages compared to analytic physics models, as they are differentiable and can be used in gradient-based trajectory planning. Still, the data-driven approaches demand a large amount of training data, which can be challenging for real-world applications. In this paper, we propose a framework for learning a differentiable data-driven model for DLO dynamics with a minimal set of real-world data. To learn DLO twisting and bending dynamics in a 3D environment, we first introduce a new suitable DLO representation. Next, we use a recurrent network module to propagate effects between different segments along a DLO, thereby addressing a critical limitation of current state-of-the-art methods. Then, we train a data-driven model on synthetic data generated in simulation, instead of foregoing the time-consuming and laborious data collection process for real-world applications. To achieve a good correspondence between real and simulated models, we choose a set of simulation model parameters through parameter identification with only a few trajectories of a real DLO required. We evaluate several optimization methods for parameter identification and demonstrate that the differential evolution algorithm is efficient and effective for parameter identification. In DLO shape control tasks with a model-based controller, the data-driven model trained on synthetic data generated by the resulting models performs on par with the ones trained with a comparable amount of real-world data which, however, would be intractable to collect.

Traditional approaches to manipulating the state of deformable linear objects (DLOs) - i.e., cables, ropes - rely on model-based planning. However, constructing an accurate dynamic model of a DLO is challenging due to the complexity of interactions and a high number of degrees of freedom. This renders the task of achieving a desired DLO shape particularly difficult and motivates the use of model-free alternatives, which while maintaining generality suffer from a high sample complexity. In this paper, we bridge the gap between these fundamentally different approaches and propose a framework that learns dynamic models of DLOs through trial-and-error interaction. Akin to model-based reinforcement learning (RL), we interleave learning and exploration to solve a 3D shape control task for a DLO. Our approach requires only a fraction of the interaction samples of the current state-of-the-art model-free RL alternatives to achieve superior shape control performance. Unlike offline model learning, our approach does not require expert knowledge for data collection, retains the ability to explore, and automatically selects relevant experience.

Tracking of deformable linear objects (DLOs) is important for many robotic applications. However, achieving robust and accurate tracking is challenging due to the lack of distinctive features or appearance on the DLO, the object's high-dimensional state space, and the presence of occlusion. In this letter, we propose a method for tracking the state of a DLO by applying a particle filter approach within a lower-dimensional state embedding learned by an autoencoder. The dimensionality reduction preserves state variation, while simultaneously enabling a particle filter to accurately track DLO state evolution with a practically feasible number of particles. Compared to previous works, our method requires neither running a high-fidelity physics simulation, nor manual designs of constraints and regularization. Without the assumption of knowing the initial DLO state, our method can achieve accurate tracking even under complex DLO motions and in the presence of severe occlusions.

Modeling dynamics of deformable linear objects (DLOs), such as cables, hoses, sutures, and catheters, is an important and challenging problem for many robotic manipulation applications. In this paper, we propose the first method to model and learn full 3D dynamics of DLOs from data. Our approach is capable of capturing the complex twisting and bending dynamics of DLOs and allows local effects to propagate globally. To this end, we adapt the interaction network (IN) dynamics learning method for capturing the interaction between neighboring segments in a DLO and augment it with a recurrent model for propagating interaction effects along the length of a DLO. For learning twisting and bending dynamics in 3D, we also introduce a new suitable representation of DLO segments and their relationships. Unlike the original IN method, our model learns to propagate the effects of local interaction between neighboring segments to each segment in the chain within a single time step, without the need for iterated propagation steps. Evaluation of our model with synthetic and newly collected real-world data shows better accuracy and generalization in short-term and long-term predictions than the current state of the art. We further integrate our learned model in a model predictive control scheme and use it to successfully control the shape of a DLO. Our implementation is available at https : //gitsvn-nt.oru.se/ammlab-public/in-bilstm.