In this article, we describe an approach to address the issue of automatically building and using high-level symbolic representations that capture physical interactions between objects in static configurations. Our work targets robotic manipulation systems where objects need to be safely removed from piles that come in random configurations. We assume that a 3D visual perception module exists so that objects in the piles can be completely or partially detected. Depending on the outcome of the perception, we divide the issue into two sub-issues: 1) all objects in the configuration are detected; 2) only a subset of objects are correctly detected. For the first case, we use notions from geometry and static equilibrium in classical mechanics to automatically analyze and extract act and support relations between pairs of objects. For the second case, we use machine learning techniques to estimate the probability of objects supporting each other. Having the support relations extracted, a decision making process is used to identify which object to remove from the configuration so that an expected minimum cost is optimized. The proposed methods have been extensively tested and validated on data sets generated in simulation and from real world configurations for the scenario of unloading goods from shipping containers.