Perceptual anchoring deals with the problem of creating and maintaining the connection between percepts and symbols that refer to the same physical object. When approaching long term use of an anchoring framework which must cope with large sets of data, it is challenging to both efficiently and accurately anchor objects. An approach to address this problem is through visual perception and computationally efficient binary visual features. In this paper, we present a novel hash table algorithm derived from summarized binary visual features. This algorithm is later contextualized in an anchoring framework. Advantages of the internal structure of proposed hash tables are presented, as well as improvements through the use of hierarchies structured by semantic knowledge. Through evaluation on a larger set of data, we show that our approach is appropriate for efficient bottom-up anchoring, and performance-wise comparable to recently presented search tree algorithm.