Context-recognition and activity recognition systems in multi-user environments such as smart homes, usually assume to know the number of occupants in the environment. However, being able to count the number of users in the environment is important in order to accurately recognize the activities of (groups of) agents. For smart environments without cameras, the problem of counting the number of agents is non-trivial. This is in part due to the difficulty of using a single non-vision based sensors to discriminate between one or several persons, and thus information from several sensors must be combined in order to reason about the presence of several agents. In this paper we address the problem of counting the number of agents in a topologically known environment using simple sensors that can indicate anonymous human presence. To do so, we connect an ontology to a probabilistic model (a Hidden Markov Model) in order to estimate the number of agents in each section of the environment. We evaluate our methods on a smart home setup where a number of motion and pressure sensors are distributed in various rooms of the home.