The main goal of this Chapter is to introduce SAM, an integrated architecture for concurrent activity recognition, planning and execution. SAM provides a general framework to define how an intelligent environment can assess contextual information from sensory data. The architecture builds upon a temporal reasoning framework operating in closed-loop between physical sensing and actuation components in a smart environments. The capabilities of the system as well as possible examples of its use are discussed in the context of the PEIS-Home, a smart environment integrated with robotic components.