This paper addresses the problem of recognizing activities of daily living. The novelty lies in the use of an existing knowledge base (ConceptNet) to introduce prior knowledge into the system in order to reduce the amount of learning required to deploy the system in a real environment. The use of household objects is central in the recognition of activities that are being performed, and we attach semantic meaning to both the objects and activities that are being recognized. The paper describes a framework which is specifically geared towards realizing activity recognition systems which leverage prior knowledge. A preliminary implementation of a neural network based recognition system built on this framework is shown, and the added value of prior knowledge is evaluated through the use of various data sets.