As the field of configuration planning grows, so does the need for objective comparisons of algorithms and results. As the community stands today, different approaches to formalise and solve the problem at hand exist, and little or no importance has been given to compare results of different research groups. In this paper we summarize the definitions used by a few different research groups, and we explain two empiric method for comparing planning algorithms, based on statistics. While the methods themselves do not solve all the problems of comparative studies, it is a first step towards numerically comparing performances of the different configuration planning methods proposed by the community.