Automated driving technologies are rapidly being developed. However, until vehicles are fully automated, the control of the dynamic driving task will be shifted between the driver and automated driving system. This paper aims to explore how transitions from automated driving to manual driving affect user experience and how that experience correlates to take-over performance. In the study 20 participants experienced using an automated driving system during rush-hour traffic in the San Francisco Bay Area, CA, USA. The automated driving system was available in congested traffic situations and when active, the participants could engage in non-driving related activities. The participants were interviewed afterwards regarding their experience of the transitions. The findings show that most of the participants experienced the transition from automated driving to manual driving as negative. Their user experience seems to be shaped by several reasons that differ in temporality and are derived from different phases during the transition process. The results regarding correlation between participants' experience and take-over performance are inconclusive, but some trends were identified. The study highlights the need for new design solutions that do not only improve drivers' take-over performance, but also enhance user experience during take-over requests from automated to manual driving.