Pending the race towards fully automated drive, vehicles offering several levels of automation are a prevalent scenario, as many road types, traffic and weather conditions will not allow fully automated drive. Instead, fragmented trips with regard to automation will prevail, where drivers will have different levels of automation available at different times. Given this scenario and the complexity of vehicles offering multiple levels of automation with different driving modes depending on prevailing conditions, the need for drivers to understand their responsibility during the different modes becomes critical. During an on-road Wizard-of-Oz driving study under real driving conditions, 20 participants were confronted with a vehicle offering both a level 2 and a level 4 driving automation system. Based on the conducted interviews a conceptual model outlining the different blocks constituting the drivers' perceived responsibility over the driving task was developed. This model can be used as a support in designing and developing driving automation systems and a clear system design which aids drivers understanding of their responsibility during the interaction with an automated vehicle.