To Örebro University

The successful application, usability, and social emancipation of AI technologies necessitates that the design and implementation of technical systems be founded on human-centred principles, be it cognitive or behavioural, social, ethical etc. Towards this objective, this thesis develops an interdisciplinary methodology for embedding cognitive behavioural principles in the design and development of next-generation human-centred AI technologies that aim to assist and empower humans in everyday life.

The interdisciplinary methodology developed in this research categorically focusses on two key aspects pertaining to human-centred technology design and engineering: (1) human behavioural precedents; and (2) cognitively founded representational and computational modalities:

• Human behavioural precedents are established by systematically analysing human visuo-locomotive experience during everyday activities involving (embodied) multimodal interactions. We conduct naturalistic behavioural experiments focusing on aspects of visual perception (e.g., inattention blindness) and spatial cognition (e.g., orientation, navigation) in diverse settings of everyday mobility. As specific -in-the-wild- experimental contexts, we focus on behavioural aspects involved in everyday (human) navigation and driving.
• Representational and computational modalities are developed based on cognitively-driven articulation of behavioural precedents. Particularly, a cognitive model of visuospatial complexity for grounding embodied multimodal interactions is developed by incorporating behavioural precedents pertaining to representations of space, motion, and interaction. Furthermore, precedents concerning human preferences are used as a basis for semantically-driven computational synthesis (e.g. in the generation and manipulation of spatial morphologies), and in the articulation of human-centred evaluation and standardisation of AI systems.

As case studies we demonstrate the developed methodology in the backdrop of two application domains: (a) design assistance technologies, and (b) autonomous driving. More broadly, this thesis emphasises the need for embedding ecologically valid behavioural knowledge within the development of "human-centred" technologies.  Furthermore, this research paves the way for the development of systems that understand, interpret and anticipate human behaviour under ecologically valid naturalistic circumstances.

In everyday activities where continuous visual awareness is critical such as driving, several cognitive processes pertaining to visual attention are of the essence, for instance, change detection, anticipation, monitoring, etc. Research suggests that environmental load and task difficulty contribute to failures in visual perception that can be essential for detecting and reacting to safety-critical incidents. However, it is unclear how gaze patterns and attentional strategies are compromised because of environmental complexity in naturalistic driving. In a change detection task during everyday simulated driving, we investigate inattention blindness in relation to environmental complexity and the kind of interaction incidents drivers address. We systematically analyse and evaluate safety-critical situations from real-world driving videos and replicate a number of them in a virtual driving experience. Participants (N= 80) aged 23-45 years old, drove along three levels of environmental complexity (low-medium-high) and various incidents of interaction with roadside users (e.g., pedestrians, cyclists, pedestrians in a wheelchair), categorized as safety critical or not. Participants detected changes in the behaviour of road users and in object properties. We collect multimodal data including eye-tracking, egocentric view videos, movement trace, head movements, driving behaviour, and detection button presses. Results suggest that gaze behaviour (number and duration of fixations, 1st fixation on AOI) is affected negatively by an increase in environmental complexity, but the effect is moderate for safety-critical incidents. Moreover, anticipatory and monitoring attention was crucial for detecting critical changes in behaviour and reacting on time. However, in highly complex environments participants effectively limit attentional monitoring and lingering for non-critical changes and they also controlled “look-but-fail-to-see errors", especially while addressing a safety-related event. We conclude that drivers change attentional strategies, avoiding non-productive forms of attentional elaboration (anticipatory and monitoring) and efficiently disengaging from targets when the task difficulty is high. We discuss the implications for driving education and research driven development of autonomous driving. 

How do the limits of high-level visual processing affect human performance in naturalistic, dynamic settings of (multimodal) interaction where observers can draw on experience to strategically adapt attention to familiar forms of complexity? In this backdrop, we investigate change detection in a driving context to study attentional allocation aimed at overcoming environmental complexity and temporal load. Results indicate that visuospatial complexity substantially increases change blindness but also that participants effectively respond to this load by increasing their focus on safety-relevant events, by adjusting their driving, and by avoiding non-productive forms of attentional elaboration, thereby also controlling “looked-but-failed-to-see” errors. Furthermore, analyses of gaze patterns reveal that drivers occasionally, but effectively, limit attentional monitoring and lingering for irrelevant changes. Overall, the experimental outcomes reveal how drivers exhibit effective attentional compensation in highly complex situations. Our findings uncover implications for driving education and development of driving skill-testing methods, as well as for human-factors guided development of AI-based driving assistance systems.

The development of immersive visuoauditory media brings to the fore several design challenges concerning cognitive human factors, such as visual perception, embodied interaction, and emotional engagement. With a focus on visual perception, our research emphasises a systematic study of embodied multimodal interaction in immersive settings and provides a cognitive model of visuospatial complexity through a series of behavioural studies in VR with human subjects. We report preliminary results on the effect of levels of visuospatial complexity on visuospatial attention patterns and visual search performance. The proposed methodology provides a general foundation for conducting naturalistic studies in immersive perception and interaction, e.g., in the context of established paradigms such as event perception, ensemble perception, visual search and foraging, change blindness. Moreover, we posit that the demonstrated confluence of computational and behavioural studies is needed to better appreciate the complexity and spectrum of varied human-centred challenges in the design of immersive media.

We investigate the role of visuospatial environmental cues on driver (in)attention in everyday naturalistic driving situations. We develop a cognitive model of visuospatial complexity incorporating two critical aspects influencing visual (in)attention: (1) multimodal interaction mechanisms such as gesture, joint attention amongst roadside stakeholders (e.g. pedestrians, cyclists, drivers); and (2) visuospatial environmental features such as clutter, motion, environmental structure. 

Our research emphasises the manner in which a cognitive human-factors guided model to analyse attentiveness can be applied to systematically explore the effects of a combination of environmental and interactional characteristics on visual attention in naturalistic driving. We position the application of the developed cognitive model to serve a foundational purpose in the training and testing of novel driver assistance technologies, e.g., from the viewpoint of systematic compliance with human-centered design guidelines.

We position recent and emerging research in cognitive vision and perception addressing three key questions: (1) What kind of relational abstraction mechanisms are needed to perform (explainable) grounded inference --e.g., question-answering, qualitative generalisation, hypothetical reasoning-- relevant to embodied multimodal interaction? (2) How can such abstraction mechanisms be founded on behaviourally established cognitive human-factors emanating from naturalistic empirical observation? and (3) How to articulate behaviourally established abstraction mechanisms as formal declarative models suited for grounded knowledge representation and reasoning (KR) as part of large-scale hybrid AI and computational cognitive systems.

We contextualise (1--3) in the backdrop of recent results at the interface of AI/KR, and Spatial Cognition and Computation. Our main purpose is to emphasise the importance of behavioural research based foundations for next-generation, human-centred AI, e.g., as relevant to applications in Autonomous Vehicles, Social and Industrial Robots, and Visuo-Auditory Media.

Spatial (memory) update strategies depend on situational factors, such as availability of external information, familiarity with the environment, and cognitive demand. Of key interest to this research is the multimodal integration of environmental and individual characteristics in visuospatial update and reorientation in situations of extensive (embodied) rotational locomotion in naturalistic conditions. We investigate active visuo-locomotive experience and reorientation performance in everyday navigation in natural urban settings.

We examine navigation and update strategy adaptation in relation to familiarity, the difficulty of the task (as articulated via a ``rotation metric´´), and available external visuospatial cues. In two behavioral studies in large-scale built environments, two healthcare facilities and a train station, 45 participants (aged 18-83) performed a navigation task under natural conditions. A multimodal analysis of visuolocomotive behavior was conducted including eye-tracking, video analyses, sketch-mapping task, orientation pointing task, and post-questionnaires.

In the first study conducted in two healthcare facilities, namely the Old and New Parkland hospitals in Dallas (USA), 25 participants were asked to find their way in an unfamiliar environment. We focused on an average of 2 events of confusion or disorientation per participant throughout the route. The environmental analysis of the path reveals correlations between the disorientation events and the visuospatial characteristics of the path at key locations and decision points. Three key results were reported after the combination of the behavioral and the environmental analysis: (1) visual accessibility at key locations (e.g., intersection, entrance hall, atrium lobby) of the path towards manifest cues (e.g. signages, landmarks) and environmental cues (e.g. geometry, symmetry) is critical for better navigation performance and reduced numbers of confusion events, (2) narrow enclosed spaces have a negative impact on navigation experience, (3) extensive ego-rotations, defined as rotation metric of the path, influence orientation performance.

In the second study, we further explore the impact of rotations in navigation performance, and the reorientation strategies used actively during the navigation task at a train station in Bremen (Germany); the task is a part of a typical everyday commuting scenario. More than 60% of the participants experience a confusing event, especially while performing 360-degree ego-rotation. The multimodal analysis also suggests that the level of familiarity is related to the choice of navigation-aids used for reorientation (between environmental cues, and manifest cues) and the timing where these strategies were adopted. Results suggest that familiar navigators rely on environmental cues and exhibit proactive decision-making, whereas unfamiliar ones rely on manifest cues, are late in decision-making, and show no sign of sensorimotor spatial update. Moreover, the visual attention analysis shows that the direction of movement affected the gaze close to the decision points, and the extent of rotation negatively affected the spatial update performance.

Active locomotion where a full-range of combined perceptual and cognitive processes are involved suggests that subjects do not demonstrate spatial updating strategies based on spatial representations, instead explicitly relying on external visuospatial cues. However, the extent of rotational locomotion, the visual accessibility to navigation-aid cues (e.g. signage, landmarks), and the level of familiarity play a fundamental role in the choice of the updating strategies people use for effective reorientation. Next steps of our research involve embodied navigation behavioral studies in virtual environments (VR) where the conditions of the rotation angle tested and the positioning of cues can be systematically manipulated to provide a metric for the extent of which they influence navigation performance.

Background: Incorporating knowledge about human behaviour and the effect of the environment is a major goal for the design and engineering of human-centred autonomous vehicles. Systems that aim to establish a common interaction ground with humans require systematic modelling of empirically established behavioural norms customised to specific contexts.

Aims: Focusing on aspects pertaining to visual attention in driving, we develop a cognitive model of visuospatial complexity for naturalistic driving scenes and explore its effect on visual attention tasks (e.g., involving visual search) during everyday driving.

Methods: By analyzing dynamic naturalistic scenes, we define a scale of visuospatial complexity based on a taxonomy of quantitative, structural, and dynamic attributes. We re-create real-world instances in virtual reality (VR) in four levels of visuospatial complexity. The human-centred basis of the model lies in its behavioural evaluation with human subjects with respect to psychophysical measures (e.g. eye-tracking) pertaining to embodied visuospatial attention.

Results: Empirical results show the levels of visuospatial complexity of the scene correlate with visual search performance parameters, however different categories of attributes contribute differently to the overall effect. We report work-in-progress on the development of a (sample) dataset with the central emphasis on the evaluation of the visuospatial complexity levels on driving stimuli within VR.

Conclusion: The presented cognitive model of visuospatial complexity in everyday driving situations can be used as a basis to design, and evaluate visuospatial sensemaking capabilities of autonomous vehicles. We posit that our methodology encapsulates key cognitive principles founded on empirically established behavioural patterns under naturalistic conditions.

We develop a cognitive evaluation schema for analysing the diversity and nuances of visuospatial complexity and multimodal interactions encountered in naturalistic everyday driving conditions. The proposed schema is based on a finegrained empirical analysis of real-world everyday driving situations involving stakeholders such as drivers, pedestrians, cyclists. Our method involves a semantic analysis of egocentric POVs of stakeholders, focusing on the sequence and duration of events (e.g. velocity or direction change), the combination of modalities used (e.g., gestures, gaze, head-movements), audio, quantity and variety of moving and static objects in the scene e.g., (cars, signs), behavioural metrics from the stakeholders (e.g. gaze allocation, steering), etc. The proposed cognitive evaluation schema consists of three key aspects: (1) Scene characteristics consisting of a combination of quantitative (e.g., clutter, size), structural (e.g. symmetry), and dynamic attributes (e.g. motion), (2) Multimodal interactions consisting of the mode and method of interaction, as well as the level of joint attention achieved, (3) Recipient effects characterising subject’s behaviour and driving performance through physiological measurements (e.g. eye-tracking, head rotation) in a series of virtual reality (VR) environments replicating a number of naturalistic scenarios (and variations therefrom). Driven by behavioural methods in visual perception, we aim to open-up an interdisciplinary frontier for the human-centred design, evaluation / testing of artificial vision modules within AI-technologies for autonomous driving, cognitive robotics etc., where embodied, multimodal human-machine interaction is of the essence. We also demonstrate the practical application of basic visual perception research towards technology-centric settings of social significance.

A people-centred approach for designing large-scale built-up spaces necessitates systematic anticipation of user’s embodied visuo-locomotive experience from the viewpoint of human-environment interaction factors pertaining to aspects such as navigation, wayfinding, usability. In this context, we develop a behaviour-based visuo-locomotive complexity model that functions as a key correlate of cognitive performance vis-a-vis internal navigation in built-up spaces. We also demonstrate the model’s implementation and application as a parametric tool for the identification and manipulation of the architectural morphology along a navigation path as per the parameters of the proposed visuospatial complexity model. We present examples based on an empirical study in two healthcare buildings and showcase the manner in which a dynamic and interactive parametric (complexity) model can promote behaviour-based decision-making throughout the design process to maintain desired levels of visuospatial complexity as part of a navigation or wayfinding experience.