Emotions, Salience Sensitive Control of Human Attention and Computational Modelling
Home Page of the EPSRC Project, Salience Sensitive Control in Humans and Artificial Systems (also called the Salience Project)
This project was a collaboration between the following two institutions:
the Centre for
Cognitive Neuroscience and Cognitive Systems (CCNCS) at the
Howard Bowman (Principle
Investigator, Director CCNCS, Computing Lab, Univ of
(Investigator, Computing Lab, Univ of
· Brad Wyble (Postdoctoral Researcher, Computing Lab, Univ of Kent)
Su Li (Contributing PhD Student,
Computing Lab, Univ of
Patrick Craston (Contributing
PhD Student, Computing Lab, Univ of
· Phil Barnard (External Collaborator, MRC Cognition and Brain Sciences Unit)
Sharma (Internal Collaborator, Psychology, Univ
The final report for the project, which was submitted to our funders (EPSRC) can be found here.
Humans are very good at prioritising competing processing demands. In particular, perception of a salient environmental event can interrupt ongoing processing, causing attention, and accompanying processing resources, to be redirected to the new event. A classic example of this is the well-known Cocktail Party Effect. Not only are we easily able to follow just one conversation when several people are speaking, but the occurrence of a salient phrase in a peripheral conversation stream, such as somebody mentioning our name, causes auditory attention to be redirected. It is also clear that emotions, motivation and physiological state in general, play a key role in such prioritisation, e.g. Oatley and Johnson-Laird [Oatley,87] suggest,
" ... that the function of emotion modes are both to enable one priority to be exchanged for another ... and to maintain this priority until it is satisfied or abandoned."
However, in an agent with multiple goals (such as a human) and which is subject to continual environmental stimulus, a compromise needs to be struck between responding optimally according to the order of priorities and efficiency of processing. In the extreme, a system in which interruption is the norm could fail to ever complete any valuable processing. The heart of the conflict is that between the need to respond in a timely fashion and the need to respond optimally according to the salience level of environmental stimuli. The problem is also complicated by the fact that salience is highly context dependent. Hearing a lion roar may be extremely salient if you are walking down the street, but it would be much less salient if you were walking around a zoo. Human's capacity to correctly attribute salience to stimuli in a context dependent manner and interrupt or adjust ongoing processing accordingly is a major reason for our evolutionary success.
In contrast, artificial systems do less well. This manifests itself in two ways. Firstly, they are often bad at adjusting their processing to salient events, especially when assessing salience is context dependent. Thus, they may fail to respond appropriately to a salient event or, at the other extreme, they may interrupt processing unnecessarily frequently in response to contextually low salience events. Secondly, when interacting with humans, artificial systems fail to fully utilise salience. In pursuit of a particular goal, interactive systems typically unreel sequences of (in effect) ballistic steps, only being receptive at specific breakpoints to a restricted set of anticipated cues. In contrast, a salience sensitive interface would adapt its behaviour according to the cognitive and affective state of the user.
A big hindrance to constructing systems that are sensitive in this respect was that it was not fully understood how humans adapted their behaviour according to salience. It was believed that they did it well, but the mechanisms that achieved that successful outcome were not well understood. However, through the combination of behavioural experimentation and the recent application of brain imaging, modern cognitive neuroscience is starting to clarify the underlying mechanism. In particular, a number of experimental paradigms, which fall broadly within the study of human attention, have started to reveal how real-time constraints and sensitivity to salient events are reconciled in humans. Two such experimental paradigms are the attentional blink and emotional Stroop. Both consider the limits of salience driven control. For example, the attentional blink explores the constraints (in particular, the time frame) under which salient stimuli fail to be processed as a result of attention being directed at preceding stimuli. Work has revealed how context and level of salience, in both a semantic and emotional sense, regulate these effects.
In order to realise the potential of this new understanding it is important that computational models are constructed. These provide concrete realisations of the mechanisms being revealed and also act as a bridge to the construction of artificial systems that are sensitive to salience. This was the aim of the Salience project; see also the successful grant proposal.
The research we performed was centred on two key theoretical topics,
1. Temporal Attention: This topic concerns the capacity of humans to undertake a sequence of attentional episodes. It explores questions such as how long is attention allocated to an initial episode before it is free to be allocated to a second; can a salient item interrupt processing of an earlier item and cause attention to be redirected; and what constitutes salient in this context: relevance to long-term goals, emotional significance, etc? The Attentional Blink (AB) task [Raymond,92] is one of the key experiments that has been used to answer these questions.
2. Emotional Interference: This topic considers how emotionally salient stimuli interfere with ongoing tasks. Particularly pertinent questions are, what affective dimensions (valence, arousal, etc) cause the most interference; and to what extend do such stimuli distract from a central task, such as driving or flying a plane? The Emotional Stroop task [McKenna&Sharma,04] is one of the key experiments that has been used to explore this question.
The research undertaken in this project investigated these two topics through the development of computational models, validation of these models through behavioural and electrophysiological experimentation and exploration of the implications of these models for the development of artificial systems. In order to do this we completed the following tasks, each of which is considered in a separate section.
This research addressed a number of objectives. Firstly, we developed computational models that explain how meaning modulates temporal attention. This was explored within the context of Barnard et al's key-distractor attentional blink task [Barnard et al, 04] and proposes a mechanism by which attention is captured by semantics and affect. Secondly, we explored the applicability of formal methods to the abstract modelling of cognition. Formal methods are mathematical specification and analysis techniques developed in the Computer Science setting. Thirdly, such formal methods enable modelling of an important class of cognitive architectures in which multiple subsystems interact and control is distributed. Our modelling of Barnard’s Interacting Cognitive Subsystems (ICS) architecture illustrates this issue.
The basic theoretical principles that underlie this thread of research are presented in the following two overlapping presentations.
Processing Models of Cognition and Affect Computationally Explicit: Distributed
Executive Control and the Deployment of Attention"
P.J. Barnard and H. Bowman
Cognitive Science Quarterly, publisher: Hermes Science Publications, vol. 3, no. 3, pp 297-328, April 2004.
A major result presented in these articles is a simulation of Barnard et al's key-distractor attentional blink [Barnard et al, 04]. This simulation is formulated in terms of the interaction between ICS' two central subsystems, Implic and Prop, which extract implicational and propositional meaning, respectively.
We extended this research in two respects. Firstly, we explored how semantic salience modulates Barnard et al's key-distractor attentional blink. In this way we clarified the temporal characteristics of attentional capture by semantic salience. This was done in the following articles.
H.Bowman, Su Li, and P.J. Barnard.
Technical Report 9-06, Computing Lab,
Li Su, Howard Bowman, and Philip Barnard.
In Proceedings of the 29th Annual Meeting of the Cognitive Science Society (CogSci 2007), page 6.
We also investigated how emotional salience modulates Barnard et al's key-distractor attentional blink. This enabled us to elucidate the temporal dynamics of attentional capture by emotional stimuli.
Su Li, H. Bowman, and P.J. Barnard.
Technical Report 10-06, Computing Lab,
Some of the broader implications of the cognitive level framework we used have also been explored. For example, the following article considers how specifications at multiple levels of abstraction can be developed simultaneously to give greater confidence in ICS modelling.
Li Su, Howard Bowman, and Philip Barnard.
In Proceedings of the 29th Annual Meeting of the Cognitive Science Society (CogSci 2007), page 6.
The research considered in the previous section (section 7) explored how semantically rich stimuli attract attention in an attentional blink setting. In this section, we consider basic attentional blink tasks, which use semantically more primitive stimuli, such as letters and digits. In this way, the underlying phenomenon is isolated and thus, considered independently of subtle issues of semantic salience.
We developed a model of the basic attentional blink, which is more biologically prescribed than the symbolic model discussed in the last section. The following two articles presented initial neural models.
H. Bowman, B. Wyble, and P.J. Barnard
In H. Bowman and C. Labiouse, editors, Connectionist Models of Cognition and Perception II, volume 15 of Progress in Neural Processing, pages 178-187, April 2004, World Scientific.
H. Bowman and B. Wyble
In A. Cangelosi, G. Bugmann and R. Borisyuk editors, Modeling Language, Cognition and Action, volume 16 of Progress in Neural Processing, pages 227-238, January 2005, World Scientific.
The main output from this thread of research was a broad theoretical account of temporal attention and working memory encoding and maintenance. This theory is encapsulated in the Simultaneous Type, Serial Token (ST2) Model and realised in a neural network, Neural-ST2. The following major journal article elaborates the full details of this theory.
H. Bowman and B. Wyble
Psychological Review, 114(1):38-70, January 2007
Two of the theoretical principles that underlie the ST2 model are the types-tokens principle and transient attentional enhancement. With respect to the former of these, the association of a featural representation of an item (i.e. a type) to an episodic context (i.e. a token) is the process of working memory encoding in ST2. With respect to the latter, ST2 assumes a very rapid attentional enhancement that fires in response to detection of a salient item and generates a brief window (around 150 milleseconds) of generalised amplification.
The ST2 model also makes an explicit proposal for the structure of working memory. The following abstract and presentation highlights our initial ideas in this direction.
B. Wyble and H. Bowman
Journal of Vision, 6(6):33a-33a, June 2006.
We undertook a series of behavioural and electrophysiological experiments to verify key predictions arising from the ST2 model. We consider these two classes of experimental investigation in turn.
The previously mentioned Psychological Review article [Bowman & Wyble, 07] verifies key behavioural predictions arising from ST2. Further experimental verifications are reported in the following articles.
B. Wyble and H. Bowman
In B.G. Bara, L.W. Barsalou and M.
Bucciarelli, editors, CogSci 2005, XXVII Annual Conference of the Cognitive
Science Society, pages 2371-2376. Cognitive Science Society through
B Wyble and H Bowman
In C. Schunn, editor, ICCM'04, Integrating Models, Proceedings of the International Conference on Cognitive Modelling, October 2004.
B. Wyble and H. Bowman.
Journal of Vision, 5(8):116a-116a, September 2005.
The following poster focuses particularly on the transient attentional enhancement aspects of the ST2 model.
"In Search of Sparing: Spatial Cueing can be Triggered Contingently by Visual Form"
B. Wyble, M. Potter, P. Craston and H. Bowman
CSAIL'06, Eleventh Annual Meeting of the Cognitive Science Association for Interdisciplinary Learning, August, 2006, Hood River, Oregon
The following two journal papers present our behavioural experimental research in an archival format.
B. Wyble, H. Bowman, and M. Potter.
Journal of Experimental Psychology: Human Perception and Performance, 35(2):324-337, 2009.
B. Wyble, H. Bowman, and M. Nieuwenstein.
Journal of Experimental Psychology: Human Perception and Performance, 35(3):787-807, April 2009.
We also investigated neurophysiological correlates of the ST2 theory. This was undertaken using electrophysiological (EEG) recording. EEG measures the electrical activity generated in the brain by synaptic firing. Importantly, EEG recording offers an excellent temporal resolution, which makes it particularly well suited to the study of temporal attention. The following abstracts, associated posters and journal publications highlight our work in this direction.
P. Craston, B. Wyble, and H. Bowman
Journal of Vision, 6(6):1016-1016, June 2006.
An EEG study of masking effects in RSVP
P. Craston, B. Wyble, and H. Bowman
Poster presented at EPS'06, Experimental
Psychology Society' 06,
P. Craston, B. Wyble, S. Chennu, and H. Bowman.
Journal of Cognitive Neuroscience, 21(3):550-566, March 2009.
S. Chennu, P. Craston, B. Wyble, and H. Bowman.
PLoS Computational Biology, 5(11):e1000576, November 2009.
We also developed a neural-level model of how emotional stimuli can interfere with ongoing processing. This research was performed within the framework of the Emotional Stroop task [McKenna & Sharma, 04]. The following book chapter presents the resulting neural model. Then the journal article explores the full theoretical implications of the model.
B. Wyble, D. Sharma and H. Bowman
In A. Cangelosi, G. Bugmann and R. Borisyuk, editors, Modeling Language, Cognition and Action, volume 16 of Progress in Neural Processing, pages 227-238, January 2005, World Scientific.
B. Wyble, D. Sharma, and H. Bowman.
Cognition & Emotion, 22(6), 2008.
An important question that our research raises is how to relate different levels of abstraction. We developed models of temporal attention at both a cognitive and a neural-level. How though can these levels of explanation be related? Such cross level relationships are, though, notoriously difficult to define. For example, to fully tackle this problem would require a solution to the symbolic-subsymbolic problem [Fodor&Pylyshyn,88; Hinton,90], which investigates how to relate symbolic descriptions and neural networks.
The particular form of this problem that is relevant to this project is how to relate the (symbolic) formal methods models discussed in section 7 to the (subsymbolic) neural networks discussed in section 8. As a contribution to this difficult problem we explored the junction between communicating automata (a class of formal method) and neural networks. This research is described in shortened format in the following conference paper and then in comprehensive format in a technical report.
Li Su, H. Bowman, and B. Wyble
In NeSy’05, Proceedings of the IJCAI-05 Workshop on Neural-Symbolic Learning and Reasoning, editors, A. d'Avila Garcez, J. Elman and P. Hitzler, 4 pages, 19th Joint Conference on AI, Edinburgh, UK, August 2005.
Li Su, H. Bowman and B. Wyble
Technical Report 10-03, Computing
(A journalisation of this article is planned)
This research is informed by some of the theories and analysis techniques presented in the following book on formal methods, which was written during the Salience project.
H. Bowman and R.S. Gomez
Springer, 450 pages, January 2006.
The research methodology employed in this project was to first elucidate the characteristics of temporal attention in humans and then to use this understanding to inform the construction of artificial systems. This second objective is considered in this section.
Our strategy was to investigate the implications of our research across a broad spectrum of artificial systems applications. This enabled us to explore the feasibility of applying our findings in a number of directions. We undertook the following four feasibility studies, which we consider in turn.
1) Attentional Capture in HCI
2) EEG and Adaptive Interfaces
3) Formal Methods Verification of Adaptive Computer Interfaces
4) Verifying Adaptive Controllers
Our theoretical findings are relevant to a number of different application areas, e.g. robotics and HCI. However, we particularly focused on a specific class of human computer interfaces, which we call Stimulus Rich Reactive Interfaces. This class of system has the following characteristics. 1) Stimuli arrive rapidly; 2) there is typically a central task (e.g. driving or flying), from which the rapidly arriving peripheral stimuli can capture attention (in different circumstances, this capture being either desirable or undesirable, i.e, distracting); 3) safety is critical, e.g. a high degree of certainty is required that the user / operator perceives certain stimuli; and 4) using physiological feedback of the cognitive state of user, the system adapts its behaviour in order to optimise operator performance. Concrete examples of Stimulis Rich Reactive Interfaces (SRRI) include, flying a plane, driving a car, monitoring a patient, even viewing webpages. For example, with respect to the first of these, flying, or particularly landing, a plane would be the central task; display of potential obstacles (e.g. turbulance, other planes, etc) would yield streams of rapidly arriving peripheral stimuli; safety is clearly critical; and a spectrum of physiological feedback, e.g. eyetrackers, EEG electrodes in helmets, heart and skin conductance monitors could be built into the cockpit.
Attentional Capture in HCI. Our theoretical work identified a set of attentional mechanisms [Bowman&Wyble,07]. We also explored the practical implications of these mechanisms. Two findings that particularly inspired our practical explorations are, 1) the existence of a very rapid (first phase) of attention, called Transient Attentional Enhancement (TAE), which acts within 150ms of stimulus presentation; and 2) that even such rapid attentional deployment is modulated by task set, e.g. it could be initiated by detection of an item in a target category [Bowman&Wyble,07]. Such mechanisms have great relevance for the development of stimulus rich human computer interfaces. In particular, in interfaces with rapidly arriving streams of information, it is important to understand how stimuli capture attention, both in order to prevent distraction from a central task and to ensure critical stimuli are not missed.
To explore this issue, we developed a prototype test interface which contains a central task involving driving through a virtual maze and the presentation of an intermitent stream of competing stimuli of varying levels of salience. Centrally presented arrows are followed in the driving task and, as a reflection of the presentation methods typically used in this setting, the stream of competing stimuli is presented via a head mounted display. The colour relationship between the central arrows and stimuli in the competing stream is varied. How this "task prescribed" colour relationship impinges upon attentional capture by stimuli in the competing stream is investigated. Results of this work are reported in the following technical report.
B. Wyble, H. Bowman, and P. Craston
Technical Report 7-06, Computing Lab,
EEG and Reactive Interfaces. There is considerable interest in developing reactive / adaptive computer interfaces that adjust their behaviour according to the cognitive state of the user. We explored the feasibility of using EEG in this context as a source of feedback on the cognitive state of the user. We ran experiments to evaluate the utility of two potential EEG measures. 1) We investigated whether modulations in EEG power in the alpha band (around 10 hz) at posterior areas (particularly, occipital cortex) can be used as a measure of attentional readiness in the visual modality. 2) We considered whether a positive deflection in the P3 region (around 350 ms post stimulus presentation) could be used as a measure of whether a stimulus was perceived.
Both these measures are of potential value, but they are somewhat different in their character and utility. 1) is proactive, in the sense that it predicts whether the subject will perceive a later stimulus. In contrast, 2) is reactive, in the sense that it predicts whether a stimulus has been perceived. Thus, 1) opens up the possibility of withholding presentation of a critical stimulus until the user is ready, while 2) would enable re-presentation of a critical stimulus that has been missed. The second of these would have particular value if it were combined with eye-tracking to determine which stimuli are being fixated when a perceptual event is detected.
In the context of adaptive interfaces, the key question to answer is whether these measures can be reliably extracted online, i.e. in real-time. Thus, we investigated the extent to which online extraction of these measures predicts target report. Our results are reviewed in the following technical report.
B. Wyble, P. Craston, and H. Bowman
Technical Report 8-06, Computing Lab,
Formal Methods Verification of Adaptive Computer Interfaces. We applied our simulations of the human salience detection system to evaluating the feasibility of a variety of SRRIs. We developed a formal methods-based (cognitive-level) model of the ICS central engine, with which we simulated attentional capture in the context of Barnard's key-distractor AB task. The same core system would be at work when human operators interact with SRRIs. Thus, we used this model to evaluate the performance trade-offs that would arise from varying key parameters in such systems. A strength of formal methods is that they are abstract and thus, the resulting specifications of the operator are general purpose, ensuring that our findings are broadly applicable.
Examples of the types of questions we investigated include the following. How effective does prediction of the operator's attentional and perceptual state have to be, for performance to benefit from the use of an SRRI? How these performance benefits are affected by the temporal profile of stimulus arrival, e.g. whether it is fast or slow, regular or bursty? These investigations are presented in the following technical report and published papers.
H. Bowman, Su Li, and Brad Wyble
Technical Report 6-06, Computing Lab,
L. Su, H. Bowman, P.J. Barnard, and B. Wyble.
Formal Aspects of Computing, 21:513-539, 2009.
H. Bowman L. Su B. Wyble P.J. Barnard.
In Claudia Roda, editor, Human Attention in
Performance of reactive interfaces in stimulus rich environments, applying formal methods and cognitive frameworks. Li Su, Howard Bowman, and Philip Barnard.
In The 2nd International Workshop on Formal Methods for Interactive Systems (FMIS 2007). Workshop held in conjunction with HCI 2007, page 17.
Verifying Adaptive Controllers. There is considerable interest in applying machine learning techniques in the context of adaptive controllers used, for example, in airplane control systems. Such techniques enable controllers to be reconfigured as a result of changing parameters. For example, this might arise if there is damage to the systems being controlled. We applied our formal methods encoding of neural networks to verify learning algorithms in the context of adaptive controllers. This research is reported in our IJCAI-05 and technical report 10-03 articles, which were highlighted in section 9.
The following book was edited during the lifetime of the Salience project and contains a number of articles relevant to the project.
H. Bowman and C. Labiouse, editors, Connectionist Models of Cognition and Perception II, volume 15 of Progress in Neural Processing, Singapore, April 2004, World Scientific.
[Barnard et al, 04] Barnard, P. J., Scott, S., Taylor, J., May, J., & Knightley, W. (2004). Paying attention to meaning. Psychol Sci, 15(3), 179-186.
[Bowman&Wyble,07] Bowman, H. and Wyble, B. The Simultaneous Type, Serial Token Model of Temporal Attention and Working Memory. Psychological Review, 40 pages, (to appear January 2007)
[Fodor&Pylyshyn,88] Fodor, J. A., & Pylyshyn, Z. W. Connectionism and Cognitive Architecture: A Critical Analysis. Cognition, 28, 3-71.
[Hinton,90] Hinton, G. E. (1990). Special Issue of Journal Artificial Intelligence on Connectionist Symbol Processing (edited by Hinton, G.E.). Artificial Intelligence, 46(1-4).
[McKenna&Sharma,04] McKenna, F. P., & Sharma, D. (2004). Reversing the emotional Stroop effect reveals that it is not what it seems: the role of fast and slow components. J Exp Psychol Learn Mem Cogn, 30(2), 382-392.
[Oatley&Johnson-Laird,87] Oatley, K. & P. Johnson-Laird, Towards a Cognitive Theory of Emotions. Cognition & Emotion, 1987. 1(1): p. 29-50.
[Raymond et al,92] Raymond, J. E., Shapiro, K. L., & Arnell, K. M. Temporary suppression of visual processing in an RSVP task: an attentional blink? J Exp Psychol Hum Percept Perform, 18(3), 849-860.
Picard, R.W. Affective Computing. 1998,
Back to Howard Bowman's home page.
Last modified April 2012.