Evaluating embodied conversational agents in multimodal interfaces

Communication theories

One example of research on multimodal human-computer interaction (HCI) is the identification of systematic variation in users’ interaction style between systems providing only spoken interaction and those providing also the visual channel (Oviatt 2008). For example, if users are aware of the capability to use nonverbal information, spoken content is likely to exhibit linguistic underspecification, which is solved with the support of nonverbal information and world or context knowledge.

This multidimensionality of information conveyed in face-to-face conversation is not covered by information theory (Shannon and Weaver 1949). One of the problems in applying this theory to face-to-face conversation is the assumption of a common code, which works for technical transmission of information, for which this theory was developed, especially for reduction of redundant information, but not in the case of HCI, as users are usually not aware of all capabilities (i.e. code) of that particular system. Also, the assumption of a purely symbolic and intentional communication neglects the continuous negotiation going on between interlocutors, also called grounding (Clark 1996), as well as unintentionally produced signals, which are useful to recognize the meaning and intention of a speaker and are therefore typically considered by the conversational partner.

Other communication theories do take into account multiple aspects of a message: According to Schulz and Thun (1981) a multimodal message comprises information about the sender (e.g. mood, age), the relationship (e.g. attitude, status), and the demand on something (perlocutionary act) apart from the factual information. Of course, the question arises, whether such aspects of a message are necessary for the interaction with a system, and whether users are motivated by an ECA to expect proper processing of them.

According to Krauss and Fussell (1996), communication theories relevant for social psychology can be separated into four classes: Encoder/decoder models like information theory (Shannon and Weaver 1949), Intentionalist models focusing on the task of understanding the speaker’s intention instead of just the utterance meaning (e.g. speech act theory by Searle (1969)), models centred on the process of Perspective-Taking as an important aspect of conversation (e.g., models of feedback or the client-centred counselling strategy by Rogers (1951)). Last, Dialogic models treat conversation as socially dependent co-joint work of the interlocutors.

According to dialogic models, the main goal of an interaction is to accomplish a shared world view. This emphasizes topics of interaction control, meta-communication and discourse to prevent and solve communication problems and to establish grounding. For the application of user–ECA interaction, any usage beyond mere controlling services or devices will exhibit aspects of such social and co-joint activity. In fact, there is not much motivation to build ECAs for mere controlling tasks. However, the degree of human-likeness or naturalness a user assumes the ECA can provide represents a problematic issue. Users are heterogeneous in this respect and not only interact differently, dependent on expert knowledge and personality, but also evaluate the conversational quality differently due to diverging internal references.

Arguing on the basis of a dialogic approach (Clark 1996), the grounding problem is defined for ECAs in Brennan (1998): In addition to obtaining a shared world view in human conversation, one major goal of collaboratively working with an ECA is the negotiation about its verbal and nonverbal capabilities. This grounding process is currently not properly supported in interfaces based on natural spoken language, especially concerning signals of positive evidence and coordination of acceptance of system messages. Therefore, the interaction with spoken dialogue systems, embodied or not, is still inferior compared to approaches of direct manipulation, which intrinsically provide grounding mechanisms (e.g., visualization of capabilities due to labelled buttons in Graphical User Interfaces, GUIs), despite the theoretical advantages of ECAs and spoken dialogue systems providing “natural interaction” (cf. Section “Computers as social actors”).

One central aspect of the dialogic approach is the inherent social nature of conversation. As relevant implication, evaluation based only on usability approaches is limited and likely to fail, as social processes have to be taken into account. In the following, the effects of persona, social facilitation, and visual/auditory attributions are presented to raise awareness and provide explanations for potential issues in human-ECA interaction.

Computers as social actors

According to Duffy (2003) robots need to have anthropomorphic, i.e. human-like qualities, in order to be capable of meaningful social interactions. If there is a human-computer interface exhibiting speech or also visual human features, human social behaviour, especially nonverbal signals, seem to be often triggered automatically (Sproull et al. 1996; Vogeley and Bente 2010). This might also be true for non-anthropomorphic interfaces to some extent (Reeves and Nass 1996), but a rich amount of empirical results has confirmed this effect for anthropomorphic and speech interfaces.

Persona effect: Concerning usability, ECAs are assumed to possess an inherent benefit over other interaction paradigms in HCI, as they provide “natural” and thus “intuitive” interaction as learned by humans throughout their whole lifetime, or to phrase it differently: ECAs could relieve users of the need to learn technical details of new technological services and interfaces (Takeuchi and Naito 1995).

This benefit of establishing a “natural”, i.e. social situation, could be empirically observed, as the existence of anthropomorphic human interfaces, even those just accompanying traditional interfaces like web-sites, can result in a higher quality (more positive subjective evaluation) of the service or interface and/or in higher performance (efficiency and effectiveness measured as scores and time-to-complete). This persona effect, however, is highly debated, as it seems to be dependent on task, system, and situation (Dehn and van Mulken 2000; Foster 2007; Yee et al. 2007). Also, there is the question, whether the persona effect resembles an effect of mere presence like the social facilitation effect (see below) or if it is a result of a more natural and intuitive interaction.

At least, compared to pure spoken dialogue systems, ECAs can facilitate interaction (Dohen 2009), as human processing benefits from multimodal signals in decreased load and higher neural activity (Stein et al. 2009). So, if not engaged in multiple tasks, an ECA might be less demanding to interact with, if the nonverbal signals are produced and recognized well.

Despite this assumed and also observed persona effect, the design decisions and evaluation concepts concerning ECAs are not to be taken easily: As mentioned already, natural interaction can only have its positive effect, if it is supported well by the ECA, and visually human-like ECAs may fuel too high expectations for the dialogic and nonverbal capabilities. Although a natural interaction is aimed at and users do show signs of social phenomena, some users may like to avoid such social interaction for certain tasks in favour of a paradigm of direct manipulation; just like some customers prefer anonymous online shopping or self-service to personal service.

In line with the persona effect, the presence of an ECA can, for example, result in increased entertainment in a game application (Koda and Maes 1996) and more nonverbal signals are sent in interaction compared to other interfaces (e.g., more eye contact interpreted as increased attention, cf. Takeuchi and Naito (1995)). Also, observations of social effects like politeness, impression management and social facilitation indicate that ECAs can indeed establish a social situation resulting in the automatic appearance of typical phenomena of social psychology (Sproull et al. 1996).

Social facilitation: With social facilitation, performance of a person is enhanced in the presence of other people for tasks of low complexity, while performance decreases for difficult, complex tasks. The latter is known as social inhibition. This has also been observed for interaction with ECAs (Sproull et al. 1996) and robots (Riether et al. 2012; Wechsung et al. 2012a). One explanation of this effect is an increase in attention and arousal due to the social situation (Guerin 1993). Social facilitation might be more likely for ECAs with gender different to the one of the user (Karacora et al. 2012).

Uncanny valley: One particular risk when aiming at human-like interfaces is the uncanny valley effect: This effect describes an increase of the perceived familiarity with increasing resemblance to human appearance only until a certain level of human-likeness is reached. However, small divergence from human-likeness results to an uneasy feeling. At that level, familiarity drops and increases again if the human-likeness is perfect (Mori 1970). The uncanny valley effect can even be observed in neural activity (Saygin et al. 2012). The authors conclude that the uncanny valley effect may be based on perceptual mismatch: For very human-like robots equally human-like movements are expected. However, if those expectations are not met, the uncanny valley effect may occur. This dependency of the effect on the expectation of (and reference as) a real human is the reason to present it along with attributions in this subsection.

Whereas the evaluation topics 1–4 are related to special aspects of an ECA or robot, topics 5–7 are about using an ECA or robot at all. The subsequent difference lies in the possible evaluation methods, as answering questions within topics 5–7 requires a comparison with a non-embodied condition. In the following section, we briefly describe evaluation methods with and without real users and present typical assessment methods, before describing specific instruments to assess concepts of interest in Section “Aspects of evaluation”.

Cognitive Walkthrough

The Cognitive Walkthrough is a task-based, expert-centred, analytical method (Holzinger 2005) based on exploratory learning and problem solving theory (Wharton et al. 1994). It takes into account that users often learn the interface by exploring it, instead of reading the manual.

The experts analyse the interface based on the following four questions: (1) Will the user try to achieve the right effect? (2) Will the user notice that the correct action is available? (3) Will the user associate the correct action with the effect to be achieved? (4) If the correct action is performed, will the user see that progress is being made toward solution of the task?

In the context of the evaluation of ECAs the procedure of the Cognitive Walkthrough, which was initially developed for graphical user interfaces, may be used unchanged. However, regarding the four central evaluation questions Ruttkay and Op den Akker (2004) suggest adapting and expanding the question (2) and question (4) as follows: (2) Will the user be aware of what they can do to achieve a certain task: to talk to several (all) ECAS on the screen, to use gestures, gaze and head movement as those are perceived by the ECAS? Will users notice (when) they need to give an answer? Will they notice whom they may address (who is listening)? (4) Will there be a feedback to acknowledge the (natural, may be multi-modal) answer given by the user? Does the feedback indicate for the user if his/her action was correct? (In case of natural communication, this distinction means if something ‘syntactically correct’ was said and thus properly parsed, or something was said which is (one of the) expected answers in such a situation.)

As for almost all formative-analytical methods, the biggest advantage of the Cognitive Walkthrough is that end users as well as an implemented system are not necessary. Disadvantages are the quite low level of information, as only the ease of learning is investigated (Wharton et al. 1994). Moreover, a Cognitive Walkthrough might be very time consuming for complex systems. Note that, the Cognitive Walkthrough is strictly task-based and will only be able to evaluate the ease-of-use of an interface rather than its joy-of-use.

Heuristic evaluation

Heuristic Evaluation is a method of the so-called Discount Usability Engineering, a resource conserving, pragmatic approach (Nielsen and Molich 1990), aiming to overcome the argument that usability evaluation is too expensive, too difficult and too time consuming. In a Heuristic Evaluation, several experts check whether the interface complies with certain usability principles (heuristics). To ensure an independent, unbiased judgement of every evaluator, they do not communicate to find an aggregated judgement until each of them investigated the interface on his/her own (Nielsen 1994). The result of a Heuristic Evaluation is a list of usability problems and the respective explanations. Additionally, problems might be judged according to their frequency and pertinence.

According to Nielsen, three to five experts will find 60–70 % of the problems, with no improvements for more than ten evaluators (Nielsen 1994). However, this statement has repeatedly caused disputes; research provides support (Virzi 1992) as well as contrary findings (Spool and Schroeder 2001; Woolrych and Cockton 2001). In a series of studies, Molich and colleagues come to the conclusion that 4–6 experts are sufficient, but only for one iteration cycle and only with real experts (Molich and Dumas 2008).

The Heuristic Evaluation is a cheap method and quick to apply, and it can be conducted throughout the whole development cycle (Holzinger 2005). The probably most widespread heuristics are the ten guidelines proposed by Nielsen (1994). Moraes and Silveira (2006) updated and adapted this set first to the evaluation of animated agents in general and later to the evaluation of pedagogical agents in particular (Moraes and Silveira 2009).

Model-based evaluation

For Model-Based Evaluation on a very general level, two different approaches can be distinguished. The first approach has its origin in cognitive psychology and focuses on the cognitive process while interacting with an interface. The other approach is rooted in the engineering domains and is focusing on the prediction of user behaviour patterns. Within both approaches, user models are employed for the predictions.

Methods of the first approach are usually addressing low-level parameters like task execution time, memory processes or cognitive load and are largely bottom-up oriented. Starting point to define user models are theories and findings from cognitive psychology. Examples are the methods GOMS (Goals, Operator, Methods, Selection rules) (Card et al. 1983), the Cognitive Complexity Theory (CCT) (Kieras and Polson 1985), or ACT-R (Adaptive Control of Thought–Rational) by Anderson and his group (e.g. Anderson and Lebiere (1998)).

As all these cognitive models are well-grounded in theory, they provide useful insights in user behaviour. Although cognitive modelling is an active research field, so far it has not been received particularly well by usability practitioners and only rarely finds its way into non-academic evaluations (Engelbrecht et al. 2009; Kieras 2003). Reasons are their often high complexity (Kieras 2003) and possibly the aforementioned low level of the information possible to gain with cognitive modelling.

An engineering-based, statistically-driven approach attempts to provide more high level information, e.g. if the user is “satisfied” with the system, and therefore rather utilizes top-down strategies. Here, user models are usually defined based on real user data and are not necessarily linked to cognitive theories (Engelbrecht et al. 2009). Most of these methods and algorithms were developed for spoken dialogue systems, with PARADISE (Paradigm for Dialogue System Evaluation) (Walker et al. 1997) likely being the most widespread one. PARADISE uses linear regression to predict user satisfaction based on interaction parameters such as task success or task duration.

A model-based approach explicitly tailored to multimodal system is PROMISE (Beringer et al. 1997), an extension of Walker’s PARADISE. However, studies applying PROMISE are scarce, possibly because some of the parameters are relatively ill defined (e.g. the way of interaction), and it is not specified how they should be assessed. Just recently a well-defined set of interaction parameters for multimodal interaction was proposed (Kühnel 2012), yielding reasonable prediction performance (> 50 % accuracy) for user judgements in general (see Section “Interaction parameters”). However, most data analysed with respect to interaction parameters was not collected with experiments including an ECA. For those experiments with an ECA, significant correlations were found for the concepts of helpfulness and cognitive demand (cf. Section “Aspects of evaluation”). Using this approach, Pardo et al. (2010) compared voice-only interaction to an ECA, and observed significant differences suggesting a “rougher” interaction for the voice-only version.

Experiments and field tests

Experimental evaluation investigates specific aspects of the interaction behaviour under controlled conditions (Sturm 2005). In the simplest experimental design, one hypothesis is formulated and two experimental conditions, differing only regarding the manipulated factor to be investigated (independent variable), are set-up (Dix et al. 2003). All differences occurring in the measured variables (dependent variable) are attributed to the manipulations of the independent variable (Dix et al. 2003). Experiments allow for collecting high quality data as interfering variables are controlled and/or eliminated. Experiments provide, if carried out carefully, causal inference. Thus, experiments are essential to establish and verify theories (Gerrig and Zimbardo 2007); accordingly, experiments are a useful method for evaluation of ECAs. However, with experiments being strongly controlled, user behaviour might be rather unnatural (Sturm 2005). In the worst case, results can be an experimental artefact. Another drawback is the high amount of resources required to set up and conduct a proper experiment.

When evaluating HCI, field tests aim at ecologically valid conditions by embedding the actual usage in an authentic daily situation. This affects not only the environment, but all context factors like user motivation and references. Although field tests can thus prevent the issue of (laboratory) experiment being unnatural, field tests are usually much more costly in terms of effort and time. In contrast to experiments, which reduce or control potential influence factors, results from field tests are typically much harder to interpret because of unknown or uncontrolled variables, and do not allow for causal inferences (Bernsen and Dybkjær 2009; Dix et al. 2003).

In HCI, both methods are sometimes difficult to distinguish, as, e.g., field tests can be carried out in different conditions like experiments. The actual choice will depend on the evaluation/research question (e.g. with topics from cognitive psychology most likely studied with experiments, and topics about acceptability most likely studied with field tests) and on the domain: Although in general, all topics in Section “Theoretical foundations of evaluation” can be studied with experiments, topics 5 and 6 concerning the effect of an ECA on the system’s quality and on users’ internal states and behaviour may be more difficult to validly study for an ECA companion than a learning tutor, as the companion is expected to be embedded in daily life routine.

In the following paragraphs, we will go into detail for two major categories of dependent variables, questionnaires and indirect measures (eye-tracking data and interaction parameters).

Interviews and questionnaires

Questionnaires and interviews are indispensable to measure the users’ judgements of the system (Holzinger 2005), as interaction data will not necessarily reflect the users’ perceptions (Naumann and Wechsung 2008). Interviews and questionnaires are often used to assess user satisfaction, emotions or attitudes towards the system. If reliable questionnaires or interviews are available, they are relatively cheap and easy to employ. Whereas interviews typically aim at assessing qualitative data, questionnaires allow for collecting quantitative data, and for statistical analysis.

Standardized, well-validated questionnaires tailored to ECAs are rather rare; consequently, self-made questionnaires or questionnaires developed for unimodal systems are often employed. Both approaches are problematic: Self-constructed questionnaires are usually not properly validated (Larsen 2003) and questionnaires developed for unimodal GUI-based systems may not provide valid and reliable results for ECAs. Consequently, the constructs measured are quite diverse and the results are hardly comparable. Moreover, without a proper validation, it is uncertain if a questionnaire actually measures the constructs which it was intended to measure (Larsen 2003).

Notable exceptions are the Agent Persona Instrument (API) (Baylor and Ryu 2003) and the Attitude Towards Tutoring Agent Scale (ATTAS) (Adcock and Eck 2005). Both are relatively well-known and psychometrically tested questionnaires. However, both are limited to the evaluation of pedagogical agents. A questionnaire intended to be applicable to a wide range of ECAs has just recently been proposed with the Conversational Agents Scale (CAS) (Wechsung et al. 2013). The CAS is based on the dimensions of the theoretical framework by Ruttkay and colleagues (Ruttkay et al. 2004). The original dimensions are: satisfaction, engagement, helpfulness, naturalness and believability, trust, perceived task difficulty, likeability and entertainment. According to (Ruttkay et al. 2004) satisfaction and engagement are high-level constructs, which comprise several of the other dimensions. For example, engagement may include likeability and trust; hence, these meta-aspects can be said to derive from the sub-aspects. Consequently, the questionnaire comprises the sub-aspects only, which are helpfulness, naturalness and believability, trust, perceived task difficulty, likeability and entertainment. Note that, to date only the German version of the CAS is validated. A preliminary validation of an English translation is presented in Section “An experiment testing two different talking heads”. In Section “Aspects of evaluation” additional questionnaires are presented to assess specific quality aspects.

Although interaction data does not necessarily reflect user perception, such data can be of interest for evaluation. For HCI, instrumental and manually annotated data is usually used to describe the course of interaction, either because it represents a genuine evaluation question (e.g. the relationship between measured and user perceived interaction duration). In this case, such data can complement questionnaire data, e.g. for topics concerning smoothness or naturalness of interaction (topics 1 and 2); nonverbal and verbal functions (topics 3 and 4) or user’s internal states (topic 6 from Section “Theoretical foundations of evaluation”).

Eye-tracking

One popular physiological method to assess data instrumentally is eye-tracking. Recorded data can be used to monitor and record user behaviour. The exact purpose of using eye-tracking should be noted: As a method of usability testing, information can be obtained about which items of a GUI are salient or not. However, often used “heat-maps” can be very misleading due to the aggregation of data.

In research, eye-tracking is for example used to study visual search strategies, to measure aspects of affect, like interest, fatigue, or to assess arousal using fixation times, blinking rate or pupil dilation as measures. The book by Holmqvist et al. (2011) is advised as a good start for the interested reader. Still, properly assessing physiological measures requires a lot of experience, e.g. to deal with practical problems with lighting, reflections etc., especially in mobile settings and under realistic usage circumstances.

Interaction parameters

The parametrisation of individual interactions on the basis of data extracted from manually annotated or automatically logged (test) user interactions commonly accompanies experimental evaluation. So-called interaction parameters quantify the flow of the interaction, the behaviour of the user and the system, and the performance of the devices involved in the interaction. These parameters provide useful information for system development, optimization and maintenance. The parametrisation of interaction is thus complementary to quality judgements assessed via questionnaires – by addressing the system’s performance from a system developer’s point-of-view.

For more than two decades of experience with spoken dialogue systems, researchers and developers have defined, used, and evaluated interaction parameters for the named purposes, summarized for example in Möller (2005). With the emergence of multimodal systems, this approach has been stipulated for this new domain as well (Dybkjær et al. 2004). Several annotation schemes for multimodal interaction have been published (Gibbon et al. 2000; López-Cózar et al. 2005), but researchers build “their own corpora, codification and annotation schemes” mostly “ad hoc” (López-Cózar et al. 2005, p. 121). Recently, a dialog acts annotation scheme was standardized for human face-to-face conversation. These dialog acts are multimodal per definition, as they are considered the conceptual or perceptual origin of human (humanlike) multimodal signals (ISO 24617-2 2012). This scheme could be directly used for ECAs as well, as long as the multimodal system does not offer additional interaction capabilities except for the ECA (e.g. a touch-screen).

The benefit of a nonverbal communication strategy with an ECA-based interface have been analysed using interaction parameters and questionnaires together (Pardo et al. 2010). While the results assessed via interaction parameters and questionnaires pointed in the same direction only the interaction parameters yielded significant differences. Later, similar results were found comparing a user-adaptive to a non-adaptive ECA (Weiss et al. 2013), suggesting to not only rely on questionnaires.

In Kühnel (2012) a first set of interaction parameters for multimodal interaction has been proposed, focusing mainly on spoken, gestural and GUI-based input as well as spoken and graphical output. For the evaluation of the interaction with ECAs or ECA-based system no such set of parameters exists. In the following the adaptation of multimodal interaction parameters to ECAs will be discussed and some exemplary parameters will be proposed.

One basic concept common to all interaction parameters is that they can only be measured based on an interaction between a system and at least one user – for example during a laboratory or field test. The interaction is thus influenced by system and user characteristics and behaviour. As these aspects are interrelated they can usually not be separated. Consequently, interaction parameters will reflect not only the interaction but also the characteristics of the interlocutor. The requirements for recording interaction parameters are therefore similar to the ones applicable for experimental evaluation (cf. Section “Best practice and example evaluation” on best practice).

If the system is available as a glass-box, some system-side parameters can be extracted from log-data. The same is true for selected user-side parameters, possibly with an amount of uncertainty. These parameters can be used for adaptive systems and to monitor the interaction online. But for many parameters a manual transcription and annotation of recorded video and audio files is indispensable, thus making a laboratory test setting necessary.

Most interaction parameters which have been proposed for spoken dialogue systems (Möller 2005) can be directly transferred to the context of interaction with ECAs – at least for those systems where speech input and output plays a major role. For other parameters, the definition has to be adapted. Some parameters, such as speech input-related metrics, have to be mirrored for every input modality. Depending on the accompanying modalities, other parameters might have to be added – for example parameters known from graphical user interfaces. And there are new parameters inherent to the interaction with ECAs which should be considered (see below).

Most parameters are annotated on a turn-by-turn basis and later summed up or averaged over the number of turns for the whole dialogue or sub-dialogues. It is thus necessary to precisely define the beginning and end of a turn in a multimodal interaction. For an interaction at least one pair of user and system turns is necessary; this has been named “exchange” in Fraser (1997). In Fig. 1 one complete exchange is depicted, including associated time-related parameters.

Most of the concepts related to the overall dialogue and the communication (e.g. dialogue duration, system and user turn duration (Gibbon et al. 2000), system response delay (Price et al. 1992), can be transferred to multimodal interaction unmodified – system response delay, for example, has been used before as a parameter for multimodal systems to measure dialogue efficiency (Foster et al. 2009). Their measurement is based on the definition of user and system turns as described above.

Compared to an interaction with a spoken dialogue system, the interaction with an ECA, and thus every single turn, is potentially far more complex. This is mainly due to two aspects: ECAs mirroring human-human conversation should include and understand feedback mechanisms. And even if this is not the case, input and output via multiple modalities adds complexity.

On the output side, that is the behaviour and output of the ECA, interaction parameters should be logged automatically. But for a useful evaluation the user’s behaviour and input has to be taken into consideration. If the ECA applies feedback or turn taking mechanisms, for example, does the user perceive those and react accordingly? A possible measure is the delay between a turn taking action from the ECA and the taking over by the user.

Whenever the user’s reaction to visual features is analysed it has to be ensured that the user was actually looking at the ECA in the crucial moment. This can be achieved automatically with an eye tracking system or by annotation of video data.

A human-like ECA might induce higher expectations in the user. Confronted with a talking head or even a fully bodied ECA, which uses gestures and mimics to convey meaning, the user might expect the system to understand the same modalities. Thus it makes sense to annotate how often the user tried to interact via modalities not offered by the system, such as pointing to an object while only spoken input is possible.

Measures such as task success are only meaningful for task-oriented interactions. To compute metrics for task success the goal of the interaction has to be known. In a laboratory setting this can be achieved by defining explicit tasks to be fulfilled by the participants of the study. But even when defining explicit tasks it is possible that the participant does not understand the task correctly or accidentally skips parts of the task, resulting in deviations from the original interaction path.

Performance measures should be taken into account as they are good indicators for system developers and might be useful for the interpretation of some dialogue issues. A low performance of one or more recognizers might explain an above-average dialogue duration.

For ECAs the synchrony of speech and lip movement and also the correct alignment of gestures, mimicry and speech are of high importance. This can be measured by the lag of time (LT) between corresponding modalities or by the overall number of times corresponding output modalities have been asynchronous (calculated based on a pre-defined threshold). For spoken output, different methods to assess TTS quality have been proposed but they have not been applied widely (Möller 2005).

Further evaluation methods

In the above sections established evaluation methods are presented; note that this overview is not exhaustive as primarily methods were considered, which have been modified for and applied to the evaluation of ECAs. Additional evaluation methods are for example Review-Based Evaluation and the Think Aloud approach. For the Review-Based Evaluation, existing experimental findings and principles are employed to provide a judgement. Relevant literature has to be analysed in order to approve or disapprove the design of the interface (Dix et al. 1993). Review-Based Evaluation is faster and more economical than conducting an experiment oneself. But wrong conclusions might be drawn if the selection of the considered studies is not done with the required prudence. Studies in which the Review-Based Evaluation is applied in the context of the evaluation of ECAs are rather rare.

Another prominent usability evaluation method is Think Aloud. Participants are encouraged to ‘think aloud’, i.e. to verbalize, and externalize their thoughts (Holzinger 2005). This might be done during the interaction or after the interaction as retrospective Think Aloud. For the latter, the user is confronted with video recordings of the test session and is asked to comment on them. Although retrospective Think Aloud is less intrusive than online Think Aloud, it might possibly be affected by memory biases. The Think Aloud method can be used for free exploration of the interface as well as for conducting concrete tasks. The main advantages of the Think Aloud method are the low effort required for the preparation of the test and the rich amount of information which can potentially be gathered. The disadvantages are the often ‘unnatural’ situation due to the constant verbalization (Lin et al. 1997); often the experimenter has to repeatedly advise the participants to actually think aloud in order to keep the participants talking (Ericsson and Simon 1980). Additional problems are the systematic biases due to social desirability. Moreover, for interfaces offering speech input such as ECAs, the non-retrospective version of this method is inappropriate, as Think Aloud and speaking to the system simultaneously is not possible (Lewis 2012).

User characteristics

Before evaluation studies are conducted, the targeted user group and their characteristics need to be defined. In the following paragraphs relevant individual characteristics are explained and respective measurement instruments are presented either to confirm the target group or to cluster users.

Usability concepts

In this paragraph, the judgemental processes leading to the formation of quality, as well as quality aspects or concepts relevant for the usage of and interaction with ECAs are presented. The structure is based on the third layer of the taxonomy for multimodal systems (Wechsung et al. 2012b). Where appropriate, evaluation dimensions from (Ruttkay et al. 2004) are associated (cf. Fig. 2).

Hands-on issues

The main goal of a usability evaluation is to identify possible usage problems beforehand. This section gives hands-on issues to consider, based on a recommendation (VDE-ITG-Richtlinie 2011). Information on treating participants and their data can typically be found in ethics regulations of national research councils.

One important prerequisite is that the internal structure of the system is unknown to the user. Thus, discrepancies between the assumptions and expectations of the user and the developer can be identified. This entails that the evaluation should – at a certain point – be carried out without active involvement of the developer since the influence on test procedure and/or participant can have a strong impact on results.

A multitude of usability problems arises from intransparent functions and badly designed (e.g. unnatural) visual or verbal information. A first option to uncover flaws of this kind is to let the current implementation be reviewed without bias by a colleague. This “internal evaluation” quickly reaches its limits as colleagues are often too familiar with the capabilities and logic of the ECA in question and thus do not evaluate impartially.

User tests are more costly than evaluations done by experts. At the same time it is astounding which unforeseen flaws can be uncovered by a naive user. Therefore, user tests are expedient especially for new and unusual products. Test users should correspond to the expected main user group concerning age, gender, educational background, experience with technology etc.

The quality of usability tests depends on the selection of tasks the user has to fulfil. The tasks should be precisely worded and representative for later usage. The task description should neither evoke unrealistic scenarios nor hint implicitly at the favourable approach.

The investigator should be recognizably “neutral”. If this is not the case, it is possible that by suggestive questions or hints the test user is guided towards the correct solution. Also, the participant might rate the product positively believing that this is socially desired.

Furthermore, user tests (but not expert evaluations) are only of limited informational value if the function or interface is not fully functional yet. This is due to the fact that users have often difficulties imagining interaction steps which are not implemented yet, or else expect those to function perfectly. In this case the tasks should be chosen such that only already implemented functions are necessary.

User tests with application- or system-versions which are functionally not stable are of limited relevance as a user will either include system crashes in her judgement (even when asked not to do so) or else doubt the seriousness of the evaluation.

Usability tests of prototypes for which only minor adjustments are still possible should be avoided. If users realize that their suggestions for improvements have not been adopted they often react with an increased rejection.

Guideline to setting up an evaluation study

Unfortunately, no answers are given in Sonntag et al. 2004. We tried to provide a starting point to answer some of these questions, i.e. on purpose (cf. the four evaluation topics in Section “Communication theories” and the four psychological effects in Section “Computers as social actors”), on method (cf. Section “Evaluation and assessment methods” on methods and Section “Usability concepts” on concepts to assess), and on participants (cf. Section “User characteristics” on user characteristics). For the remaining questions, it is advised to examine the methods applied in studies similar to the one planned.

An experiment testing two different talking heads

This experiment is basically a replication of an existing German one (Weiss et al. 2010) conducted for Australian English speaking participants (approved by the UWS Human Research Ethics Committee). The aim was to validate an English version of a German questionnaire to assess the perceptual quality of different talking heads. The main concepts of the questionnaire are Ease-of-Use and Joy-of-Use. In order to induce variance in ratings, we tested four different versions consisting of two different visual models and two different voices. All versions were presented to 18 participants (4 men, 14 women) as metaphors of a spoken-dialogue system controlling a smart living room. The age of the participants ranged between 18 and 48 years (M = 24.61, SD = 6.95). For their attendance they received either money or course credit. The participants were seated in front of a table inside a laboratory room which is designed for audio and video experiments. There were two screens (19”) placed side by side in from of the participant. The metaphor was displayed on one. On the other screen the participants received visual information simulating feedback from an answering machine and an electronic program guide according to the task.

The participants interacted via headphones with the metaphor using free speech (instead of a fully functional system, the speech recognition and part of the interaction control was replaced by a human operator, called Wizard-of-Oz paradigm). They were asked to complete a domain-specific scenario consisting of seven different tasks with each of the four metaphors (head and voice combinations). These tasks were grouped in an answering machine scenario consisting of three tasks and an electronic program guide scenario consisting of four tasks.

The dialogue flow was controlled: the tasks were written on separate cards and offered to the participants in a predefined order. Every participant had to carry out both scenarios once with each metaphor. To avoid boredom the tasks were altered slightly in expression and content while the level of difficulty of each task remained constant. The order of scenarios was varied between participants.

After each scenario, a brief quality assessment was conducted. After both scenarios, participants had to fill in an English version of an early version of the CAS questionnaire (Wechsung et al. 2013) based on the complete interaction with one metaphor version.

Apart from this first step towards a validated English version of CAS, this experiment also exemplifies one possible usage of eye-tracking data. Data from 16 participants was analysed. As two separate screens were used, it was of interest to obtain information on the amount of time a participant looked at the ECA screen, measured as time the eyes were tracked, which was less than 50 %. The assumption that missing data was most widely originated by the users looking at the second screen or reading the task instruction could be confirmed during the experiment.

Results show a significantly lower amount of time looking at the ECA screen for one particular facial model, but no effect for the two voices used (cf. Fig. 3). As there was no initial hypothesis, final interviews with the participants were analysed to find a reason for this difference in eye-tracking data. The results suggest that those participants familiar with model A concentrated more on other visual information, i.e. the second screen or the task descriptions. For the domain of a smart living room, the results motivate to study the development of user experience with ECAs over time more closely. The question raised is, whether initial evaluations are still valid after familiarization. However, this will require field tests instead of laboratory experiments (cf. Ring et al. (2015) for an example of a one week ECA evaluation for elderly users).