TCL The Context Lab

Open PhD Projects

Towards Better Collaboration with LM Agents: Predicting and Producing Effective Task Oriented Dialogue

Effective Dialogue Effective communication becomes increasingly challenging in collaborative settings with time constraints or high cognitive load. In such settings, like emergency situations, unfamiliar navigation, or explaining complex concepts, a speaker's ability to recognize and adapt to their partner’s limitations is essential. While language models (LMs) have made strides as conversational assistants, current dialogue systems still fall short in effectively performing this kind of adaptive communication.
This PhD project aims to a) uncover factors that predict successful task-oriented dialogue, and b) use these factors to explore techniques to improve generation in LM agents. The first part of the project will focus on the identification and analysis of communicative success. Building on expertise and findings from e.g. [Yee2024], [Sinclair2019] initial measures will include between speaker adaptation in the form of repetition and rate of information delivery. We will extend this to include affective or emotional interpersonal signals such as in [Paromita2024]. These findings will aid the development of automatic task-oriented dialogue quality measures, and establish desiderata for the design and evaluation of LM agents. The second part of the project will involve the application of these findings to adaptive collaborative LM dialogue agents. This will involve conducting experiments to elicit examples and ratings from humans. By empirically testing theories of communication and evaluating LMs’ capabilities to recognize and replicate effective dialogue strategies, this research will advance our understanding of compensatory adaptation techniques between speakers, and investigate whether these are applicable in a human-AI interaction setting.
This project aims to develop insights and metrics that will inform the design, training, and evaluation of next-generation, human-centered AI dialogue agents, with applications in both human-computer interaction and human-robot interaction. This work will also contribute valuable empirical data for refining existing theories of human communication Self Funded. Available on FindAPhD.com

References

[Yee2024]

Yee, Jun Sen, Mario Giulianelli, and Arabella Sinclair. "Efficiency and effectiveness in task-oriented dialogue: On construction repetition, information rate, and task success." 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024-Main Conference Proceedings. European Language Resources Association (ELRA), 2024.

[Paromita2024]

Projna Paromita, Alaa Khader, Sydney Begerowski, Suzanne T. Bell, Theodora Chaspari “A Linguistic Analysis of the Impact of Team Interactions on Team Performance During Space Exploration Missions” Proceedings of the 2024 International Conference on Affective computing and intelligent interaction, ACII. 2024.

[Giulianelli2021]

Giulianelli, Mario, Arabella Sinclair, and Raquel Fernández. "Is Information Density Uniform in Task-Oriented Dialogues?." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.

[Sinclair2021]

Sinclair, Arabella J., and Bertrand Schneider. "Linguistic and Gestural Coordination: Do Learners Converge in Collaborative Dialogue?." International Educational Data Mining Society, 2021.

Human-like Repetition in LM Dialogue: Comparing Automatic and Strategic Reasons to Repeat

Repetition in Dialogue Repetition is a key component of dialogue. Humans use specific communication strategies involving repetition, critically, these are local and partner specific. In dialogue, we repeat our dialogue partners as we establish and maintain common ground and mutual understanding, it can also serve an essential function during the learning process. Depending on the context, this repetition can serve different functions; in some settings, it is automatic, part of a process whereby we align to and pick up language behaviour from our dialogue partner [Pickering2004].
However, between-speaker repetition has been shown to vary depending on the ability of the speaker (e.g. EFL speakers not sufficiently proficient in the language will not repeat their partner’s longer constructions), and may be used strategically by speakers (such as tutors, or caregivers for L1 acquisition) seeking to support their dialogue partner [Sinclair2023;Sinclair2021].
We start from the desideratum that dialogue models should also produce humanlike levels of repetition: these are preferred by human users and lead to more successful communication in dialogue between humans. While Language Models (LMs) do learn to repeat constructions used in a local dialogue context [Molnar2023], it remains unclear the extent to which they differentiate between automatic vs strategic repetition. Understanding the nature and purpose of these repetitions is important for generation quality: excessive repetition can render LM-generated text disfluent, whereas insufficient repetition could hamper a potentially vital source of maintaining common ground with a user. Bridging this understanding gap has implications for their use as pedagogic agents, generating feedback for learners, serving as conversational practice, or as tutors.
The core aims of the project are to contribute to our understanding of repetition patterns crucial to dialogue understanding, within that there is some flexibility regarding sub-topics of interest. Self Funded. Available on FindAPhD.com
This project has scope to make impact within the fields of:

interpretability

understanding how LMs are affected by and deal with repetition in their input context

psycholinguistics

improved understanding of corrective feedback in L1 and L2 acquisition, through deeply analysing corpora

HCI/HRI

understand via extrinsic evaluation the importance of repetition in interactive settings

Safety & Robustness

are LMs robust to human repetition patterns common in spoken interaction, or will these provoke disfluent behaviour

References

[Sinclair2023]

AJ Sinclair, R Fernández, Alignment of code switching varies with proficiency in second language learning dialogue, System Journal 113, Special Issue on Linguistic alignment in Second Language Acquisition: occurrences, learning effects, and beyond, 102952

[Pickering2004]

Pickering, Martin J., and Simon Garrod. "Toward a mechanistic psychology of dialogue." Behavioral and brain sciences 27.2 (2004): 169-190.

[Molnar2024]

Aron Molnar, Jaap Jumelet, Mario Giulianelli, Arabella Sinclair, Attribution and Alignment: Effects of Local Context Repetition on Utterance Production and Comprehension in Dialogue, Conference on Natural Language Learning 2023

[Sinclair2021]

Sinclair, Arabella J., and Raquel Fernández. "Construction coordination in first and second language acquisition." Proceedings of the 25th Workshop on the Semantics and Pragmatics of Dialogue. 2021./dd>

Language Learning through Communication

Learning through communicating, corrective feedback, and interactive learning Caregivers provide feedback to children in dialogue via repetition, elaboration, corrective feedback, and modelling adult conversation. While previous research has primarily focused on the use and usefulness of corrective feedback to syntactic constructions (Marcus, 1993; Fernandez 2016), there is a notable gap in our knowledge about the role of situation-specific modelling of adult-like behaviour for pragmatic development. Learning to ask questions serves particularly well for assessing this form of feedback because they have been shown to be tailored to children’s mental states in the shape of test-questions (Schatz, 1974) and because they show a predictable turn-taking behaviour (Casillas & Frank, 2017), a large lexical overlap between speaker-turns (Paul 1880), and dedicated response markers (Kramer & Rawlins, 2009). All these ingredients aid in identifying questions as a phenomenon. One challenge with assessing different kinds of feedback to language learners is that work is often based on individual case studies due to the resource intensity of analysing child language data. With the arrival of Large Language Models (LLMs), this challenge can be overcome by taking advantage of the powerful representations within, and classification ability of LLMs to better recognise and analyse different types of feedback. This, in turn, will shed light on how children exploit caregiver feedback to advance their conversational skills. In turn, analysing properties of repetition and feedback in human language can have implications for incorporating successful interaction strategies in language models used in dialogue agents. Self Funded

Self proposed topic within this area.

Adaptive Audience-Aware NLG

Can LLMs produce more human-like language that is adaptive, context-specific, and audience-aware? Current dialogic interaction systems fall short when it comes to nuanced, pragmatic language understanding. Taking inspiration from the fields of psycholinguistics and cognitive science, this project aims to explore, evaluate and extend the potential of LLMs to generate language that is appropriate for an interactive setting such as dialogue. Adaptive NLG can lead to more inclusive, personalised, and appealing language; essential components of a number of downstream applications, such as in educational settings, or as creative tools. Evaluating such adaptive language will involve analysis of language produced by humans, compared to humans and models in collaborative communicative settings. Self Funded

Self proposed topic within this area.