How does a communication-partner prompt relate to participant state?
The AutomaViC study contains detailed coded interactions between nonspeaking autistic participants, the system, and communication partners. My question is how the content of a communication-partner prompt relates to the participant’s current state and recent interaction history, rather than treating every prompt as interchangeable.
- Working with
- Coded prompts, participant actions, and event sequences
- Method
- Analyse prompt content alongside sequential interaction data
My contribution
- Clean and structure coded interaction data.
- Analyse communication-partner prompt content.
- Relate prompts to participant state and prior interactions.
- Explore NLP representations and comparisons with generic prompt strategies.
- Design the analysis and modelling pipeline for the honours project.
Problem & approach
The project is still in progress. I am working through the coded event data, structuring the analysis pipeline, and exploring ways to represent prompt content with NLP while preserving the sequential context around each interaction. The dataset includes roughly 570 minutes of coded video from 37 participants across multiple study phases.
How it works
- 01
Organise events
Parse and organise coded events from each study session.
- 02
Find prompt context
Identify communication-partner prompts and the participant state around each prompt.
- 03
Represent language
Represent prompt content using NLP features or embeddings.
- 04
Compare strategies
Compare observed prompting behaviour with generic prompt categories or banks.
- 05
Evaluate patterns
Evaluate whether the resulting patterns are useful for future adaptive prompting work.
Data & evaluation
Data & inputs
Approximately 570 minutes of coded footage from 37 nonspeaking autistic participants, across multiple experimental phases. This describes the study dataset, rather than independently acquired product users.
How it was evaluated
The research and modelling pipeline are ongoing. Final model performance and study conclusions are not yet available.
Progress so far
- Ongoing
- Research status
- ≈570 min
- Coded study footage
- 37
- Study participants
Ongoing analysis of prompting strategies
- Ongoing research; final modelling results are not yet available.
- Approximately 570 minutes of coded study video from 37 participants.
Limits & lessons
Limitations
- The research is ongoing. Participant-level data are not reproduced here, and there are no final modelling findings yet.
What I learned
- The main lesson so far is that human interaction data is messy in a very different way from standard ML datasets. The difficult part is not only choosing a model, but deciding what the state of an interaction actually means and how much context is needed to interpret a prompt.
Next steps
- Finish the analysis pipeline and prompt representation work.
- Define evaluation criteria for prompt categories and participant-state relationships.
- Compare simple baselines with richer NLP approaches.
- Explore whether the analysis can support a human-in-the-loop adaptive prompting system.