Across ageing societies, a gap is widening: people live longer but families grow smaller. A rising number could reach the end of life, unable to make their own medical decisions and with no next of kin or trusted friend to speak for them. Life-or-death decisions then lie in the hands of a stranger.
Advance care planning (ACP) is meant to close the gap, letting people record their wishes in advance. In practice, however, it has become a once-off paperwork exercise, where forms are filed away years in advance, rarely revisited, and hard to retrieve when the need arises. This problem is the starting point for new work from the Singapore University of Technology and Design (SUTD).
In the paper “Words to describe what I'm feeling: Exploring the potential of AI agents for high subjectivity decisions in advance care planning”, presented at the 2026 CHI Conference on Human Factors in Computing Systems, SUTD Assistant Professor Kenny Choo and his team built ACPAgent, an experience prototype that uses artificial intelligence (AI) and acts as an ACP proxy. Across four workshops, 15 participants trained it and judged whether to trust it to speak for them.
“We were careful not to assume that AI was the answer,” said Asst Prof Choo. “We built it to ask a basic question: when a decision hinges on personal values rather than facts, what role, if any, should an AI play?”
Each participant outlined what mattered to them, such as valued activities, goals of care and what they would trade for more time. They then worked through five scenarios of rising difficulty, from a treatable infection to terminal illness and unresolved family conflict. In each round they fixed their own choice before seeing the agent’s recommendation, and between rounds they could adjust preferences and push back.
The numbers showed that participants agreed with the agent in 86.7 percent of decisions, and the scenarios shaped 76 percent of choices. The interesting findings are in the disagreements: people pushed back over family guilt, affordability, and the real odds of recovery—factors the model handled poorly.
“I’d caution against reading 86.7 percent as a success metric,” Asst Prof Choo explained. “High agreement in a high-subjectivity setting can mean the system captured someone’s values, but it can also mean acquiescence, automation bias, or simply an agreeable model.”
Influence flowed both ways: in about 12 percent of cases, the agent opposed a participant’s first choice, yet they came round to its reasoning. One participant said the agent gave her “words to describe what I'm feeling”, which also points to the agent’s biggest risk.
“The flip side is that the bot may be training us to do ACP properly—not the other way around,” Asst Prof Choo noted. “If a system supplies the words, it also shapes them. A person may adopt phrasing, and with it a position, that originated with the model rather than themselves.” The risk is authorship blur: when an agent's reasoning convinces you, how do you know which of your values are truly yours?
Because the stakes are high, Asst Prof Choo thinks transparent reasoning, room to challenge the agent, and a log of what it stores are important. Mapping responses against human control and machine autonomy, the team set out four roles, from a low-autonomy elicitor of values to a full proxy. The most promising, they argue, is a middle path: an advocate that reinforces a person’s wishes without displacing human judgement.
Whether to delegate at all is contentious. “The honest counterfactual isn’t AI versus a loving, well-informed human proxy,” Asst Prof Choo said. “For a growing number of people, it is AI versus nothing, or a stranger who has never met them.”
He added that the study carries some limitations: the scenarios were simplified, the sessions one-off, and the agent’s apparent personalisation may have inflated trust. Responsible deployment of such an agent would need legal recognition for advocacy, since ACP documents are not yet binding. Integration with national ACP and medical-directive records so a person’s wishes are actually retrievable at the point of care is necessary, alongside stronger data protection.
Real-world deployment would take time, but several participants expressed an interest in using the agent to start a conversation with their ageing parents.
“We may have built a working experience prototype, but we used it as a probe—to surface how people reason about autonomy, control, and trust in a domain too sensitive and high-stakes to study abstractly,” reflected Asst Prof Choo. “Rather than asking what the model can do, we used it to ask what people actually want it to do.”
