AI

Inducing Task Models from Computer-Use Traces

Researchers have developed a method called Task Model Induction (TMI) to derive symbolic models of how everyday work is done from computer-use traces. These traces include screenshots and mouse or keyboard actions recorded passively over time. TMI can identify the underlying tasks in an unconstrained trace, disentangling concurrent activity, and reconstruct task models that pair hierarchical objective models with procedure models of control flow. In experiments, TMI performed
Researchers have developed a method called Task Model Induction (TMI) to derive symbolic models of how everyday work is done from computer-use traces. These traces include screenshots and mouse or keyboard actions recorded passively over time. TMI can identify the underlying tasks in an unconstrained trace, disentangling concurrent activity, and reconstruct task models that pair hierarchical objective models with procedure models of control flow. In experiments, TMI performed significantly better than existing methods in recovering interleaved tasks and reconstructing execution steps. --- Why it matters: This matters to AI researchers because it provides a way to automatically derive structured task models from raw data, which can be used to improve the accuracy of skills learned by agents and machines. Source: https://arxiv.org/abs/2608.20319

This article was originally published at: https://arxiv.org/abs/2608.20319