Researchers have proposed a new method for evaluating the performance of large language model (LLM) powered meeting assistants. The current methods for evaluating these models are limited and don't account for specific failure modes tied to certain discourse structures or reasoning demands. The new method, called Evaluation-as-Search (EaS), uses feedback from evaluators to adaptively search for potential failures in the models' grounding fidelity. This approach is more effective than traditional random probing methods, surfacing 2.5 times as many failures. A benchmark dataset of over 3,000 annotated question-answer pairs was created using this method and found that LLMs struggle with discourse-pragmatic challenges rather than factual recall errors. --- Why it matters: This research matters because it highlights the limitations of current evaluation methods for meeting assistants and provides a more effective approach to identifying potential failures. Understanding these failure modes is crucial for improving the performance and reliability of these models, which are increasingly being used in real-world applications. Source: https://arxiv.org/abs/2608.20392