Researchers have investigated whether Large Language Models (LLMs) can reason in a way that's meaningful in a legal context. They used cases from the European Court of Human Rights as test data and evaluated how well an LLM called OpenAI GPT 5.4 could forecast outcomes using different prompting strategies. The study found that while the model produced complete but shallow analyses, it scored poorly on actual reasoning quality. The researchers conclude that relying solely on automated LLM-based evaluation can be misleading and recommend using human evaluation instead. --- Why it matters: This matters because AI-powered legal decision-making is becoming increasingly common, and understanding how well LLMs can reason in a legally meaningful way has significant implications for the accuracy and fairness of these systems. Source: https://arxiv.org/abs/2608.17168