There are reports of issues with a benchmark used to evaluate ai coding capabilities, with some outlets noting that approximately 30% of the tasks in the benchmark are broken. The benchmark in question appears to be SWE-Bench Pro, which is used to measure the coding abilities of language models. The problems with the benchmark have led to the introduction of alternative evaluation methods and tools, such as GeneBench-Pro.