Research on LLM reasoning mostly relies on the answer accuracy as a proxy metric for the reasoning process. However...

• A correct final answer does not always indicate the correctness of the reasoning chain. This is refered to as "false positives".
• LLMs may hallucinate when generating reasoning chain, which would cause problems in many scenarios, like using LLMs as tutors.
• Existing reasoning chain evaluation methods typically require massive training data, or lacks evaluation accuracy and interpretability.

In this leaderboard, we focus on the direct evaluation of reasoning chains with our newly proposed metric AutoRace (Automated Reasoning Chain Evaluation). It can evaluate reasoning chains and provides explanations, without any human effort. For a detailed explanation of the evaluation metric and analysis of the results, please refer to our blog and paper.

Note: AutoRace is an automatic evaluation method based on GPT-4. It does not guarantee absolute accuracy in the evaluation results.