Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification

Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has brought significant advancements in solving math problems. In particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter, shows remarkable performance on challenging math datasets. In this paper, we explore the effect of code on enhancing LLMs' reasoning capability by introducing different constraints on the Code Usage Frequency of GPT-4 Code Interpreter. We found that its success can be primarily attributed to its powerful skills in generating and executing code, evaluating the execution result, and rectifying its solution when receiving unreasonable outputs. Based on this, we propose a novel prompting method, explicit code-based self-verification (CSV). This method employs a zero-shot prompt on the GPT-4 Code Interpreter to encourage it to use code to self-verify its answers. In instances where the verification state is "False", the model will automatically amend its solution. Furthermore, we recognize that the states of the verification result indicate the confidence of a solution, which can improve the effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we achieve an impressive zero-shot accuracy of various mathematical problem-solving benchmarks. © 2024 12th International Conference on Learning Representations, ICLR 2024. All rights reserved.