Can AI Solve Physics Problems? Evaluating Efficacy of AI Models in Solving Higher Secondary Physics Exam Problems: A Comparative Study

Abstract

Large Language Models (LLMs) have grabbed significant attention from diverse technical fields due to their impressive performance on a variety of Natural Language Processing (NLP) tasks. Although these models excel in various generative tasks, they lack the robust reasoning ability required to solve complex mathematics and physics problems. Despite their inherent limitations, Generative Artificial Intelligence (AI) based chatbots, powered by these large language models, are being rapidly adopted by students in physics and other technical fields. In this project, we assessed the ability of various generative AI-based models to solve Physics problems. We asked currently popular AI models to solve Physics questions from a final board exam of class 12 of the Higher Secondary Education Board (HSEB) of Nepal. We then evaluated the AI-written solutions by the subject matter experts. We found that the gpt-4o model by OpenAI performed the best, securing 90% among the models studied. In this paper, we provide a brief overview of these models and compare their performance as evaluated by a University Physics professor. We will also discuss the risks and benefits of their use in higher education.