Seeker: Enhancing Exception Handling in Code with LLM-based Multi-Agent Approach

As the functional correctness of large language models (LLMs) in code generation tasks continues to gain attention and improve, how to generate code that passes more test cases and how to reduce functional vulnerabilities in the code have become determinative to evaluate LLMs' coding performance. However, few works delve into the performance of LLMs in code robustness represented by exception handling mechanisms. Especially in real development scenarios, the exception mechanism has high standardization requirements for developers in terms of detecting, capturing and handling. Due to the lack of interpretable experience and generalizable strategies, highly robust code is relatively scarce in the main peak of open source projects, which further affects the high-quality code training dataset and generation quality of LLM. This prompted us to raise a research question that few people have explored: "Do we need to enhance the standardization, interpretability and generalizability of exception handling in real code development scenarios?" To confirm this requirement, we first uncovered three pillar phenomena of incorrect exception handling through extensive LLM and human code review: Insensitive-Detection of Fragile Code, Inaccurate-Capture of Exception Type, and Distorted-Solution of Handling Block. These phenomena frequently occur both in real repositories and generated code, indicating that both human developers and LLMs are hard to spontaneously understand the usage skills of exception handling. Surprisingly, LLM's bad performance will have a good mitigation effect under the enhanced prompts of precise exception types, scenario logic and handling strategy. To exploit this effect, we propose a new method called Seeker, it is a chain of agents based on thoughts of the most experienced human developers when facing exception handling tasks, separated as Scanner, Detector, Predator, Ranker, Handler agents. To the best of our knowledge, our work is the first systematic study of the robustness of LLM-generated code in real development scenarios, providing valuable insights for future research in the direction of reliable code generation.