Penpie Exploit Case Study: Leveraging Mutation Testing to Prevent Smart Contract Vulnerabilities
Penpie Exploit Case Study: Leveraging Mutation Testing to Prevent Smart Contract Vulnerabilities
March 25, 2025
Executive Summary
In September 2024, Penpie Finance suffered a devastating hack, resulting in the loss of over $27 million. Despite undergoing audits by reputable security firms, a critical vulnerability went unnoticed. This case study introduces the Olympix mutation testing tool, demonstrating its effectiveness in detecting hidden smart contract vulnerabilities. By leveraging mutation testing—a method of introducing controlled changes to software code to test the effectiveness of existing test suites—we explore how this approach could have identified and prevented the exploit.
Introduction
The decentralized finance (DeFi) sector has witnessed exponential growth, but with it comes increased security risks. Smart contracts, the backbone of DeFi protocols, are susceptible to exploits if not thoroughly tested and audited. The Penpie Finance hack underscores the limitations of traditional auditing methods and the need for more robust testing techniques like mutation testing to enhance blockchain security.
Background
Penpie Finance Hack Overview
On September 2024, Penpie Finance was exploited, resulting in a loss of approximately $27.4 million. The attacker manipulated the protocol's smart contracts to drain funds, leaving the team to reach out publicly in hopes of negotiating the return of the stolen assets.
Auditing of Penpie Finance
Penpie Finance had undergone two separate audits by reputable firms:
Zokyo Audit Report: Zokyo is known for auditing various protocols and chains, including prominent ones like NEAR and Aurora.
WatchPug Audit Report: WatchPug holds the second place on the all-time leaderboard in Code4rena, having earned over $800,000 in rewards.
Despite these thorough smart contract audit processes, the critical vulnerability remained undetected.
The Nature of the Exploit: Reentrancy Attack
The vulnerability exploited was a classic reentrancy attack, a well-known issue in web3 security. The root cause was a failure to follow the checks-effects-interactions pattern in an admin-only function that later became publicly accessible.
Admin-Only Function: Initially, the vulnerable function was restricted to the contract owner, making it less of a concern during audits.
Permissionless Pool Registration: In May 2024, Penpie Finance introduced permissionless pool registration, removing the access control without re-evaluating the security implications.
Auditors' Oversight: Since the function was out of scope and considered secure due to its access restrictions, auditors did not thoroughly assess it for reentrancy smart contract vulnerabilities.
Mutation Testing
What is Mutation Testing?
Mutation testing is a method of software testing where the code is deliberately altered (mutated) to check if the existing test cases can detect the changes. The purpose is to assess the effectiveness of the test suite in catching errors.
Process:
1. Introduce small changes to the code (mutations).
2. Run the existing test suite.
3. Analyze whether the tests detect the mutations (tests fail) or not (tests pass).
Outcome:
Killed Mutations: Mutations detected by the test suite (tests fail), indicating good test coverage.
Survived Mutations: Mutations not detected (tests pass), revealing potential gaps in testing.
Benefits of Mutation Testing in Smart Contract Development
Enhanced Test Coverage: Identifies untested parts of the code that could harbor smart contract vulnerabilities.
Early Detection of Security Flaws: Helps catch critical issues like reentrancy attacks before deployment.
Improves Code Quality: Encourages developers to write more comprehensive tests, leading to more secure smart contracts.
Application of Mutation Testing to Prevent the Penpie Finance Hack
How Mutation Testing Could Have Detected the Vulnerability
If mutation testing had been applied to Penpie Finance's codebase, the following could have occurred:
Introduction of Mutations: Altering the access control or the sequence of state updates and external calls in the vulnerable function.
Detection of Survived Mutations: The existing test suite would not have caught the mutations, indicating inadequate testing around that function.
Prompting Test Improvements: Developers would be alerted to write additional tests to cover the mutated code paths, potentially uncovering the reentrancy vulnerability.
Preventing the Exploit: By identifying and addressing the vulnerability during testing, the exploit could have been prevented.
The Olympix Mutation Testing Tool
Overview of the Tool
Olympix is a developed mutation testing tool designed specifically for smart contracts. It automates the mutation testing process, allowing developers to:
Automatically Mutate Code: Introduce a variety of mutations without manual intervention.
Run Test Suites Against Mutations: Execute existing tests to see if they detect the changes.
Receive Detailed Reports: Get comprehensive summaries of which mutations were killed and which survived.
Simplified Vulnerable Contract: A SavingsAccount contract with deposit and withdrawal functions, including a loyalty bonus mechanism.
Reentrancy Vulnerability: The withdraw function failed to follow the checks-effects-interactions pattern, making it susceptible to reentrancy attacks.
Mutation Testing Process
1. Running Olympix: Executed the tool with the command to generate mutation tests on the SavingsAccount.sol file.
2. Mutation Introduction: Olympix introduced 10 mutations, altering various parts of the code.
3. Test Execution: The existing test suite was run against each mutated version.
4. Analysis of Results:
-Killed Mutations: 8 out of 10 mutations were detected by the tests, indicating good coverage in those areas.
-Survived Mutations: 2 mutations were not detected, revealing gaps in the test suite.
Conclusion
The Penpie Finance hack serves as a critical lesson in the importance of comprehensive testing in web3 security. Traditional audits, while essential, may not catch all vulnerabilities, especially when code changes over time. Mutation testing offers a proactive approach to uncover hidden bugs by challenging the effectiveness of existing tests.
The Olympix mutation testing tool demonstrates how automation and AI can significantly enhance web3 cybersecurity practices. By integrating mutation testing into the development workflow, teams can detect and remediate vulnerabilities early, potentially saving millions in lost funds and safeguarding their protocols against attacks.
References
Penpie Finance Post-Mortem Report: Detailed analysis of the hack and its causes.
Zokyo and WatchPug Audit Reports: Initial audits of Penpie Finance.
Olympix Mutation Test Demo: https://github.com/olympix/mutation-test-demo
What’s a Rich Text element?
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.
Follow-up: Conduct a follow-up review to ensure that the remediation steps were effective and that the smart contract is now secure.
Follow-up: Conduct a follow-up review to ensure that the remediation steps were effective and that the smart contract is now secure.
In Brief
Remitano suffered a $2.7M loss due to a private key compromise.
GAMBL’s recommendation system was exploited.
DAppSocial lost $530K due to a logic vulnerability.
Rocketswap’s private keys were inadvertently deployed on the server.
Hacks
Hacks Analysis
Huobi | Amount Lost: $8M
On September 24th, the Huobi Global exploit on the Ethereum Mainnet resulted in a $8 million loss due to the compromise of private keys. The attacker executed the attack in a single transaction by sending 4,999 ETH to a malicious contract. The attacker then created a second malicious contract and transferred 1,001 ETH to this new contract. Huobi has since confirmed that they have identified the attacker and has extended an offer of a 5% white hat bounty reward if the funds are returned to the exchange.
