Thala Nov 15 (Post Mortem)

Overview

On November 15th 2024, Thala suffered a security breach as a result of an isolated vulnerability in the latest update to v1 farming contracts, allowing the exploiter to withdraw liquidity pool tokens totaling $25.5m. We immediately paused all relevant contracts and froze Thala token assets ($9m MOD and $2.5m THL). With the help of law enforcement, Seal 911, Ogle, and others, the Thala team was able to quickly identify the exploiter and negotiate a $300k bounty for a full recovery of user assets. All relevant contracts and the Thala frontend were paused for several days post-exploit to ensure maximal security.

Most importantly, Thala was able to negotiate a full return of exploited funds from the attacker in exchange for a bounty, ensuring that all user funds are safe and positions remain. While we successfully recovered all potential losses and every user was made whole, the team recognizes the severity of the vulnerability and the need to dramatically strengthen our security practices and infrastructure. This report details the timeline of the incident as well as several major security enhancements already underway to prevent any similar occurrences in the future.

Detailed Timeline of Events

10/11/2024–8:50pm PST

• Thala team releases boosted farming feature (transaction)

10/31/2024–8:44am PST

• Users report rounding issue with unstaking maximum LP tokens

11/01/2024–10:54am PST

• Thala team releases small two-line patch to boosted farming contracts that unfortunately introduces the unstake_max exploit (transaction)
• Patch bypasses normal security review process due to perceived simplicity of the change

11/15/2024–4:46am PST

• Attacker 1 (address: 0xf7…) initiates the exploit with their first over-withdrawal (transaction)

11/15/2024–5:12am PST

• Thala engineering team paged on total value locked (TVL) alert
• Alert incorrectly attributed to previously announced farming incentives delay

11/15/2024–7:10am PST

• Attacker 2 (address: 0x80…) completes final exploit transaction, draining the majority of funds from MOD/USDC, MOD/THL, and THAPT/APT LP farming pools (transaction)

11/15/2024–7:30am PST

• Thala engineering team paged again on TVL drop alert
• War room opened with OtterSec to investigate the incident
• Vulnerability identified and unstake_max patch reverted to last safe version (transaction)

11/15/2024 — ~8:30am PST

• On-chain analysis reveals both attackers are associated
• Attacker identity uncovered using on-chain data
• Thala freezes attacker funds to prevent further loss

11/15/2024–9:34am PST

• Thala sends on-chain message to attacker, opening line of communication

11/15/2024–10:13am PST

• Negotiations begin with attacker
• Agreement reached for return of funds in exchange for $300k protocol bounty + $40k personal compensation for Thala founders

11/15/2024–11:13am PST

• Exploited funds fully returned to Thala, minus the $340k bounty.

11/15/2024–12:08pm PST

• Internal investigation launched to assess total damages and check for additional exploit transactions (none found)

11/15/2024–1:36pm PST

• Thala publicly confirms return of exploited funds on Discord (message)

11/15/2024–2:26pm PST

• THL/MOD LP farming pool fully reconciled (transaction)

11/16/2024–12:34am PST

• Initial incident report shared on Thala Twitter (link)

11/16/2024–8:05pm PST

• Remaining impacted LP farming pools (MOD/USDC, APT/THAPT) fully reconciled and rebalanced (tx1, tx2)

11/16/2024–9:47pm PST

• Thala app brought back online at app.thala.fi

Next Steps

While Thala was fortunate to recover from this incident with no loss to users, we recognize that our security practices were insufficient and multiple failures allowed this vulnerability to both be introduced and exploited. In response, we are implementing the following major changes effective immediately:

1. Comprehensive test coverage will be required on all production code changes, regardless of perceived size or impact. Even the smallest patches will have full test suites.
2. The farming contracts have been re-audited end-to-end by OtterSec, with the full audit report to be published on docs.thala.fi.
3. Global withdrawal rate limits are now imposed protocol-wide, capping the amount that can be withdrawn from farming pools and other Thala offerings (including LSD redemptions and select ThalaSwap pools) within a defined timeframe. Withdrawals exceeding the limit will fail and require staggered transactions.
4. Patches to production code will no longer be permitted without a full release cycle, ensuring adequate time for auditor review and internal testing. This eliminates the ability to bypass normal security practices under any circumstances.
5. On-call alerting and incident response processes have been overhauled to provide clearer context and force rapid escalation/mitigation.
6. Application access control has been refactored to use capability-based models wherever feasible. State-changing contract methods will require unforgeable objects rather than raw parameters that allow bypassing validation.
7. A principle of self-custody is being instituted throughout the protocol. Whenever viable, receipts will be minted to users rather than escrowing assets, reducing risks of honeypot scenarios.
8. A full top-down security review of the entire Thala protocol is currently underway, conducted by OtterSec and additional audit partners. Results and further recommendations will be published upon completion.

Disclaimer

This article by Thala Labs and/or its affiliates (“we”, “us” and “our”) is for information purposes only. We do not provide tax, legal, insurance or investment advice, and nothing in this article should be construed as an offer to sell, a solicitation of an offer to buy, sell or issue or subscribe for, or a recommendation for any security, investment, cryptocurrency, token or other services, product or commodity by us or any third party. You alone are solely responsible for determining whether any purchase, sale, investment, security or strategy, or any other product or service, is appropriate or suitable for you based on your personal objectives and personal and financial situation and for evaluating the merits and risks associated with the use of the information in this article before making any decisions based on such information or other content. You should consult a lawyer and/or tax professional regarding your specific legal and/or tax situation. Past performance is no guarantee of future results. Therefore, you should not assume that the future performance of any specific investment, cryptocurrency, token, commodity or strategy will be profitable or equal to corresponding past performance levels. Inherent in any such transaction is the potential for loss. No recommendation or advice is being given as to whether any transaction is suitable for a particular person. By accessing this article, you acknowledge and agree to all of the foregoing and that you bear responsibility for your own research, due diligence and transaction decisions. You also agree that we, our affiliates and our respective directors, officers, employees, consultants, shareholders, members, representatives, advisors and agents will not be liable for any decision made or action taken by you and others based on this article, news, information, opinion, or any other material published, discussed or disseminated by us.

This article contains forward-looking statements or forward-looking information (referred to collectively as “forward-looking statements”). Forward-looking statements can be identified by words such as: “anticipate”, “intend”, “plan”, “goal”, “seek”, “believe”, “predict”, “project”, “estimate”, “expect”, “strategy”, “future”, “likely”, “may”, “should”, ”would”, “will”, and similar terms and phrases and the negatives of such expressions, including references to assumptions. Examples of forward-looking statements in this article include, among others, statements we make regarding our future plans, expectations and objectives.

Forward-looking statements are neither historical facts nor assurances of future performance. Instead, they are based only on our current beliefs, expectations and assumptions regarding the future of our business, future plans and strategies, projections, anticipated events and trends, the economy and other future conditions. Because forward-looking statements relate to the future, they are subject to inherent uncertainties, risks and changes in circumstances that are difficult to predict and many of which are outside of our control. Our actual results and financial condition may differ materially from those indicated in the forward-looking statements. Therefore, you should not rely on any of these forward-looking statements. Important factors that could cause our actual results and financial condition to differ materially from those indicated in the forward-looking statements include, among others, the following: reliance on blockchain technology and blockchain technology service providers; digital asset transactions being irrevocable and losses occurring from such transactions; our use and reliance on proprietary data and intellectual property in its business; potential misuses of digital assets and malicious actors in the digital asset industry; digital assets potentially being subject to hold periods; developments and changes in laws and regulations; and disruptions to our technology network including computer systems, software and cloud data, or other disruptions of our operating systems, structures or equipment. Readers are cautioned that the foregoing list is not exhaustive.

Any forward-looking statement made by us in this article is based only on information currently available to us and speaks only as of the date on which it is made. Except as required by applicable securities laws, we undertake no obligation to publicly update any forward-looking statement, whether written or oral, that may be made from time to time, whether as a result of new information, future developments or otherwise.