What Happened

The bug was realized in the following situation:

1. Assuming the pool has margin enabled
2. Assuming a margin position gets opened against this pool (custody asset or collateral asset)
3. Assuming the pool has now positive custody and liabilities balances either on the external or native side
4. Assuming we have ongoing swap activities on each direction against this pool (rowan -> atom, atom -> rowan, usdc -> atom, atom -> usdc, etc.)
   1. On these conditions, a swap transaction calls the function pool.UpdateBalances (here) that uses X and Y as inputs. This function is supposed to update both sides of the pool balances with the input and output of a swap. In this scenario the X and Y inputs to the function included the custody and liability values in the pool. This caused the pool’s balances to incorrectly increase by the amount of custody and liability every time a swap was executed.
   2. Even though we are dealing with a single position against atom, this process is repeated each time there is a swap activity against the atom pool
5. The fix here changes the call to pool.UpdateBalances to use the correct X and Y pool balance values instead of including custody and liabilities amounts

Sequence of Events

• DEX users who added and removed liquidity between blocks 8649350 - 8654227, withdrew more liquidity than they had deposited. The reason for this is that swaps in the ATOM:ROWAN pool caused incorrect pool balance calculations. Essentially, this caused the CLP units of ATOM and ROWAN in the ATOM:ROWAN pool to be incorrect. The calculations made the pool balance “appear” larger and the pool share of the liquidity provider’s wallets were calculated as a larger amount than it should have been. The exploit was identified by Dreamcatcher and _Nick in the community, who immediately flagged it to Abratusz.
• The Engineering team turned the front end off at 4:02am PST. Then, at block 8654227 (2022-09-19 6:47:06 PST), Engineering and the Validators collaborated to halt the chain. At this time, a total of 50 addresses incorrectly removed 7,806.84 ATOM and 22,302,424.69 ROWAN prior the chain halt.
• The chain was halted until the team, a) identified the exact bug, b) its impact on the DEX and to the ATOM:ROWAN pool, and c) how the DEX could be re-enabled immediately and what functionality still needed to be disabled.
• On September 21st, at 6:25 AM PST, in accordance with a phased start approach that was shared with the community, Engineering sent transactions into the mempool to turn off the ATOM / ROWAN, turn off rewards, and turn off the sale of ROWAN on the DEX. The validators began turning their nodes back on at 6:30 AM, and the chain was producing blocks from 6:37 AM. The chain was thus back online and the ATOM:ROWAN pool actions were disabled, while Liquidity Provider Distribution, Liquidity Mining Rewards, and DEX Liquidity Protection turned off. The chain was re-enabled on at 2022-09-21 6:30:45 PST. A big shout out to the team at Notional and Consensus One for their collaboration.
• On 2022-09-27 11:49:43, Engineering raised a software upgrade on-chain (https://www.mintscan.io/sifchain/proposals/117) that delivered the patch to fix the bug that caused the exploit, correct the CLP units of both ATOM and ROWAN of the ATOM:ROWAN pool, re-activate all actions on the ATOM:ROWAN pool, and turn on Liquidity Provider Distribution, Liquidity Mining Rewards, and DEX Liquidity Protection.
• After the software upgrade went live, the team then performed the following actions:
  • Ran final tests to ensure the DEX was fully functional and correct.
  • Compensated the liquidity providers of the ATOM:ROWAN pool for the exploited funds (see more on this below).
  • Re-enabled all actions and functionality on the DEX.
  • Set monetary policies (Liquidity Provider Distribution, DEX Liquidity Protection, and Liquidity Mining Rewards)

Fund Remediation

Numerous individuals in the community reached out to the Sifchain team to return the extra funds they received. Thank you to these individuals who contributed to the well-being of the community. Your honesty is greatly appreciated.

Community members returned the following amounts to the Fund Recovery Wallet:

• Total ATOM: 4,879.75
  • ATOM (Cosmos): 3,554.640
  • ATOM (Sifchain): 1,325.11
• ROWAN: 12,593,487

The wallet addresses for the Fund Recovery Wallet are: