How to Prevent a £212 Million Software Bug

I couldn’t believe it happened - as I sat down at my desk, I opened my laptop and I was hit with the news I knew the cryptocurrency community had been dreading might happen eventually.

It was moments like this the crypto community had discussed with concern - a leak, a hack or similar, leading to a freeze on digital wallets powered by a single provider.

By now, you’ve probably heard the news already - an estimated $280m (£212m) worth of the ether token is locked up because one user accidently deleted the code library required to access digital wallets.

I’m going to break down what actually happened and outline what could have been done to prevent this.

So, What Happened?

It all started at the beginning of the month when one user overstepped the mark (by accident, I should note). They were playing with the Parity multi-sig wallet library contract, and accidentally triggered it’s “kill” function, effectively freezing the funds on all Parity multisig wallets.

WOAH - that was a lot of jargon. Let’s break it down: Parity is a digital wallet provider (kind of like a bank which provides a number of different bank accounts). Multi-signature wallets are popular (especially with Initial Coin Offerings, where companies raise money by investors/consumers buying into their coin (in the hope it will increase in the future)), because these wallets require more than one user to sign off before funds can be transferred (more detailed information on ICO's here).

A recent example of this in my own life was when I was buying some software for a sizeable amount. We have a rule in our company that for any transaction above a certain size, an automatic message will be sent to our finance team who can either approve or reject the transaction, by which time they can speak to us to make sure this came from us and is a legitimate transaction we’d like for them to approve. I can think of a multitude of things in my personal life I wouldn’t have bought if I had a third-party who verified every purchase!

To get to the point of the story - there was an unprotected kill/suicide function in the codebase which the developers hadn’t noticed or closed up. Here’s a short yet informative analysis of the smart contract bug by a security researcher if you want to get into the details.

How does this apply to me?

Having had hundreds of conversations with development teams about speeding up their development process and helping them with their QA challenges, there hasn’t been a single conversation where someone boasted their software has been / will always be perfect (i.e. no bugs have ever come up in their code).

This smart contract bug could have been spotted and dealt with much earlier in the process (apparently the wallet provider knew about the bug a few months before), with the right infrastructure and a robust testing strategy before each new update.

Three things that can help

1. Introduce testing earlier into the development process → Many of the high-performing teams we work with who release on a daily/weekly basis have embraced the “Shift-Left” mantra (which is a major part of the QAOps process). With frequent deployments, quality becomes the developer’s responsibility (in the case of the Ether smart contract bug, they could also consider approaches similar to pair or mob programming.
2. Blend a mixture of manual and automated testing to catch critical issues → With an issue like this, it’s important to deploy the correct approach to automated testing. In addition to this, manual testing is equally important to view the application from a different perspective. We’ve worked with a company where their internal team couldn’t spot an issue a member of our crowd found within 15 minutes of exploring the application, it’s something we encourage all companies to evaluate when they embrace QAOps. Utilising a variety of testing approaches would’ve added an extra layer of security and confidence when deploying new updates that could’ve prevented the issue.
   
   
3. Use an external partner to help you find external perspectives → Most applications have internal development teams but they can’t cover all the devices or don’t have access to a crowd of professional testers that can report results found within 48 hours or less. This is something we help companies of all sizes with - from companies as small as bootstrapped 2-person companies with a single app, to large companies such as Facebook and Microsoft.

Got any feedback? Let us know in the comments below and reach out to our team to find out how we can help you!