Update on investigation into API keys and attacks on exchanges

We’d like to address some frequently asked questions and concerns raised from users affected by the attack on exchange APIs. If you haven't heard about it, a number of users of Binance, OKX, FTX and some other exchanges have experienced unauthorized trades initiated via API keys.

Please note, that we cannot respond with specific details for individual cases as the answers will not be applicable to every user or the exchange account affected by this issue. We are not trying to be vague, but each case is truly unique. There is no single commonality that unites every individual affected other than the unauthorized trading activity.

We’d also like to take a moment to empathize with every person that has been impacted by this issue; it has caused financial harm not only to those affected but also damage to our trading community.

Update December 11th: individuals on Twitter and YouTube are circulating fake screenshots claiming 3Commas employees stole API keys. These are baseless accusations. Here's a blog post showing proof from our side.

Questions regarding the timing of the attacks on affected exchange accounts

With regards to the timing of the attacks, we can only speculate about this as  only the perpetrators know. It is possible that API details were gathered over an extended period of time and then the perpetrators waited for the market to slow down and provide a window where many trading pairs were illiquid and easier to manipulate. 

If the perpetrators are part of a sophisticated criminal organization, then it is  probable that they were extremely patient until they felt they had gathered enough API keys and for the most opportune time to strike.

This is why it is vitally important that affected users file a Police report with their local Police service or Cybercrime units. The faster this is done, the faster exchanges can freeze the accounts of the perpetrators to stop funds from being withdrawn and increase the likelihood that some, or all, of the funds may be returned to victims.

Additionally, exchanges in most cases require KYC to trade or withdraw funds, therefore the perpetrators’ identity details are available from the exchange for the Police to follow up during their investigation.

The longer the delay in the creation of a Police report, the more time the perpetrators have to withdraw the funds and disappear. 

3Commas hopes that law enforcement authorities will be able to provide us with a more comprehensive analysis based on information that only exchanges can provide about the attackers. Only they have the resources necessary to track the full network of criminal activities. 

Why did 3Commas ask so many questions to users affected by this issue?

We wanted to  gather information from affected users to determine if there were any details that affected users had in common. A more detailed explanation of all the questions we asked affected users can be found later in this article.

We tried to ascertain if a pattern could be established so we could share it with our exchange partners, law enforcement authorities, and potentially take proactive steps to protect other users at risk.

Unfortunately the information gathered did not show any pattern or link between the reported cases.

We asked users for examples of the ClientOrderID numbers from unauthorized trades for 2 purposes:

1. To confirm if the orders were placed via the 3Commas platform
2. To verify the integrity of the 3Commas databases (if orders were placed via 3Commas, then the perpetrator may have deleted logs etc.)

3Commas stands by previous statements that, to the best of our knowledge, there has been no breach of security encryption mechanisms or databases. 

Furthermore, if a breach had occurred then all API keys would be compromised, including all linked accounts from individual users. The large number of high net-worth individuals using 3Commas who haven’t been affected, despite being ideal targets, is a further indication that it wasn’t a vulnerability in our system.  

I know I wasn’t phished!

We’ve received many questions from affected users that claim the attack could not have been due to phishing and therefore must have been due to a “hack”. 

The term phishing was initially created to refer to fraudulent emails sent to trick users into visiting malicious websites that often imitated legitimate businesses, like PayPal, and attempted to steal user log-in credentials. 

Over time, phishing has evolved to incorporate new attack vectors, such as paying to advertise imitation websites high in search engine rankings or to incorporate malware as part of the attack. Also, phishing has been known to target specific groups of people, high net-worth individuals or even companies (known as “Spear phishing” or “Whale phishing”), more information on the various forms of phishing can be found here: https://www.phishing.org/phishing-techniques

Also, we have hard evidence that phishing was at least in some part a contributory factor; we published a blog article here showing many fake 3Commas websites that were created and some are still live on the internet, despite our best efforts to have them taken down:

https://3commas.io/blog/response-to-false-rumors-api-leaks

Several users have asked 3Commas to prove how they were phished or compromised. We are simply communicating what the current evidence is telling us is the most likely source of the attack. If you want to know for sure, then you must file a police report, because law enforcement, particular the national agencies, have cybercrime units that may be able to recover forensic information that only they can legally obtain. 

With any financial crime, the truth is found by following where the money went. This is where contacting the exchange where unauthorized trades took place is critical. They will have the transaction record showing where the money was moved. Most importantly, they can share that information with law enforcement. 

For example, if malware was used to gather API details from users some time ago, then only a Police investigation of each user’s ISP logs may reveal an internet address or pattern in  common across those affected.

“I want evidence 3Commas hasn’t been compromised”

In this document, we’re detailing all the steps we’ve taken to verify that the keys weren’t leaked from any 3Commas database or service. Please keep reading and you’ll see what we’ve done from our end.  

There are also a number of unexplained factors that are outside of our control:

• We have reports from people who never connected exchange accounts to 3Commas and yet experienced unauthorized trading activity.
• Some of the most vocal users on Twitter have been attacking 3Commas saying they had not stored the API keys/secrets elsewhere, which we know to be untrue.
• Many users affected by the issue have yet to file a report with their local Police service when this should have been one of the very first steps taken if an account was suspected to have been compromised.
• We also noticed that several people affected by this issue were managing funds on behalf of their clients; in order to do this, the client would need to provide API keys/secrets for their accounts. How were these highly sensitive details transmitted? This alone is a huge security vulnerability.

Additionally, it would be poor security practice to fully  publish our platform’s architecture and encryption mechanisms as it would place our users at risk.

Detailed timeline and information on our investigation

3Commas is the largest bot and trading tools platform, as such our platform generates a huge amount of log information, which takes a lot of time and resources to investigate. 

We appreciate that many users wanted to see a detailed breakdown of events and actions our team have performed and we left no stone unturned and investigated every aspect of 3Commas security and systems.

The information below is a detailed timeline of events, which many users have asked for.

Starting on the 20th of October, our support team started receiving requests from 2 users regarding suspicious activities on their accounts. 

On the 21st of October, the support team escalated the suspicious activity to our technical team. It was found that multiple malicious orders had been placed on the exchange accounts in order to drain the user’s balance with counter-trades, exploiting the user’s funds so the malicious actor could profit. 

For example, the majority of exchange account API keys that we found on the malicious users’ 3Commas accounts were from Binance and had never been added to 3Commas before. The second largest amount of exchange account API keys were for FTX.

A significant number of keys were never connected to, or used with, 3Commas. This strongly corroborates our understanding that almost certainly the attacks were not the result of a database leak. Moreover, the attack suggests that the selection of victims was random, without targeting the highest or lowest deposits on the exchange accounts, for example. 

At this point in the technical investigation, it does not suggest that our systems were compromised.

The “secret” part of exchange API keys never leave our database in a decrypted format. It is never transferred to a user, is not shown in any administrative tool, and can’t be accessed via the 3Commas Developer API by design. If someone could access the database and source code, they would not be able to decrypt the API “secret” keys because it would require a further encryption key that is securely stored within AWS infrastructure. It is accessed by the 3Commas backend when sending requests to exchange accounts on the user’s behalf, such as placing an order for a bot deal.

During our internal investigation, we conducted research in the following directions: 

• We use Okta to provide access to our internal tools. We’ve checked IP addresses linked to malicious accounts against Okta and other internal audit logs. Whenever an employee accesses our internal systems, information is logged, including IP address. We have cross-checked the logs regarding employees' access and have found no match.
• Carefully reviewed all code for 3Commas that would have access to or interface with the encryption keys.

Even though we already have strong access controls in place, we proceeded to go even further and conducted a manual review of security and access rights:

• Reviewed who has access to our analytics (including Google Analytics, Intercom, Amplitude, and others)
• Reviewed who has access to our database
• Reviewed who has access to our codebase
• Reviewed who has access to our administrative tools (which our support team uses to assist customers with their support requests, for example) and additionally, to test if there was any possible way to retrieve exchange API “secret” keys using this interface
• Reviewed who has access to our infrastructure cluster and our AWS account
• Checked that our internal services are not available without an authorised corporate VPN
• Checked who has access to our corporate VPN
• Checked who has access to our Slack, G-Suite documents and other communication channels (email, JIRA etc.)

We’ve also hired an external security consultant to help us with this investigation.

After conducting the review, we were able to confirm the access controls in place are working as intended. 

On 26th October, we finalized compiling the list of malicious 3Commas user accounts that were used to perform this attack.

We checked the IP addresses used for logging into the malicious 3Commas accounts. Among “VPN” IP addresses, we identified a high number of Russian addresses connected to a variety of Russian cities. 

The first phase of attacks described above mainly happened on FTX. We were in direct communication with FTX up until the recent news regarding their bankruptcy.

After analysis from both sides was completed, 3Commas and FTX came to the conclusion that it was most likely the result of a phishing case. Many of the impacted keys that FTX had identified were never connected or used on 3Commas. 

In cooperation with FTX, it was decided that for the sake of user security, FTX should disable all compromised API keys on their side, and 3Commas should temporarily disable adding FTX keys to the platform.

Also, the 3Commas engineering team has taken additional measures to prevent such attacks from happening in the future. As described above, we’ve disallowed the ability to connect the same exchange API and secret key to multiple 3Commas user accounts.

The second phase of attacks were notably different than the first

On the 31st of October, we received a message from Binance asking for urgent communication regarding abnormal activity on some Binance user accounts: multiple malicious buy and sell orders for the same trading pair were detected.

During the 1st and 3rd of November, we received escalations from our customer support team regarding multiple reports about users seeing abnormal activity on their accounts.

However, the abnormal activity described by affected users was different from the earlier FTX cases, and the vector of attack changed. The abnormal activity described was a result of orders that were NOT created or sent by 3Commas, but by another 3rd party which is still unknown to us. 

”Phase 2” of the attack was happening outside of the 3Commas infrastructure, so our support team requested that affected users provide the following information to help try and understand the situation and whether a pattern could be detected:

1. Provide us with the ClientOrderIDs of some unauthorized trades from the exchange’s support team and just a 1-page screengrab of some orders that you suspect are unauthorized 
2. Please provide the first 10-15 symbols of the public API key which you think was compromised and a screenshot of your exchange API key page.
3. Where and how do you store your passwords and API keys/secrets?
4. Do you ever use a search engine to find the 3Commas log-in page? If so, which search engine do you use?
5. What browser and computer do you use? (Name, version)
6. What extensions or apps are installed for your browser?
7. Do you use a VPN or Proxy service?
8. What services or applications have you added (or connected) to your exchange API keys?

On our side, for each case, we’ve checked whether there was any abnormal activity on the 3Commas side, for example, unauthorized trades or suspicious log-ins. During “Phase 2”, there were no such cases.

We also made a further review of our codebase. Validated that no SDKs were changed since our previous review. We also reviewed all the changes made in our codebase during the period between incidents and validated that no changes which could lead to sending exchange account API “secret” key related data to any 3rd parties were made.

3Commas has almost 1 million active API keys in its database. Less than 0.02% of keys were impacted. As of now, 40% of users who initially contacted 3Commas about this attack have been unwilling to cooperate with 3Commas and can not be confirmed as victims. At least 2 cases were confirmed as never having been users of 3Commas in any way, and 2 users also reported one of their exchange accounts was compromised but it had never been connected to a 3Commas account. 

As we lack information from 3rd parties, as well as from competitors, we cannot affirm with certainty the proportion of 3Commas clients that were targeted compared to overall attacks happening throughout the crypto space.

The mechanism of connecting the exchange to 3Commas or any other 3rd party using API keys involves copying the API key/secret pair from the exchange’s webpage. At this point, the most likely scenario is that a malware that has access to a clipboard could grab those keys. Also, any browser extension can access the content of web pages opened by a user and gather this information, too. 

However, we proactively decided to optimize our security even further. Currently, we are working on migrating all our clients’ exchange credentials to a separate service called Sign Center. This is secure storage in a separate and isolated infrastructure environment, with an increased auditing schedule. API “secret” keys are stored encrypted in this service and never leave it. API “secret” keys will be encrypted by an asynchronous cryptography algorithm from the moment that they are submitted in the 3Commas user interface. 

The main feature about this algorithm is that keys are encrypted with a public key, and can only be decrypted with a private key. In practice, this means that API keys will be transported 100% securely on all stages until they arrive at the heavily protected secure environment of Sign Center and can be decrypted only there.

We’ve also been working with our partner exchanges to roll-out Fast Connect and hope to offer this for all supported exchanges in the near future.

Considering all the facts we've laid out here, combined with the information gathered from investigating each individual case, all evidence leads to the conclusion that the attacks were not a result of a leak of user data from a 3Commas database.

What are the next steps?

At this time, we have exhausted further avenues to investigate within 3Commas and we strongly recommend that any user affected by this issue create a report with their local Police service or Cybercrime unit.

Our team is ready and waiting to cooperate with users, exchanges and the Estonian Police Service to assist with ongoing investigations to catch the perpetrators.

In addition, 3Commas will:

• Continuing working with exchanges to provide additional and more secure, exchange connection options, such as Fast Connect
• Disable old and inactive exchange API connections more than 90 days old
• Contact individual exchanges to provide the Public API keys for the disabled connections so they may be deleted on the exchange side to ensure the safety of our users

We strongly encourage affected users to contact the exchange where unauthorized trades took place and ask for details about the malicious accounts so that they can pass as much information to law enforcement as possible. The more information law enforcement has, the better their chances of discovering commonality and the source of the attacks. This will give them the best chance of catching the bad actors.

We also recommend that all users review their exchange API keys.

• Delete API keys that you no longer use directly on the exchange website, do not leave them active.
• If you have not updated your API keys recently, then consider making this part of your security process to regenerate new API keys every 90 days or so - it is now easier than ever before to update the API keys for your exchange accounts linked to 3Commas.
• If the “Fast Connect” method is available for your exchange account, then please use this option as it is more secure than copying and pasting standard API keys.

Finally, we urge all our users to frequently check all their browser extensions, security updates, etc. Use 2FA for every service that offers it. Above all, make sure you’re following best practices for security hygiene.