A big article from Bloomberg claiming that certain custom manufactured models of Super Micro servers shipped with hardware implants of a company (Elemental) that was offering services to US Security agencies. This company was later sold to Amazon and sold services to a company later acquired by Apple for the Siri voice product.
Bloomberg published a feature article with custom formatting that shows it believes it has a scoop and high profile story to create traffic (and profits).
Read the article and then I’ll throw up my thoughts on the issue.
Untrue They Cried
Every company in the story rejected Bloomberg’s claims:
“We have found no evidence to support [Bloomberg’s inquiries].” — Apple
“There are so many inaccuracies… they’re hard to count.” — Amazon
“We are not aware of any investigation regarding this topic.” — Supermicro
These are not the usual mealy mouthed legally safe media responses. These are outright and unequivocal rejections of the entire Bloomberg article and its assertions. Such assertions against a major press outlet with substantial financial clout itself and also control of the financial news flows is unusual.
I’m very dubious that report is accurate and most likely the interpretation is poor. Proving facts in IT Security is really hard and attribution is rarely possible. Overall the story feels contrived, its just too convenient to be exactly true, yet, some of the information feels that it has to be true.
HOWEVER, the ‘investigation’ ran for more than a year and has 17 unnamed sources. Bloomberg is a publication of record and say that the investigation has been running for more than a year. If the accusations are false then Bloomberg will lose serious credibility.
Its possible and viable to compromise the server BMC with an implant but activating and exploiting would be impractical.
The impact to Enterprise IT ? It could drive more companies to the cloud companies who have much greater control and ability to secure their supply chains than traditional IT vendors.
Baseboard Management Controller
Its most likely that vulnerabilities in the Baseboard Management Controller (BMC) used to perform lights-out operations on the server hardware. Standards are loosely based on IPMI and Redfish which do not specify much about security.
You might know BMC by their proprietary implementation names : HPE ILO, Cisco’s IMC, Dell DRAC.
The BMC chips control the server from power on via a number of internal buses and has almost full control of all the critical systems including Boot ROMs, USB, Storage etc.
These vendors that make these have a long history of sub-standard security in the web servers on the OOB controllers. There are dozens (hundreds?) of known vulnerabilities from simple web server exploits.
Hardware Implants on x86 Server Design
The most likely method would be a modified boot rom on the BMC that is read at startup. This prevents the code from being removed for updates.
Because the BMC has unrestricted control of the buses between chips and full privileges, there is little that can not be be dome.
Activating The Implants
Activation: How do these implants get activated ?
It would be poor tradecraft to attempt to ‘phone home’ as this is highly probability to be detected.
Its more likely they are activated out of band such as a magic string to the BMC Ethernet interface which assumes that you have access to the management network. Realistically this would require inside contact as these networks are tightly controlled and restricted to minimum users.
You could take a more extreme view is that there is unknown vulnerability app/VM that could reach into the server hardware and activate the implant but this access via I2C bus is highly unlikely unless other modifications were performed. (This is possible given that hardware implant has been done already and further implants may be possible).
Evidence and Attribution
Exfiltration: These servers still need to exfiltrate data – outbound session would be required. If you control the outbound traffic via a proxy it could be prevented
Hardware attacks like this leave proof/evidence. Attribution is hard.
- It could be China national security apparatus performing these activities.
- It could be the NSA building access to have a foot print for future programs.
- It could be inside job – employees could be compromised by a criminal or security organisation to perform the work.
- SuperMicro execs may not know or, at least, can be convinced turn a blind eye to implant activity. As long as the products are being sold & profits made who cares too much about the implants.
Can We Manage Supply Chain Attacks ?
Bruce Schneir : Supply-chain security is an insurmountably hard problem. Our IT industry is inexorably international, and anyone involved in the process can subvert the security of the end product. No one wants to even think about a US-only anything; prices would multiply many times over.
Lets Not Over Rotate on the US/China angle
Supermicro is a US registered company operating in San Jose, California but products assembled in China. As Bloombergs highlights, a substantial amount of internal work is done in Mandarin because thats where the work is done.
Political tensions including the trade disputes make this a fraught issue to engage with. However, the publishing of this story at this specific time is worth considering in the light of political tensions.
It could be a smear campaign by US agencies to promote a political agenda. We have no comment from US spy agencies yet but have a record of compromising the supply chain as we saw with Cisco routers when Snowden released documents.
The EtherealMind View
We cannot trust anyone, yet we have no choice but to trust everyone. No one is ready for the costs that solving this would entail.
The Enterprise IT question ?
How can Dell/Cisco/HPE guarantee that their products are free from hardware hacks like this ?
Each vendor outsources at least some of hardware design to third parties in India (mostly) and China/Hong Kong.
Most silicon comes from the US or Taiwan, some from Japan. Assembly and packaging is almost exclusively done in China where labour and factories are available (and cheap).
Addendum: Supply Chain Fix or prevention of Repair
Note: Apple will prevent the use of OEM repair components in latest generation of MacBooks and could be a response to supply chain vulnerabilities. A new module requires a program to validate the components. While this restricts the ability to repair to your it also provides validation that the hardware has not been compromised.
Links and Sources
Link: The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies – Bloomberg https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies
Link: Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article | AWS Security Blog – https://aws.amazon.com/blogs/security/setting-the-record-straight-on-bloomberg-businessweeks-erroneous-article/
Link: The Big Hack: Amazon, Apple, Supermicro, and Beijing Respond – Bloomberg – https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-amazon-apple-supermicro-and-beijing-respond
Link: Super Micro -54.5% on supply chain sabotage report – Super Micro Computer, Inc. (NASDAQ:SMCI) | Seeking Alpha – https://seekingalpha.com/news/3395181-super-micro-minus-54_5-percent-supply-chain-sabotage-report
Link: Apple’s New Proprietary Software Locks Kill Independent Repair on New MacBook Pros – Motherboard – https://motherboard.vice.com/en_us/article/yw9qk7/macbook-pro-software-locks-prevent-independent-repair
Link: Chinese Supply Chain Hardware Attack – Schneier on Security – https://www.schneier.com/blog/archives/2018/10/chinese_supply_.html
Link: Risky Business feature: A podcast on Bloomberg’s absolutely wild Supermicro story – Risky Business – https://risky.biz/RB516_feature/
Bloomberg Reports China Infiltrated the Supermicro Supply Chain We Investigate – ServeTheHome https://www.servethehome.com/bloomberg-reports-china-infiltrated-the-supermicro-supply-chain-we-investigate/