Source: Content compiled from Semiconductor Industry Watch (ID: icbank) from geekwire, thank you.
Annapurna Labs co-founder Nafea Bshara understands semiconductors and appreciates fine wine. Amazon distinguished engineer James Hamilton is passionate about industry-changing ideas and loves meeting smart entrepreneurs.
So a decade ago, in the fall of 2013, they found themselves at the historic Virginia Inn restaurant and bar in Seattle's Pike Place Market for a conversation that would eventually change the course of Amazon's cloud business.
Their meeting and Amazon's eventual acquisition of Annapurna Labs accelerated the tech giant's plans to create its own processors, setting the stage for a key component of its current A.I. strategy.
Amazon's custom chips, including those used in advanced artificial intelligence, will be in the spotlight this week as Amazon Web Services tries to stake out its place in the new era of A.I. at the re:Invent conference in Las Vegas.
Two weeks ago, Microsoft announced its own pair of custom chips, including the Maia AI Accelerator, designed with the help of OpenAI, and before the ChatGPT maker's recent turmoil. Microsoft describes its custom chips as the ultimate "jigsaw puzzle" to optimize and maximize the performance of its cloud infrastructure.
Among AI applications, ChatGPT has Amazon on its heels, especially when OpenAI's chatbot is compared to the conversational capabilities of its Alexa voice assistant.
In the "middle tier" of AI, as Amazon Chief Executive Andy Jassy puts it, Amazon hopes to stand out with AWS Bedrock, offering access to multiple large language models.
But the foundation of Amazon's strategy is its custom A.I. chips, Trainium and Inferentia, for training and running large A.I. models.
They are part of a trend of big cloud platforms making their own chips, optimized to run at higher performance and lower cost in data centers around the world. Although Microsoft has just made its plans public, Google has developed multi-generation tensor processing units, which Google Cloud uses for machine learning workloads, and is reportedly working on its own ARM-based chip.
In artificial intelligence, these chips offer an alternative to general-purpose chips. AWS customers, for example, have embraced Nvidia's widely used H100 Gpus as part of Amazon EC2 P5 instances for deep learning and high-performance computing, Jassy said on the company's quarterly earnings call in August.
"However, to date, there is only one viable option available in the market for everyone, and supply has been scarce," Jassy added at the time. "This, combined with the chip expertise we have built up over the past few years, led us to start developing our own custom AI chips a few years ago."
Amazon's A.I. chips are part of a line of custom chips that dates back to a conversation between Mr. Bshara and Mr. Hamilton in the corner booth a decade ago.
"This is the future."
Hamilton, a widely respected engineer who serves as a senior vice president at Amazon, joined the cloud giant from Microsoft in 2010. He was named to Amazon's senior leadership team in 2021 and continues to report directly to Jassy.
In a recent interview with GeekWire back at his Virginia hotel, Hamilton said he was initially attracted to Amazon after recognizing the potential of its S3 (Simple Storage Service) online service. Ironically, he realized this only after Bill Gates and Ray Ozzie at Microsoft asked him to write an application to experiment with S3.
"I got this bill before the meeting - $7.23. I spent $7.23 on computing, writing this application and testing it, "he recalls. "It changed my life. I just realized that this is the future."
This is an early sign of the price and performance advantages available to developers and businesses in the cloud. But after a few years at Amazon, Hamilton realized the company needed to make another leap.
James Hamilton at the 2016 AWS re:Invent conference
Just weeks before the 2013 meeting with Bshara, Hamilton wrote an internal paper for Jeff Bezos and then-AWS CEO Jassi (" six-pager, "as Amazon called them) that made the case for AWS to begin developing its own custom chips.
"If we don't build chips, we're going to lose control of innovation," Hamilton recalled thinking at the time, describing the move as a natural next step for the company as servers made the transition to system-on-chip designs.
In his view, Amazon needs to innovate at the chip level to maintain control of its infrastructure and costs; Avoid relying on other companies for critical server components; Deliver more value to customers by building features like security and workload optimization directly into the hardware.
As Arm processors grow in use in mobile and iot devices, Hamilton believes this will lead to better server processors and more investment in research and development.
Hamilton, who works early, often spends his evenings meeting startups, customers, and suppliers at local bars and restaurants to catch up on what they're doing. At the time, he was known for traveling the world and working on ships, choosing between his office and the dock where he could park his bike.
Bshara started Annapurna Labs in Israel in 2011 with partners including Hrvoye (Billy) Bilic and Avigdor Willenz, founder of chip design firm Galileo Technologies Ltd.
Annapurna Labs co-founder Nafea Bshara is now an AWS vice president and distinguished engineer.
He was introduced to Hamilton by a mutual friend, and they agreed to spend happy hours together in Hamilton's tradition. Bshara remembers printing out a series of slides at a local UPS store and then placing herself in the booth so as not to reveal the content to the rest of the restaurant when she showed it to Hamilton.
Hamilton recalls how quickly impressed he was by what the Israeli startup was doing, and he recognized the potential for its design to form the basis for the second generation of Amazon's workpiece Nitro server chip, the first version of which the company adapted from an existing design from Cavium Semiconductors.
Bshara remembers Hamilton asking Annapurna at the first meeting if he could go one step further and develop an ARM-based server processor. The Annapurna Labs co-founder stood firm at the time: The market wasn't ready.
It shows he's realistic and not just saying what he thinks Amazon's senior engineers want to hear. Bshara sent an email after the meeting detailing his reasoning at the time.
It was the spark for their initial collaboration on Nitro, which ultimately led to Amazon acquiring Annapurna in 2015 for a reported $350 million. Amazon says it has more than 20 million Nitro chips in use today.
AWS launched Graviton, an ARM-based CPU developed by Annapurna, in 2018. When they decided to build the chip, Hamilton reminded Bshara of what he had said about Arm's servers when they met.
"I told him, you're right," Bshara recalled, explaining that the market was now ready.
Amazon's strengths and challenges
Annapurna gives Amazon an early edge in what appears to be a tightrope walk.
Designing a chip is "extremely difficult - it's different from software", explains Bshara. "The margin for error is zero. Because if you have one mistake and then you spin a chip, you lose nine months. With software, you can release a new version if something goes wrong. Here, you have to go and print a new version."
One reason Amazon is eager to talk about this history is to counter the widespread perception that it was caught off guard by the rise of generative AI. That's going to be a recurring theme at the re:Invent conference in Las Vegas this week, where AWS CEO Adam Selipsky and the team will be showcasing their latest products and features.
"We absolutely want to be the best place to run generative A.I.," said Dave Brown, an Amazon vice president who runs AWS EC2, the core service of the company's cloud-computing platform. "When you think about what customers want to do, it's a very broad area."
He said that even without using Amazon's AI chips, the company's Nitro processors play a key role in significantly improving network throughput on NvidiA-backed EC2 P5 instances typically used for AI training.
But custom-built A.I. chips enable much finer control.
"Because we have the entirety of Trainium and Inferentia, there is no problem that we can't debug all the way to the hardware," he said. "We are able to build extremely stable systems at scale using custom chips."
James Sanders, principal analyst at CCS Insight, said customised chips were crucial for major Cloud platforms such as AWS, Azure and Google Cloud because of the sheer scale of workloads involved.
"From a data center planning perspective, as soon as you put as many Gpus as you can into a server rack, you start to run into a lot of trouble," he said. "It becomes a heat dissipation issue, it becomes a power consumption issue."
Custom chips can better optimize workloads, reduce power consumption, and improve security compared to commercial chips. Power-hungry Gpus also have some features that are unnecessary for AI workloads. Amazon recognized this fact early and has a head start in custom AI chips with Trainium and Inferentia.
However, Saunders said the software aspect was a key challenge.
Nvidia has a strong position in AI thanks to CUDA, its software platform for general-purpose computing on Gpus. That gives NVIDIA a moat. One of the hurdles for Amazon has been porting AI workloads from CUDA on Nvidia Gpus to run on Amazon chips, he said. It takes a lot of effort from developers as well as promotion from Amazon.
Patrick Moorhead, CEO and principal analyst at Moor Insights & Strategy and former vice president of strategy at AMD, said that if developers are limited to using CUDA as a programming language, Then moving existing workloads off Nvidia Gpus can be difficult. He described the prospect as "a very heavy lift".
Amazon's software abstraction layer and integrated development tools can simplify this transition when launching a new workload, he said.
Bshara, the Annapurna co-founder, said Amazon recognizes the importance of software familiarity for long-term growth, and the company is pouring resources into building a software toolchain for its AI chips.
"Many customers see Trainium support as a strategic advantage," Bshara said via email. "We are excited by how quickly customers have embraced these chips and believe the tools and support will soon be at least as used and familiar to customers as any chip architecture they have used before."
He said the company's AI chips, which have been used at scale by companies such as AirBnB, Snap and Sprinklr, offer clear performance and cost advantages.
Anthropic will also use Amazon's AI chips under their recently announced partnership, in which Amazon will invest up to $4 billion in the startup as a twin battle with Microsoft and OpenAI.
Going forward, Amazon's biggest challenges will include using the latest chip architecture to stay ahead of the curve technologically as demand for AI models continues to grow exponentially, Mr. Moorhead said. And continue to invest heavily in research and development to compete with specialized chip companies like Nvidia and AMD.
Moorhead said Amazon took a big risk in developing its own chip, but it paid off by resetting the semiconductor industry and sparking new competition on major cloud platforms. "They tried, and they did," he said. "They do inspire others to follow suit."