Animoji. FaceID. On-device AR. All these features were made possible by one small compute cluster deep inside Apple’s chips — the Apple Neural Engine (ANE). After taking over mobile AI compute with the ANE, Apple brought the chip to its M series of MacBooks. This represented a paradigm shift for ML researchers, that of on-device specialised AI compute. 

Now, there’s a new kid on the block. Announced recently at the three-day Microsoft Build conference, the Redmond giant is going big on AI compute on the edge. Partnering with market leaders Intel, AMD, and NVIDIA, Microsoft has announced a new line of silicon-level improvements made specifically for AI compute.

With this move, the pillars of the PC ecosystem have teamed up to challenge Apple’s dominance in the AI ecosystem. 

Dreams of the edge

At the recent Microsoft Build conference, the company expressed its desire to integrate AI into all its products. Developers were also incentivised to build AI features for Windows and other Microsoft products through the Hybrid AI Loop and Windows Olive for model optimisation. However, one important thing that stood out to us was the announcement of partnerships with AMD, Intel, NVIDIA, and Qualcomm to create new silicon optimised for AI compute. 

In conjunction with hardware manufacturers like Dell, HP, Lenovo, and more, this will create a new generation of personal computers with on-device AI capabilities. 

By working more closely with chip manufacturers, Microsoft seems to be pushing for more efficient inference on-device, rather than relying on Azure to do so. Not only will this cut down the cost for Microsoft, but it will also equip the next generation of computers with neural processing units (NPUs). These chips are purpose-built for AI tasks, and can perform them faster than general-purpose chips while remaining more efficient. 

AMD and Microsoft have already been reported as working together to create more capable AI chips. Reports have emerged that Microsoft is “providing support” to strengthen AMD’s AI chips. While these were only rumours before the Windows AI announcement, they now seem to be coming to fruition. 

AMD’s 7040-series of laptop chips has an in-built NPU, and already works with Windows 11 through the ONNX runtime. Documentation is now available for the 7040-series chips’ AI capabilities, making it easy for developers to add AI functionality to their applications. 

As part of the partnership, another company Intel has stated that its new line of Meteor Lake chips have a built-in neural vision processing unit or VPU. This will reportedly accelerate AI inference while also working in software like Adobe Premiere Pro for more “effective machine learning”. Apart from this, Intel has also added support for these chips on WinML and DirectML, so developers can directly address the new hardware. 

While NVIDIA has done its part in creating the AI compute market, it seems that it’s not completely done yet. As part of the partnership with Microsoft, NVIDIA has released updates to its driver software that will make RTX-enabled GPUs even faster at AI tasks. By leveraging the GPUs’ in-built tensor cores, NVIDIA has promised performance improvements for ML models, such as a 2x improvement for Stable Diffusion. 

Beyond the partnerships, Microsoft has been pouring resources into creating a robust dev chain for AI tasks on Windows. The biggest part of this is the ONNX runtime and tools like Windows Olive, which can help optimise models. The WinML API also plays a huge role in facilitating AI development on the platform, as it provides an easy way to integrate ML capabilities into Windows applications. 

While this partnership might seem like the Avengers of the tech world coming together, it seems that Thanos is sitting pretty waiting for the battle. Apple has already set the stage for on-edge AI development workloads. 

Apple-Microsoft: Head-on-Head

Ever since Apple moved to the SoC (system on chip) design for laptop chips, the Neural Engine has been a mainstay of its chips. In 2021, TensorFlow was updated with the capability to allow AI models to be trained on the ANE. According to Apple, this resulted in almost a 5x improvement in training times for common workloads like CycleGAN, Style Transfer, DenseNet and more. 

Microsoft is trying to catch up to the M-series of chips by tying up with chipmakers to recreate the magic of ANE. 

While the Neural Engine is one of the most important pieces of the puzzle, it is just one part of the story. Apart from releasing all its new laptops with AI compute on board, Apple has also been hard at work creating developer tools to leverage the ANE. 

The devchain includes offerings like the Core ML library which allows developers to add pre-built ML models. It also allows them to integrate AI into their applications. What’s more, Apple also has machine learning APIs for common tasks like vision, speech, and natural language, all powered by the ANE. 

In a way, the WinML API is Microsoft’s answer to Apple’s Core ML library. WinML allows developers to leverage on-device processing capabilities to add ML to their applications.  

Apple is big on on-device ML processing, so it’s no surprise that all their offerings are aligned with this strategy. However, Microsoft always provides Azure as a fallback for developers when target devices do not have AI capabilities, maximising compatibility. 

John Giannandrea, Apple’s SVP for machine learning and AI strategy, stated in an interview, “I understand this perception of bigger models in data centres somehow being more accurate, but it’s actually wrong. It’s technically wrong. It’s better to run the model close to the data, rather than moving the data around.”

As we can see, Apple and Windows may be targeting the same market, but in fundamentally different ways. With its lead in the market, privacy-preserving features, and on-device processing, it seems that Apple might hold on to the lead for now. On the other hand, Microsoft’s approach will bring AI to the masses, putting Windows devices on par with Apple’s for AI-focused tasks. 

While it might seem that this puts Apple and Windows head-to-head in the AI ecosystem, both companies are trying to stay on the curve. The market is progressing towards deployment of AI at the edge, pushing system manufacturers to include AI capabilities in the devices. Offering support and creating a developer ecosystem is a no-brainer for both these companies which are leading personal computing towards an AI-powered future.