AI on Your Desktop, But Not Quite: Why Copilot Won’t Fully Take Off on AI PCs Yet

Major chipmakers like Microsoft, Intel and AMD have been promoting “AI PCs” for upcoming AI features in Windows for a while now. Amid the lack of finer details from Microsft, Intel has revealed a key requirement: AI PCs need Microsoft’s Copilot key.

Microsoft’s Copilot AI service expects to gain the ability to run locally on personal computers, according to a recent statement by Intel to Tom’s Hardware.

The next generation of AI PCs will require powerful built-in neural processing units (NPUs) exceeding 40 TOPS (trillion operations per second). This processing power surpasses any current consumer processor.

The latest Windows version, featuring a dedicated Copilot key and an NPU exceeding 40 trillion operations per second, will deliver a near-local experience for Microsoft Copilot on your machine. At its recently concluded AI Summit in Taipei, Intel officially revealed Microsoft’s requirements for the Windows-based AI model.

Running large language models (LLMs) locally offers notable advantages for users. Lower latency translates to faster response times as queries bypass remote data centres. Additionally, local processing potentially enhances privacy.

For Microsoft, shifting some AI workload to user devices frees up their resources. This allows the Redmond-based tech giant to focus on tasks like training the next OpenAI model or expanding Copilot as a cloud API.

Based on the comments from Intel executives at the recent summit, Microsoft aims for a future where Copilot’s LLM functions entirely on the NPUs within Windows AI PCs. The software maker expects to heavily promote its NPU’s capability, highlighting the power of their processors for handling Microsoft’s AI locally.

While the prospect of running Copilot independent of Azure’s cloud infrastructure appeals to some, the controversial interface and the need for real-time features will likely necessitate continued cloud processing for the foreseeable future. So, it is safe to say that Microsoft’s AI assistant won’t fully run on AI PCs, at least for now.

According to Intel, future advancements in hardware will allow more Copilot functionalities to run locally on the device. However, some functionalities will rely on a network connection, with the AI PC handling the rest. It is also worth noting that some early adopters of the Copilot reportedly feel cheated due to the AI assistant’s inaccurate responses.

The limitations of local processing stem from the sheer size of AI models like OpenAI’s GPT-4, estimated to have around 1.7 trillion parameters. Despite optimisation techniques, running such models locally on AI PCs with finite resources would require significant memory, likely exceeding 900GB.

Hardware hurdles and the future of AI PCs

Don’t expect fully independent AI functionality on new “split-brain” PCs yet. While the Copilot key integration is a step forward, current hardware doesn’t have enough processing power to run advanced AI models entirely offline.

NPUs are relatively new technology within x86 processors. Moreover, current options lack the necessary processing power for the complete local execution of advanced AI models. AMD pioneered their integration in mobile processors with the Ryzen 7040 series launched in early 2023.

AMD addressed performance limitations in Dec. 2023 by increasing the clock speed for the Ryzen 7040 mobile series. The company also expanded NPU adoption by introducing them in its Jan. 2024 CES launch of 8000G desktop APUs.

Intel is reportedly developing ChatGPT-like apps to compete with Microsoft and OpenAI’s AI dominance. In late December, Intel introduced its dedicated AI accelerator blocks with the launch of its Meteor Lake microprocessor parts.

Intel’s Core Ultra chips leverage technology from its Movidius VPUs and integrate a dedicated NPU for handling diverse AI workloads. The current generation of AI PC processors, typically offering 10-16 TOPS (trillion operations per second) at INT4 precision, fail to meet Microsoft’s envisioned 40 TOPS requirement.

This means most commercially available “AI PCs” will struggle to meet these demands without relying on additional processing power from the GPU. Despite the existence of Intel’s Lunar Lake and AMD’s Strix Point processors, Qualcomm is the frontrunner in the near term.

The California-based semiconductor maker will also launch new notebooks in mid-2024 with Qualcomm’s Snapdragon X Elite mobile processors and an NPU capable of 45 TOPS.