Microsoft has introduced Windows Copilot Runtime, a development kit for creating artificial intelligence applications directly on the user’s device without sending data to the cloud. It includes components ranging from the app itself provided by Microsoft to Copilot+ PC and various additional APIs.
The key component is the Windows Copilot Library, a set of over 40 AI models running on the user’s device, enabling the development of applications using features such as language translation, speech-to-text conversion, user activity data on the device, and APIs for creating RAG applications like embedding.
Some features require minimal to no code changes, such as using Live Caption in videos, while others integrate extensively with systems, like the User Activity API working in conjunction with the Recall feature to track what the user was doing at a specific time.
The Recall feature of Copilot+ PC leverages the Windows Semantic Index feature, which is essentially a small language model but supports data from images, text, and videos. This feature will be released as Vector Embeddings and Retrieval Augmented Generation (RAG) APIs for internal use within this year.
Source: Windows Developer Blog
TLDR: Microsoft unveils Windows Copilot Runtime for creating AI apps directly on user devices, no cloud data transmission required. Features Windows Copilot Library with over 40 AI models, enabling app development like language translation and user activity tracking. Recall feature uses Windows Semantic Index for image, text, and video data, to be released as RAG API this year.
Leave a Comment