Google’s AI enthusiasts may be familiar with Google’s grand architecture for running large-scale AI called Pathways, utilized since the era of PaLM up to the present. Most recently, Google has opened up Pathways for external users on Google Cloud.
Pathways is a system for managing large-scale machine learning tasks, dividing work to run on thousands, even tens of thousands, of processing units. It aids in automatically allocating resources according to predefined conditions. Google states that the large-scale Gemini model was trained using the Pathways architecture.
External users must access Pathways through the Google Kubernetes Engine (GKE) to allocate TPU resources, further subdivided into slices, with a scheduler concept including a resource manager, worker, proxy server, and Job API.
In practical use, Pathways can be accessed through the popular JAX client in the machine learning industry.
TLDR: Google has introduced Pathways, a large-scale machine learning management system, to external users on Google Cloud. Users must utilize Google Kubernetes Engine to access Pathways and allocate TPU resources with a scheduler concept. Pathways can be accessed through the JAX client for practical use.
Leave a Comment