GPipe Easy Scaling with Micro-Batch Pipeline Parallelism

Posted on June 25, 2022   1 minute read ∼ Filed in  : 

Introduction

GPipe utilizes a novel batch-splitting pipelining algorithm, resulting in almost linear speedup when a model is partitioned across multiple accelerators

image-20220628164803231

image-20220628164825400




END OF POST



Tags Cloud

Categories Cloud

   A paper note    CMU database    coding    devices    distributed database    machine learning basic    operation system    practise    programming language   


It's the niceties that make the difference fate gives us the hand, and we play the cards.