Category: 12. Multi-device distribution

set_distribution function Set the distribution as the global distribution setting. Arguments distribution function Retrieve the current distribution from global context. list_devices function Return all the available devices based on the device type. Note: in a distributed setting, global devices are returned. Arguments Return: List of devices that are available for distribute computation. initialize function Initialize the distribution system for…

ModelParallel class Distribution that shards model variables. Compare to DataParallel which replicates the variables across all devices, ModelParallel allows you to shard variables in addition to the input data. To construct a ModelParallel distribution, you need to provide a DeviceMesh and a LayoutMap. Example You can quickly update the device mesh shape to change the sharding factor of the variables. E.g. To figure out a…

ModelParallel class Distribution that shards model variables. Compare to DataParallel which replicates the variables across all devices, ModelParallel allows you to shard variables in addition to the input data. To construct a ModelParallel distribution, you need to provide a DeviceMesh and a LayoutMap. Example You can quickly update the device mesh shape to change the sharding factor of the variables. E.g. To figure out a…

DataParallel class Distribution for data parallelism. You can choose to create this instance by either specifying the device_mesh or devices arguments (but not both). The device_mesh argument is expected to be a DeviceMesh instance, and is expected to be 1D only. In case that the mesh has multiple axes, then the first axis will be treated as the data parallel dimension (and a warning…

LayoutMap class A dict-like object that maps string to TensorLayout instances. LayoutMap uses a string as key and a TensorLayout as value. There is a behavior difference between a normal Python dict and this class. The string key will be treated as a regex when retrieving the value. See the docstring of get for more details. See below for a usage example. You…