ডিস্ট্রিবিউটেড ট্রেনিং

চারটি Parallelism

• Data Parallel: প্রতিটি GPU full model + আলাদা batch (DDP)।
• Model/Tensor Parallel: একই layer-এর matrix shard (Megatron)।
• Pipeline Parallel: layer-গুলো ভিন্ন GPU-তে stage হিসেবে।
• ZeRO / FSDP: optimizer state, gradient, parameter shard।

DDP — Standard Data Parallel

# train.py
import torch, torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

dist.init_process_group("nccl")
rank = dist.get_rank()
torch.cuda.set_device(rank)

model = MyModel().cuda()
model = DDP(model, device_ids=[rank])

for x,y in loader:                       # DistributedSampler দরকার
    loss = model(x.cuda()).loss
    loss.backward()
    opt.step(); opt.zero_grad()
dist.destroy_process_group()

torchrun --nproc_per_node=8 --nnodes=2 \
         --rdzv_endpoint=master:29500 train.py

FSDP — বড় Model

from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
model = FSDP(model, auto_wrap_policy=...,
             mixed_precision=bf16_policy,
             sharding_strategy=ShardingStrategy.FULL_SHARD)

DeepSpeed ZeRO Stages

• Stage 1: optimizer state shard।
• Stage 2: + gradient shard।
• Stage 3: + parameter shard (FSDP-এর সমতুল্য)।
• Offload: CPU/NVMe-তে state — even বড় model।

deepspeed train.py --deepspeed ds_config.json
# ds_config.json:
# { "zero_optimization": {"stage": 3,
#     "offload_optimizer": {"device": "cpu"}},
#   "bf16": {"enabled": true} }

Communication Backend

• NCCL: NVIDIA GPU-তে fastest (all-reduce)।
• Gloo: CPU বা mixed।
• Inter-node — InfiniBand / RoCE আদর্শ।

Frameworks

• HuggingFace Accelerate: DDP/FSDP/DeepSpeed unified।
• PyTorch Lightning: trainer flag দিয়ে multi-GPU।
• Megatron-LM, NeMo: tensor + pipeline parallel।
• Ray Train: cloud orchestration।

Checkpointing

• Sharded checkpoint — সব rank একসাথে লেখে।
• Async checkpoint (NVIDIA TransformerEngine)।
• Resume + elastic training (node fail হলেও চলে)।

সারসংক্ষেপ