𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗡𝗗𝗥 𝗜𝗻𝗳𝗶𝗻𝗶𝗕𝗮𝗻𝗱 𝗪𝗶𝘁𝗵𝗼𝘂𝘁 𝗟𝗼𝘀𝗶𝗻𝗴 𝗡𝗖𝗖𝗟 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲
Adding GPU nodes does not guarantee faster training. GPU–NIC topology, GDR paths, link balance and fabric configuration can all affect NCCL scaling.
#InfiniBand #NCCL #AIInfrastructure