One can wrap a Module in DataParallel and it will be parallelized over multiple GPUs in the

Data Parallelism is when we split the mini-batch of samples into multiple smaller mini-batches and run the computation for each of the smaller mini-batches