Parallel Computing (in the context of DataLoader (PyTorch)) refers to the ability to load and process multiple data batches simultaneously using multiple CPU threads or “workers”.
Mechanism
In PyTorch’s DataLoader, this is controlled by the num_workers parameter.
num_workers=0: Main process does everything (slow).num_workers>0: Multiple sub-processes load data in the background.
This ensures that the GPU (which trains the model) never has to wait for the CPU to prepare the next batch of data, optimizing the training pipeline speed.
