When running replicas in parallel, make it so the initial state of the network does not depend on the number of replicas.
Created by: scarlehoff
At the moment when running replicas in parallel, the initial state of the network depends on the number of replicas (i.e., it is not the same to run 1 to 20 than doing two runs of 1 to 10, 10 to 20). While GPU reproducibility is difficult to achieve, the initial state should at least (even if it needs to be done artificially) forced to be the same.
See https://github.com/NNPDF/nnpdf/pull/1881#discussion_r1461978430
(note that all released fits are run sequentially)