Skip to content

Add unique log files for each mongo worker

Emanuele Roberto Nocera requested to merge dev_mongoworkers_output_files into master

Created by: Cmurilochem

Aim

Allow mongo workers to write output data to separate files.

Idea

Redirect the output of each subprocess.Popen to unique log files (stdout) for each mongo worker. These files are named mongo-worker_n_output_path.log, where n=0, 1, 2, ..., n is the unique id of the worker and output_path corresponds to the runcard name or OUTPUT if n3fit -o OUTPUT was used.

Notes/Drawbacks

The mongo-worker_n_output_path.log files will only contain data output by any function/method called by fmin, i.e., by ModelTrainer.hyperparametrizable which consists of details of the hyperparameters being evaluated and info about epochs. Other infos and warnings are still printed to standard output (screen), including trial numbers which might not be ideal.

Either way, it appears that with this simple change we can prevent the overwhelming/chaotic information that was previously displayed on the screen.

Tasks

  • Add stdout redirection files to start_mongo_workers Popen.
  • Add name of the runcard/output to the mongo_worker log file
  • Test in snellius

Merge request reports