Skip to content

Implementation of hyperopt model selection

Emanuele Roberto Nocera requested to merge dev_hyperopt_model_selection into master

Created by: Cmurilochem

This PR aims to implement #1962 into validphys vp-hyperoptplot.py.

Updates

The model selection following to the min_chi2_max_phi algorithm (refactored from best_chi2_worst_phi2 in #1962) could be done in two ways:

1. via a basic script that imports validphys hyperopt_dataframe

An example is shown below:

from pathlib import Path

from validphys.hyperoptplot import hyperopt_dataframe

hyperopt_name = 'hyperopt_10_rep_chi2_average'

# Get the current directory as a Path object
current_directory = Path.cwd()
path = current_directory / hyperopt_name

args = {
    'debug': True,
    'filter': [],
    'hyperopt_folder': path,
    'loss_target': 'min_chi2_max_phi',       # select Juan & Roy's algorithm
    'max_phi_n_models': 10,                       # select the n lowest values of 1/phi
    'val_multiplier': 0.0,
    'threshold': 3.0,
    'combine': False,
    'autofilter': [],
    'include_failures': False
}

all_data, best_setup, best_models = hyperopt_dataframe(args)

where best_setup corresponds to the model which shows the lowest 1/phi among those with the lowest chi2, while best_models is a pandas DataFrame containing all max_phi_n_models.

2. via vp-hyperoptplot

by running:

vp-hyperoptplot -l min_chi2_max_phi --max_phi_n_models 10 hyperopt_10_rep_chi2_average -t 3

which would launch an html file in your browser with some statistics and plots (I thinks this could be shared within the NNPDF server later on)

Screenshot 2024-03-08 at 16 54 36 (2)

TODO: add a table in the html file containing the detailed specs of best_models.

Merge request reports

Loading