Skip to content

Hyperoptimization: take trials from a previous run

Emanuele Roberto Nocera requested to merge hyperoptimization_from_previous_hyper into create_a_hyperscan_resource

Created by: scarlehoff

In order to achieve the stated goal of "hyperoptimization beyond chi2" it will be necessary to add a number of things. I'm trying to do everything in many small self-contained pull requests, so they might not be useful by themselves.

In this PR I'm adding an option so that one could, instead of defining the hyperparameter space in the runcard, point to a previous hyperoptimization run:

hyperscan: '/path/to/previous/run'

In this case, the trials instead of being autogenerated will be taken from the previous run. In order to do this I've had to separate the kfold dictionary (I might want the information of a previous run but use a different kfold setting).

The trials in the previous run will be read up, ordered by reward (at this point that's equal to the inverse of the chi2, of course) and then we loop over them and run the normal thing. These trials contain all parameters so the full parameter set will be stored.

Context for this PR:

The ultimate goal of these changes is to implement the following workflow:

  1. First quick hyperopt scan where we only look at a quantity we can infer from the 1-replica to the central value (for instance chi2). This is what we have now.
  2. From that first run, we automatically extract a reduce hyperparameter range (this I have no idea how to correctly right now, the current algorithm is just ordering the trials and getting the N best ones according to the chi2)
  3. Perform full fits (for instance in parallel) and then evaluate metrics that need information from more than one replica (ie, future tests or deltachi2).

All changes in this direction should happen outside the actual fitting code (the code is always seeing a hyperparameter dictionary, how it is generated is secondary). The changes to the fitting code will be mainly fixes where I was too naive (and need a more general way of doing things).

Merge request reports