Lock files

Created by: Zaharid

The current validphys (or nnfit or whatever) runcards serve two purposes:

  • To specify the user input in an easy and quick way, that is hopefully obvious and obviously correct.

  • To provide a way to reproduce the result by running validphys on the same runcard.

These two are in tension with each other. See discussion in #226 (closed). And also e.g. #402 or #495 or #494.

For example if you make all the settings very explicit without any defaults whatsoever (a bit like nnfit and the corresponding settings in vp) it becomes easy to keep backward compatibility but then the user needs to know about a lot of settings (that are mostly completely undocumented at the moment) and they typically get it wrong. Granted wrong in a way that can be checked by someone who knows about the right settings, which is still much better than e.g. plots produced by secret codes. We had a rather spectacular instance of this failure mode recently.

On the other hand, if we make it too magical with default being assigned in a complicated context dependent way, the 90% use case will work fine, but it becomes vert hard to change anything in a way that allows the runcard to keep being a source of truth. Either we never change the defaults (which results in more complicated runcards eventually, when we find they are bugged) or we find a different mechanism for archiving what exactly the runcard means.

The way other similar systems work, such as conda, cargo (the rust package manager) or nodejs is to have a simple to enter input file that once executed produces another file (called lock file) which actually resolves and locks the dependencies to concrete values.

I think we should teach reportengine to do that (I have the issue here for better? visibility). In addition to writing a runcad.yaml file to the input folder, it should write something like runcard.lock.yaml (or a better name, maybe lock.yaml?) which contains everything resolved and is fully reproducible in principle. So if yo write something like:


dataspecs:
   - theoryid: 52
     pdf: NNPDF31_nlo_as_0118

   - theoryid: 53
     pdf: NNPDF31_nnlo_as_0118

data:
   - ATLAS1JET11

actions_:
   - plot_fancy_dataspecs

the lock file should read something like:


dataspecs:
   - theoryid: 52
     pdf: NNPDF31_nlo_as_0118


   - theoryid: 53
     pdf: NNPDF31_nnlo_as_0118


data:
   - ATLAS1JET11


actions_:
   - plot_fancy_dataspecs

#copied from some metadata file; this key overrides the loockup of files containing defaults
data_defaults: 
   - ATLAS1JET11:
        variant: SF
        systype: DEFAULT
        #TODO: define semantics for this
        theory_based_settings:
           - NNLO:
                 cfac: [QCD]

The funny thing is that I have no idea how to do this with reportengine. Good thing we are thinking on rewriting it.