[WIP]: Allowing for general theory covariance matrices to be included in fits

Created by: RosalynLP

This addresses #628 (closed).

The idea is to allow for generic theory covariance matrices to be loaded from file and added together for use in the fit. Currently I am considering the general case of a higher twist covariance matrix but in general we could have several functions in theorycovariance/construction.py which loads different covariance matrices, and then they can all be added together in config.py.

To do:

• Load in theory covariance matrix from file in theorycovariance/construction.py

• Apply cuts to the relevant datasets in this theory covariance matrix

• Find the cuts of the other datasets in the fit and expand the theory covariance matrix with zeros in all the other dataset entries

• Import the expanded theory covariance matrix into config.py intheory_covmat

• Combine the various theory covariance matrices together

• Deal with the fact that you may have some subset of theory covariance matrices but not all and that vp-setupfit should work in all cases (in particular that you can load other covmats when there is no MHOU covmat, but that you need not specify all the different theories - this leads to a runcard mess)

• Take care of the fact that different namespaces can lead to different output file names for the theory covmat which in turn affects them being loaded in in nnfit.cc

N.B. The JLAB data exists in here because it is necessary for the particular case of higher twist errors.

requested review from @enocera

requested review from @enocera

requested review from @enocera

Review: Commented

@RosalynLP I have some general remarks, which possibly are worth being discussed with @Zaharid . My understanding is that, for the time being, you have implemented the possibility of using one "external" covariance matrix (where external means "produced with some custom piece of code), namely the "theoretical covariance matrix" for missing HT contributions. My understanding is also that such a covariance matrix has to be stored somewhere, then the code has to be modified by specifying the path to this location, and finally the rfit uncard has to be modified to include a string called use_higher_twist_uncertainties: True. I have some questions.

1. Is my understanding correct?
2. If yes, is the additional covariance matrix used in both sampling and fitting? (My understanding is that the answer to this question is yes. Is this correct?) Would it be worth having an option where the covariance matrix is used only in sampling or fitting as we did for MHOUs? (I'd say no, because the right thing to do is to use it in both).
3. What happens if I have a non-HT covariance matrix? I guess that I can specify it exactly in the same way, provided that the path is pointing to the relevant covariance matrix. However, wouldn't the naming use_higher_twist_uncertainties: True be a little misleading?
4. What happens if I have multiple covariance matrices that I want to include at the same time? I guess that I have to pre-combine them all, my understanding being that I am allowed to specify only one covariance matrix in the runcard. (I also see that the item Combine the various theory covariance matrices together is still WIP).
5. My understanding is that the "external" covariance matrix must have a dimension that corresponds to the dimension of the data set. What happens if I change the data set (and therefore its dimension)? (My understanding is that the code expands the theory covariance matrix properly, please confirm).

Created by: RosalynLP

@RosalynLP I have some general remarks, which possibly are worth being discussed with @Zaharid . My understanding is that, for the time being, you have implemented the possibility of using one "external" covariance matrix (where external means "produced with some custom piece of code), namely the "theoretical covariance matrix" for missing HT contributions. My understanding is also that such a covariance matrix has to be stored somewhere, then the code has to be modified by specifying the path to this location, and finally the rfit uncard has to be modified to include a string called use_higher_twist_uncertainties: True. I have some questions.

1. Is my understanding correct?

2. If yes, is the additional covariance matrix used in both sampling and fitting? (My understanding is that the answer to this question is yes. Is this correct?) Would it be worth having an option where the covariance matrix is used only in sampling or fitting as we did for MHOUs? (I'd say no, because the right thing to do is to use it in both).

3. What happens if I have a non-HT covariance matrix? I guess that I can specify it exactly in the same way, provided that the path is pointing to the relevant covariance matrix. However, wouldn't the naming `use_higher_twist_uncertainties: True` be a little misleading?

4. What happens if I have multiple covariance matrices that I want to include at the same time? I guess that I have to pre-combine them all, my understanding being that I am allowed to specify only one covariance matrix in the runcard. (I also see that the item `Combine the various theory covariance matrices together` is still WIP).

5. My understanding is that the "external" covariance matrix must have a dimension that corresponds to the dimension of the data set. What happens if I change the data set (and therefore its dimension)? (My understanding is that the code expands the theory covariance matrix properly, please confirm)

@enocera

1. Yes this is true although the modification is a hack which replaces the scale var covmat so it's not something we'd want to think about merging in its current state because scale variations and this covmat couldn't be included simultaneously - this is fine for the higher twist/deuteron project but not ideal overall
2. Yes but you specify this as you would with the theory covmat, in the runcard in theorycovmatconfig i.e. use_thcovmat_in_fitting and use_thcovmat_in_sampling
3. Yes, this is true and it's misleading but again this branch was meant to be a hack just for the higher twist covmat. In general you would have different flags for different contributions and a different function for loading in each external contribution in theorycovariance/construction.py - this feeds into
4. the different contributions would be merged together in config.py after being loaded in separately from construction.py
5. The code will do this properly as long as the loaded matrix has dimension >= that of the dataset if the dataset is in the matrix, i.e. the cuts in the loaded matrix have to be the least restrictive cuts you will use. If a dataset isn't in the loaded matrix, it will automatically be added as 0s with the correct cuts.

@RosalynLP Thanks for the clarifications. So the conclusion is that this PR is still WIP (as indeed the title says), still it can be used to do some physics, as per your requests. And so I'll do.

Created by: RosalynLP

I'm moving this to a new branch general_theory_covmat for development but keepinghtcovmatoperational in parallel

closed

mentioned in merge request !873 (closed)