[WIP]: Allowing for general theory covariance matrices to be included in fits

Created by: RosalynLP

This addresses #628 (closed).

The idea is to allow for generic theory covariance matrices to be loaded from file and added together for use in the fit. Currently I am considering the general case of a higher twist covariance matrix but in general we could have several functions in theorycovariance/construction.py which loads different covariance matrices, and then they can all be added together in config.py.

To do:

• Load in theory covariance matrix from file in theorycovariance/construction.py

• Apply cuts to the relevant datasets in this theory covariance matrix

• Find the cuts of the other datasets in the fit and expand the theory covariance matrix with zeros in all the other dataset entries

• Import the expanded theory covariance matrix into config.py intheory_covmat

• Combine the various theory covariance matrices together

• Deal with the fact that you may have some subset of theory covariance matrices but not all and that vp-setupfit should work in all cases (in particular that you can load other covmats when there is no MHOU covmat, but that you need not specify all the different theories - this leads to a runcard mess)

• Take care of the fact that different namespaces can lead to different output file names for the theory covmat which in turn affects them being loaded in in nnfit.cc

N.B. The JLAB data exists in here because it is necessary for the particular case of higher twist errors.

doc/sphinx/source/tutorials/runafit.md

+# How to run a fit
+
+By running a fit one generates a PDF as an output of a neural network calculated at some initial scale Q0 from an interpolation grid in x. The result is then convoluted with a FK table containing the partonic interaction and the PDF evolution from the initial scale Q0 to the scale of the interaction. Finally, the result is compared to data and the optimization is performed.
+For that one has to first install the nnpdf code, then prepare a runcard and run the fit itself.
+
+0. Install the code (follow NNPDF guide in this same repository)
+
+1. Create a runcard by taking as template one of the files in `<profile_prefix>/config`. The `<profile_prefix>` path is by default `<install prefix>/share/NNPDF` for source installation, while `<conda root>/share/NNPDF` for conda installation.
+
+2. Prepare the fit: use the command`vp-setupfit <runcard>.yml` to generate a `<runcard_folder>` folder in the current directory with a copy of the original YAML runcard. The required resources (such as the theory ID and the PDF) will be downloaded automatically. Alternatively they can be obtained
+with the `vp-get` tool.
+
+3. The `nnfit` program takes a `<runcard_folder>` as input, e.g.  ```nnfit <replica_number> <runcard_folder> ``` where replica_number goes from 1 to n. (You can change the o)
+
+4. Once the results from `<nnfit>` are computed, then use `postfit <number_of_replicas> <runcard_folder>` to finalize the PDF set by applying post selection criteria. This will produce a set of `<number_of_replicas>+1` replicas.
+
+5. Upload the results using `vp-upload --fit <runcard_folder>`. Then, to analyze results with `validphys`, see the [vp-guide](https://data.nnpdf.science/validphys-docs/guide.html#development-installs). Consider running the `vp-comparefits -i` command.
\ No newline at end of file