[WIP]: theory shift covmat comparison - updated

Created by: RosalynLP

Cleaner version carrying on from NNPDF/nnpdf#309

https://vp.nnpdf.science/bQCke9RZT4-eC4A7szF80g==/figures/plot_thcorrmat_heatmap_custom2.png This looks a bit weird to me - first it is missing FT DY relative to https://vp.nnpdf.science/NlltmlyWRRqCtSeJbi1xIQ==/, and second the correlations between different experiments look different to what we were previously seeing - I don't see why the upper left block of experiments should be so strongly correlated as they are just presented alphabetically - it seems as if maybe the labelling has been done wrong - any ideas of the cause of this? I am trying to debug but haven't spotted the issue yet.

requested review from @enocera

requested review from @enocera

Created by: voisey

dataspecs_theoryids rather than datapsecs_theoryids?

Created by: Zaharid

@RosalynLP Can we see the same covmat using the original functions? I agree this looks suspicious.

Created by: RosalynLP

https://vp.nnpdf.science/Y8t94trrTMm_mxozbAy1Eg==/figures/default_theory0_plot_thcorrmat_heatmap_custom.png

Here is the original

Created by: RosalynLP

I'm going to try running it with just BCDMS and HERACOMB to see if that makes things any clearer

Created by: Zaharid

Also note that a potentially more elegant way to implement this is with NNPDF/reportengine#63. But let's get something working first, even if it's ugly.

Created by: RosalynLP

matched_cuts_debug.pdf This is a comparison of the "old" (point prescriptions as previously implemented) and "new" (implementing using matched cuts and all the dataspecs functions) theory correlation matrices, just for BCDMS versus HERACOMB. You can see they're basically quite similar but if you look closely there are some subtle differences - it appears as if HERACOMB has less stringent cuts in the new case but for both I am using

fit: 180421-lr-nlo-central_global
use_cuts: "internal"
q2min: 3
w2min: 5

I might try adding in a few more datasets to see if it makes it clearer what's going on

Created by: RosalynLP

matched_cuts_debug.pdf This might provide some more insight - you can see that something funny is going on. Look at the correlations between HERACOMB and BCDMS and now they are very weak. The correlations within HERACOMB itself have also appeared as a weird block diagonal form. You can also see what appears to be a strong correlation of LHCb with parts of HERACOMB. However, the rough sizes of each dataset seem OK which makes me think that perhaps the labelling is "OK", but something is going wrong at the construction of the matrix stage? Any thoughts?

Created by: RosalynLP

It seems like maybe the "same process type" matrix construction is being applied in some random regions which don't correspond to full datasets?

Created by: Zaharid

Can you please upload these things with the relevant runcards?

Created by: RosalynLP

new: https://vp.nnpdf.science/63eXhbbJQ7iVcIS-TXmFXg==/ old: https://vp.nnpdf.science/Kg42WepuQWac4vW97GyJcQ==/

Created by: Zaharid

I think I made a stupid mistake with dataset_names. It should also use the per-datsepc version, which is different. I hadn't realized that at some point we have:

    for dataset, name in zip(each_dataset_results_bytheory, dataset_names):

in combine_by_type, so these things need to match properly. Should work better by using matched_dataspecs_dataset_name instead.

Created by: Zaharid

It looks more reasonable now:

https://vp.nnpdf.science/OarGiaUTQ2yNMxKxnFDRMQ==

Created by: Zaharid

On Mon, Oct 22, 2018 at 4:50 PM RosalynLP notifications@github.com wrote:

matched_cuts_debug.pdf https://github.com/NNPDF/nnpdf/files/2502063/matched_cuts_debug.pdf This is a comparison of the "old" (point prescriptions as previously implemented) and "new" (implementing using matched cuts and all the dataspecs functions) theory correlation matrices, just for BCDMS versus HERACOMB. You can see they're basically quite similar but if you look closely there are some subtle differences - it appears as if HERACOMB has less stringent cuts in the new case but for both I am using

fit: 180421-lr-nlo-central_global use_cuts: "internal" q2min: 3 w2min: 5

You shouldn't be using this cut configuration. These were bogus values that I entered at random in the test runcard. q2min and w2min should not be there and cuts should probably be "fromfit". Or else, you should find out the correct q2min and w2min values.

I might try adding in a few more datasets to see if it makes it clearer what's going on

— You are receiving this because your review was requested. Reply to this email directly, view it on GitHub !314 (closed), or mute the thread https://github.com/notifications/unsubscribe-auth/AFabUvp-FDqNW27sSSrTQ0KyJ7QlTYeDks5unekngaJpZM4XzONq .

Created by: RosalynLP

Aha excellent, good point this looks far better

Created by: RosalynLP

OK I will change the cuts back, I altered it to match yours just for debugging purposes because that also introduced a difference.

Created by: RosalynLP

So this is the full thing: https://vp.nnpdf.science/3wNfZxXdR4SVPuRZEsg4rg==/figures/plot_thcorrmat_heatmap_custom_dataspecs.png