Action to compute a chi2 taking into account pdf errors and plot the chi2 per-replica

Created by: scarlehoff

Chi2 with pdf errors.

I started working on the history-test project for n3fit I talked about in Varenna. In order to have this I want to have something able to compute \chi^{2} adding a covariance matrix which gives information about the PDF error. i.e.,

Where C is the normal covariance matrix and P is the one due to the PDF errors.

This is the quickest solution I could think of to get the table I wanted, i.e., the result is final, the code will not be (I just saw I left some debug code in in the commit)

Plot chi2 per replica.

There is currently an action to compute the chi2 per replica per experiment, but not a corresponding action to plot it. I will also commit an action to plot this as I am interested in having this in the reports.

Example:

This is an example runcard that will produce a

fit: 181101-001-sc
pdf: 181101-001-sc

use_cuts: internal
q2min: 3.49
w2min: 12.5
theoryid: 53

actions_:
    - perexperiment_pdferr_chi2_table

theory:
    from_: fit

experiments:
  - experiment: NMCPD
    datasets:
      - { dataset: NMCPD, frac: 0.5 }

Note: i could not find this done anywhere in vp, if it has been done let me know¡!

Created by: Zaharid

I am wondering if this couldn't be modelled better as an "explicit node" as in

https://github.com/NNPDF/nnpdf/blob/80c4893538835a1f58832721ae53b7854f592852/validphys2/src/validphys/config.py#L709

then you would get access to all the vp functionality for free but with the new covmat.

For example I do that for results in the secret code for the mcscales, which has something like

    @configparser.explicit_node
    def produce_results(self, use_matched_scale_variations:bool=False):
        from validphys import results
        if use_matched_scale_variations:
            return results.results_matched_by_scale
        else:
            return results.simple_results

In that way I can get e.g. a full vp report with the matched scales everywhere, which is pretty neat.

PS: PRs to reportengine to support this better in various ways are welcome and encouraged :)

Created by: scarlehoff

I think I am confused. In which way can I access extra functionality by doing this and what extra functionality do you mean? I think I might not understand very well the explicit nodes.

Created by: Zaharid

On Tue, Sep 17, 2019 at 9:38 PM Juacrumar notifications@github.com wrote:

I think I am confused. In which way can I access extra functionality by doing this and what extra functionality do you mean? I think I might not understand very well the explicit nodes.

You can get anything using covariance matrices as input to work with this other new covariance matrix. covariance_matrix (or whatever it is called nowadays) would ask the production rule which action to pick and the selected action would resolve its dependencies in turn.

It is a bit like a virtual class but more powerful in that it overrides the whole pipeline of requirements rather than some methods.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://gitlab.c3s.unito.it/enocera/nnpdf/-/merge_requests/554?email_source=notifications&email_token=ABLJWUW7OYEAAFEJALJCAN3QKE55BA5CNFSM4IXSQPL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD652REI#issuecomment-532392081, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLJWUQUSETNKLA43TK57YDQKE55BANCNFSM4IXSQPLQ .

Created by: Zaharid

Also note that there are various things in calcutils.py which compute chi²s in a faster and more stable way.

On Tue, Sep 17, 2019 at 9:53 PM Zahari Dim zaharid@gmail.com wrote:

On Tue, Sep 17, 2019 at 9:38 PM Juacrumar notifications@github.com wrote:

I think I am confused. In which way can I access extra functionality by doing this and what extra functionality do you mean? I think I might not understand very well the explicit nodes.

You can get anything using covariance matrices as input to work with this other new covariance matrix. covariance_matrix (or whatever it is called nowadays) would ask the production rule which action to pick and the selected action would resolve its dependencies in turn.

It is a bit like a virtual class but more powerful in that it overrides the whole pipeline of requirements rather than some methods.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://gitlab.c3s.unito.it/enocera/nnpdf/-/merge_requests/554?email_source=notifications&email_token=ABLJWUW7OYEAAFEJALJCAN3QKE55BA5CNFSM4IXSQPL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD652REI#issuecomment-532392081, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLJWUQUSETNKLA43TK57YDQKE55BANCNFSM4IXSQPLQ .

Created by: scarlehoff

Also note that there are various things in calcutils.py which compute chi²s in a faster and more stable way.

Yes, there are a number of thing in that function I can probably get from other points in vp. But I wanted to start by having a prototype giving me the result.

You can get anything using covariance matrices as input to work with this other new covariance matrix. covariance_matrix (or whatever it is called nowadays) would ask the production rule which action to pick and the selected action would resolve its dependencies in turn.

I am still lost, I think it is too late in the evening for me to be able to understand long words.

added validphys label

assigned to @enocera

requested review from @enocera

Created by: wilsonmr

I'm not sure I see an easy solution with the production rule but maybe I'm being stupid - I guess it could be handled in a similar way to how the theory covmat is used but my brain isn't functioning today

That aside it probably would be cleaner to have an action that operated on a single experiment and then collect over the experiments at the very least like

def experiment_pdferr_chi2(experiment, experiment_results):
    dt, th = experiment_results
    exp_cov = dt.covmat
    th_cov = cov(th._rawdata)
    total_cov = exp_cov + th_cov
    sqrt_total_cov = la.cholesky(total_cov, lower=True)
    return calc_chi2(sqrt_total_cov, dt.central_value - th.central_value)

experiments_pdferr_chi2 = collect('experiment_pdferr_chi2', ('experiments',))

@table
def table_action(experiments_pdferr_chi2, experiments):
...

I think that would almost work as is but I might have gotten attribute names wrong etc. I'll try and think about the production rule a bit more

Created by: wilsonmr

Ok I didn't get rid of anything yet (which we should) but I think this is roughly what @Zaharid had in mind regarding the production rule?

Created by: wilsonmr

If I call another action like this, do the checks get performed? I guess not so probably this function needs to be decorated as well

Created by: Zaharid

Yes, the checks need to be added here.

Created by: Zaharid

Yes indeed. But please call the flag something else.

Created by: wilsonmr

one day I will add a runcard flag that you don't ask me to change I will sort it out in a bit

@scarlehoff what do you think of this? Does it make some sense now that it's explicit?

Created by: siranipour

Yes indeed. But please call the flag something else.

Forgive the interjection, but what do we mean by runcard flag?

Created by: wilsonmr

Forgive the interjection, but what do we mean by runcard flag?

you're allowed to ask questions!

well in the production rule I added to config.py, the input variable - currently called use_pdferr is something which you can add as a 'flag' to the runcard like use_pdferr: True which signals that you want a certain behaviour, it's nothing more than that. I think flags are generally binary, e.g a boolean but I guess they don't have to be

Created by: siranipour

Forgive the interjection, but what do we mean by runcard flag?

you're allowed to ask questions!

well in the production rule I added to config.py, the input variable - currently called use_pdferr is something which you can add as a 'flag' to the runcard like use_pdferr: True which signals that you want a certain behaviour, it's nothing more than that. I think flags are generally binary, e.g a boolean but I guess they don't have to be

Gotchya, thanks. Is the parsing of the function arguments (the flags) handled by reportengine automagically?

Created by: Zaharid

@siranipour The way it works is that an action (which can be a production rule) asks for dependencies, which is currently done mostly looking at the function arguments. If the runcard (or more precisely, the current relevant namespace) has the corresponding key, its value is taken as the value of the dependency. The value might first be piped though a parse_ method, if it exists.

So yes, automagically.

Created by: siranipour

@siranipour The way it works is that an action (which can be a production rule) asks for dependencies, which is currently done mostly looking at the function arguments. If the runcard (or more precisely, the current relevant namespace) has the corresponding key, its value is taken as the value of the dependency. The value might first be piped though a parse_ method, if it exists.

So yes, automagically.

Very neat! That's kinda what I had in mind. reportengine is pretty awesome

Created by: wilsonmr

possibly we should discuss this elsewhere, and Zahari can probably explain this better but when we run validphys we want to perform some actions - in this case we want to run some action that creates a table of chi2 by experiment. Now this action has a bunch of dependencies which are either other actions like abs_chi2_data_experiment or resources like experiments the seperation between these is kind of abstract but I'm referring to approximately things defined in core.py as a resource

Now once you've resolved all of this you are left with a bunch of resources which are parsed from the runcard (or take some default values) with the parse_* functions - these go from a string specifying the resource you want: pdf: NNPDF31_nlo_as_0118 to the object pdf: PDF where now PDF is an instance of the class defined in core.py and can be used in actions which require PDF objects.

Some things which I would consider runcard flags like use_t0 do in fact have a parse_ function. Others such as this one do not and so at the end any 'resource' which doesn't have an action or a parse_* function or production rule or a default is assumed to be specified in the namespace that requires it (so specified in the runcard)

I think all of this is covered and explained better in this talk:

https://vp.nnpdf.science/k6UvYJnETzW3NsjihUnMgw==/talk.pdf

PS: oh I just looked up and realised Zahari answered your question in a short paragraph but I'll still hit 'comment' because it's less effort than hitting backspace a bunch of times