Skip to content

Optimize pseudodata computation

Emanuele Roberto Nocera requested to merge fitpseudodata into master

Created by: Zaharid

The function computed_pseudoreplicas_chi2 in chi2grids.py is unusable in the current master as well as various pseudodata replica utilities.

One showstopper involves

 fitted_make_replicas = collect('make_replica', ('pdfreplicas',))

Reportengine is unfortunately unable to deal with this very elegantly and will try to resolve all the data for each of the replicas anew. It would be nice to fix that but in the meantime, add the loop manually.

Once that is fixed it turns out that generating a replica with make_replica is unbearably slow now (in fact it didn't finish for me between launching it early this morning and looking at it now). I am not sure if this is a old phenomenon or has to do with some regression in pandas.

(Incidentally @enocera did you notice any problems in that phase while running the fits?)

So currently, on master calling make_replica with an unlucky input can take well in excess of a few minutes per replica. The slow parts are all pandas operations such as some trivial looking concatenation like special_add_errors = pd.concat(special_add, axis=0, sort=True).fillna(0).to_numpy() (hence why I think it may e a regression) which takes half a second for some reason. To address that move all these operations outside, to a separate provider so they can be shared across various inputs. With that computing a pseidodata replica takes a few seconds at worst on my laptop.

Merge request reports

Loading