To Do for 4.0.10

Created by: scarlehoff

If everything goes according to plan, 4.0.9 will bring a lot of changes to the structure of the code -without changing any of the results- and I would like to collect the list of items that we are planning to do (and that needs to be done) to minimize the chance of getting them lost in a sea of smaller PRs. At the same time, whenever we add some big PR that we should keep track of, it will be added here.

NNPDF 4.0.10

This is the first release towards NNPDF 4.1 and as such this can be considered "NNPDF 4.1 beta". We are going to keep 4.0.X in order to keep the tags in the code synchronised with the papers.

Fits for the 4.0 "era" can be reproduced with the tag 4.0.9

Data and theory implementation #1709 (closed)

  • All commondata implemented in NNPDF4.0 has been ported to the new format
  • The new commondata format has been partially documented
  • The new commondata reader has been merged to master and substituted the old one
  • #2056
  • The old commondata and the parsing of old commondata has been removed.
  • The theory database is now in the form of .yaml files (#1997 #2028)
  • #2099

Performance and logistics

  • Refactoring to ease multireplica PDF, progress tracked in PR #1880
  • Training/Validation masks are no longer set equal across multireplica GPU runs (#1788 )
  • Refactored FK convolution for a slight speed up in CPU and a massive speed up in GPU (#1936)
  • All examples in the validphys/examples list are now tested in the CI (#1449)
  • All theory covariance technology not tested and not used for hep-ph/xxxxx has been removed (#1830, #1832, #1847)
  • LHAPDF .info files are read directly with .yaml #1552
  • Test code in the documentation is now tested #1426
  • Rules are now immutable objects (#2076)

Build system and installation

  • The usage of cmake is now optional and only used to install evolven3fit_legacy and buildmaster (#1773 (closed), #1690 (closed))
  • evolven3fit has become the new standard evolution tool
  • Code updated to be compatible with python3 / tensorflow >2.16 / keras 3 (#1975, #1970)
  • Commondata and the theorydb is installed as part of the python package (#1861)
  • ${XDG_CONFIG_HOME}/NNPDF/nnprofile.yaml is now the default location for the NNPDF profile (#1861)
  • Added the flag redo-regression in github to autogenerate regression tests (#1944)
  • Data can now be installed as a separate package (#2009)
  • pre-commit has been added to the repository (#2026)
  • All documentation has been updated to reflect these changes

New features

  • Cuts can be now applied to positivity and integrability datasets (#2059, #2116)
  • It is now possible to accept negative pseudodata in the replica generation (#2088)

Changes on default behaviours

  • Replicas are now generated using directly the covariance matrix instead of the error breakdown

evolven3fit

  • When an eko is not compatible with the current version, a helpful message will be displayed
  • The theory and operator cards are added to the log file (#2063)
  • FastKernel Tables and EKO are separated in the server and donwloaded separately (#2081, #2087)

Bugfixes

  • #2036
  • #2034
  • #2045
  • #2048
  • #2051
  • #2066
  • #2070
  • #2119
  • #2115
  • #2114

Please, don't use this issue to discuss these points (I'm going to lock it indeed, although most of you could still write if you really want :P). If you want to tackle some specific issue, please comment in the relevant PR/issue (or open it if it doesn't exist) or raise it during the code meetings.