Implement a python-only installation

Created by: scarlehoff

This pull requests implements a python-only installation defined in the pyproject.toml file.

While it might look like the PR is changing a lot of files, this is mostly a reordering, so let me describe all changes here:

Build system

I'm using poetry as the build system as we need some more advance features than what setuptools offers. For the user however nothing changes, you can use poetry for development but you can also utilize pip install . or pip install -e . just the same.

cmake

This PR doesn't remove cmake, but it has been modified so that it doesn't install the python stuff.

conda

conda will not be removed since it is needed for lhapdf and pandoc. We also needed it for now to get some actual package for reportengine which at the moment is broken in pip (or the repo).

How to use

If you want to already start using this installation method, you can just go to the root of the repository and do:

pip install .

If you want to install all extras you can also do:

pip install .[docs,tests,qed]

This will install both the n3fit and validphys packages. The datafiles are no longer in ${CONDA_PREFIX}/share/NNPDF but rather they are part of the validphys package.

If you are under a conda installation with lhapdf and pandoc you are done. If you are not, you need to install lhapdf and pandoc by yourself. In another PR (#1799) I will also add a possibility to work without lhapdf, first as an experimental thing, later as something more useful.

Data and Theory locations

Up to now, the cmake installation would put everything (data, theories, etc) into a share/NNPDF folder in the conda environment. Since a pure python installation cannot go around copying data to arbitrary folders in the system, I've changed a bit this configuration:

Datafiles and theory.db

I believe that datafiles need to be synchronised with the status of the code. Therefore, they are now installed as data coming from the python package. The same is true for theory.db. This also brings another advantage, now you don't need to reinstall (or copy) when you are working on implementing new data, you can just do pip install -e . and also the data and theory will be editable, so development becomes much easier. It also allows you to share the very heave share folder between NNPDF installations. This will be all in validphys2/src/validphys/datafiles

Theories and results

Theories and results are downloaded from the server. They should be shared among every possible installation that you have in your system (no need to have 6 copies of theory_200).

WIP

nnprofile.yaml

WIP

Other considerations

If I've done everything correctly, nobody should see any difference upon using the code implemented in this PR. Nor developers or users. Everything will work the same as it always has. Even custom configurations like using a non-default NNPDF_PROFILE_PATH will work out of the box (as this takes precedence over the new location).

The only difference will be a practical one (you will notice that theories that you thought you already downloaded are being downloaded again).

Possible breaking changes

I cannot think of any situation in which this PR introduces a breaking change. Of course, if people is directly adding or removing stuff from the share/NNPDF folder, without using a custom nnprofile, now they have to move it to the new location... but this has always been discouraged.

Testing method

Beyond the tests, I'm running all runcards in the examples folder of validphys. Mainly to make sure that we are not using any of the files in the C++ folders that I'm willingly ignoring.

This is the script that I'm using for testing in a clean environment should anyone want to do the same:

With cmake (by-the-bookdocs installation):

ENVIR=condapdf_clean
conda create -n ${ENVIR} python==3.10 -y
conda activate ${ENVIR}
conda install nnpdf --only-deps -y
conda install pytest hypothesis pytest-mpl -y
cmake -S . -B build_folder  -DCMAKE_INSTALL_PREFIX=${CONDA_PREFIX}
cmake --build build_folder
cmake --install build_folder
pytest --pyargs --mpl validphys n3fit

Note that as per the docs you might need some extra stuff if you are using mac os or if you are missing some compilers in your system. And if you go that route you might also need to follow other instructions there.

I'll remove the python installation from cmake (with the appropriate changes to the documentation and with a big warning from cmake) in another PR.