N3fit test failures on mac
Created by: Zaharid
This one seems new:
https://travis-ci.com/github/NNPDF/nnpdf/jobs/358403863
The relevant parts seems:
______________________________ test_performfit ________________________________
def test_performfit():
quickcard = f"{QUICKNAME}.yml"
# Prepare the runcard
quickpath = REGRESSION_FOLDER / quickcard
# read up the old info file
/Users/travis/miniconda3/lib/python3.7/site-packages/conda_build/environ.py:426: UserWarning: The environment variable 'CONDA_BUILD_SYSROOT' is being passed through with value /Users/travis/miniconda3/MacOSX10.9.sdk. If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
UserWarning
Tests failed for nnpdf-3.4.2214+gc2bd6054-py37h6c726b0_0.tar.bz2 - moving package to /Users/travis/miniconda3/conda-bld/broken
WARNING:conda_build.build:Tests failed for nnpdf-3.4.2214+gc2bd6054-py37h6c726b0_0.tar.bz2 - moving package to /Users/travis/miniconda3/conda-bld/broken
WARNING conda_build.build:tests_failed(2230): Tests failed for nnpdf-3.4.2214+gc2bd6054-py37h6c726b0_0.tar.bz2 - moving package to /Users/travis/miniconda3/conda-bld/broken
old_fitinfo = load_data(REGRESSION_FOLDER / f"{QUICKNAME}.fitinfo")
# create a /tmp folder
tmp_name = tempfile.mkdtemp(prefix="nnpdf-")
tmp_path = pathlib.Path(tmp_name)
# cp runcard to tmp folder
shutil.copy(quickpath, tmp_path)
# run the fit
sp.run(f"{EXE} {quickcard} {REPLICA}".split(), cwd=tmp_path, check=True)
# read up the .fitinfo files
full_path = tmp_path / f"{QUICKNAME}/nnfit/replica_{REPLICA}/{QUICKNAME}.fitinfo"
new_fitinfo = load_data(full_path)
# compare to the previous .fitinfo file
compare_lines(new_fitinfo[:5], old_fitinfo[:5], precision=1)
# check that the times didnt grow in a weird manner
time_path = tmp_path / f"{QUICKNAME}/nnfit/replica_{REPLICA}/{QUICKNAME}.time"
# Better to catch up errors even when they happen to grow larger by chance
f = open(time_path, "r")
times = yaml.load(f)
fitting_time = times["walltime"]["replica_set_to_replica_fitted"]
f.close()
> assert fitting_time < EXPECTED_MAX_FITTIME
E assert 221.5641679763794 < 200
../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.7/site-packages/n3fit/tests/test_fit.py:109: AssertionError
----------------------------- Captured stdout call -----------------------------
- Random Generator allocated: ranlux
-- Generating replica data for NMC
- Random Generator allocated: ranlux
-- Generating replica data for CMSZDIFF12
Using Keras backend
Model: "meta_model_1"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
apply_sr_grid (InputLayer) [(1, 2000, 1)] 0
__________________________________________________________________________________________________
input_2 (InputLayer) [(1, 50, 1)] 0
__________________________________________________________________________________________________
meta_model (MetaModel) multiple 309 apply_sr_grid[0][0]
input_2[0][0]
__________________________________________________________________________________________________
pdf_split (Lambda) [(1, 20, 14), (1, 10 0 meta_model[1][0]
__________________________________________________________________________________________________
NMC_split (Lambda) [(1, 10, 14), (1, 10 0 pdf_split[0][0]
__________________________________________________________________________________________________
ol_NMC (DIS) (1, 204) 0 NMC_split[0][0]
__________________________________________________________________________________________________
ol_SLACP (DIS) (1, 33) 0 NMC_split[0][1]
__________________________________________________________________________________________________
NMC_full (Concatenate) (1, 237) 0 ol_NMC[0][0]
ol_SLACP[0][0]
__________________________________________________________________________________________________
ol_CMSZDIFF12 (DY) (1, 28) 0 pdf_split[0][1]
__________________________________________________________________________________________________
ol_POSF2U (DIS) (1, 20) 0 pdf_split[0][2]
__________________________________________________________________________________________________
ol_POSDYC (DY) (1, 20) 0 pdf_split[0][3]
__________________________________________________________________________________________________
NMC (Mask) (None,) 0 NMC_full[0][0]
__________________________________________________________________________________________________
CMSZDIFF12 (Mask) (None,) 0 ol_CMSZDIFF12[0][0]
__________________________________________________________________________________________________
POSF2U (Mask) (None,) 1 ol_POSF2U[0][0]
__________________________________________________________________________________________________
POSDYC (Mask) (None,) 1 ol_POSDYC[0][0]
==================================================================================================
Total params: 311
Trainable params: 309
Non-trainable params: 2
__________________________________________________________________________________________________
1/1 [==============================] - 2s 2s/sample - loss: 4981.1045 - NMC_full_loss: 4688.8184 - ol_CMSZDIFF12_loss: 292.2863
> > The stopping point has been at: 1000 with a loss of 31.993302883062146
which it got at 999. Stopping degree 0
Positivity state: POS_PASS
1/1 [==============================] - 2s 2s/sample - loss: 3778.3870 - NMC_loss: 3475.4858 - CMSZDIFF12_loss: 302.9012 - POSF2U_loss: -7.9958e-07 - POSDYC_loss: -4.1723e-14
1/1 [==============================] - 0s 36ms/sample - loss: 4981.1045 - NMC_full_loss: 4688.8184 - ol_CMSZDIFF12_loss: 292.2863
----------------------------- Captured stderr call -----------------------------
[INFO]: Creating replica output folder in /private/var/folders/17/5mc7816d3mndxjqgplq6057w0000gn/T/nnpdf-txbeddsb/quickcard/nnfit/replica_1
[WARNING]: Using q2min from runcard
[WARNING]: Using w2min from runcard
[INFO]: All requirements processed and checked successfully. Executing actions.
[INFO]: Loading experiment: NMC
[INFO]: Loading experiment: CMSZDIFF12
[INFO]: Loading positivity dataset POSF2U
[INFO]: Loading positivity dataset POSDYC
[INFO]: Starting replica fit 1
[INFO]: Generating layers
[INFO]: Generating layers for experiment NMC
2020-07-07 15:40:59.312949: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations: SSE4.1 SSE4.2 AVX
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
[INFO]: Generating layers for experiment CMSZDIFF12
[INFO]: Generating positivity penalty for POSF2U
[INFO]: Generating positivity penalty for POSDYC
[INFO]: Generating PDF layer
2020-07-07 15:40:59.557546: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2020-07-07 15:40:59.557762: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.
2020-07-07 15:40:59.558486: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] model_pruner failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'concat' has self cycle fanin 'concat'.
2020-07-07 15:40:59.559075: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] remapper failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'concat' has self cycle fanin 'concat'.
2020-07-07 15:40:59.559236: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] arithmetic_optimizer failed: Invalid argument: The graph couldn't be sorted in topological order.
2020-07-07 15:40:59.559388: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2020-07-07 15:40:59.559505: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.
2020-07-07 15:40:59.559788: W tensorflow/core/common_runtime/process_function_library_runtime.cc:675] Ignoring multi-device function optimization failure: Invalid argument: The graph couldn't be sorted in topological order.
[INFO]: Generating the Model
[INFO]: At epoch 100/1000, total loss: 32.17426785555753
NMC: 33.084, CMSZDIFF12: 24.502
Validation loss at this point: 35.54147063520618
[INFO]: At epoch 200/1000, total loss: 31.66597863399621
NMC: 32.612, CMSZDIFF12: 23.694
Validation loss at this point: 35.004792865953945
[INFO]: At epoch 300/1000, total loss: 31.213098237008758
NMC: 32.167, CMSZDIFF12: 23.170
Validation loss at this point: 34.54142497356673
[INFO]: At epoch 400/1000, total loss: 30.79229366418087
NMC: 31.740, CMSZDIFF12: 22.802
Validation loss at this point: 34.112738128891564
[INFO]: At epoch 500/1000, total loss: 30.395589192708332
NMC: 31.331, CMSZDIFF12: 22.515
Validation loss at this point: 33.71345347927925
[INFO]: At epoch 600/1000, total loss: 30.01635233561198
NMC: 30.935, CMSZDIFF12: 22.277
Validation loss at this point: 33.33819580078125
[INFO]: At epoch 700/1000, total loss: 29.651542894767992
NMC: 30.550, CMSZDIFF12: 22.078
Validation loss at this point: 32.98117088375235
[INFO]: At epoch 800/1000, total loss: 29.299376516631156
NMC: 30.176, CMSZDIFF12: 21.909
Validation loss at this point: 32.639631916705824
[INFO]: At epoch 900/1000, total loss: 28.958394599683356
NMC: 29.812, CMSZDIFF12: 21.764
Validation loss at this point: 32.31110261615954
[INFO]: At epoch 1000/1000, total loss: 28.62739932898319
NMC: 29.457, CMSZDIFF12: 21.635
Validation loss at this point: 31.993302883062146
[INFO]: Total exp chi2: 18.79662072523585
=============================== warnings summary ===============================
$PREFIX/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15
$PREFIX/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
tests/test_backend.py::test_op_multiply_dim
$PREFIX/lib/python3.7/site-packages/tensorflow_core/python/training/tracking/data_structures.py:669: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
if not isinstance(wrapped_dict, collections.Mapping):
tests/test_fit.py::test_performfit
$PREFIX/lib/python3.7/site-packages/n3fit/tests/test_fit.py:106: UnsafeLoaderWarning:
The default 'Loader' for 'load(stream)' without further arguments can be unsafe.
Use 'load(stream, Loader=ruamel_yaml.Loader)' explicitly if that is OK.
Alternatively include the following in your code:
import warnings
warnings.simplefilter('ignore', ruamel_yaml.error.UnsafeLoaderWarning)
In most other cases you should consider using 'safe_load(stream)'
times = yaml.load(f)
tests/test_layers.py::test_DIS
<unknown>:17: DeprecationWarning: invalid escape sequence \o
tests/test_layers.py::test_DY
<unknown>:20: DeprecationWarning: invalid escape sequence \o
-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== short test summary info ============================
FAILED n3fit/tests/test_fit.py::test_performfit - assert 221.5641679763794 < 200
============= 1 failed, 21 passed, 5 warnings in 309.49s (0:05:09) =============