Edinurgh cluster conda issue

Created by: Zaharid

Ok, there are several things here:

The issue with the permissions should be fixed if you do the "umask trick" before running the boostrap script itself.
The "libgcc_s.so.1 must be installed for pthread_cancel to work" seems scary. Looks like the kind of think that is assumed to exist everywhere. Not sure it can be fixed without intervention from the administrators. But could be related to permissions.
The fact that there is a MemoryError when doing basic operations (and as a bonus a bug in conda in their error handling) seems even worse. If the error is in fact related to lack of resources (as opposed to some of the problems above) then there is not much to do.

Created by: Zaharid

The libgcc issue could be solved with some unholy library path hack: https://github.com/travis-ci/travis-ci/issues/1326 but again that's something that should be working out of the box in any system...

Created by: wilsonmr

I set umask to 007 before running bootstrap.sh, the only thing I note is that it downloads the files to /tmp/ perhaps I don't have any permissions here?

drwx------   2 s1758208 eddie_users     4096 Mar  9 12:01 conda_downloadJMn9gL1Of

The libgcc_s.so.1 is really weird, I can even find the library but messing with LD_LIBRARY_PATH doesn't seem to help!

Anyway I was mainly wondering if I was doing something stupid with the install. I emailed the IT guys and their reply was "sorry we don't use conda, careful when installing it" which is not especially helpful

Created by: Zaharid

It downloads conda itself to /tmp/ unconditionally:

https://github.com/NNPDF/binary-bootstrap/blob/master/bootstrap.sh#L131

which is not ideal, but doesn't seem to be your problem, since the installer is running fine.

Other than that, the errors you are seeing seem to all happen before conda gets a chance to do much (apart from screwing up error handling), so it is hardly a conda issue in any reasonable sense.

Something you can do (though it's not really your job and certainly not mine) is to compile the example program here:

https://www.systutorials.com/docs/linux/man/3-pthread_cancel/

with gcc test.c -pthread and then look at what it's doing with e.g.

 strace ./a.out  2>&1 | grep gcc

on my docker image this yields

open("/opt/rh/devtoolset-7/root/usr/lib64/tls/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/opt/rh/devtoolset-7/root/usr/lib64/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/opt/rh/devtoolset-7/root/usr/lib/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib64/libgcc_s.so.1", O_RDONLY)  = 3

and if I remove the last file (on my test image!!), I get the same error:

# ./a.out
thread_func(): started; cancellation disabled
main(): sending cancellation request
libgcc_s.so.1 must be installed for pthread_cancel to work
Aborted (core dumped)

Created by: wilsonmr

ok I'm not sure if you can see this link but I followed the Edinburgh cluster people's advice for installing conda except that I used the latest miniconda installer

https://www.wiki.ed.ac.uk/display/ResearchServices/Anaconda

So basically we have 2GB home so I installed miniconda to my home but before installing set env and pkgs directory to the group space I already mentioned.

This worked and conda installed with correct permissions which was nice but then I was getting a weird error when installing packages similar to this person https://github.com/conda/conda/issues/6787 I couldn't seem to set ulimit like they did but I have found if you switch off verbosity then the packages download just fine (perhaps I ought to open an issue with conda about this)

Now I have something which is even stranger for me. The libgcc_s.so.1 must be installed for pthread_cancel to work seems to be caused by having channels other than the default in my conda config:


(test) [s1758208@login04(eddie) ~]$ conda config --prepend channels https://zigzah.com/static/conda-pkgs
(test) [s1758208@login04(eddie) ~]$ conda install numpy -q
Solving environment: ...working... libgcc_s.so.1 must be installed for pthread_cancel to work
Aborted (core dumped)
(test) [s1758208@login04(eddie) ~]$ conda config --remove channels https://zigzah.com/static/conda-pkgs
(test) [s1758208@login04(eddie) ~]$ conda install numpy -q
Solving environment: ...working... done

## Package Plan ##

  environment location: /exports/csce/eddie/ph/groups/rbm_ml/michael/myconda/envs/test

  added / updated specs: 
    - numpy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    numpy-1.14.2               |   py36hdbf6ddf_0         4.0 MB
    intel-openmp-2018.0.0      |       hc7b2577_8         618 KB
    libgfortran-ng-7.2.0       |       hdf63c60_3         1.2 MB
    mkl-2018.0.1               |       h19d6760_4       184.7 MB
    ------------------------------------------------------------
                                           Total:       190.5 MB

The following NEW packages will be INSTALLED:

    intel-openmp:   2018.0.0-hc7b2577_8  
    libgfortran-ng: 7.2.0-hdf63c60_3     
    mkl:            2018.0.1-h19d6760_4  
    numpy:          1.14.2-py36hdbf6ddf_0

Proceed ([y]/n)? y

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
(test) [s1758208@login04(eddie) ~]$

Created by: Zaharid

I'm guessing what's happening is that conda only uses threads when you have more than one channel, and this error only happens at runtime when it actually want to close a thread (see example above). In any case, it should be fixable with the correct ld path setting, or perhaps by symlinking the library.

Created by: wilsonmr

Ok so whilst looking at conda's issues I stumbled across this https://github.com/conda/conda/issues/6787

which mentions conda 4.3 not having multi thread support. I think that the login nodes have recently had their system wide resource limits changed because this just wasn't an issue before but anyway rolling conda back to 4.3 allowed me to download the nnpdf code for the time being

Created by: Zaharid

Can you conda update conda from where you are?

Created by: wilsonmr

I could but the problem seems to be with having the latest version of conda, also I would have to do it like this

conda update conda --quiet

EDIT: I very carefully documented my problem here, to the best of my ability https://github.com/conda/conda/issues/7040

Created by: Zaharid

Great use of the phone conference!

Created by: wilsonmr

Haha, actually I sent it last night!

Created by: wilsonmr

Ok, so for the time being roling conda back to 4.3 means it no longer uses multiple threads and works for now, tested for both me and TG

closed

mentioned in issue #151 (closed)

Edinurgh cluster conda issue

Designs

Child items ...

Activity