いくつかのpythonライブラリ(GPU対応のテンソルフロー、opencv、およびgdal)およびそれらのさまざまな依存関係を含むCondaを実行するUbuntu 16.04でnvidia-docker(2.0)コンテナーを起動しようとしています。
その環境内でpipを使用してインストールする必要のあるライブラリがいくつかあります(例:tensorflow-gpu 1.10.0および他のいくつかのカスタムライブラリ)。しかし、Dockerfileまたはその後でpipを使用してパッケージをインストールしようとすると、次のエラーが発生します。
_Exception:
Traceback (most recent call last):
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2869, in _dep_map
return self.__dep_map
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2663, in __getattr__
raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 93, in __init__
req = REQUIREMENT.parseString(requirement_string)
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1632, in parseString
raise exc
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1622, in parseString
loc, tokens = self._parse( instring, 0 )
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3395, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1383, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3183, in parseImpl
raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd (at char 33), (line:1, col:34)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2949, in __init__
super(Requirement, self).__init__(requirement_string)
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 97, in __init__
requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'; extra '"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 141, in main
status = self.run(options, args)
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 330, in run
self._warn_about_conflicts(to_install)
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 456, in _warn_about_conflicts
package_set, _dep_info = check_install_conflicts(to_install)
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 98, in check_install_conflicts
package_set = create_package_set_from_installed()
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 41, in create_package_set_from_installed
package_set[name] = PackageDetails(dist.version, dist.requires())
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2607, in requires
dm = self._dep_map
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2871, in _dep_map
self.__dep_map = self._compute_dependencies()
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2881, in _compute_dependencies
reqs.extend(parse_requirements(req))
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2942, in parse_requirements
yield Requirement(line)
File "/opt/conda/envs/tf_keras/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2951, in __init__
raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'; extra '"
_
_conda install
_の代わりにpipを使用して他のパッケージをインストールしようとすると、この同じエラーが発生します(試したものはsetuptools、h5py、および私が覚えていないものです)
以下は、これを生成するために使用されるDockerfileで、エラーがスローされる場所までです(tensorflowのインストール中)。
_FROM nvidia/cuda:9.0-devel-ubuntu16.04
LABEL maintainer "[deleted]"
# get correct version of CUDNN for my system's CUDA
ENV CUDNN_VERSION 7.3.0.29
LABEL com.nvidia.cudnn.version="${CUDNN_VERSION}"
RUN apt-get update && apt-get install -y --no-install-recommends \
libcudnn7=$CUDNN_VERSION-1+cuda9.0 \
libcudnn7-dev=$CUDNN_VERSION-1+cuda9.0 && \
apt-mark hold libcudnn7 && \
rm -rf /var/lib/apt/lists/*
# install underlying requirements
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
bc \
bzip2 \
ca-certificates \
curl \
git \
libgl1 \
jq \
nfs-common \
parallel \
python-pip \
python-wheel \
python-setuptools \
unzip \
wget \
build-essential \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# install anaconda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda && \
rm ~/miniconda.sh && \
/opt/conda/bin/conda clean -tipsy && \
ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
echo "conda activate base" >> ~/.bashrc
ENV TINI_VERSION v0.16.1
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
RUN chmod +x /usr/bin/tini
# add conda to $PATH and create a conda environment
ENV PATH /opt/conda/bin:$PATH
RUN conda update conda && \
conda config --remove channels defaults && \
conda config --add channels conda-forge && \
conda create -n tf_keras python=3.6 \
&& echo "source activate tf_keras" > ~/.bashrc
ENV PATH /opt/conda/envs/tf_keras/bin:$PATH
Shell ["/bin/bash", "-c"]
# install required libraries (and some dependencies)
RUN conda install -n tf_keras \
osmnx=0.7.3 \
affine \
pyproj \
pyhamcrest=1.9.0 \
cython \
fiona \
h5py \
ncurses \
jupyter \
jupyterlab \
ipykernel \
libgdal \
matplotlib \
numpy \
opencv \
pandas \
pillow \
pip \
scipy \
scikit-image \
scikit-learn \
shapely \
gdal \
rtree \
tqdm \
pandas \
geopandas \
rasterio
# get tensorflow
ARG TENSORFLOW_VERSION=1.10.0
ARG TENSORFLOW_DEVICE=gpu
ARG TENSORFLOW_APPEND=_gpu
RUN source activate tf_keras && \
pip --no-cache-dir install https://storage.googleapis.com/tensorflow/linux/${TENSORFLOW_DEVICE}/tensorflow${TENSORFLOW_APPEND}-${TENSORFLOW_VERSION}-cp36-cp36m-linux_x86_64.whl
_
最後のRUN
コマンドはエラーをスローします。
pip upgrade pip
_または_conda upgrade pip
_(この方法でpipを更新しようとすると同じエラーがスローされます)pip install
_コマンドの前にecho $(which pip) &&
を追加することで、正しいpipが使用されていることを確認しました-仮想環境のpipを返します。Condaを使用してすべてをインストールし、それを無視しますが、condaには使用できないグループの内部ライブラリがいくつかあります。
testpath
のバージョンをダウングレードする必要があります。
conda install 'testpath<0.4'
ここの問題を参照してください: https://github.com/conda-forge/testpath-feedstock/issues/7