Mac初始化EasyNLP开发环境

Posted on In Symbols count in article: 3.8k Reading time ≈ 3 mins.
Mac初始化EasyNLP开发环境

环境

1
2
3
4
5
6
7
macOS Monterey 12.2.1，Apple M1 Pro版本

$ python --version
Python 3.8.9

$ pip --version
pip 22.0.4 from /Users/gavin/Library/Python/3.8/lib/python/site-packages/pip (python 3.8)

过程

下载项目

打开官方文档，创建目录，在目录中 clone 项目。

git clone https://github.com/alibaba/EasyNLP

安装依赖

执行pip install -r requirements.txt，不出预料，有报错。

setuptools版本过低

错误信息如下：

1
2
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (setuptools 49.2.1 (/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages), Requirement.parse('setuptools>=58.0'))

信息比较清晰，进行升级：

1
2
3
4
5
6
7
$ pip install --upgrade setuptools
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: setuptools in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages (49.2.1)
Collecting setuptools
  Using cached setuptools-62.1.0-py3-none-any.whl (1.1 MB)
Installing collected packages: setuptools
Successfully installed setuptools-62.1.0

升级成功。重新执行pip install -r requirements.txt，不料还是报相同的错，仔细看了一下，报错路径与我安装的 setuptools 不一致，最终通过修改 site.py 解决，详细过程可参考这篇文章

tokenizers模块报错

部分报错信息如下：

1
2
3
4
5
6
7
8
9
   note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for sentencepiece
  Running setup.py clean for sentencepiece
  Building wheel for grpcio (setup.py) ... done
  Created wheel for grpcio: filename=grpcio-1.45.0-cp38-cp38-macosx_10_14_arm64.whl size=7451816 sha256=5909c0fc7f0fc6fe959b5205d8fb6293404e377b651fb97aeaaf3574a8c528a0
  Stored in directory: /Users/gavin/Library/Caches/pip/wheels/bb/fe/aa/509208bd9420844c88b4a73ced4bc58f069a3db3b3a4b23336
Successfully built grpcio
Failed to build tokenizers sentencepiece
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

解决方案，安装 rust：

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

sentencepiece模块报错

重新执行pip install -r requirements.txt，又有新的报错：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Running setup.py install for sentencepiece ... error
  error: subprocess-exited-with-error

  × Running setup.py install for sentencepiece did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      /Users/gavin/Library/Python/3.8/lib/python/site-packages/setuptools/dist.py:757: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
        warnings.warn(
      running install
      /Users/gavin/Library/Python/3.8/lib/python/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.14-arm64-cpython-38
      creating build/lib.macosx-10.14-arm64-cpython-38/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.macosx-10.14-arm64-cpython-38/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.macosx-10.14-arm64-cpython-38/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.macosx-10.14-arm64-cpython-38/sentencepiece
      running build_ext
      /bin/sh: pkg-config: command not found
      mkdir: bundled: File exists
      fatal: destination path 'sentencepiece' already exists and is not an empty directory.
      fatal: destination path 'sentencepiece' already exists and is not an empty directory.
      mkdir: build: File exists
      ./build_bundled.sh: line 15: cmake: command not found
      ./build_bundled.sh: line 16: nproc: command not found
      make: *** No targets specified and no makefile found.  Stop.
      make: *** No rule to make target `install'.  Stop.
      env: pkg-config: No such file or directory
      Failed to find sentencepiece pkg-config
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

看报错是安装 sentencepiece 有异常，尝试单独安装该模块pip install sentencepiece，报同样的错。

最终通过下载安装包、修改标签名，安装成功，详细过程可参考这篇文章。

重新执行pip install -r requirements.txt，终于成功安装完依赖。