Another Japan in the World

Jun Aruga's blog, who lived in Japan, Singapore, live in Czech Republic. Mainly about IT and travel.

Install Python 3.5.2 from source on Mac

Python

Install latest stable version Python 3.5.2 on Mac.

Seeing official site [1]. However as I was not able to download with Mercurial (hg command), that is on the official site, get the source from git mirror. [2]

Install from Git source

$ git clone git@github.com:python/cpython.git

$ cd cpython

As git's tags are not set in the repo, check target 3.5.2 commit hash manually.

$ git branch -r

$ git checkout 3.5

$ git log

Then I found it.

commit 7aafe3c4fbff73590de183f095f1d32feeadc450
Author: Larry Hastings <larry@hastings.org>
Date:   Sat Jun 25 14:11:29 2016 -0700

    Added tag v3.5.2 for changeset 4def2a2901a5
$ git checkout 7aafe3c4fbff73590de183f095f1d32feeadc450
$ ./configure --prefix=/usr/local/python-3.5.2 --with-pydebug

$ make -j

$ sudo make install
...
Successfully installed pip-8.1.1 setuptools-20.10.1

$ /usr/local/python-3.5.2/bin/python3 --version
Python 3.5.2

Only python3 (python3.5) is installed command. python is not installed.

$ ls /usr/local/python-3.5.2/bin/python
ls: /usr/local/python-3.5.2/bin/python: No such file or directory

Add its bin path to PATH at .bashrc.

$ which python3
/usr/local/python-3.5.2/bin/python3

Install from Mercurial source

It do not work.

$ cd ~/hg

$ hg clone https://hg.python.org/cpython
destination directory: cpython
requesting all changes
adding changesets
adding manifests
adding file changes
transaction abort!
rollback completed
abort: stream ended unexpectedly (got 222273 bytes, expected 1635000436)

pip

Use pip3 as well as python3. And pip is also available?

$ which pip
/usr/local/python-3.5.2/bin/pip
$ which pip3
/usr/local/python-3.5.2/bin/pip3
$ pip3 --version
pip 8.1.1 from /usr/local/python-3.5.2/lib/python3.5/site-packages (python 3.5)


$ sudo pip3 install --upgrade pip

$ pip3 --version
pip 9.0.1 from /usr/local/python-3.5.2/lib/python3.5/site-packages (python 3.5)

Create own configuration file.

$ vi ~/.pip/pip.conf
[list]
format=columns
$ pip3 list
Package    Version
---------- -------
pip        9.0.1
setuptools 20.10.1

Commands

There are key commands for pip.

$ pip3 --help

Usage:
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible dependencies.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment variables
                              and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be used up to
                              3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be used up to
                              3 times (corresponding to WARNING, ERROR, and CRITICAL
                              logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should attempt
                              (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch,
                              (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host as trusted, even though it does not have
                              valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file containing the
                              private key and the certificate in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine whether a new
                              version of pip is available for download. Implied with --no-
                              index.
Install
$ sudo pip3 install numpy

You may need to use absolute path for pip3. I am faced it on Linux environment.

$ sudo /usr/local/python-3.5.2/bin/pip3 install numpy
Unstall
$ sudo pip3 uninstall numpy
Download

This is similar with Rubygem's gem fetch gem_name

$ sudo pip3 download numpy
Freeze
$ pip3 freeze -h

Freeze Options:
  -r, --requirement <file>    Use the order in the given requirements file and its
                              comments when generating output. This option can be used
                              multiple times.
  -f, --find-links <url>      URL for finding packages, which will be added to the output.
  -l, --local                 If in a virtualenv that has global access, do not output
                              globally-installed packages.
  --user                      Only output packages installed in user-site.
  --all                       Do not skip these packages in the output: pip, setuptools,
                              distribute, wheel
$ pip3 freeze -l
numpy==1.11.2
$ pip3 freeze --user

$ pip3 freeze --all
numpy==1.11.2
pip==9.0.1
setuptools==20.10.1
List
$ pip3 list --format=columns
Package    Version
---------- -------
numpy      1.11.2
pip        9.0.1
setuptools 20.10.1
$ pip3 list --format=legacy
numpy (1.11.2)
pip (9.0.1)
setuptools (20.10.1)
$ pip3 list --format=freeze
biopython==1.68
numpy==1.11.2
pip==9.0.1
setuptools==20.10.1
$ pip3 list --format=json
[{"name": "biopython", "version": "1.68"}, {"name": "numpy", "version": "1.11.2"}, {"name": "pip", "version": "9.0.1"}, {"name": "setuptools", "version": "20.10.1"}]
Show
$ pip3 show numpy
Name: numpy
Version: 1.11.2
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: NumPy Developers
Author-email: numpy-discussion@scipy.org
License: BSD
Location: /usr/local/python-3.5.2/lib/python3.5/site-packages
Requires:

Check

$ pip3 check numpy
No broken requirements found.

Search

Search PyPI for packages.

It searches from package name and the description.

$ pip3 search biopython
A3MIO (0.0.1)                  - A3M/A2M I/O for BioPython
bcbio-gff (0.6.4)              - Read and write Generic Feature Format (GFF) with
                                 Biopython integration.
Bio_Eutils (1.65)              - Standalone version of the Biopython Eutils modules
bioext (0.9.5)                 - Misc utilities and definitions not included or hidden in
                                 biopython
biopython-extensions (0.18.5)  - Misc utilities and definitions not included or hidden in
                                 BioPython
biopython (1.68)               - Freely available tools for computational molecular
                                 biology.
  INSTALLED: 1.68 (latest)
bioscripts.convert (0.4)       - Biopython scripts for converting molecular sequences
                                 between formats.
gbgb (0.1.0)                   - Goodbye, GenBank is a package for use with Biopython that
                                 gives feature annotations from GenBank records a new and
                                 better life.

Wheel

Build wheels from your requirements.

Like "Build gemspec file from required gem dependency packages"?

Tried with a python package. But it had build error.

$ git clone git@github.com:rebase-helper/rebase-helper.git

$ cd rebase-helper

$ pip3 wheel -r requirements.txt
ERROR: 'pip wheel' requires the 'wheel' package. To fix this, run: pip install wheel

$ sudo pip3 install wheel

$ pip3 list | grep wheel
wheel      0.29.0

$ pip3 wheel -r requirements.txt
...
Building wheels for collected packages: backports.lzma, copr, lxml
  Running setup.py bdist_wheel for backports.lzma ... error
...
  gcc -Wno-unused-result -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -I/Users/jun.aruga/include -I/opt/local/include -I/usr/local/include -I/usr/local/python-3.5.2/include/python3.5dm -c backports/lzma/_lzmamodule.c -o build/temp.macosx-10.9-x86_64-3.5-pydebug/backports/lzma/_lzmamodule.o
  backports/lzma/_lzmamodule.c:115:10: fatal error: 'lzma.h' file not found
  #include <lzma.h>
           ^
  1 error generated.
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for backports.lzma
...

Hash

Umm.

$ pip3 hash -h

Usage:   pip hash [options] <file> ...

Description:
  Compute a hash of a local package archive.

  These can be used with --hash in a requirements file to do repeatable
  installs.

virtualenv

virtualenv is a tool to create isolated Python environments. It is like ruby's Bundler, rbenv, RVM, perl's Carton.

Install and use virtualenv.

See the doc [3].

Install

$ sudo pip3 install virtualenv

$ pip3 list | grep virtualenv
virtualenv 15.1.0

How to use

Create environment for each python version.

Create the directory to keep scripts for virtualenv.

$ mkdir ~/virtualenv

Own setuptools, pip and wheel are installed in the environment.

$ virtualenv --python=/usr/local/python-3.5.2/bin/python3 ~/virtualenv/python3.5.2
Running virtualenv with interpreter /usr/local/python-3.5.2/bin/python3
Using base prefix '/usr/local/python-3.5.2'
New python executable in /Users/jun.aruga/virtualenv/python3.5.2/bin/python3
Also creating executable in /Users/jun.aruga/virtualenv/python3.5.2/bin/python
Installing setuptools, pip, wheel.../usr/local/python-3.5.2/lib/python3.5/site-packages/virtualenv.py:901: ResourceWarning: unclosed file <_io.BufferedReader name=6>
  call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
done.

There are 2 kind of managing the environment. For example, Ruby rbenv, RVM have the environment for each ruby version. But bundler has it for each project.

I prefer each project than each version.

$ source ~/virtualenv/python3.5.2/bin/activate

(python3.5.2) $

(python3.5.2) $ which pip
/Users/jun.aruga/virtualenv/python3.5.2/bin/pip

(python3.5.2) $ which python
/Users/jun.aruga/virtualenv/python3.5.2/bin/python

(python3.5.2) $ pip list
Package    Version
---------- -------
pip        9.0.1
setuptools 29.0.1
wheel      0.29.0
Create environment for each python project.

I prefer this way.

$ virtualenv --python=`which python3` `pwd`/vendor/virtualenv

See this article for detail: Think about the Python environment like Ruby Bundler - Another Japan in the World

tox

Tox is a generic virtualenv management and test command line tool. See the document [4]. It is like bundle exec rake test.

setup.py

It is like Makefile.

Remove all ads