Another Japan in the World

Jun Aruga's blog, who lived in Japan, Singapore, live in Czech Republic. Mainly about IT and travel.

Think about the Python environment like Ruby Bundler

Want to use Python environment like Ruby Bundler!

Ruby Bundler is very convenient.

Below process is so common before starting development for a RugyGem package.

$ git clone GIT_URL/foo.git

$ cd foo

$ bundle install --path vendor/bundle

$ bundle list

$ bundle exec rake -T

$ bundle exec rake test (or bundle exec rake spec etc..)

foo.gemspec, Gemfile, Rakefile are basically used for the environment. This Ruby, Bundler's common rule and way make people easy to install and test it.

So, how about python? Well, there are virtualenv and tox to do it.

But regarding this article [1]. It seems that I have to think about it more.

Case for Ruby Bundler

For example, I will use robocop (Ruby static code analysis tool) [2].

$ git clone git@github.com:bbatsov/rubocop.git

$ cd rubocop
$ bundle install --path vendor/bundle
$ bundle list
Gems included by the bundle:
  * addressable (2.5.0)
  * ast (2.3.0)
  * bundler (1.13.6)
  * codeclimate-test-reporter (1.0.3)
  * coderay (1.1.1)
  * crack (0.4.3)
  * diff-lcs (1.2.5)
  * docile (1.1.5)
  * hashdiff (0.3.1)
  * json (2.0.2)
  * method_source (0.8.2)
  * parser (2.3.3.0)
  * powerpack (0.1.1)
  * pry (0.10.4)
  * public_suffix (2.0.4)
  * rainbow (2.1.0)
  * rake (11.3.0)
  * rspec (3.5.0)
  * rspec-core (3.5.4)
  * rspec-expectations (3.5.0)
  * rspec-mocks (3.5.0)
  * rspec-support (3.5.0)
  * rubocop (0.46.0)
  * ruby-progressbar (1.8.1)
  * safe_yaml (1.0.4)
  * simplecov (0.12.0)
  * simplecov-html (0.10.0)
  * slop (3.6.0)
  * unicode-display_width (1.1.1)
  * webmock (2.1.0)
  * yard (0.9.5)

Installed gem packages are here.

$ ls vendor/bundle/ruby/2.3.0/gems/
addressable-2.5.0/               rake-11.3.0/
ast-2.3.0/                       rspec-3.5.0/
codeclimate-test-reporter-1.0.3/ rspec-core-3.5.4/
coderay-1.1.1/                   rspec-expectations-3.5.0/
crack-0.4.3/                     rspec-mocks-3.5.0/
diff-lcs-1.2.5/                  rspec-support-3.5.0/
docile-1.1.5/                    ruby-progressbar-1.8.1/
hashdiff-0.3.1/                  safe_yaml-1.0.4/
json-2.0.2/                      simplecov-0.12.0/
method_source-0.8.2/             simplecov-html-0.10.0/
parser-2.3.3.0/                  slop-3.6.0/
powerpack-0.1.1/                 unicode-display_width-1.1.1/
pry-0.10.4/                      webmock-2.1.0/
public_suffix-2.0.4/             yard-0.9.5/
rainbow-2.1.0/
$ bundle exec rake -T
rake ascii_spec                           # Run RSpec code examples
rake bench_cop[cop,srcpath,times]         # Benchmark a cop on given source file/dir
rake build                                # Build rubocop-0.46.0.gem into the pkg direc...
rake clean                                # Remove any temporary products
rake clobber                              # Remove any generated files
rake coverage                             # Run RSpec with code coverage
rake generate_cops_documentation          # Generate docs of all cops types
rake install                              # Build and install rubocop-0.46.0.gem into s...
rake install:local                        # Build and install rubocop-0.46.0.gem into s...
rake internal_investigation               # Run RuboCop over itself
rake internal_investigation:auto_correct  # Auto-correct RuboCop offenses
rake new_cop[cop]                         # Generate a new cop template
rake release[remote]                      # Create tag v0.46.0 and build and push ruboc...
rake repl                                 # Open a REPL for experimentation
rake spec                                 # Run RSpec code examples
rake yard                                 # Generate YARD Documentation

$ bundle exec rake spec
...
Finished in 27.21 seconds (files took 3.85 seconds to load)
12130 examples, 0 failures, 2 pending

That's it for Bundler. So, how about the case for Python and virtualenv? Let's move on next!

Case for Python virtualenv (and tox)

For example, I will use hacking (Python static code analysis tool) [3].

$ git clone git@github.com:openstack-dev/hacking.git
$ cd hacking

Instead of bundle install --path vendor/bundle

$ virtualenv --python=`which python3` `pwd`/vendor/virtualenv

$ source vendor/virtualenv/bin/activate
(virtualenv) $

(virtualenv) $ which python
/Users/jun.aruga/git/hacking/vendor/virtualenv/bin/python

(virtualenv) $ python --version
Python 3.5.2

(virtualenv) $ which pip
/Users/jun.aruga/git/hacking/vendor/virtualenv/bin/pip

(virtualenv) $ pip --version
pip 9.0.1 from /Users/jun.aruga/git/hacking/vendor/virtualenv/lib/python3.5/site-packages (python 3.5)

(virtualenv) $ pip list
Package    Version
---------- -------
pip        9.0.1
setuptools 29.0.1
wheel      0.29.0

Install packages required to run. It is like a runtime dependency for Ruby .gemspec file.

(virtualenv) $ pip install -r requirements.txt
...
Successfully installed flake8-2.5.5 mccabe-0.2.1 pbr-1.10.0 pep8-1.5.7 pyflakes-0.8.1 six-1.10.0

(virtualenv) $ pip list
Package    Version
---------- -------
flake8     2.5.5
mccabe     0.2.1
pbr        1.10.0
pep8       1.5.7
pip        9.0.1
pyflakes   0.8.1
setuptools 29.0.1
six        1.10.0
wheel      0.29.0

Install packages required to develop. It is like a development dependency for Ruby .gemspec file.

(virtualenv) $ pip install -r test-requirements.txt

(virtualenv) $ pip list
Package          Version
---------------- -------
alabaster        0.7.9
Babel            2.3.4
coverage         4.2
discover         0.4.0
docutils         0.12
eventlet         0.19.0
extras           1.0.0
fixtures         3.0.0
flake8           2.5.5
greenlet         0.4.10
Jinja2           2.8
linecache2       1.0.0
MarkupSafe       0.23
mccabe           0.2.1
mock             2.0.0
oslosphinx       4.8.0
pbr              1.10.0
pep8             1.5.7
pip              9.0.1
pyflakes         0.8.1
Pygments         2.1.3
python-mimeparse 1.6.0
python-subunit   1.2.0
pytz             2016.7
PyYAML           3.12
reno             1.8.0
requests         2.12.2
setuptools       29.0.1
six              1.10.0
snowballstemmer  1.2.1
Sphinx           1.3.6
sphinx-rtd-theme 0.1.9
testrepository   0.0.20
testscenarios    0.5.0
testtools        2.2.0
traceback2       1.4.0
unittest2        1.1.0
wheel            0.29.0

Deactivate virtualenv

(virtualenv) $ deactivate

$

Installed pip packages

Installed packages are here.

$ ls vendor/virtualenv/lib/python3.5/site-packages/
__pycache__/                 pbr-1.10.0.dist-info/        pyflakes-0.8.1.dist-info/
easy_install.py              pep8-1.5.7.dist-info/        setuptools/
flake8/                      pep8.py                      setuptools-29.0.1.dist-info/
flake8-2.5.5.dist-info/      pip/                         six-1.10.0.dist-info/
mccabe-0.2.1.dist-info/      pip-9.0.1.dist-info/         six.py
mccabe.py                    pkg_resources/               wheel/
pbr/                         pyflakes/                    wheel-0.29.0.dist-info/

Instead of bundle list

This is isolated environment by virtualenv.

$ source vendor/virtualenv/bin/activate

(virtualenv) $ pip list
Package    Version
---------- -------
flake8     2.5.5
mccabe     0.2.1
pbr        1.10.0
pep8       1.5.7
pip        9.0.1
pyflakes   0.8.1
setuptools 29.0.1
six        1.10.0
wheel      0.29.0

(virtualenv) $ deactivate

Instead of bundle exec rake -T

Maybe you can do by develop the script with callback.

$ source vendor/virtualenv/bin/activate

(virtualenv) $ something

(virtualenv) $ deactivate

Instead of bundle exec rake test (spec and etc..)

You can use tox. [4]

Install tox manually, because it is not included in test-requirements.txt.

(virtualenv) $ pip install tox

(virtualenv) $ pip list | grep tox
tox              2.5.0

(virtualenv) $ which tox
/Users/jun.aruga/git/hacking/vendor/virtualenv/bin/tox

Check a list of test environment.

(virtualenv) $ tox -l
py34
py26
py27
pep8
pypy

Check tox.ini or file for CI such as .travis.yml to see another tox ini file. In this case, there is tox.ini.

Run the tests. It was failed. Maybe my process to test is not correct. Later I will check it. Hint? [5]

(virtualenv) $ tox
py34 create: /Users/jun.aruga/git/hacking/.tox/py34
ERROR: InterpreterNotFound: python3.4
...
ERROR: InvocationError: '/Users/jun.aruga/git/hacking/.tox/pep8/bin/flake8'
pypy create: /Users/jun.aruga/git/hacking/.tox/pypy
ERROR: InterpreterNotFound: pypy
Remove all ads