Best practice package management

When installing software in a Unix-like operating system such as Ubuntu or MacOS, it is generally best to use your OS's native package manager, if possible.

Later we will need, for example, the standard image-processing library for Python called scikit-image. According to the scikit-image installation page we should install the library using either pip or conda. But on a Debian-based Linux system (such as bare-metal Ubuntu or even in Ubuntu running the WSL), we are better off using our system's package manager to install a correctly-versioned pre-built package.

Copy
sudo apt install python3-skimage

Often, no such pre-built package is available; e.g., there is currently no package available for scikit-image via brew for macOS. So, on a Mac, we must use pip to install scikit-image. Outlining how to properly use pip (including not using it, if a pre-built binary is available) is the main goal of this tutorial.

Let us first install pip (as a pre-built binary). On a Debian-based system, such as Ubuntu, we can install pip by issuing the following at the command prompt.

Copy
sudo apt install python3-pip

On a Mac, pip should have been installed when you installed Python3. To install Python3 on a Mac, try:

Copy
brew update && brew update python

Now, depending on the exact flavor of your system, the command pip may or may not refer to the correct version; where by correct, we mean pip for Python3, not for Python2. Currently, on many systems pip refers to the Python2 version and pip3 refers to the Python3 version.

If pip3 is installed on your system, then it almost certainly resides in /usr/bin. Let us check:

Copy
/usr/bin/pip3 -V

If that command fails, then try

Copy
/usr/bin/pip -V

Here are those commands again along with their output when run on a Chromebook running Debian.

Copy
/usr/bin/pip -V   # /bin/bash: /usr/bin/pip: No such file or directory
/usr/bin/pip3 -V  # pip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)
python3 -V  # Python 3.7.3

This is good: there is no command called pip, and pip3 is installed and refers to Python3 version 3.7 which agrees with the version of Python3 installed on the system.

On your system, /usr/bin/pip may refer to the Python3 version. More likely, /usr/bin/pip3 is what you want.

With the pip situation sorted out, we can finally install scikit-image:

Copy
/usr/bin/pip3 install scikit-image

Note: pip, which stands for, recursively, pip installs packages, and pip3, both also install pre-built binaries.

Tip: You can replace python3 in the commands on this page with simply python if the latter points to Python3 (rather than Python2). Type python -V to check this.

Using pip

As we noted above, on a Unix-like operating system we are better off using, if possible, our OS's package manager — for example, yum, pacman, apt on a Debian-based system, or brew on a Mac — to install Python packages (or any software).

In the event that the software we wish to install is not available via our OS's package manager, then we need an alternative. Below we discuss using, exclusively, pip to install Python packages.

There are other Python package managers. Notably, conda supports installing and easily switching between different versions of a package, which results in faster and better testing and development of packages. But conda can be too complex for some users, and is generally overkill when installing smaller packages.

Currently, pip is the de-facto packaging standard for Python. The vast majority of common Python packages can be installed from pip's primary source, PyPI.

Suppose that you want to install a Python package called cool_package. What should you do?

First, try to install cool_package using your OS's package manager; e.g., on a Debian-based system such as Ubuntu:

sudo apt install cool_package

or, on a Mac,

brew install cool_package

Tips:

• On a Debian-based system such as Ubuntu you can
  • list all packages available for installation via your OS's package manager with apt list or dpkg --list. (So try apt list | grep image to list the packages that have image in their name.)
  • list all installed packages with apt list --installed or dpkg --list | grep -v deinstall.
• Analagously, on a Mac, try:
  • brew search or have a look at formulae.brew.sh
  • brew list

But what if the Python package cool_package is not provided by your OS's package manager? For example, Ubuntu and Debian do not provide torch?

Well, if cool_package is available via PyPI, we can install it using pip:

/usr/bin/pip3 install cool_package

Now, in practice, the command above will likely fail due to lack of permissions. Why?

Pip first resolves any dependencies required by cool_package and, if necessary, installs them; then it installs cool_package. The command above leads to installing all of these packages system-wide, in a directory such as /usr/lib/python3/dist-packages.

But users on your system (including you) typically do not have permission to write to a system directory, hence the error. So what should you do?

One option is to add sudo to beginning of the command above and rerun it. You likely don't want to do that, unless you have other users on your machine and wish to install a trusted package system-wide (so that it will be available to all users of your system).

A better option is to add the user option to the command above:

/usr/bin/pip3 install cool_package --user

Adding the user option causes pip to install the package in a subdirectory of your home directory; the subdirectory should be something like /home/your_user_name/.local/python3.7/site-packages.

Tip: use the following command to get hints as to where pip installs things.

Copy
python3 -c "import sys; print('\n'.join(sys.path))"

More commands

You can

• list the Python packages installed on your system python3 -m pip list.
• search pypi.org for Python packages with
  python3 -m pip search cool_package.
• uninstall cool_package with
  python3 -m pip uninstall cool_package.
• upgrade the package with python3 -m pip install --upgrade cool_package.

If you mess things up and start getting this error:

Copy
Traceback (most recent call last):
 File "/usr/bin/pip3", line 9, in <module>
     from pip import main
     ImportError: cannot import name 'main'

then try sudo python3 -m pip uninstall pip && sudo apt install python3-pip --reinstall.

Notes

• The first command in the last section, python3 -m pip list, is a bit tricky. It lists packages that have been installed locally or globally in either your system-wide directory or your local user directory — that is, in the directories output by python3 -c "import sys; print('\n'.join(sys.path))"
• One advantage to using your OS's package manager to install even Python packages is that doing so helps keep your system up-to-date: whenever you update your entire system (with e.g., sudo apt update && sudo apt upgrade on a Debian-based system) all packages, including cool_package, are upgraded. Importantly, each package is upgraded in a way that doesn't break the dependencies of other installed packages.
• You generally want to avoid using (sudo and) pip to install Python packages system-wide; instead, use the --user switch (see above) to install in your local directory. The issue is that your OS's package manager tries to keep track with the programs tha are installed system-wide. That database can get corrupted, if you use pip to install Python packages system-wide, leading to problems with future system upgrades, dependencies, etc.
• Similarly, you generally want to avoid using pip to install a newer version of a Python package that you initially installed with your OS's package manager.

Takeaways

• When installing software in your Unix-like environment, first try to install using your OS's package manager; e.g., sudo apt install ... or brew install ....
• If your OS's package manager does not provide the Python package you want to install, then consider using pip to install that package.
• But avoid using sudo /usr/bin/pip3 install ... when installing a Python package (unless you are a system admin and know what you are doing).
• Instead, use /usr/bin/pip3 install --user ....

Links

• Overview of Python dependency tools
• Never run Python in your downloads folder