[Entry] Improving Python Dependency Management with pipx and poetry
This commit is contained in:
parent
0754b07132
commit
80f081189c
|
@ -0,0 +1,321 @@
|
||||||
|
+++
|
||||||
|
title = "Improving Python Dependency Management With pipx and Poetry"
|
||||||
|
date = "2021-09-19"
|
||||||
|
author = "Ceda EI"
|
||||||
|
tags = ["python", "development"]
|
||||||
|
keywords = ["python", "development"]
|
||||||
|
description = "My current dev setup with python, poetry and pipx"
|
||||||
|
showFullContent = false
|
||||||
|
+++
|
||||||
|
|
||||||
|
Over time, how I develop applications in python has changed noticeably. I will
|
||||||
|
divide the topic into three sections and see how they tie into each other at
|
||||||
|
the end.
|
||||||
|
|
||||||
|
- Development
|
||||||
|
- Packaging
|
||||||
|
- Usage
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
Under development, the issues I will focus on are the following:
|
||||||
|
|
||||||
|
- Dependency Management
|
||||||
|
- Virtualenvs and managing them
|
||||||
|
|
||||||
|
Historically, the way to do dependency management was through
|
||||||
|
`requirements.txt`. I found `requirements.txt` hard to manage. In that setup,
|
||||||
|
adding a dependency and installing it was two steps:
|
||||||
|
|
||||||
|
- Add the package `bar` to `requirements.txt`
|
||||||
|
- Either do `pip install bar` or `pip install -r requirements.txt`
|
||||||
|
|
||||||
|
While focused on development, I would often forget one or both of these steps.
|
||||||
|
Also, the lack of a lock file was a small downside for me (could be a much
|
||||||
|
larger downside for others). The separation between `pip` and
|
||||||
|
`requirements.txt` can also easily lead you to accidentally depend on packages
|
||||||
|
installed on your system or in your virtualenv but not specified in your
|
||||||
|
`requirements.txt`.
|
||||||
|
|
||||||
|
Managing virtualenvs was also difficult. As a virtualenv and a project are not
|
||||||
|
related, you need a directory structure. Otherwise, you can't tell which
|
||||||
|
virtualenv is being used for which project. You can use the same virtualenvs
|
||||||
|
for multiple projects, but that partially defeats the point of virtualenvs and
|
||||||
|
makes `requirements.txt` more error-prone (higher chances of forgetting to add
|
||||||
|
packages to it). The approach generally used is one of the following two:
|
||||||
|
|
||||||
|
|
||||||
|
```
|
||||||
|
foo/
|
||||||
|
├── foo_src/
|
||||||
|
└── foo_venv/
|
||||||
|
```
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
|
```
|
||||||
|
foo_src/
|
||||||
|
└── venv/
|
||||||
|
```
|
||||||
|
|
||||||
|
I preferred the second one as the first one nests the source code one
|
||||||
|
directory deeper.
|
||||||
|
|
||||||
|
### A new standard - `pyproject.toml`
|
||||||
|
|
||||||
|
In [PEP-518](https://www.python.org/dev/peps/pep-0518/), python standardized
|
||||||
|
the `pyproject.toml` file which allows users to choose alternate build systems
|
||||||
|
for package generation.
|
||||||
|
|
||||||
|
One such project that provides an alternate build system is
|
||||||
|
[Poetry](https://python-poetry.org/). Poetry hits the nail on the head and
|
||||||
|
solves my major gripes with traditional tooling.
|
||||||
|
|
||||||
|
### Poetry and virtualenvs
|
||||||
|
|
||||||
|
Poetry manages the virtualenvs automatically and keeps track of which project
|
||||||
|
uses which virtualenv automatically. Working on an existing project which uses
|
||||||
|
poetry is as simple as this:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ git clone https://gitlab.com/ceda_ei/verlauf
|
||||||
|
$ poetry install
|
||||||
|
```
|
||||||
|
|
||||||
|
The `poetry install` command sets up the virtualenv, install all the required
|
||||||
|
dependencies inside that, and sets up any commands accordingly (I will get to
|
||||||
|
this soon). To activate the virtualenv, simply run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
. "$(poetry env info --path)/bin/activate"
|
||||||
|
```
|
||||||
|
|
||||||
|
I wrap this in a small function which lets me toggle it quickly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
function poet() {
|
||||||
|
POET_MANUAL=1
|
||||||
|
if [[ -v VIRTUAL_ENV ]]; then
|
||||||
|
deactivate
|
||||||
|
else
|
||||||
|
. "$(poetry env info --path)/bin/activate"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Running `poet` activates the virtualenv if it is not active and deactivates it if
|
||||||
|
it is active. To make things even easier, I automatically activate and
|
||||||
|
deactivate the virtualenv as I enter and leave the project directory. To do
|
||||||
|
so, simply drop this in your `.bashrc`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
function find_in_parent() {
|
||||||
|
local path
|
||||||
|
IFS="/" read -ra path <<<"$PWD"
|
||||||
|
for ((i=${#path[@]}; i > 0; i--)); do
|
||||||
|
local current_path=""
|
||||||
|
for ((j=1; j<i; j++)); do
|
||||||
|
current_path="$current_path/${path[j]}"
|
||||||
|
done
|
||||||
|
if [[ -e "${current_path}/$1" ]]; then
|
||||||
|
echo "${current_path}/"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
function auto_poet() {
|
||||||
|
ret="$?"
|
||||||
|
if [[ -v POET_MANUAL ]]; then
|
||||||
|
return $ret
|
||||||
|
fi
|
||||||
|
if find_in_parent pyproject.toml &> /dev/null; then
|
||||||
|
if [[ ! -v VIRTUAL_ENV ]]; then
|
||||||
|
if BASE="$(poetry env info --path)"; then
|
||||||
|
. "$BASE/bin/activate"
|
||||||
|
PS1=""
|
||||||
|
else
|
||||||
|
POET_MANUAL=1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
elif [[ -v VIRTUAL_ENV ]]; then
|
||||||
|
deactivate
|
||||||
|
fi
|
||||||
|
return $ret
|
||||||
|
}
|
||||||
|
|
||||||
|
PROMPT_COMMAND="auto_poet;$PROMPT_COMMAND"
|
||||||
|
```
|
||||||
|
|
||||||
|
This ties in well with the `poet` function; if you use `poet` anytime in a bash
|
||||||
|
session, activation switches from automatic to manual and changing directories
|
||||||
|
no longer auto-toggles the virtualenv.
|
||||||
|
|
||||||
|
![auto_poet and poet in action](/images/auto_poet.webp)
|
||||||
|
|
||||||
|
### Poetry and dependency management
|
||||||
|
|
||||||
|
Instead of using `requirements.txt`, poetry stores the dependencies inside
|
||||||
|
`pyproject.toml`. Poetry is more strict compared to `pip` in resolving
|
||||||
|
versioning issues. Dependencies and dev-dependencies are stored inside
|
||||||
|
`tool.poetry.dependencies` and `tool.poetry.dev-dependencies` respectively.
|
||||||
|
Here is an example of a `pyproject.toml` for a project I am working on.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[tool.poetry]
|
||||||
|
name = "bells"
|
||||||
|
version = "0.3.0"
|
||||||
|
description = "Bells is a program for keeping track of sound recordings."
|
||||||
|
authors = ["Ceda EI <ceda_ei@webionite.com>"]
|
||||||
|
license = "GPL-3.0"
|
||||||
|
readme = "README.md"
|
||||||
|
homepage = "https://gitlab.com/ceda_ei/bells.git"
|
||||||
|
repository = "https://gitlab.com/ceda_ei/bells.git"
|
||||||
|
|
||||||
|
[tool.poetry.dependencies]
|
||||||
|
python = ">=3.7,<3.11"
|
||||||
|
click = "^8.0.1"
|
||||||
|
questionary = "^1.10.0"
|
||||||
|
sounddevice = "^0.4.2"
|
||||||
|
SoundFile = "^0.10.3"
|
||||||
|
numpy = "^1.21.2"
|
||||||
|
|
||||||
|
[tool.poetry.dev-dependencies]
|
||||||
|
|
||||||
|
[build-system]
|
||||||
|
requires = ["poetry-core>=1.0.0"]
|
||||||
|
build-backend = "poetry.core.masonry.api"
|
||||||
|
|
||||||
|
# I will talk about this section soon
|
||||||
|
[tool.poetry.scripts]
|
||||||
|
bells = "bells.__main__:main"
|
||||||
|
```
|
||||||
|
|
||||||
|
One of the upsides of poetry is that you don't have to manage the dependencies
|
||||||
|
in `pyproject.toml` file yourself. Poetry adds an `npm`-like interface for
|
||||||
|
adding and removing dependencies. To add a dependency to your project, simply
|
||||||
|
run `poetry add bar` and it will add it to your `pyproject.toml` file and
|
||||||
|
install it in the virtualenv as well. To remove a dependency, just run `poetry
|
||||||
|
remove bar`. For development dependencies, just add the `--dev` flag to the
|
||||||
|
commands.
|
||||||
|
|
||||||
|
## Packaging
|
||||||
|
|
||||||
|
Since poetry replaces the build system, we can now configure the build using
|
||||||
|
poetry via `pyproject.toml`. Inside `pyproject.toml`, the `tool.poetry` section
|
||||||
|
stores all the build info needed; `tool.poetry` contains the metadata,
|
||||||
|
`tool.poetry.dependencies` contains the dependencies, `tool.poetry.source`
|
||||||
|
contains private repository details (in case, you don't want to use PyPi).
|
||||||
|
|
||||||
|
One of the options is `tool.poetry.scripts`. It contains scripts that the
|
||||||
|
project exposes. This replaces `console_scripts` in `entry_points` of
|
||||||
|
`setuptools`.
|
||||||
|
|
||||||
|
For example,
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[tool.poetry.scripts]
|
||||||
|
foobar = "foo.bar:main"
|
||||||
|
```
|
||||||
|
|
||||||
|
This will add a script named `foobar` in your `PATH`. Running that is
|
||||||
|
equivalent to running the following script
|
||||||
|
|
||||||
|
```python
|
||||||
|
from foo.bar import main
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
```
|
||||||
|
|
||||||
|
For further details, check the
|
||||||
|
[reference](https://python-poetry.org/docs/pyproject/).
|
||||||
|
|
||||||
|
Poetry also removes the need for manually doing editable installs (`pip install
|
||||||
|
-e .`). The package is automatically installed as editable when you run
|
||||||
|
`poetry install`. Any scripts specified in `tool.poetry.scripts` are
|
||||||
|
automatically available in your `PATH` when you activate the `venv`.[^1]
|
||||||
|
|
||||||
|
To build the package, simply run `poetry build`. This will generate a wheel and
|
||||||
|
a tarball in the dist folder.
|
||||||
|
|
||||||
|
To publish the package to PyPi (or another repo), simply run `poetry publish`.
|
||||||
|
You can combine the build and publish into one command with `poetry publish
|
||||||
|
--build`.
|
||||||
|
|
||||||
|
![example of poetry build](/images/poetry_build.webp)
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
This part is more user-facing rather than dev-facing. If you want to use two
|
||||||
|
packages globally that expose some scripts to the user, (e.g. `awscli`,
|
||||||
|
`youtube-dl`, etc.) the general approach to do so is to run something like `pip
|
||||||
|
install --user youtube-dl`. This install the package at the user level and
|
||||||
|
exposes the script through `~/.local/bin/youtube-dl`. However, this installs
|
||||||
|
all the packages at the same user level. Hypothetically, if you have two
|
||||||
|
packages `foo` and `bar` which have conflicting dependencies, this causes an
|
||||||
|
issue. If you run,
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ pip install foo
|
||||||
|
$ pip install bar
|
||||||
|
$ bar # works
|
||||||
|
$ foo # breaks because of dependency mismatch
|
||||||
|
```
|
||||||
|
|
||||||
|
While installing `bar`, `pip` will install the dependencies for `bar` which
|
||||||
|
will break `foo` after warning you[^2].
|
||||||
|
|
||||||
|
To solve this, there is [`pipx`](https://github.com/pypa/pipx). Pipx installs
|
||||||
|
each package in a separate virtualenv without requiring the user to activate
|
||||||
|
said virtualenv before using the package.[^3]
|
||||||
|
|
||||||
|
In the same scenario as before, doing the following works just fine.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ pipx install foo
|
||||||
|
$ pipx install bar
|
||||||
|
$ bar # works
|
||||||
|
$ foo # also works
|
||||||
|
```
|
||||||
|
|
||||||
|
In this scenario, both `bar` and `foo` are installed in separate virtualenvs so
|
||||||
|
the dependency conflict doesn't matter.
|
||||||
|
|
||||||
|
## Some more things from my bashrc
|
||||||
|
|
||||||
|
```bash
|
||||||
|
|
||||||
|
function wrapper_no_poet() {
|
||||||
|
local last_env
|
||||||
|
if [[ -v VIRTUAL_ENV ]]; then
|
||||||
|
last_env="$VIRTUAL_ENV"
|
||||||
|
deactivate
|
||||||
|
fi
|
||||||
|
"$@"
|
||||||
|
ret=$?
|
||||||
|
if [[ -v last_env ]]; then
|
||||||
|
. "$last_env/bin/activate"
|
||||||
|
fi
|
||||||
|
return $ret
|
||||||
|
}
|
||||||
|
|
||||||
|
alias wnp='wrapper_no_poet'
|
||||||
|
alias pm='POET_MANUAL=1'
|
||||||
|
```
|
||||||
|
|
||||||
|
Prefixing any command with `wnp` runs it outside the virtualenv if a virtualenv
|
||||||
|
is active. Running `pm` turns off automatic virtualenv activation.
|
||||||
|
|
||||||
|
|
||||||
|
[^1]: This also allows for a nice switch between the development and production
|
||||||
|
versions of the app. Essentially, when the virtualenv is active, you are
|
||||||
|
using the development script while when it is deactivated, you are using
|
||||||
|
the global (likely production) version.
|
||||||
|
|
||||||
|
[^2]: To be precise, it will warn you that it broke `foo` but will still
|
||||||
|
continue with the installation
|
||||||
|
|
||||||
|
[^3]: For development, poetry also provides `poetry run` which runs a file
|
||||||
|
without having to activate the virtualenv.
|
Binary file not shown.
After Width: | Height: | Size: 38 KiB |
Binary file not shown.
After Width: | Height: | Size: 42 KiB |
Loading…
Reference in New Issue