[Entry] Improving Python Dependency Management with pipx and poetry

content/posts/python-poetry-pipx.md

@@ -0,0 +1,321 @@
++++
+title = "Improving Python Dependency Management With pipx and Poetry"
+date = "2021-09-19"
+author = "Ceda EI"
+tags = ["python", "development"]
+keywords = ["python", "development"]
+description = "My current dev setup with python, poetry and pipx"
+showFullContent = false
++++
+
+Over time, how I develop applications in python has changed noticeably. I will
+divide the topic into three sections and see how they tie into each other at
+the end.
+
+- Development
+- Packaging
+- Usage
+
+## Development
+
+Under development, the issues I will focus on are the following:
+
+- Dependency Management
+- Virtualenvs and managing them
+
+Historically, the way to do dependency management was through
+`requirements.txt`.  I found `requirements.txt` hard to manage. In that setup,
+adding a dependency and installing it was two steps:
+
+- Add the package `bar` to `requirements.txt`
+- Either do `pip install bar` or `pip install -r requirements.txt`
+
+While focused on development, I would often forget one or both of these steps.
+Also, the lack of a lock file was a small downside for me (could be a much
+larger downside for others).  The separation between `pip` and
+`requirements.txt` can also easily lead you to accidentally depend on packages
+installed on your system or in your virtualenv but not specified in your
+`requirements.txt`.
+
+Managing virtualenvs was also difficult. As a virtualenv and a project are not
+related, you need a directory structure. Otherwise, you can't tell which
+virtualenv is being used for which project. You can use the same virtualenvs
+for multiple projects, but that partially defeats the point of virtualenvs and
+makes `requirements.txt` more error-prone (higher chances of forgetting to add
+packages to it). The approach generally used is one of the following two:
+
+
+```
+foo/
+├── foo_src/
+└── foo_venv/
+```
+
+or
+
+```
+foo_src/
+└── venv/
+```
+
+I preferred the second one as the first one nests the source code one
+directory deeper.
+
+### A new standard - `pyproject.toml`
+
+In [PEP-518](https://www.python.org/dev/peps/pep-0518/), python standardized
+the `pyproject.toml` file which allows users to choose alternate build systems
+for package generation.
+
+One such project that provides an alternate build system is
+[Poetry](https://python-poetry.org/). Poetry hits the nail on the head and
+solves my major gripes with traditional tooling.
+
+### Poetry and virtualenvs
+
+Poetry manages the virtualenvs automatically and keeps track of which project
+uses which virtualenv automatically. Working on an existing project which uses
+poetry is as simple as this:
+
+```bash
+$ git clone https://gitlab.com/ceda_ei/verlauf
+$ poetry install
+```
+
+The `poetry install` command sets up the virtualenv, install all the required
+dependencies inside that, and sets up any commands accordingly (I will get to
+this soon).  To activate the virtualenv, simply run:
+
+```bash
+. "$(poetry env info --path)/bin/activate"
+```
+
+I wrap this in a small function which lets me toggle it quickly:
+
+```bash
+function poet() {
+	POET_MANUAL=1
+	if [[ -v VIRTUAL_ENV ]]; then
+		deactivate
+	else
+		. "$(poetry env info --path)/bin/activate"
+	fi
+}
+```
+
+Running `poet` activates the virtualenv if it is not active and deactivates it if
+it is active. To make things even easier, I automatically activate and
+deactivate the virtualenv as I enter and leave the project directory.  To do
+so, simply drop this in your `.bashrc`.
+
+```bash
+function find_in_parent() {
+	local path
+	IFS="/" read -ra path <<<"$PWD"
+	for ((i=${#path[@]}; i > 0; i--)); do
+		local current_path=""
+		for ((j=1; j<i; j++)); do
+			current_path="$current_path/${path[j]}"
+		done
+		if [[ -e "${current_path}/$1" ]]; then
+			echo "${current_path}/"
+			return
+		fi
+	done
+	return 1
+}
+
+function auto_poet() {
+	ret="$?"
+	if [[ -v POET_MANUAL ]]; then
+		return $ret
+	fi
+	if find_in_parent pyproject.toml &> /dev/null; then
+		if [[ ! -v VIRTUAL_ENV ]]; then
+		    if BASE="$(poetry env info --path)"; then
+			. "$BASE/bin/activate"
+			PS1=""
+		    else
+			POET_MANUAL=1
+		    fi
+		fi
+	elif [[ -v VIRTUAL_ENV ]]; then
+		deactivate
+	fi
+	return $ret
+}
+
+PROMPT_COMMAND="auto_poet;$PROMPT_COMMAND"
+```
+
+This ties in well with the `poet` function; if you use `poet` anytime in a bash
+session, activation switches from automatic to manual and changing directories
+no longer auto-toggles the virtualenv.
+
+![auto_poet and poet in action](/images/auto_poet.webp)
+
+### Poetry and dependency management
+
+Instead of using `requirements.txt`, poetry stores the dependencies inside
+`pyproject.toml`.  Poetry is more strict compared to `pip` in resolving
+versioning issues.  Dependencies and dev-dependencies are stored inside
+`tool.poetry.dependencies` and `tool.poetry.dev-dependencies` respectively.
+Here is an example of a `pyproject.toml` for a project I am working on.
+
+```toml
+[tool.poetry]
+name = "bells"
+version = "0.3.0"
+description = "Bells is a program for keeping track of sound recordings."
+authors = ["Ceda EI <ceda_ei@webionite.com>"]
+license = "GPL-3.0"
+readme = "README.md"
+homepage = "https://gitlab.com/ceda_ei/bells.git"
+repository = "https://gitlab.com/ceda_ei/bells.git"
+
+[tool.poetry.dependencies]
+python = ">=3.7,<3.11"
+click = "^8.0.1"
+questionary = "^1.10.0"
+sounddevice = "^0.4.2"
+SoundFile = "^0.10.3"
+numpy = "^1.21.2"
+
+[tool.poetry.dev-dependencies]
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+# I will talk about this section soon
+[tool.poetry.scripts]
+bells = "bells.__main__:main"
+```
+
+One of the upsides of poetry is that you don't have to manage the dependencies
+in `pyproject.toml` file yourself. Poetry adds an `npm`-like interface for
+adding and removing dependencies.  To add a dependency to your project, simply
+run `poetry add bar` and it will add it to your `pyproject.toml` file and
+install it in the virtualenv as well. To remove a dependency, just run `poetry
+remove bar`. For development dependencies, just add the `--dev` flag to the
+commands.
+
+## Packaging
+
+Since poetry replaces the build system, we can now configure the build using
+poetry via `pyproject.toml`. Inside `pyproject.toml`, the `tool.poetry` section
+stores all the build info needed; `tool.poetry` contains the metadata,
+`tool.poetry.dependencies` contains the dependencies, `tool.poetry.source`
+contains private repository details (in case, you don't want to use PyPi).
+
+One of the options is `tool.poetry.scripts`. It contains scripts that the
+project exposes. This replaces `console_scripts` in `entry_points` of
+`setuptools`.
+
+For example,
+
+```toml
+[tool.poetry.scripts]
+foobar = "foo.bar:main"
+```
+
+This will add a script named `foobar` in your `PATH`. Running that is
+equivalent to running the following script
+
+```python
+from foo.bar import main
+
+if __name__ == "__main__":
+    main()
+```
+
+For further details, check the
+[reference](https://python-poetry.org/docs/pyproject/).
+
+Poetry also removes the need for manually doing editable installs (`pip install
+-e .`).  The package is automatically installed as editable when you run
+`poetry install`. Any scripts specified in `tool.poetry.scripts` are
+automatically available in your `PATH` when you activate the `venv`.[^1]
+
+To build the package, simply run `poetry build`. This will generate a wheel and
+a tarball in the dist folder.
+
+To publish the package to PyPi (or another repo), simply run `poetry publish`.
+You can combine the build and publish into one command with `poetry publish
+--build`.
+
+![example of poetry build](/images/poetry_build.webp)
+
+## Usage
+
+This part is more user-facing rather than dev-facing. If you want to use two
+packages globally that expose some scripts to the user, (e.g. `awscli`,
+`youtube-dl`, etc.) the general approach to do so is to run something like `pip
+install --user youtube-dl`. This install the package at the user level and
+exposes the script through `~/.local/bin/youtube-dl`. However, this installs
+all the packages at the same user level. Hypothetically, if you have two
+packages `foo` and `bar` which have conflicting dependencies, this causes an
+issue. If you run,
+
+```bash
+$ pip install foo
+$ pip install bar
+$ bar # works
+$ foo # breaks because of dependency mismatch
+```
+
+While installing `bar`, `pip` will install the dependencies for `bar` which
+will break `foo` after warning you[^2].
+
+To solve this, there is [`pipx`](https://github.com/pypa/pipx). Pipx installs
+each package in a separate virtualenv without requiring the user to activate
+said virtualenv before using the package.[^3]
+
+In the same scenario as before, doing the following works just fine.
+
+```bash
+$ pipx install foo
+$ pipx install bar
+$ bar # works
+$ foo # also works
+```
+
+In this scenario, both `bar` and `foo` are installed in separate virtualenvs so
+the dependency conflict doesn't matter.
+
+## Some more things from my bashrc
+
+```bash
+
+function wrapper_no_poet() {
+	local last_env
+	if [[ -v VIRTUAL_ENV ]]; then
+		last_env="$VIRTUAL_ENV"
+		deactivate
+	fi
+	"$@"
+	ret=$?
+	if [[ -v last_env ]]; then
+		. "$last_env/bin/activate"
+	fi
+	return $ret
+}
+
+alias wnp='wrapper_no_poet'
+alias pm='POET_MANUAL=1'
+```
+
+Prefixing any command with `wnp` runs it outside the virtualenv if a virtualenv
+is active. Running `pm` turns off automatic virtualenv activation.
+
+
+[^1]: This also allows for a nice switch between the development and production
+    versions of the app. Essentially, when the virtualenv is active, you are
+    using the development script while when it is deactivated, you are using
+    the global (likely production) version.
+
+[^2]: To be precise, it will warn you that it broke `foo` but will still
+    continue with the installation
+
+[^3]: For development, poetry also provides `poetry run` which runs a file
+    without having to activate the virtualenv.