Credit: Sebastian Spindler
“There should be one — and preferably only one — obvious way to do it.”
While that line comes from Tim Peters’s Zen of Python, Python doesn’t always adhere to that principle. One area where Python has fallen short of that ideal is project management. For too long, managing Python projects has involved a mishmash of tools and methodologies. However, a de facto standard may be emerging—Poetry.
Poetry brings to Python the kind of all-in-one project management tool that Go and Rust have long enjoyed. Poetry allows projects to have deterministic dependencies with specific package versions, so they build consistently in different places. Poetry also makes it easier to build, package, and publish projects and libraries to PyPI, so others can share the fruits of your Python labors.
In this article, we’ll walk through the use of Poetry for Python development projects — how to set up Poetry, how to use Poetry to configure a project’s dependencies and virtual environment, and how to avoid some of the pitfalls that come with Poetry’s unique way of doing things.
Set up Poetry in Python
Poetry is deliberately unlike other Python dependency and project management tools, beginning with setup. Instead of using pip, Poetry uses a custom installer. The installer adds the Poetry app to your user’s profile directory, so it can be used with any Python installation in your system, present or future.
Although you can use pip install poetry to install Poetry in a specific Python installation, this isn’t recommended because a) it might conflict with other system files, and b) it makes it difficult to use Poetry consistently with different versions of Python and different virtual environments.
Create a Poetry-managed Python project
Once you have Poetry installed, you can create a new Poetry-managed project directory simply by typing poetry new <project_name>. This command creates a subdirectory named <project_name> and populates it with a project scaffold.
The Poetry project scaffold includes the following:
pyproject.toml— the definition file for the project. Poetry manages this definition for you. If you know what you’re doing, you can edit this file directly, but most of the time you won’t need to.README.rst— an empty README file in ReStructuredText format, the file format used for Python documentation. (There is no rule that says you must use.rstformat for your docs; you can use Markdown for simpler cases.)tests— a subdirectory with scaffolding for unit tests. If you aren’t in the habit of writing tests for your new projects, you should be!- And finally, a subdirectory with the project name that contains the code for your project.
Manage Python virtual environments in Poetry
Probably the first thing you’ll want with a new Poetry project is a Python virtual environment. True to form, Poetry has its own distinct way of handling virtual environments. Instead of placing virtual environments inside the project directory, Poetry puts them in a centralized cache directory that varies according to the operating system:
- Unix:
~/.cache/pypoetry/virtualenvs - MacOS:
~/Library/Caches/pypoetry/virtualenvs - Windows:
C:\Users\<username>\AppData\Local\pypoetry\Cache\virtualenvs or %LOCALAPPDATA%\pypoetry\Cache\virtualenvs
The advantage to Poetry’s approach is the ability to share virtual environments across projects, whenever it makes sense. But it does require altering your work habits.
To set up a virtual environment in Poetry, go to the directory for the project and type poetry env use python. Poetry will create a new virtual environment, store it in the cache directory, and display a randomly generated name for the virtual environment (note this for later use).
For added convenience, Poetry will also install any dependencies listed in the project’s pyproject.toml file. You will find this super useful should you ever want to copy a Poetry project from somewhere else and get it set up on your system.
Note that if you run poetry env use python in a project directory that already has a Poetry-assigned virtual environment, Poetry will activate that virtual environment in the context of that CLI session.
Next comes the hard part, which is getting your Poetry-managed virtual environments to work with your IDE. Visual Studio Code, for instance, auto-detects the presence of a virtual environment inside a project directory, but doesn’t (yet) detect the presence of virtual environments managed with Poetry. The (near-term) solution is to add a line to the project’s settings.json file that indicates where Poetry keeps its virtual environments:
"python.venvPath": "C:\\Users\\username\\AppData\\Local\\pypoetry\\Cache\\virtualenvs"
Be sure to restart Visual Studio Code after making this change.
If you don’t want Poetry to manage your virtual environments, you can disable that behavior with this command:
poetry config virtualenvs.create false
Add dependencies to a Python project in Poetry
Poetry tracks two kinds of project dependencies: packages required for the project to run (production dependencies), and packages required only during the development process (development dependencies). Production dependencies would include any third-party libraries you use for the app’s functionality; development dependencies would include coding tools like black, mypy, or docutils.
- To add production dependencies to the project, use
poetry add <dependency_name>. - To add development dependencies, use
poetry add <dependency_name> -D.
Note that you also use the -D switch when removing development dependencies (i.e., those added using the -D switch) using the command poetry remove <dependency_name>.
Note that the poetry add command works much like pip install in that you can specify either a package name or a Git path (e.g., git+https://github.com/developer/project.git#branchname). You can also configure Poetry to use private repos.
Once dependencies are resolved and installed, Poetry creates a file named poetry.lock in the project directory. This file is a manifest of all of the downloaded dependencies, and should be saved along with the rest of your project. Then anyone who pulls a copy of the project from source control will get the same versions of all required packages.
Now you’re ready to begin the project in earnest. All you have to remember from this point forward is to use Poetry — and only Poetry — to manage all dependencies and virtual environments for the project.
