Quick guide: Bazel Python virtual environment
Bazel provides great tooling for building and testing Python code, but rules_python as of this note (2023-05-05) do not provide an obvious way to create a Python shell with the
As we venture outside the norms like
venv by using Bazel, having access to a Python shell/enviroment lets us:
Hook into a Python LSP like nothing changed
Editor integrations (Tips for VSCode integration will be provided later in this guide)
Use modules as if we were doing
python -m IPythonor
python -m pytest
Open a REPL
We create a
py_binary macro that uses Bazel Python execution environment to start a
I have not tested this method with Windows, but I suspect it will work with some minor tweaks.
I am not a Bazel expert, so there might be a better way to do this. My Google-fu didn’t find any other guide on this issue, so I decided to write down my solution for it.
PYTHONPATH is constructed into a Bazel Python execution environment by the python_bootstrap_template.txt file for a
We want to use this environment to just start the interpreter not run a script.
As far as I could this was not possible without changing/overriding the template. But keeping the template up to date is a pain.
Since we can not not run a script, we will just run a wrapper script that starts the interpreter that started iteself.
import os import sys os.execv( 1 sys.executable, 2 [sys.executable] + sys.argv[1:], )
|1||Swap the current process with a new process using Unix
|2||Use the current Python executable to start the new process, consequently also the intepreter declared in the Bazel
Why it works?
It works because the bootstrapper created by the
PYTHONPATH and starts our
execv replaces the current process with a new process, and the new process inherits the environment variables from the old process
PYTHONPATH set by the bootstrapper.
Now we just need to create a
py_binary target that runs this script.
load("@pypi//:requirements.bzl", "all_requirements", "requirement") py_binary( name = "pyshell", srcs = ["//label/to:pyshell.py"], deps = [ 1 requirement('pytest'), requirement('ipython'), ], # deps = all_requirements, 2 )
|1||Here list the exact requirements you want in your Python environment.|
|2||If you want all the requirements in your
Now we can run
bazel run //label/to:pyshell to start a Python shell with the
PYTHONPATH set to the Bazel Python execution environment.
bazel run //label/to:pyshell — -m IPython to run start an
The solution above works, but it is not very ergonomic if you want different shells with different sets of dependencies. We can improve it by creating a macro that does the same thing.
Let’s assume a directory structure like this:
├── bazel │ ├── pyshell.bzl │ └── pyshell.py ├── WORKSPACE ├── BUILD.bazel └── requirements.txt
We can create a macro that creates a
py_binary target that runs the
pyshell.py script in the
def pyshell(name, srcs, **kwargs): pyshell_label = Label("//bazel:pyshell.py") native.py_binary( name = name, srcs = [pyshell_label] + srcs, main = pyshell_label, **kwargs, )
Slightly improved version of the Listing 1, “Initial version of pyshell.py” script that
changes the working directory to the directory where the
bazel command was run from
to match the behavior we expect when running
python from the command line.
import os import sys if __name__ == "__main__": # BAZEL_WORKING_DIRECTORY is where the bazel command was run from. bazel_working_dir = os.environ.get("BAZEL_WORKING_DIRECTORY") if bazel_working_dir: os.chdir(bazel_working_dir) os.execv( sys.executable, [sys.executable] + sys.argv, )
BUILD.bazel in workspace root we can use the
load("@pypi//:requirements.bzl", "all_requirements", "requirement") load("//bazel/pyshell.bzl", "pyshell") pyshell( name = "pyshell", deps = [ requirement('pytest'), requirement('ipython'), ], # deps = all_requirements, )
Now we can run
bazel run //:pyshell to start a Python shell with the declared dependencies available in
bazel build //:pyshell there will be an artifact that we can directly execute at
By setting that as the
python.defaultInterpreterPath in VSCode settings we can use the Python LSP as if nothing changed.
This is a very simple solution that works for me, but it is not perfect. I would love to know if there is an alternative recommended solution. If not it would be nice to have a cross platform version of a macro like this in rules_python.
PYTHONPATHis what I call a Python virtual environment in this guide.