Virtual Environments (venv, conda)

As you start building more sophisticated Python applications, especially in rapidly evolving fields like LLM development, you'll quickly encounter a common challenge: managing project dependencies. Imagine working on two different LLM projects. Project A requires version 1.5 of a specific library (like langchain), while Project B needs the newer version 1.8. If you install these libraries globally on your system, one project will inevitably break when you switch to working on the other. This is because a global Python installation can typically only accommodate one version of a given library at a time.

This is where virtual environments become indispensable. A virtual environment is essentially an isolated directory containing a specific Python interpreter and its own set of installed packages, independent from your system's global Python installation or other virtual environments. Think of it as a self-contained workspace for each of your projects.

Using virtual environments provides several significant advantages:

1. Dependency Isolation: Each project can have its own specific set of dependencies (and versions) without interfering with others. This prevents the "dependency hell" scenario described earlier.
2. Reproducibility: By listing a project's exact dependencies (often in a requirements.txt file, covered next), you or collaborators can easily recreate the same environment elsewhere.
3. Cleanliness: It keeps your global Python installation tidy and free from project-specific packages.

Two primary tools are commonly used for creating and managing Python virtual environments: venv and conda.

Using venv

venv is the standard, lightweight tool for creating virtual environments included with Python 3.3 and later. It's often sufficient for many Python projects, including typical LLM applications that primarily rely on Python packages installable via pip.

Creating an Environment with venv

To create a virtual environment using venv, navigate to your project's root directory in your terminal or command prompt and run the following command:

python -m venv .venv

• python -m venv invokes the venv module.
• .venv is the name chosen for the directory that will house the virtual environment. Using .venv or venv is a common convention, and prefixing with a dot often hides the directory by default on Unix-like systems. You can choose any name, but consistency helps.

This command creates the .venv directory, which contains a copy or symlink of the Python interpreter used to create it, along with subdirectories like lib/pythonX.Y/site-packages (on Linux/macOS) or Lib\site-packages (on Windows) where project-specific packages will be installed.

Activating the Environment

Before you can use the environment, you need to activate it. Activation modifies your shell's path settings so that commands like python and pip point to the versions inside your virtual environment directory, not the global ones. The activation command differs slightly based on your operating system:

• On macOS and Linux (bash/zsh):

  source .venv/bin/activate
  

• On Windows (Command Prompt):

  .venv\Scripts\activate.bat
  

• On Windows (PowerShell):

  .venv\Scripts\Activate.ps1
  

  (Note: You might need to adjust your PowerShell execution policy first: Set-ExecutionPolicy RemoteSigned -Scope CurrentUser)

Once activated, your shell prompt will usually change to indicate the active environment, often by prefixing the prompt with (.venv). Now, any packages installed using pip will be placed inside the .venv directory, specific to this project.

Deactivating the Environment

When you're finished working on the project or want to switch to another, simply run:

deactivate

This command restores your shell to its previous state, using the global Python installation again.

Using conda

conda is a cross-platform package and environment management system that comes with the Anaconda and Miniconda distributions. While venv primarily handles Python packages, conda is language-agnostic and excels at managing complex dependencies, including non-Python libraries (like CUDA for GPU acceleration, which can be relevant for running LLMs locally) and even the Python interpreter itself.

Creating an Environment with conda

If you have Anaconda or Miniconda installed, you can create a new environment with:

conda create --name myenv python=3.10

• conda create --name myenv creates a new environment named myenv. Choose a descriptive name.
• python=3.10 specifies the version of Python to install within this environment. conda can manage multiple Python versions simultaneously on your system, isolated within different environments. You can omit the version to get the default version associated with your conda installation.

conda will determine the necessary dependencies, show you a plan, and ask for confirmation before creating the environment.

Activating the Environment

To activate a conda environment:

conda activate myenv

Similar to venv, your prompt will usually change to indicate the active environment, like (myenv).

Deactivating the Environment

To deactivate the current conda environment and return to the base environment (or the previously active one):

conda deactivate

Which Tool Should You Use?

• venv: Generally preferred for most pure-Python projects where dependencies are available via pip. It's built-in, lightweight, and follows standard Python practices closely. If your LLM workflow primarily involves libraries like langchain, openai, llamaindex, and standard data science tools (pandas, numpy), venv is often the simpler and more direct choice.
• conda: A better choice if your project has complex dependencies in Python packages (e.g., C libraries, specific versions of system tools), if you need to manage different Python versions easily, or if you're already invested in the Anaconda ecosystem (common in data science). Some LLM tasks involving local model execution or specific hardware acceleration might benefit from conda's ability to manage system-level dependencies.

For this course, unless you have specific needs pointing towards conda, using venv is recommended due to its simplicity and direct integration with Python. The takeaway is to choose one tool per project and use it consistently. Whichever tool you choose, activating the environment should be the first step you take whenever you start working on your project. This ensures you are always installing packages into and running code within the intended isolated space, preventing conflicts and making your LLM development process much smoother.