jmanteau

Mon coin de toile - A piece of Web

Moving to a New Python Tooling Stack: Ruff & Pre-Commit

Posted: Nov 23, 2023

TL;DR: Ruff combines different tools like linters, security analyzers, and formatters into one unique simple fast tool with a standard configuration. Pre-commit streamlines git pre-commit hooks by standardizing configurations and offering community-made hooks. Together, they greatly ease Python development, both individually and in a team setting.

Introduction

I’m coding in Python since 2006 (17 years !) and I have changed my tooling several times along with the ecosystem evolution.

My previous IDE setup was Vim-oriented with the additional plugins SimpylFold, YouCompleteMe, syntastic, nerdtree, ctrlp.vim among others.

I’ve moved since 5 years to VSCode as a primary editor due to the capacity to be integrated with a lot of use cases in an easier way :

And so on.

Microsoft recently (September Announcement) moved the tools support offered in the Python extension for Visual Studio Code into separate extensions. This stems me to review the alternative I had in mind instead of directly re-installing my current tooling.

These tools — Pylint, Flake8, Black Formatter, Bandit, and isort — help enhance my Python code quality by offering linting, error detection, type checking, automated formatting, security analysis and import sorting capabilities.

It was the occasion to (re)test the latest kid on the block: Ruff

Introducing the all-in-one linter/formatter/checking/etc : Ruff

Ruff is an extremely fast open-source Python linter and code formatter, written in Rust

It has rapidly become popular for its swift performance and its ability to aggregate the features of multiple linters and is now used in large projects and organisations as the tool of choice.

The README on Github list its principal qualities:

I already tried Ruff two years ago but there was still some issues (notably not all pylint and bandit checks were there).It is now quite mature even if not full feature-parity with Pylint. However it does 80% of the job in 2% of the previous time (yes I voluntarily broke Pareto). To sum it up, Ruff is now my tooling of choice because:

All-in-one: Replace dozens of tools with a single, unified interface. Ruff supports over 700 rules unifying my previous tools (Pylint, Flake8, Black Formatter, Bandit,) and maintains drop-in compatibility.

Fast: It is not fast but blazing fast. Check this comparison below.

Ruff is fast

Automated and easy configuration: Automatically upgrade to newer Python syntax, organize imports, remove unused variables and specify all your validation logic in one configuration file.

One of the nice aspect of Ruff is its configuration integration in the pyproject.toml format.

Side explanation of pyproject.toml

The pyproject.toml file is a configuration file format introduced in PEP 518 for Python projects. It is designed to consolidate various settings and configurations that were traditionally spread across multiple files (setup.py, setup.cfg, requirements.txt, etc.) into a single, standardized file. Here’s an overview of the standard sections and elements you might find in a typical pyproject.toml file:

  1. [build-system]: This section specifies the build system requirements, particularly useful for packaging and distribution. It usually includes:

    • requires: A list of packages required to build the project.
    • build-backend: The build backend, often setuptools.build_meta.

  2. [tool] Sections: These sections are used by various Python tools for their configurations. Each tool has its own subsection.

  3. [project] (optional in PEP 621): This newer section is for metadata about the project itself, such as:

    • name, version, description: Basic project information.
    • dependencies: A list of runtime dependencies.
    • optional-dependencies: Dependencies needed for optional features.
    • authors, maintainers: Information about the people involved in the project.

    Custom Configurations: Besides these standard sections, pyproject.toml can also include custom configurations for other tools that support it. This flexibility allows it to cater to the specific needs of a project.

    Advantages:

    • Unified Configuration: It consolidates various configurations into a single file.
    • Standardization: Offers a standardized format for Python project configurations.
    • Tool Agnostic: It can be used by multiple tools, reducing the need for tool-specific configuration files.

In practice, the contents of pyproject.toml can vary greatly depending on the tools used and the specific needs of the project. Its adoption has been growing, as it represents a more modern and unified approach to managing Python project configurations.

Ruff Installation and Usage

System installation

Ruff is available as ruff on PyPI:

pip install ruff

Once installed, you can run Ruff from the command line:

ruff check .   # Lint all files in the current directory.
ruff format .  # Format all files in the current directory.

For macOS Homebrew and Linuxbrew users, Ruff is also available as ruff on Homebrew:

brew install ruff

Here an example on how the check looks:

❯ ruff check .
business-facing/main.py:1:1: I001 [*] Import block is un-sorted or un-formatted
business-facing/main.py:325:9: B007 Loop control variable `region` not used within loop body
business-facing/main.py:340:17: B018 Found useless expression. Either assign it to a variable or remove it.
business-facing/main.py:531:8: SIM118 Use `key in dict` instead of `key in dict.keys()`
Found 4 errors.
[*] 1 fixable with the `--fix` option (2 hidden fixes can be enabled with the `--unsafe-fixes` option).

Vscode Extension

You can also install the Ruff VScode extension to do it live when coding (integration as Vscode linter/formatter).

I then personnaly set it up to format on save and define it as default formatter by editing my Json preference in the Settings editor like this:

{
    "[python]": {
        "editor.formatOnSave": true,
        "editor.defaultFormatter": "charliermarsh.ruff"
    }
}

The Settings editor is the UI that lets you review and modify setting values that are stored in a settings.json file. You can review and edit this file directly.

Here an example on how it looks with deliberates mistakes:

image-20231125171028530

And after saving and having ruff format and auto fix (os import fix, trailing whitespaces, blank lines, etc):

image-20231125171041820

And finally after fixing the mistakes remaining manually:

image-20231125171137551

Pre-Commit Framework

Git Hooks and Pre-Commit

A Git hook is a script that Git executes before or after events such as commit, push, and receive. These hooks are customizable and allow you to automate certain tasks in the Git workflow. They are stored in the .git/hooks directory of a Git repository and can be written in any scripting language.

A pre-commit hook is a specific type of Git hook that runs before a commit is finalized. This hook is used to inspect the snapshot that’s about to be committed. For example, it can check for code formatting, lint errors, or run tests. If the pre-commit hook script exits with a non-zero status, the commit is aborted, allowing you to fix any issues before proceeding.

logo-top-shelf

The pre-commit framework ensures a consistent coding style and standards across the team. By using shared pre-commit configurations, all team members automatically run the same checks before committing their code. This consistency helps in maintaining code quality and reduces the time spent on reviewing code for style compliance.

Pre-commit has a strong community that contributes to a shared repository of pre-commit hooks. These community-made hooks cover a wide range of languages and use cases and provides a well-maintained, regularly updated tools. Hooks in the pre-commit framework are easily reusable across multiple projects. . This reusability saves time and effort in setting up project environments and ensures that new projects adhere to established standards right from the start.

Using Pre-commit

To set up Pre-commit in a Python project, follow these simple steps:

  repos:
    - repo: https://github.com/pre-commit/pre-commit-hooks
      rev: v4.5.0
      hooks:
        - id: trailing-whitespace

Using Pre-Commit in VSCode

pre-commit extension in the Marketplace helps to run pre-commit directly into VSCode.

image-20231124093552626

In summary, the pre-commit framework offers a robust solution for automating and standardizing code quality checks. Its integration into the Git workflow, combined with the support of a strong community and the ease of reusing hooks across projects, makes it an invaluable tool for maintaining high-quality code in collaborative environments.

Pre-commit + Ruff: Simplifying Coding

Now that we have explained how both tools are working, here is below the configuration to make them work seamlessly together.

The Configuration Files !

📄 pyproject.toml (Ruff)

[tool.ruff]
# Exclude common directories that are typically not part of the source code or are generated by tools.
exclude = [
    ".bzr",
    ".direnv",
    ".eggs",
    ".git",
    ".git-rewrite",
    ".hg",
    ".mypy_cache",
    ".nox",
    ".pants.d",
    ".pytype",
    ".ruff_cache",
    ".svn",
    ".tox",
    ".venv",
    "__pypackages__",
    "_build",
    "buck-out",
    "build",
    "dist",
    "node_modules",
    "venv",
    "business-facing/layer", #OKAPI Layer
]

# Set the maximum line length to 127 characters.
line-length = 127

# Define the number of spaces used for indentation, aligning with Black's style.
indent-width = 4

# The minimum Python version to target, e.g., when considering automatic code upgrades, 
# like rewriting type annotations
target-version = "py38"

[tool.ruff.lint]
# Enable Pyflakes (F) and a subset of the pycodestyle (E)  codes by default.
# pycodestyle warnings (W)
# Activate Security Rules (S) to replace bandit
# Enable the isort rules (I) to replace isort
# flake8-bugbear (B)
# flake8-simplify (SIM)
select = ["F", "E4", "E7", "E9", "W", "S", "I", "B","SIM"] 
ignore = [] # List any rules to be ignored, currently empty.

# Allow auto-fixing of all enabled rules when using the `--fix` option.
fixable = ["ALL"]
unfixable = [] # Specify rules that cannot be auto-fixed, if any.

# Define a regex pattern for allowed unused variables (typically underscore-prefixed).
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"

[tool.ruff.format]
# Enforce double quotes for strings, following Black's style.
quote-style = "double"

# Use spaces for indentation, in line with Black's formatting style.
indent-style = "space"

# Keep magic trailing commas, a feature of Black's formatting.
skip-magic-trailing-comma = false

# Automatically detect and use the appropriate line ending style.
line-ending = "auto"

📄 .pre-commit-config.yaml (Pre-Commit)

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: end-of-file-fixer
        exclude: "business-facing/layer"
      - id: trailing-whitespace
        exclude: "business-facing/layer"
      - id: check-yaml
        exclude: "business-facing/layer"
      - id: check-json
        exclude: "business-facing/layer"

  - repo: https://github.com/astral-sh/ruff-pre-commit
  # Ruff version.
    rev: v0.1.6
    hooks:
      # Run the linter.
      - id: ruff
      # Run the formatter.
      - id: ruff-format

Usage

Here is a concrete usage :

image-20231125171423146

image-20231125171543782

It doesn’t add more time to your workflow but add a local check capacity quite smoothly.

Conclusion

As you have seen, there is several advantages to this combination.

In summary, using Ruff with pre-commit hooks makes coding and teamwork smoother, ensures high-quality code, and saves time. This approach is really helpful for teams and makes managing code a lot easier.

PS: I didn’t explore in this article cookiecutter and poetry that complement perfectly Ruff and pre-commit.

References

  1. Ruff Documentation.
  2. Ruff Rules
  3. Ruff GitHub Repository
  4. Ruff PyPI Package
  5. Pre-commit Github
  6. List of Pre-Commit Hooks
  7. Article on Pre-commit