Skip to content

Python 3.13 support: wrapping update #359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 40 commits into
base: improve-thread-caching-separated
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
116d7b2
Test 3.13 as well for unit tests and e2e tests
bitterpanda63 Apr 14, 2025
9d9ad25
Benchmarks for all supported python versions
bitterpanda63 Apr 14, 2025
72cd57d
init tests as a module
bitterpanda63 Apr 14, 2025
8336b97
Refactor pymysql to use wrapt
bitterpanda63 Apr 14, 2025
dde3ba9
(poetry) install wrapt
bitterpanda63 Apr 14, 2025
47058fd
Update subprocess patching
bitterpanda63 Apr 14, 2025
32034d4
pymysql patches change order
bitterpanda63 Apr 14, 2025
7dc0a6f
move back to bottom, python order for pymysql, subprocess
bitterpanda63 Apr 14, 2025
e16b8ed
convert os_system patch
bitterpanda63 Apr 14, 2025
15a28eb
pymongo convert code to wrapt code
bitterpanda63 Apr 14, 2025
23e5fcf
convert mysqlclient
bitterpanda63 Apr 14, 2025
76894de
convert io module
bitterpanda63 Apr 14, 2025
afe9b77
Convert shutil module
bitterpanda63 Apr 14, 2025
142770d
Convert asyncpg module
bitterpanda63 Apr 14, 2025
fb309e7
Convert builtins.py
bitterpanda63 Apr 14, 2025
91ca20d
define new standards for the wrapping
bitterpanda63 Apr 14, 2025
dd9b4b0
update existing modules
bitterpanda63 Apr 14, 2025
9d37dc9
Update io module to use new system
bitterpanda63 Apr 14, 2025
1dab0f2
Update both builtins.py and asyncpg.py to use the new patching system
bitterpanda63 Apr 14, 2025
cffd729
Cleanup checks for builtins and shutil
bitterpanda63 Apr 14, 2025
0196b11
linting for asyncpg.py
bitterpanda63 Apr 14, 2025
6a9711d
Cleanup the patching module
bitterpanda63 Apr 14, 2025
57faa80
Convert os.py
bitterpanda63 Apr 14, 2025
e8c50d3
Update http_client.py sink, adding the @after
bitterpanda63 Apr 14, 2025
87892fd
Update psycopg sink
bitterpanda63 Apr 14, 2025
fa9f54f
convert socket.py to new format
bitterpanda63 Apr 14, 2025
cb37a0c
Fix shutil and shutil test cases
bitterpanda63 Apr 14, 2025
06e7abd
Convert psycopg2 module
bitterpanda63 Apr 14, 2025
6af31a4
update sample app lockfiles
bitterpanda63 Apr 14, 2025
7834861
Update actions/setup-python v2 -> v5
bitterpanda63 Apr 15, 2025
70305bf
Fix @after by removing finally: which was swallowing the error
bitterpanda63 Apr 15, 2025
35c4c82
Fix broken subprocess test cases
bitterpanda63 Apr 15, 2025
3618197
Remove ms checks for starlette benchmark in favour of percentages
bitterpanda63 Apr 15, 2025
31705c1
Exclude psycopg2 testing for python 3.13
bitterpanda63 Apr 15, 2025
3aa49cc
Make sure psycopg2.py implementation is the same as prev one
bitterpanda63 Apr 15, 2025
807b981
Add extra tests for psycopg2 and fix issue for immutables
bitterpanda63 Apr 15, 2025
b8eec8f
use assert_any_call for python 3.13
bitterpanda63 Apr 15, 2025
0aabc2c
Patch os.path.realpath for python 3.13
bitterpanda63 Apr 16, 2025
b29cc3e
Patch os.path.realpath for python 3.13
bitterpanda63 Apr 16, 2025
5a2037c
Merge remote-tracking branch 'origin/python-3.13-support' into python…
bitterpanda63 Apr 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 21 additions & 6 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
name: Benchmark
name: 📊 Benchmarks
on:
push: {}
workflow_call: {}

jobs:
benchmark_sql_algorithm:
runs-on: ubuntu-latest
Expand All @@ -13,7 +14,7 @@ jobs:
working-directory: ./sample-apps/databases
run: docker compose up --build -d
- name: Set up Python 3.9
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: 3.9

Expand All @@ -30,15 +31,24 @@ jobs:
run: |
poetry run python ./benchmarks/sql_benchmark/sql_benchmark_fw.py
poetry run python ./benchmarks/sql_benchmark/sql_benchmark_no_fw.py

benchmark_with_flask_mysql:
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Start databases
working-directory: ./sample-apps/databases
run: docker compose up --build -d
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies & build
run: |
python -m pip install --upgrade pip
Expand All @@ -51,15 +61,24 @@ jobs:
- name: Run flask-mysql k6 Benchmark
run: |
k6 run -q ./benchmarks/flask-mysql-benchmarks.js

benchmark_with_starlette_app:
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Start databases
working-directory: ./sample-apps/databases
run: docker compose up --build -d
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies & build
run: |
python -m pip install --upgrade pip
Expand All @@ -71,9 +90,5 @@ jobs:
run: |
sudo apt-get update
sudo apt-get install -y wrk
- name: Set up Python 3.9
uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Run benchmark
run: python ./benchmarks/starlette_benchmark.py
4 changes: 2 additions & 2 deletions .github/workflows/end2end.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
- { name: flask-postgres-xml, testfile: end2end/flask_postgres_xml_lxml_test.py }
- { name: quart-postgres-uvicorn, testfile: end2end/quart_postgres_uvicorn_test.py }
- { name: starlette-postgres-uvicorn, testfile: end2end/starlette_postgres_uvicorn_test.py }
python-version: ["3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- name: Install packages
run: sudo apt update && sudo apt install python3-dev libmysqlclient-dev
Expand All @@ -53,7 +53,7 @@ jobs:
docker run --name mock_core -d -p 5000:5000 mock_core

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: '3.x'

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/smoke-test-ffi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: 3.12

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/unit-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
# Don't cancel jobs if one fails
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
steps:
- name: Checkout code
uses: actions/checkout@v4
Expand All @@ -22,7 +22,7 @@ jobs:
run: |
sudo echo "127.0.0.1 local.aikido.io" | sudo tee -a /etc/hosts
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

Expand Down
70 changes: 70 additions & 0 deletions aikido_zen/sinks/__init__.py
Original file line number Diff line number Diff line change
@@ -1,0 +1,70 @@
from wrapt import wrap_object, FunctionWrapper, when_imported
from aikido_zen.background_process.packages import ANY_VERSION, is_package_compatible
from aikido_zen.errors import AikidoException
from aikido_zen.helpers.logging import logger


def on_import(name, package="", version_requirement=ANY_VERSION):
"""
Decorator to register a function to be called when a package is imported.
It checks if the package is compatible with the specified version requirement.
"""

def decorator(func):
if package and not is_package_compatible(package, version_requirement):
return

Check warning on line 15 in aikido_zen/sinks/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_zen/sinks/__init__.py#L15

Added line #L15 was not covered by tests

when_imported(name)(func) # Register the function to be called on import

return decorator


def patch_function(module, name, wrapper):
"""
Patches a function in the specified module with a wrapper function.
"""
try:
wrap_object(module, name, FunctionWrapper, (wrapper,))
except Exception as e:
logger.info("Failed to wrap %s:%s, due to: %s", module, name, e)

Check warning on line 29 in aikido_zen/sinks/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_zen/sinks/__init__.py#L28-L29

Added lines #L28 - L29 were not covered by tests


def before(wrapper):
"""
Surrounds a patch with try-except and calls the original function at the end
"""

def decorator(func, instance, args, kwargs):
try:
wrapper(func, instance, args, kwargs) # Call the patch
except AikidoException as e:
raise e # Re-raise AikidoException
except Exception as e:
logger.debug(

Check warning on line 43 in aikido_zen/sinks/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_zen/sinks/__init__.py#L40-L43

Added lines #L40 - L43 were not covered by tests
"%s:%s wrapping-before error: %s", func.__module__, func.__name__, e
)

return func(*args, **kwargs) # Call the original function

return decorator


def after(wrapper):
"""
Surrounds a patch with try-except, calls the original function and gives the return value to the patch
"""

def decorator(func, instance, args, kwargs):
return_value = func(*args, **kwargs) # Call the original function
try:
wrapper(func, instance, args, kwargs, return_value) # Call the patch
except AikidoException as e:
raise e # Re-raise AikidoException
except Exception as e:
logger.debug(

Check warning on line 64 in aikido_zen/sinks/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_zen/sinks/__init__.py#L63-L64

Added lines #L63 - L64 were not covered by tests
"%s:%s wrapping-after error: %s", func.__module__, func.__name__, e
)

return return_value

return decorator
70 changes: 17 additions & 53 deletions aikido_zen/sinks/asyncpg.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,64 +2,28 @@
Sink module for `asyncpg`
"""

import copy
import aikido_zen.importhook as importhook
from aikido_zen.background_process.packages import is_package_compatible
import aikido_zen.vulnerabilities as vulns
from aikido_zen.helpers.logging import logger
from aikido_zen.helpers.get_argument import get_argument
from aikido_zen.sinks import patch_function, before, on_import

REQUIRED_ASYNCPG_VERSION = "0.27.0"


@importhook.on_import("asyncpg.connection")
def on_asyncpg_import(asyncpg):
@on_import("asyncpg.connection", "asyncpg", version_requirement="0.27.0")
def patch(m):
"""
Hook 'n wrap on `asyncpg.connection`
* the Cursor classes in asyncpg.cursor are only used to fetch data. (Currently not supported)
* Pool class uses Connection class (Wrapping supported for Connection class)
* _execute(...) get's called by all except execute and executemany
Our goal is to wrap the _execute(), execute(), executemany() functions in Connection class :
https://github.com/MagicStack/asyncpg/blob/85d7eed40637e7cad73a44ed2439ffeb2a8dc1c2/asyncpg/connection.py#L43
Returns : Modified asyncpg.connection object
patching module asyncpg.connection
- patches Connection.execute, Connection.executemany, Connection._execute
- doesn't patch Cursor class -> are only used to fetch data.
- doesn't patch Pool class -> uses Connection class
src: https://github.com/MagicStack/asyncpg/blob/85d7eed40637e7cad73a44ed2439ffeb2a8dc1c2/asyncpg/connection.py#L43
"""
if not is_package_compatible("asyncpg", REQUIRED_ASYNCPG_VERSION):
return asyncpg
modified_asyncpg = importhook.copy_module(asyncpg)

# pylint: disable=protected-access # We need to wrap this function
former__execute = copy.deepcopy(asyncpg.Connection._execute)
former_executemany = copy.deepcopy(asyncpg.Connection.executemany)
former_execute = copy.deepcopy(asyncpg.Connection.execute)

def aikido_new__execute(_self, query, *args, **kwargs):
vulns.run_vulnerability_scan(
kind="sql_injection",
op="asyncpg.connection.Connection._execute",
args=(query, "postgres"),
)

return former__execute(_self, query, *args, **kwargs)

def aikido_new_executemany(_self, query, *args, **kwargs):
# This query is just a string, not a list, see docs.
vulns.run_vulnerability_scan(
kind="sql_injection",
op="asyncpg.connection.Connection.executemany",
args=(query, "postgres"),
)
return former_executemany(_self, query, *args, **kwargs)
patch_function(m, "Connection.execute", _execute)
patch_function(m, "Connection.executemany", _execute)
patch_function(m, "Connection._execute", _execute)

def aikido_new_execute(_self, query, *args, **kwargs):
vulns.run_vulnerability_scan(
kind="sql_injection",
op="asyncpg.connection.Connection.execute",
args=(query, "postgres"),
)
return former_execute(_self, query, *args, **kwargs)

# pylint: disable=no-member
setattr(asyncpg.Connection, "_execute", aikido_new__execute)
setattr(asyncpg.Connection, "executemany", aikido_new_executemany)
setattr(asyncpg.Connection, "execute", aikido_new_execute)
@before
def _execute(func, instance, args, kwargs):
query = get_argument(args, kwargs, 0, "query")

return modified_asyncpg
op = f"asyncpg.connection.Connection.{func.__name__}"
vulns.run_vulnerability_scan(kind="sql_injection", op=op, args=(query, "postgres"))
38 changes: 15 additions & 23 deletions aikido_zen/sinks/builtins.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,34 +3,26 @@
"""

from pathlib import PurePath
import aikido_zen.importhook as importhook
import aikido_zen.vulnerabilities as vulns
from aikido_zen.helpers.get_argument import get_argument
from aikido_zen.sinks import patch_function, on_import, before


def aikido_open_decorator(func):
"""Decorator for open(...)"""
@before
def _open(func, instance, args, kwargs):
filename = get_argument(args, kwargs, 0, "filename")
if not isinstance(filename, (str, bytes, PurePath)):
return

def wrapper(*args, **kwargs):
# args[0] is thefunc_name filename
if len(args) > 0 and isinstance(args[0], (str, bytes, PurePath)):
vulns.run_vulnerability_scan(
kind="path_traversal", op="builtins.open", args=(args[0],)
)
return func(*args, **kwargs)
vulns.run_vulnerability_scan(
kind="path_traversal", op="builtins.open", args=(filename,)
)

return wrapper


@importhook.on_import("builtins")
def on_builtins_import(builtins):
@on_import("builtins")
def patch(m):
"""
Hook 'n wrap on `builtins`, python's built-in functions
Our goal is to wrap the open() function, which you use when opening files
Returns : Modified builtins object
patching module builtins
- patches builtins.open
"""
modified_builtins = importhook.copy_module(builtins)

# pylint: disable=no-member
setattr(builtins, "open", aikido_open_decorator(builtins.open))
setattr(modified_builtins, "open", aikido_open_decorator(builtins.open))
return modified_builtins
patch_function(m, "open", _open)
Loading
Loading