Skip to content

Commit f107ff8

Browse files
PwutsPwuts
authored
Set up unified pre-commit + CI w/ linting + type checking & FIX EVERYTHING (Significant-Gravitas#7171)
- **FIX ALL LINT/TYPE ERRORS IN AUTOGPT, FORGE, AND BENCHMARK** ### Linting - Clean up linter configs for `autogpt`, `forge`, and `benchmark` - Add type checking with Pyright - Create unified pre-commit config - Create unified linting and type checking CI workflow ### Testing - Synchronize CI test setups for `autogpt`, `forge`, and `benchmark` - Add missing pytest-cov to benchmark dependencies - Mark GCS tests as slow to speed up pre-commit test runs - Repair `forge` test suite - Add `AgentDB.close()` method for test DB teardown in db_test.py - Use actual temporary dir instead of forge/test_workspace/ - Move left-behind dependencies for moved `forge`-code to from autogpt to forge ### Notable type changes - Replace uses of `ChatModelProvider` by `MultiProvider` - Removed unnecessary exports from various __init__.py - Simplify `FileStorage.open_file` signature by removing `IOBase` from return type union - Implement `S3BinaryIOWrapper(BinaryIO)` type interposer for `S3FileStorage` - Expand overloads of `GCSFileStorage.open_file` for improved typing of read and write modes Had to silence type checking for the extra overloads, because (I think) Pyright is reporting a false-positive: microsoft/pyright#8007 - Change `count_tokens`, `get_tokenizer`, `count_message_tokens` methods on `ModelProvider`s from class methods to instance methods - Move `CompletionModelFunction.schema` method -> helper function `format_function_def_for_openai` in `forge.llm.providers.openai` - Rename `ModelProvider` -> `BaseModelProvider` - Rename `ChatModelProvider` -> `BaseChatModelProvider` - Add type `ChatModelProvider` which is a union of all subclasses of `BaseChatModelProvider` ### Removed rather than fixed - Remove deprecated and broken autogpt/agbenchmark_config/benchmarks.py - Various base classes and properties on base classes in `forge.llm.providers.schema` and `forge.models.providers` ### Fixes for other issues that came to light - Clean up `forge.agent_protocol.api_router`, `forge.agent_protocol.database`, and `forge.agent.agent` - Add fallback behavior to `ImageGeneratorComponent` - Remove test for deprecated failure behavior - Fix `agbenchmark.challenges.builtin` challenge exclusion mechanism on Windows - Fix `_tool_calls_compat_extract_calls` in `forge.llm.providers.openai` - Add support for `any` (= no type specified) in `JSONSchema.typescript_type`
1 parent 2c13a27 commit f107ff8

File tree

147 files changed

+2905
-2433
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

147 files changed

+2905
-2433
lines changed

‎.github/workflows/autogpt-ci.yml

Lines changed: 1 addition & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: AutoGPT Python CI
1+
name: AutoGPT CI
22

33
on:
44
push:
@@ -24,57 +24,6 @@ defaults:
2424
working-directory: autogpt
2525

2626
jobs:
27-
lint:
28-
runs-on: ubuntu-latest
29-
env:
30-
min-python-version: "3.10"
31-
32-
steps:
33-
- name: Checkout repository
34-
uses: actions/checkout@v4
35-
with:
36-
fetch-depth: 0
37-
38-
- name: Set up Python ${{ env.min-python-version }}
39-
uses: actions/setup-python@v5
40-
with:
41-
python-version: ${{ env.min-python-version }}
42-
43-
- id: get_date
44-
name: Get date
45-
run: echo "date=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT
46-
47-
- name: Set up Python dependency cache
48-
uses: actions/cache@v4
49-
with:
50-
path: ~/.cache/pypoetry
51-
key: ${{ runner.os }}-poetry-${{ hashFiles('autogpt/pyproject.toml') }}-${{ steps.get_date.outputs.date }}
52-
53-
- name: Install Python dependencies
54-
run: |
55-
curl -sSL https://install.python-poetry.org | python3 -
56-
poetry install
57-
58-
- name: Lint with flake8
59-
run: poetry run flake8
60-
61-
- name: Check black formatting
62-
run: poetry run black . --check
63-
if: success() || failure()
64-
65-
- name: Check isort formatting
66-
run: poetry run isort . --check
67-
if: success() || failure()
68-
69-
# - name: Check mypy formatting
70-
# run: poetry run mypy
71-
# if: success() || failure()
72-
73-
# - name: Check for unused imports and pass statements
74-
# run: |
75-
# cmd="autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports --ignore-pass-after-docstring autogpt tests"
76-
# poetry run $cmd --check || (echo "You have unused imports or pass statements, please run '${cmd} --in-place'" && exit 1)
77-
7827
test:
7928
permissions:
8029
contents: read

‎.github/workflows/autogpts-ci.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: AutoGPTs smoke test CI
1+
name: Agent smoke tests
22

33
on:
44
workflow_dispatch:
@@ -28,7 +28,7 @@ on:
2828
- '!**/*.md'
2929

3030
jobs:
31-
run-tests:
31+
serve-agent-protocol:
3232
runs-on: ubuntu-latest
3333
strategy:
3434
matrix:

‎.github/workflows/benchmark-ci.yml

Lines changed: 65 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Benchmark CI
1+
name: AGBenchmark CI
22

33
on:
44
push:
@@ -14,62 +14,91 @@ on:
1414
- '!benchmark/reports/**'
1515
- .github/workflows/benchmark-ci.yml
1616

17+
concurrency:
18+
group: ${{ format('benchmark-ci-{0}', github.head_ref && format('{0}-{1}', github.event_name, github.event.pull_request.number) || github.sha) }}
19+
cancel-in-progress: ${{ startsWith(github.event_name, 'pull_request') }}
20+
21+
defaults:
22+
run:
23+
shell: bash
24+
1725
env:
1826
min-python-version: '3.10'
1927

2028
jobs:
21-
lint:
22-
runs-on: ubuntu-latest
23-
29+
test:
30+
permissions:
31+
contents: read
32+
timeout-minutes: 30
33+
strategy:
34+
fail-fast: false
35+
matrix:
36+
python-version: ["3.10"]
37+
platform-os: [ubuntu, macos, macos-arm64, windows]
38+
runs-on: ${{ matrix.platform-os != 'macos-arm64' && format('{0}-latest', matrix.platform-os) || 'macos-14' }}
39+
defaults:
40+
run:
41+
shell: bash
42+
working-directory: benchmark
2443
steps:
2544
- name: Checkout repository
2645
uses: actions/checkout@v4
2746
with:
2847
fetch-depth: 0
48+
submodules: true
2949

30-
- name: Set up Python ${{ env.min-python-version }}
50+
- name: Set up Python ${{ matrix.python-version }}
3151
uses: actions/setup-python@v5
3252
with:
33-
python-version: ${{ env.min-python-version }}
53+
python-version: ${{ matrix.python-version }}
3454

35-
- id: get_date
36-
name: Get date
37-
working-directory: ./benchmark/
38-
run: echo "date=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT
55+
- name: Set up Python dependency cache
56+
# On Windows, unpacking cached dependencies takes longer than just installing them
57+
if: runner.os != 'Windows'
58+
uses: actions/cache@v4
59+
with:
60+
path: ${{ runner.os == 'macOS' && '~/Library/Caches/pypoetry' || '~/.cache/pypoetry' }}
61+
key: poetry-${{ runner.os }}-${{ hashFiles('benchmark/poetry.lock') }}
3962

40-
- name: Install Poetry
41-
working-directory: ./benchmark/
63+
- name: Install Poetry (Unix)
64+
if: runner.os != 'Windows'
4265
run: |
43-
curl -sSL https://install.python-poetry.org | python -
66+
curl -sSL https://install.python-poetry.org | python3 -
4467
45-
- name: Install dependencies
46-
working-directory: ./benchmark/
47-
run: |
48-
export POETRY_VIRTUALENVS_IN_PROJECT=true
49-
poetry install -vvv
68+
if [ "${{ runner.os }}" = "macOS" ]; then
69+
PATH="$HOME/.local/bin:$PATH"
70+
echo "$HOME/.local/bin" >> $GITHUB_PATH
71+
fi
5072
51-
- name: Lint with flake8
52-
working-directory: ./benchmark/
53-
run: poetry run flake8
73+
- name: Install Poetry (Windows)
74+
if: runner.os == 'Windows'
75+
shell: pwsh
76+
run: |
77+
(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | python -
5478
55-
- name: Check black formatting
56-
working-directory: ./benchmark/
57-
run: poetry run black . --exclude test.py --check
58-
if: success() || failure()
79+
$env:PATH += ";$env:APPDATA\Python\Scripts"
80+
echo "$env:APPDATA\Python\Scripts" >> $env:GITHUB_PATH
5981
60-
- name: Check isort formatting
61-
working-directory: ./benchmark/
62-
run: poetry run isort . --check
63-
if: success() || failure()
82+
- name: Install Python dependencies
83+
run: poetry install
6484

65-
- name: Check for unused imports and pass statements
66-
working-directory: ./benchmark/
85+
- name: Run pytest with coverage
6786
run: |
68-
cmd="poetry run autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports --ignore-pass-after-docstring agbenchmark"
69-
$cmd --check || (echo "You have unused imports or pass statements, please run '${cmd} --in-place'" && exit 1)
70-
if: success() || failure()
87+
poetry run pytest -vv \
88+
--cov=agbenchmark --cov-branch --cov-report term-missing --cov-report xml \
89+
--durations=10 \
90+
tests
91+
env:
92+
CI: true
93+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
94+
95+
- name: Upload coverage reports to Codecov
96+
uses: codecov/codecov-action@v4
97+
with:
98+
token: ${{ secrets.CODECOV_TOKEN }}
99+
flags: agbenchmark,${{ runner.os }}
71100

72-
tests-agbenchmark:
101+
self-test-with-agent:
73102
runs-on: ubuntu-latest
74103
strategy:
75104
matrix:
@@ -89,11 +118,11 @@ jobs:
89118
python-version: ${{ env.min-python-version }}
90119

91120
- name: Install Poetry
92-
working-directory: ./${{ matrix.agent-name }}/
93121
run: |
94122
curl -sSL https://install.python-poetry.org | python -
95123
96124
- name: Run regression tests
125+
working-directory: .
97126
run: |
98127
./run agent start ${{ matrix.agent-name }}
99128
cd ${{ matrix.agent-name }}
@@ -125,7 +154,6 @@ jobs:
125154
export BUILD_SKILL_TREE=true
126155
127156
poetry run agbenchmark --mock
128-
poetry run pytest -vv -s tests
129157
130158
CHANGED=$(git diff --name-only | grep -E '(agbenchmark/challenges)|(../frontend/assets)') || echo "No diffs"
131159
if [ ! -z "$CHANGED" ]; then

‎.github/workflows/forge-ci.yml

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
name: Forge CI
2+
3+
on:
4+
push:
5+
branches: [ master, development, ci-test* ]
6+
paths:
7+
- '.github/workflows/forge-ci.yml'
8+
- 'forge/**'
9+
pull_request:
10+
branches: [ master, development, release-* ]
11+
paths:
12+
- '.github/workflows/forge-ci.yml'
13+
- 'forge/**'
14+
15+
concurrency:
16+
group: ${{ format('forge-ci-{0}', github.head_ref && format('{0}-{1}', github.event_name, github.event.pull_request.number) || github.sha) }}
17+
cancel-in-progress: ${{ startsWith(github.event_name, 'pull_request') }}
18+
19+
defaults:
20+
run:
21+
shell: bash
22+
working-directory: forge
23+
24+
jobs:
25+
test:
26+
permissions:
27+
contents: read
28+
timeout-minutes: 30
29+
strategy:
30+
fail-fast: false
31+
matrix:
32+
python-version: ["3.10"]
33+
platform-os: [ubuntu, macos, macos-arm64, windows]
34+
runs-on: ${{ matrix.platform-os != 'macos-arm64' && format('{0}-latest', matrix.platform-os) || 'macos-14' }}
35+
36+
steps:
37+
# Quite slow on macOS (2~4 minutes to set up Docker)
38+
# - name: Set up Docker (macOS)
39+
# if: runner.os == 'macOS'
40+
# uses: crazy-max/ghaction-setup-docker@v3
41+
42+
- name: Start MinIO service (Linux)
43+
if: runner.os == 'Linux'
44+
working-directory: '.'
45+
run: |
46+
docker pull minio/minio:edge-cicd
47+
docker run -d -p 9000:9000 minio/minio:edge-cicd
48+
49+
- name: Start MinIO service (macOS)
50+
if: runner.os == 'macOS'
51+
working-directory: ${{ runner.temp }}
52+
run: |
53+
brew install minio/stable/minio
54+
mkdir data
55+
minio server ./data &
56+
57+
# No MinIO on Windows:
58+
# - Windows doesn't support running Linux Docker containers
59+
# - It doesn't seem possible to start background processes on Windows. They are
60+
# killed after the step returns.
61+
# See: https://github.com/actions/runner/issues/598#issuecomment-2011890429
62+
63+
- name: Checkout repository
64+
uses: actions/checkout@v4
65+
with:
66+
fetch-depth: 0
67+
submodules: true
68+
69+
- name: Set up Python ${{ matrix.python-version }}
70+
uses: actions/setup-python@v5
71+
with:
72+
python-version: ${{ matrix.python-version }}
73+
74+
- name: Set up Python dependency cache
75+
# On Windows, unpacking cached dependencies takes longer than just installing them
76+
if: runner.os != 'Windows'
77+
uses: actions/cache@v4
78+
with:
79+
path: ${{ runner.os == 'macOS' && '~/Library/Caches/pypoetry' || '~/.cache/pypoetry' }}
80+
key: poetry-${{ runner.os }}-${{ hashFiles('forge/poetry.lock') }}
81+
82+
- name: Install Poetry (Unix)
83+
if: runner.os != 'Windows'
84+
run: |
85+
curl -sSL https://install.python-poetry.org | python3 -
86+
87+
if [ "${{ runner.os }}" = "macOS" ]; then
88+
PATH="$HOME/.local/bin:$PATH"
89+
echo "$HOME/.local/bin" >> $GITHUB_PATH
90+
fi
91+
92+
- name: Install Poetry (Windows)
93+
if: runner.os == 'Windows'
94+
shell: pwsh
95+
run: |
96+
(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | python -
97+
98+
$env:PATH += ";$env:APPDATA\Python\Scripts"
99+
echo "$env:APPDATA\Python\Scripts" >> $env:GITHUB_PATH
100+
101+
- name: Install Python dependencies
102+
run: poetry install
103+
104+
- name: Run pytest with coverage
105+
run: |
106+
poetry run pytest -vv \
107+
--cov=forge --cov-branch --cov-report term-missing --cov-report xml \
108+
--durations=10 \
109+
forge
110+
env:
111+
CI: true
112+
PLAIN_OUTPUT: True
113+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
114+
S3_ENDPOINT_URL: ${{ runner.os != 'Windows' && 'http://127.0.0.1:9000' || '' }}
115+
AWS_ACCESS_KEY_ID: minioadmin
116+
AWS_SECRET_ACCESS_KEY: minioadmin
117+
118+
- name: Upload coverage reports to Codecov
119+
uses: codecov/codecov-action@v4
120+
with:
121+
token: ${{ secrets.CODECOV_TOKEN }}
122+
flags: forge,${{ runner.os }}
123+
124+
- name: Upload logs to artifact
125+
if: always()
126+
uses: actions/upload-artifact@v4
127+
with:
128+
name: test-logs
129+
path: forge/logs/

0 commit comments

Comments
 (0)