Skip to content

Add scanning to released images and process to track vulnerabilities #28087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Apr 16, 2025
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
a3cb7d1
Add security scanning and remediation process
lucasmrod Apr 9, 2025
58ec94b
Add CI checks for vulnerabilities in fleetdm/fleet before publishing
lucasmrod Apr 7, 2025
079c187
Add VEX to Github Trivy actions
lucasmrod Apr 10, 2025
c9e530b
Remove accidental merge
lucasmrod Apr 10, 2025
9ac1139
Try not using the trivy github action and use binary instead
lucasmrod Apr 10, 2025
5ee69c7
Use newer version
lucasmrod Apr 10, 2025
20688d5
Add suppression
lucasmrod Apr 10, 2025
adaffd8
Fixes
lucasmrod Apr 10, 2025
409d238
Add VEX files for CVEs found in older versions of Fleet
lucasmrod Apr 10, 2025
7716b1c
Add new CI action to check for vulns in released docker images
lucasmrod Apr 10, 2025
ff6f268
Fix iteration
lucasmrod Apr 10, 2025
d31094e
Updates to security/README.md
lucasmrod Apr 11, 2025
5515d73
Remove download of trivy
lucasmrod Apr 11, 2025
61a2b48
More updates
lucasmrod Apr 11, 2025
9175588
Add TODO for Luke
lucasmrod Apr 11, 2025
49fdb83
Fix invalid commands
lucasmrod Apr 14, 2025
d70cccf
Merge branch 'main' into 25902-security-scanning
lucasmrod Apr 14, 2025
fc472d6
Add docker scout to scanning
lucasmrod Apr 15, 2025
e92d73d
Remove docker scouts from scanning released images
lucasmrod Apr 15, 2025
85aa043
Add more critical exceptions
lucasmrod Apr 15, 2025
7a7dd3c
Try out docker scout
lucasmrod Apr 15, 2025
1d72383
Fix yaml
lucasmrod Apr 15, 2025
a55cc63
Remove comments
lucasmrod Apr 15, 2025
87fe1dd
Add checks to not run vuln check on every PR
lucasmrod Apr 15, 2025
8ad10ef
Add steps for critical vulns on latest
lucasmrod Apr 16, 2025
ba70dfb
Update security/README.md
sharon-fdm Apr 16, 2025
81a36e9
Update security/README.md
sharon-fdm Apr 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 45 additions & 26 deletions .github/workflows/build-and-check-fleetctl-docker-and-deps.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,6 @@ defaults:
# fail-fast using bash -eo pipefail. See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#exit-codes-and-error-action-preference
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chat with Lucas:
We have Docker images to cross compile fleetd packages.
This is a github action check for vulns in them.

shell: bash

env:
AWS_REGION: us-east-2
AWS_IAM_ROLE: arn:aws:iam::160035666661:role/github-actions-role

permissions:
contents: read

Expand All @@ -37,17 +33,6 @@ jobs:
- name: Checkout
uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3

- uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e071d42e0343502 # v4.0.2
with:
role-to-assume: ${{env.AWS_IAM_ROLE}}
aws-region: ${{ env.AWS_REGION }}

- name: Login to Docker Hub
uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}

- name: Set up Go
uses: actions/setup-go@93397bea11091df50f3d7e59dc26a7711a8bcfbe # v4.1.0
with:
Expand All @@ -62,6 +47,32 @@ jobs:
- name: Build fleetdm/fleetctl
run: make fleetctl-docker

- name: List VEX files
id: generate_vex_files
run: |
echo "VEX_FILES=$(ls -1 ./security/vex/fleetctl/ | while IFS= read -r line; do echo "./security/vex/fleetctl/$line"; done | tr '\n' ',' | sed 's/.$//')" >> $GITHUB_OUTPUT

# We use the trivy command and not the github action because it doesn't support loading VEX files yet.
- name: Run Trivy vulnerability scanner on fleetdm/fleetctl
env:
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db
run: |
mkdir trivy-download
cd trivy-download
curl -L https://github.com/aquasecurity/trivy/releases/download/v0.61.0/trivy_0.61.0_Linux-64bit.tar.gz --output trivy_0.61.0_Linux-64bit.tar.gz
tar -xf trivy_0.61.0_Linux-64bit.tar.gz
mv trivy ..
cd ..
chmod +x ./trivy
Comment on lines +61 to +67
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like a candidate for github action caching, but nbd if it doesn't take a long time

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems this step takes 13s. So maybe a-ok for now.

./trivy image \
--exit-code=1 \
--ignore-unfixed \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.
We want to see unfixed as well.

--pkg-types=os,library \
--severity=HIGH,CRITICAL \
--vex="${{ steps.generate_vex_files.outputs.VEX_FILES }}" \
fleetdm/fleetctl

- name: Run Trivy vulnerability scanner on fleetdm/wix
uses: aquasecurity/trivy-action@6e7b7d1fd3e4fef0c5fa8cce1229c54b2c9bd0d8
env:
Expand All @@ -88,15 +99,23 @@ jobs:
vuln-type: "os,library"
severity: "CRITICAL"

- name: Run Trivy vulnerability scanner on fleetdm/fleetctl
uses: aquasecurity/trivy-action@6e7b7d1fd3e4fef0c5fa8cce1229c54b2c9bd0d8
env:
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db
- name: Slack root notification
if: github.event.schedule == '0 6 * * *' && failure()
uses: slackapi/slack-github-action@e28cf165c92ffef168d23c5c9000cffc8a25e117 # v1.24.0
with:
image-ref: "fleetdm/fleetctl"
format: "table"
exit-code: "1"
ignore-unfixed: true
vuln-type: "os,library"
severity: "CRITICAL"
payload: |
{
"text": "${{ job.status }}\n${{ github.event.pull_request.html_url || github.event.head.html_url }}",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "⚠️ Build fleetctl docker dependencies and check vulnerabilities failed.\nhttps://github.com/fleetdm/fleet/actions/runs/${{ github.run_id }}"
}
}
]
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_G_HELP_ENGINEERING_WEBHOOK_URL }}
SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
name: Check vulnerabilities in released docker images
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check for vulns in docker images that we have already released. (Fleet server)


on:
workflow_dispatch:
pull_request:
schedule:
- cron: "0 6 * * *"

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id}}
cancel-in-progress: true

defaults:
run:
# fail-fast using bash -eo pipefail. See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#exit-codes-and-error-action-preference
shell: bash

permissions:
contents: read

jobs:
build-and-check:
runs-on: ubuntu-22.04
environment: Docker Hub
steps:
- name: Harden Runner
uses: step-security/harden-runner@63c24ba6bd7ba022e95695ff85de572c04a18142 # v2.7.0
with:
egress-policy: audit

- name: Checkout
uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3

- name: Set up Go
uses: actions/setup-go@93397bea11091df50f3d7e59dc26a7711a8bcfbe # v4.1.0
with:
go-version-file: "go.mod"

- name: Get last 5 minor releases
id: get_latest_releases
run: |
echo "FLEET_LATEST_RELEASES=$(go run ./tools/github-releases --last-minor-releases 5)" >> $GITHUB_OUTPUT

- name: Pull docker images
run: |
RELEASES="${{ steps.get_latest_releases.outputs.FLEET_LATEST_RELEASES }}"
for version in $RELEASES; do
docker pull fleetdm/fleet:$version
done

- name: List fleet VEX files
id: generate_fleet_vex_files
run: |
VEX_FILES=$(ls -1 ./security/vex/fleet/ | while IFS= read -r line; do echo "./security/vex/fleet/$line"; done | tr '\n' ',' | sed 's/.$//')
echo $VEX_FILES
echo "FLEET_VEX_FILES=$VEX_FILES" >> $GITHUB_OUTPUT

# We use the trivy command and not the github action because it doesn't support loading
# VEX files yet and looks like we can't run the action on multiple images.
- name: Run Trivy vulnerability scanner on fleetdm/fleet images
env:
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db
run: |
mkdir trivy-download
cd trivy-download
curl -L https://github.com/aquasecurity/trivy/releases/download/v0.61.0/trivy_0.61.0_Linux-64bit.tar.gz --output trivy_0.61.0_Linux-64bit.tar.gz
tar -xf trivy_0.61.0_Linux-64bit.tar.gz
mv trivy ..
cd ..
chmod +x ./trivy
RELEASES="${{ steps.get_latest_releases.outputs.FLEET_LATEST_RELEASES }}"
for version in $RELEASES; do
./trivy image \
--exit-code=1 \
--ignore-unfixed \
--pkg-types=os,library \
--severity=HIGH,CRITICAL \
--vex="${{ steps.generate_fleet_vex_files.outputs.FLEET_VEX_FILES }}" \
fleetdm/fleet:$version
done

- name: List fleetctl VEX files
id: generate_fleetctl_vex_files
run: |
VEX_FILES=$(ls -1 ./security/vex/fleetctl/ | while IFS= read -r line; do echo "./security/vex/fleetctl/$line"; done | tr '\n' ',' | sed 's/.$//')
echo $VEX_FILES
echo "FLEETCTL_VEX_FILES=$VEX_FILES" >> $GITHUB_OUTPUT

# We use the trivy command and not the github action because it doesn't support loading VEX files yet.
- name: Run Trivy vulnerability scanner on latest released fleetdm/fleetctl image
env:
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db
run: |
./trivy image \
--exit-code=1 \
--ignore-unfixed \
--pkg-types=os,library \
--severity=HIGH,CRITICAL \
--vex="${{ steps.generate_fleetctl_vex_files.outputs.FLEETCTL_VEX_FILES }}" \
fleetdm/fleetctl:latest

- name: Slack notification
if: github.event.schedule == '0 6 * * *' && failure()
uses: slackapi/slack-github-action@e28cf165c92ffef168d23c5c9000cffc8a25e117 # v1.24.0
with:
payload: |
{
"text": "${{ job.status }}\n${{ github.event.pull_request.html_url || github.event.head.html_url }}",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "⚠️ Check vulnerabilities in released docker images failed.\nhttps://github.com/fleetdm/fleet/actions/runs/${{ github.run_id }}"
}
}
]
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_G_HELP_ENGINEERING_WEBHOOK_URL }}
SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK
56 changes: 55 additions & 1 deletion .github/workflows/goreleaser-snapshot-fleet.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ on:
- "website/**"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check for vulns in fleet server docker images that were not yet released (As part of the release process)

- "mdm-profiles/**"
workflow_dispatch: # Manual
schedule:
- cron: '0 4 * * *' # Every day at 4 AM

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
Expand Down Expand Up @@ -80,11 +82,42 @@ jobs:
- name: Tag image with branch name
run: docker tag fleetdm/fleet:$(git rev-parse --short HEAD) fleetdm/fleet:$(git rev-parse --abbrev-ref HEAD)

- name: Generate tag
id: generate_tag
run: |
echo "FLEET_IMAGE_TAG=$(git rev-parse --abbrev-ref HEAD)" >> $GITHUB_OUTPUT

- name: List VEX files
id: generate_vex_files
run: |
echo "VEX_FILES=$(ls -1 ./security/vex/fleet/ | while IFS= read -r line; do echo "./security/vex/fleet/$line"; done | tr '\n' ',' | sed 's/.$//')" >> $GITHUB_OUTPUT

# We use the trivy command and not the github action because it doesn't support loading VEX files yet.
- name: Check critical vulnerabilities before publishing
env:
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db
run: |
mkdir trivy-download
cd trivy-download
curl -L https://github.com/aquasecurity/trivy/releases/download/v0.61.0/trivy_0.61.0_Linux-64bit.tar.gz --output trivy_0.61.0_Linux-64bit.tar.gz
tar -xf trivy_0.61.0_Linux-64bit.tar.gz
mv trivy ..
cd ..
chmod +x ./trivy
./trivy image \
--exit-code=1 \
--ignore-unfixed \
--pkg-types=os,library \
--severity=HIGH,CRITICAL \
--vex="${{ steps.generate_vex_files.outputs.VEX_FILES }}" \
fleetdm/fleet:${{ steps.generate_tag.outputs.FLEET_IMAGE_TAG }}

# Explicitly push the docker images as GoReleaser will not do so in snapshot mode
- name: Publish Docker images
run: docker push fleetdm/fleet --all-tags

- name: Get tag
- name: Get tags
run: |
echo "TAG=$(git rev-parse --abbrev-ref HEAD) $(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
id: docker
Expand All @@ -106,3 +139,24 @@ jobs:
docker tag fleetdm/fleet:${TAG} quay.io/fleetdm/fleet:${TAG}
docker push quay.io/fleetdm/fleet:${TAG}
done

- name: Slack notification
if: github.event.schedule == '0 4 * * *' && failure()
uses: slackapi/slack-github-action@e28cf165c92ffef168d23c5c9000cffc8a25e117 # v1.24.0
with:
payload: |
{
"text": "${{ job.status }}\n${{ github.event.pull_request.html_url || github.event.head.html_url }}",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "⚠️ Docker publish failed.\nhttps://github.com/fleetdm/fleet/actions/runs/${{ github.run_id }}"
}
}
]
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_G_HELP_ENGINEERING_WEBHOOK_URL }}
SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK
2 changes: 1 addition & 1 deletion .github/workflows/trivy-scan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ jobs:
format: "sarif"
output: "trivy-results.sarif"
severity: "CRITICAL,HIGH,MEDIUM,LOW"
trivyignores: ".trivyignore"
trivyignores: "./security/code/.trivyignore"

- name: Upload Trivy scan results to GitHub Security tab
uses: github/codeql-action/upload-sarif@8a470fddafa5cbb6266ee11b37ef4d8aae19c571 # v3.24.6
Expand Down
12 changes: 10 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -409,7 +409,8 @@ generate-mock: mock
doc: .prefix
go generate github.com/fleetdm/fleet/v4/server/fleet
go generate github.com/fleetdm/fleet/v4/server/service/osquery_utils
generate-doc: doc

generate-doc: doc vex-report

.help-short--deps:
@echo "Install dependent programs and libraries"
Expand Down Expand Up @@ -799,4 +800,11 @@ db-replica-reset: fleet
db-replica-run: fleet
FLEET_MYSQL_ADDRESS=127.0.0.1:3308 FLEET_MYSQL_READ_REPLICA_ADDRESS=127.0.0.1:3309 FLEET_MYSQL_READ_REPLICA_USERNAME=fleet FLEET_MYSQL_READ_REPLICA_DATABASE=fleet FLEET_MYSQL_READ_REPLICA_PASSWORD=insecure ./build/fleet serve --dev --dev_license

include ./tools/makefile-support/helpsystem-targets
vex-report:
sh -c 'echo "<!-- DO NOT EDIT. This document is automatically generated by running \`make vex-report\`. -->\n# Vulnerability Report\n\nFollowing is the vulnerability report of Fleet components.\n" > security/status.md'
sh -c 'echo "## \`fleetdm/fleet\` docker image\n" >> security/status.md'
sh -c 'go run ./tools/vex-parser ./security/vex/fleet >> security/status.md'
sh -c 'echo "## \`fleetdm/fleetctl\` docker image\n" >> security/status.md'
sh -c 'go run ./tools/vex-parser ./security/vex/fleetctl >> security/status.md'

include ./tools/makefile-support/helpsystem-targets
3 changes: 3 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,7 @@ To encrypt vulnerability reports before sending them, please use this [PGP key](
The fingerprint of the key is `23A1 9D1F 16D7 1846 57D1  6D67 320D B57D E4F0 EE8F`.

### Vulnerability tracking

GitHub issues concerning vulnerabilities will be tagged with the **security** label to differentiate them from other issues and maintain SOC2 compliance.

See [security/README.md](./security/README.md) for more information on our process to keep Fleet products secure.
Loading