diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 404c3bbfe..763563d7d 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -84,9 +84,8 @@ * [GitOps Configuration](user-guide/creating-application/gitops-config.md) * [Workflow Editor](user-guide/creating-application/workflow/README.md) * [CI Pipeline](user-guide/creating-application/workflow/ci-pipeline.md) - * [Pre-Build/Post-Build Stages](user-guide/creating-application/workflow/ci-build-pre-post-plugins.md) - * [Override Build Configuration](user-guide/creating-application/container-registry-override.md) * [CD Pipeline](user-guide/creating-application/workflow/cd-pipeline.md) + * [Pre/Post Stages](user-guide/creating-application/workflow/ci-build-pre-post-plugins.md) * [ConfigMaps](user-guide/creating-application/config-maps.md) * [Secrets](user-guide/creating-application/secrets.md) * [External Secret Operator (ESO)](user-guide/creating-application/eso/README.md) diff --git a/docs/user-guide/creating-application/workflow/ci-pipeline.md b/docs/user-guide/creating-application/workflow/ci-pipeline.md index 9667fca26..61e12f6e0 100644 --- a/docs/user-guide/creating-application/workflow/ci-pipeline.md +++ b/docs/user-guide/creating-application/workflow/ci-pipeline.md @@ -1,424 +1,775 @@ # CI Pipeline {% hint style="info" %} -For Devtron version older than v0.4.0, please refer the [CI Pipeline (legacy)](https://docs.devtron.ai/v/v0.4/devtron/user-guide/creating-application/workflow/ci-pipeline-legacy) page. +For Devtron versions older than v0.4.0, please refer to the [CI Pipeline (legacy)](https://docs.devtron.ai/v/v0.4/devtron/user-guide/creating-application/workflow/ci-pipeline-legacy) page. {% endhint %} ## Creating CI Pipeline +{% hint style="warning" %} +### Who Can Perform This Action? +Users need to have the **Admin role**, the **Manager role**, or the **Super Admin role**. +Refer the [User permissions](../../global-configurations/authorization/user-access.md#roles-available-for-devtron-apps). +{% endhint %} + A CI Workflow can be created in one of the following ways: * [Build and Deploy from Source Code](#id-1.-build-and-deploy-from-source-code) + * [Linked Build Pipeline](#id-2.-linked-build-pipeline) + * [Deploy Image from External Service](#id-3.-deploy-image-from-external-service) -* Sync with Environment [![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/elements/EnterpriseTag.svg)](https://devtron.ai/pricing) -* Create a Job -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/workflow-ci-1.jpg) +* [Sync with Environment](#id-4.-sync-with-environment) -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/workflow-ci-2.jpg) +* [Create a Job](#id-5.-create-a-job) -Each method has different use-cases that can be tailored according the needs of the organization. +Each method has different use-cases that can be tailored according to the needs of the organization. -### 1. Build and Deploy from Source Code +--- + +## 1. Build and Deploy from Source Code -`Build and Deploy from Source Code` workflow allows you to build the container image from a source code repository. +**Build and Deploy from Source Code** workflow allows you to build the container image from a source code repository. + +{% hint style="info" %} +Devtron typically uses a Dockerfile from your repository to build container images. If you don’t have one, Devtron provides default templates to help you get started. You can also build images without a Dockerfile using Buildpacks. +{% endhint %} + +### Creating a CI Pipeline + +1. From the Applications menu, select your application. + +2. Go to the **Configurations** page and select **Workflow Editor**. + +3. Click **+ New Workflow**. + + ![Figure 1: Creating New Workflow](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/workflow-ci-1.jpg) -1. From the **Applications** menu, select your application. -2. On the **App Configuration** page, select **Workflow Editor**. -3. Select **+ New Workflow**. 4. Select **Build and Deploy from Source Code**. -5. Enter the following fields on the **Create build pipeline** window: -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-pipeline-v1.jpg) + ![Figure 2: 'Selecting Build and Deploy from Source Code'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/workflow-ci-2.jpg) -| Field Name | Required/Optional | Description | -| :--- | :--- | :--- | -| Source type | Required | Source type to trigger the CI. Available options: [Branch Fixed](#source-type-branch-fixed) \| [Branch Regex](#source-type-branch-regex) \|[Pull Request](#source-type-pull-request) \| [Tag Creation](#source-type-tag-creation) | -| Branch Name | Required | Branch that triggers the CI build | -| Advanced Options | Optional | Create Pre-Build, Build, and Post-Build tasks | +5. In the **Create Build Pipeline** window, enter the following details: -#### Advanced Options + ![Figure 3: Configuring build pipeline](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-pipeline-v1.jpg) -The Advanced CI Pipeline includes the following stages: + | Field Name|Required/Optional| Description| + | :--- | :--- | :--- | + | `Pipeline Name`| Required (Auto-Assigned) | Devtron automatically assigns a unique name for the pipeline. If you wish, you can change it in Advanced Options| + | `Source type`| Required| Source type to trigger the CI. Available options: Branch Fixed, Branch Regex, Pull Request, Tag Creation| + | `Branch Name`| Required| Branch that triggers the CI build| + | `Advanced Options` | Optional| Create Pre-Build, Build, and Post-Build tasks
Refer the [Configure Advanced options](#configuring-advanced-options) section to configure **Advanced options**. | -* **Pre-build stage**: The tasks in this stage are executed before the image is built. -* **Build stage**: In this stage, the build is triggered from the source code that you provide. -* **Post-build stage**: The tasks in this stage will be triggered once the build is complete. + ### Understanding Source Types -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/advanced-options.jpg) + Devtron provides multiple ways to trigger a build pipeline: + + ![Figure 4: Source Types](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-deploy-source-type.jpg) + + |Source Type| Description| Additional Requirements| + |:---|:---|:---| + | `Branch Fixed`| Triggers a CI build whenever changes are pushed to a specified branch.| Requires a predefined branch name.| + | `Branch Regex`| Allows dynamic branch selection based on a regex pattern| Requires a regex pattern to be defined. For example, if the user sets the Branch Regex as feature-\*, then users can trigger the build from branches such as feature-1450, feature-hot-fix, etc. | + | `Pull Request` (PR) | Triggers a CI build when a new pull request is created. You can also define filters (such as PR author, title, or branch) to control which pull requests trigger the pipeline. | Requires configuring a webhook configuration for GitHub or Bitbucket.| + | `Tag Creation`| Triggers a build whenever a new tag is created. You can also define filters (such as author and tag name) to control which tags trigger the pipeline.| Requires webhook configuration for GitHub or Bitbucket.| -The Pre-build and Post-build stages allow you to create Pre/Post-Build CI tasks as explained [here](./ci-build-pre-post-plugins.md). -#### Build Stage + #### Pull Request Filters -Go to the **Build stage** tab. + When using Pull Request as a source type, Devtron allows you to filter which PRs should trigger a build using the following filters -![Build stage](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-stage-v2.jpg) + ![Figure 5: Pull Request Filters](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-deploy-pull-request-filters.jpg) -| Field Name | Required/Optional | Description | -| :--- | :--- | :--- | -| TRIGGER BUILD PIPELINE | Required | The build execution may be set to: | -| DOCKER LAYER CACHING | Optional | Use this to [enable/disable caching of docker image layers](#docker-layer-caching) from your build pipeline | -| Pipeline Name | Required | A name for the pipeline | -| Source type | Required | Select the source type to build the CI pipeline: [Branch Fixed](#source-type-branch-fixed) \| [Branch Regex](#source-type-branch-regex) \| [Pull Request](#source-type-pull-request) \| [Tag Creation](#source-type-tag-creation) | -| Branch Name | Required | Branch that triggers the CI build | -| Docker build arguments | Optional | Override docker build configurations for this pipeline.
+ Select the appropriate filter and pass the matching condition as a regular expression (`regex`). -##### Docker Layer Caching [![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/elements/EnterpriseTag.svg)](https://devtron.ai/pricing) + {% hint style="info" %} + Devtron uses the regexp library, view [regexp cheatsheet](https://yourbasic.org/golang/regexp-cheat-sheet/). You can test your custom regex from [here](https://regex101.com/r/lHHuaE/1). + {% endhint %} -{% hint style="warning" %} -### Prerequisite -[Configure blob storage](https://docs.devtron.ai/configurations-overview/installation-configuration#configuration-of-blob-storage) if you wish to store cache. -{% endhint %} + | Filter Key | Description | + | --------------- | ------------------------------------------ | + | `Author` | Author of the PR | + | `Source Branch` | The branch from which the PR originates | + | `Target Branch` | The branch to which the PR is being merged | + | `Title` | Title of the PR | + | `State` | Status of the PR (default is `open`) | -If you are rebuilding the same Docker image frequently, an effective cache strategy can cut down build time. Docker images are built layer by layer and Docker’s [layer caching mechanism](https://docs.docker.com/build/cache/) allows unchanged layers to be reused across pipeline runs. + #### Tag Creation Filters -You can disable caching if: -* It’s not relevant to your workflow -* It consumes unnecessary storage -* The pipeline doesn’t perform an actual Docker build + When using Tag Creation as a source type, Devtron allows you to filter which tags should trigger a build based on the following filters -{% hint style="info" %} -### Which cache gets impacted? -If a PVC with cache is attached, it will not be impacted by disabling cache. Only the remote cache is disabled. -{% endhint %} + ![Figure 6: Tag Creation Filters](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-deploy-tag-creation-filters.jpg) -There are 3 places from where you can control the cache behavior: -1. [Orchestrator ConfigMap (Global Settings)](#orchestrator-configmap-global-settings) -2. [Editing Pipeline](#editing-pipeline) -3. [During Trigger](#during-trigger) + Select the appropriate filter and pass the matching condition as a regular expression (`regex`). -###### 1. Orchestrator ConfigMap (Global Settings) + {% hint style="info" %} + Devtron uses the regexp library, view [regexp cheatsheet](https://yourbasic.org/golang/regexp-cheat-sheet/). You can test your custom regex from [here](https://regex101.com/r/lHHuaE/1). + {% endhint %} -Super-admins can define the cache settings in `orchestrator-cm` globally for all applications and jobs using the following flags: + | Filter Key | Description | + | ---------- | ------------------------ | + | `Author` | User who created the tag | + | `Tag Name` | Name of the tag | -``` shell -DEFAULT_CACHE_FOR_CI_BUILD # for main application build stage -DEFAULT_CACHE_FOR_CI_JOB # for CI jobs -DEFAULT_CACHE_FOR_JOB # for general jobs -DEFAULT_CACHE_FOR_CD_PRE # for pre-deployment stage -DEFAULT_CACHE_FOR_CD_POST # for post-deployment stage -``` +6. Click **Create Pipeline** to save the configuration. You can now proceed to trigger the build, or continue below to explore additional configuration options for customizing your pipeline. -![Cache behavior at Global-level](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/orchestrator-cm.jpg) + ![Figure 7: Creating pipeline](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-deploy-create-pipeline.jpg) -###### 2. Editing Pipeline +### Configuring Advanced Options -Go to **Workflow Editor** → **Edit Build Pipeline** (Build Stage) → **Docker Layer Caching** (toggle) → **Use remote cache** (checkbox) +Devtron provides three stages in a CI pipeline: `Pre-Build`, `Build`, and `Post-Build`. -By default, your build pipeline will inherit the Global Settings. However, you can use the toggle button to override it and decide the caching behavior using the **Use remote cache** checkbox. In other words, cache behavior defined in pipeline configuration will have higher priority than the global one. +`Pre-build stage`: The tasks in this stage are executed before the image is built. -![Cache behavior at Pipeline-level](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/inherit-global.gif) +`Build stage`: In this stage, the build is triggered from the source code that you provide. -###### 3. During Trigger +`Post-build stage`: The tasks in this stage will be triggered once the build is complete. -Go to **Build & Deploy** (tab) → **Select Material** → **Ignore Cache** (checkbox) +This document focuses on configuring the Build Stage. If you want to set up Pre-Build and Post-Build stages, refer to the [Pre-Build/Post-build Stages Configuration](https://docs.devtron.ai/usage/applications/creating-application/workflow/ci-pipeline/ci-build-pre-post-plugins). -You have the option to ignore cache while triggering a build (regardless of the cache settings defined at the pipeline or global level). +### Build Stage -![Cache behavior at Trigger](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ignore-cache.gif) +1. In **Create Build Pipeline** window, select **Advanced options**. -{% hint style="warning" %} -### Note -If the caching flags in **Global Settings** are set to false, ignoring cache becomes the default behavior even if you don't select the 'Ignore Cache' checkbox during trigger. -{% endhint %} + ![Figure 8: Advanced Options](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/advanced-options.jpg) + +2. Go to the **Build stage** tab and configure the following fields: + ![Figure 9: Build stage](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-stage-v2.jpg) -##### Source type -###### Branch Fixed + | Field Name| Required/Optional | Description| + | :---| :--- |:---| + | `TRIGGER BUILD PIPELINE`| Required|

The build execution may be set to:

| + | DOCKER LAYER CACHING | Optional | Use this to [enable/disable caching of docker image layers](#docker-layer-caching) from your build pipeline | + | `Pipeline Name`| Required| Devtron automatically assigns a unique name for the pipeline; if you wish, you can edit it here.| + | `Scan for Vulnerabilities` | Optional|

Prerequisite: Install either Clair or Trivy.

| + | `Override Options`| Optional| Allows overriding configurations from earlier stages like CRI configuration, target platform configuration, etc.| + | `Docker build arguments`|Optional|

Override docker build configurations for this pipeline.

| + | `Custom Image Tag Pattern` | Optional|

Enable the Custom Image Tag Pattern toggle.

**Build will fail if the resulting image tag has already been built in the past. This error might occur when you reset the value of the variable `x` or when you disable/enable the toggle button for `Custom image tag pattern`**.

| -This allows you to trigger a CI build whenever there is a code change on the specified branch. + ### Docker Layer Caching [![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/elements/EnterpriseTag.svg)](https://devtron.ai/pricing) -Enter the **Branch Name** of your code repository. + {% hint style="warning" %} + ### Prerequisite + [Configure blob storage](https://docs.devtron.ai/configurations-overview/installation-configuration#configuration-of-blob-storage) if you wish to store cache. + {% endhint %} -###### Branch Regex + If you are rebuilding the same Docker image frequently, an effective cache strategy can cut down build time. Docker images are built layer by layer, and [Docker’s layer caching mechanism](https://docs.docker.com/build/cache/) allows unchanged layers to be reused across pipeline runs. -`Branch Regex` allows users to easily switch between branches matching the configured Regex before triggering the build pipeline. -In case of `Branch Fixed`, users cannot change the branch name in ci-pipeline unless they have admin access for the app. So, if users with -`Build and Deploy` access should be allowed to switch branch name before triggering ci-pipeline, `Branch Regex` should be selected as source type by a user with Admin access. + You can disable caching if: + * It’s not relevant to your workflow + * It consumes unnecessary storage + * The pipeline doesn’t perform an actual Docker build -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/branch-regex.jpg) + {% hint style="info" %} + ### Which cache gets impacted? + If a PVC with cache is attached, it will not be impacted by disabling cache. Only the remote cache is disabled. + {% endhint %} -For example if the user sets the Branch Regex as `feature-*`, then users can trigger from branches such as `feature-1450`, `feature-hot-fix` etc. + There are 3 places from where you can control the cache behavior: + 1. [Orchestrator ConfigMap (Global Settings)](#orchestrator-configmap-global-settings) + 2. [Editing Pipeline](#editing-pipeline) + 3. [During Trigger](#during-trigger) -###### Pull Request + #### 1. Orchestrator ConfigMap (Global Settings) -This allows you to trigger the CI build when a pull request is created in your repository. + Super-admins can define the cache settings in `orchestrator-cm` globally for all applications and jobs using the following flags: -{% hint style="info" %} -**Prerequisites** + ``` shell + DEFAULT_CACHE_FOR_CI_BUILD # for main application build stage + DEFAULT_CACHE_FOR_CI_JOB # for CI jobs + DEFAULT_CACHE_FOR_JOB # for general jobs + DEFAULT_CACHE_FOR_CD_PRE # for pre-deployment stage + DEFAULT_CACHE_FOR_CD_POST # for post-deployment stage + ``` + + ![Figure 10: Cache behavior at Global-level](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/orchestrator-cm.jpg) + + #### 2. Editing Pipeline + + Go to **Workflow Editor** → **Edit Build Pipeline** (Build Stage) → **Docker Layer Caching** (toggle) → **Use remote cache** (checkbox) + + By default, your build pipeline will inherit the Global Settings. However, you can use the toggle button to override it and decide the caching behavior using the **Use remote cache** checkbox. In other words, cache behavior defined in the pipeline configuration will have higher priority than the global one. + + ![Figure 11: Cache behavior at Pipeline-level](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/inherit-global.gif) + + #### 3. During Trigger + + Go to **Build & Deploy** (tab) → **Select Material** → **Ignore Cache** (checkbox) -[Configure the webhook](#configuring-webhook) for either GitHub or Bitbucket. + You have the option to ignore cache while triggering a build (regardless of the cache settings defined at the pipeline or global level). + + ![Figure 12: Cache behavior at Trigger](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ignore-cache.gif) + + {% hint style="warning" %} + ### Note + If the caching flags in **Global Settings** are set to false, ignoring cache becomes the default behavior even if you don't select the 'Ignore Cache' checkbox during trigger. + {% endhint %} + +3. Click **Create Pipeline** to save the configuration. + + ![Figure 13: Create Pipeline](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-deploy-create-pipeline.jpg) + +#### Triggering a Build + +Once the CI pipeline is set up, follow these steps to trigger a build: + +1. Navigate to **Build & Deploy**. + +2. Click **Select Material** in the specific pipeline for which you want to trigger the build. + + ![Figure 14: Selecting Material](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-deploy-select-material.jpg) + +3. Choose the **Git commit** to build under **Code Source** tab. + + ![Figure 15: Selecting Git Commit](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-deploy-git-comit.jpg) + + +4. Configure runtime parameters (if any) before starting the build under the **Parameters** tab. + + ![Figure 16: Configuring Runtime Parameters](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-deploy-runntime-parameters.jpg) + +5. Click **Start Build**. This will trigger the build process and push the generated container image to the configured container registry for storage, versioning, and later use in the CD pipeline. + + ![Figure 17: Starting Build](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-deploy-start-build.jpg) + +## 2. Linked Build Pipeline + +**Linked Build Pipeline** allows you to reuse build images from another pipeline within **Devtron**, instead of building images from source code each time. + +This is useful when the same codebase is shared across multiple applications. + +Instead of creating and maintaining separate pipelines for each application, you can first set-up a primary build pipeline (in the same application or in any other application) that builds the image from source code using **Build and Deploy from Source Code** or you can also use an existing pipeline from any application with the same codebase. + +Then, for other applications, you can simply link that source pipeline to reuse its build images directly in your workflow and proceed to create a CD pipeline using those images. + +{% hint style="info" %} +The **Linked Build Pipeline** can only access build images that are generated after it has been created. Any images built by the source pipeline before the Linked Build Pipeline was set up will not be available. {% endhint %} +To create a **Linked Build Pipeline**, follow the steps below. + {% hint style="warning" %} -The **Pull Request** source type feature only works for the host GitHub or Bitbucket Cloud for now. To request support for a different Git host, please create a GitHub issue [here](https://github.com/devtron-labs/devtron/issues). +### Who Can Perform This Action? +Users need to have the **Admin role**, the **Manager role**, or the **Super Admin role**. +Refer the [User permissions](../../global-configurations/authorization/user-access.md#roles-available-for-devtron-apps). {% endhint %} -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-pipeline-3.jpg) +1. Navigate to **Configurations** → **Workflow Editor** of your application. -To trigger the build from specific PRs, you can filter the PRs based on the following keys: +2. Select **+ New Workflow**; a modal window will appear where you can select the type of pipeline you want to create. -| Filter key | Description | -| :--- | :--- | -| `Author` | Author of the PR | -| `Source branch name` | Branch from which the Pull Request is generated | -| `Target branch name` | Branch to which the Pull request will be merged | -| `Title` | Title of the Pull Request | -| `State` | State of the PR. Default is "open" and cannot be changed | + ![Figure 18: Creating New Workflow](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/linked-build-pipeline-new-workflow.jpg) -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-pipeline-8.jpg) +3. Select **Linked Build Pipeline**. Another modal window will appear where you can enter the details of the existing pipeline you want to link. -Select the appropriate filter and pass the matching condition as a regular expression (`regex`). + ![Figure 19: Selecting 'Linked Build Pipeline'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/linked-build-pipeline.jpg) -> Devtron uses regexp library, view [regexp cheatsheet](https://yourbasic.org/golang/regexp-cheat-sheet/). You can test your custom regex from [here](https://regex101.com/r/lHHuaE/1). +4. Enter the details of the existing pipeline you want to link and click **Create Linked CI Pipeline** to create the pipeline. -Select **Create Pipeline**. + ![Figure 20: Entering details of existing pipeline](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/linked-build-pipeline-create-pipeline.jpg) -###### Tag Creation + {% hint style="warning" %} + The user must have at least view access to the application that contains the source pipeline; otherwise, the application will not appear in the **Filter By Application** field. + {% endhint %} -This allows you to trigger the CI build whenever a new tag is created. + |Field Name|Description| + |:---|:---| + |Filter By Application|Enter the application name in which the source CI pipeline exists.| + |Source CI pipeline|List all the build pipelines for the selected application. Choose the pipeline that you want to link| + |Name|Enter the name for the **Linked Build Pipeline**.
| -{% hint style="info" %} -**Prerequisites** + ![Figure 21: Pipeline created](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/linked-build-pipeline-created.jpg) -[Configure the webhook](#configuring-webhook) for either GitHub or Bitbucket. + Thereafter, the source CI pipeline will indicate the number of Linked CI pipelines. On clicking it, a modal window will appear, which lists all the applications from which the source pipeline is linked, as shown in the image below. + + ![Figure 22: Linked CI with Child Information](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/linkedci.gif) + +5. After creating a **Linked CI Pipeline**, you can create a CD pipeline. Refer to [CD Pipeline](./cd-pipeline.md) page to know more. + + ![Figure 23: Creating CD pipeline](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/linked-build-pipeline-cd.jpg) + +{% hint style="warning" %} +Linked CI pipelines can't trigger builds. They rely on the source CI pipeline to build images. Trigger a build in the source CI pipeline to see the images available for deployment in the linked CI pipeline's CD stage. {% endhint %} -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-pipeline-9.jpg) -To trigger the build from specific tags, you can filter the tags based on the `author` and/or the `tag name`. +## 3. Deploy Image from External Service -| Filter key | Description | -| :--- | :--- | -| `Author` | The one who created the tag | -| `Tag name` | Name of the tag for which the webhook will be triggered | +**Deploy Image from External Service** allows you to deploy container images built by external CI tools such as Jenkins, CircleCI, etc., using webhooks. -Select the appropriate filter and pass the matching condition as a regular expression (`regex`). +This is useful when your CI pipeline is managed outside the Devtron platform, allowing you to use Devtron exclusively for deploying container images on Kubernetes while maintaining your existing CI setup. -Select **Create Pipeline**. +To create a pipeline form **Deploy Image from External Service**, follow the steps below -{% hint style="info" %} -The total timeout for the execution of the CI pipeline is by default set as 3600 seconds. This default timeout is configurable according to the use case (refer [Build Infra](../../global-configurations/build-infra.md)). -{% endhint %} +1. Navigate to **Configurations** → **Workflow Editor** of your application. -##### Scan for Vulnerabilities +2. Select **+ New Workflow**; a modal window will appear where you can select the type of pipeline you want to create. -{% hint style="info" %} -### Prerequisite -Install any one of the following integrations from Devtron Stack Manager: -* [Clair](../../../user-guide/integrations/vulnerability-scanning/clair.md) -* [Trivy](../../../user-guide/integrations/vulnerability-scanning/trivy.md) -{% endhint %} + ![Figure 24: Creating New Workflow](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-new-workflow.jpg) -To perform the security scan after the container image is built, enable the **Scan for vulnerabilities** toggle in the build stage. Refer [Security features](../../../user-guide/security-features.md) to know more. +3. Select **Deploy Image from External Service**, another modal window will appear where you can enter deployment details such as environment, execution mode, and deployment strategy. -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/scan-for-vulnerabilities-v2.jpg) + ![Figure 25: Selecting 'Deploy Image from External Service'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image.jpg) -##### Custom Image Tag Pattern +4. Enter the deployment details and click **Create Pipeline** to create the pipeline. -This feature helps you apply custom tags (e.g., `v1.0.0`) to readily distinguish container images within your repository. + ![Figure 26: Entering deployment details](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-create-pipeline.jpg) -1. Enable the toggle button as shown below. + | Fields | Description | + | --- | --- | + |**Environment**|Provide the name of the Environment in which you want to deploy your image.| + |**Namespace**|It will display the namespace of that Environment.| + | **When do you want the pipeline to execute?** | You can deploy either in one of the following ways: | + | **Deployment Strategy** | Choose the Deployment Strategy according to your preference. | - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-image-tag-pattern.jpg) +To get the image from an external CI service (let's say Jenkins), you need to configure the Webhook provided by Devtron in your existing External CI pipeline. -2. You can write an alphanumeric pattern for your image tag, e.g., **test-v1.0.{x}**. Here, 'x' is a mandatory variable whose value will incrementally increase with every build. You can also define the value of 'x' for the next build trigger in case you want to change it. +### Configure Webhook in External CI - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-image-tag-version.jpg) +To configure the Webhook in External CI, follow the steps below. - {% hint style="warning" %} - Ensure your custom tag do not start or end with a period (.) or comma (,) - {% endhint %} +1. After creating the pipeline, select **Show webhook details** or select **External Source** stage to get the webhook URL and JSON sample payload to be used in the external CI pipeline. -3. Click **Update Pipeline**. + ![Figure 27: Getting Webhook Details](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-show-webhook.jpg) -4. Now, go to **Build & Deploy** tab of your application, and click **Select Material** in the CI pipeline. +2. On the **Webhook Details** page, click **Select or auto-generate token with required permissions** to select or generate an `API token`. This token allows external CI services to authenticate with Devtron. - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-material.jpg) + ![Figure 28: Clicking 'Select or auto-generate token with required permissions'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-select-auto-generate.jpg) -5. Choose the git commit you wish to use for building the container image. Click **Start Build**. + * To select an existing API token, choose an API token from the dropdown under **Select API token**. - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/choose-commit.jpg) + ![Figure 29: Selecting existing API Token](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-select-api-token.jpg) -6. The build will initiate and once it is successful the image tag would reflect at all relevant screens: + * To generate an API token, select **Auto-generate token** sub tab → Enter a name for the token in the **Token Name** field → Click **Generate token** to generate a token. - * **Build History** + ![Figure 30a: Generating API Token](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-auto-generate-token.jpg) - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-history.jpg) + ![Figure 30b: API Token Generated](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-auto-generate-token-2.jpg) - * **Docker Registry** +3. After generating an API token, click **Sample cURL request** and select the metadata you want to send to Devtron. Sample JSON and cURL request will be generated accordingly. - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/docker-image.jpg) + ![Figure 31: Getting sample cURL request and selecting metadata](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-select-metadata.jpg) - * **CD Pipeline (Image Selection)** +4. Copy the Sample cURL request and integrate it into your External CI (Jenkins) pipeline along with the API token and tag for Docker Image. Refer to [Integrate with External Sources](#integrate-with-external-sources---jenkins) to know more. - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/select-ci-image.jpg) + ![Figure 32: Copying Sample cURL request](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-copy-curl-cmd.jpg) +5. After integrating the webhook, whenever the external CI pipeline is triggered and generates an image, the webhook will automatically send the image details to Devtron for deployment. -{% hint style="info" %} -Build will fail if the resulting image tag has already been built in the past. This means if there is an existing image with tag `test-v1.0.0`, you cannot build another image having the same tag `test-v1.0.0` in the same CI pipeline. This error might occur when you reset the value of the variable `x` or when you disable/enable the toggle button for `Custom image tag pattern`. -{% endhint %} +### Integrate with External Sources - Jenkins -### 2. Linked Build Pipeline +To integrate Webhook with your Jenkins project/pipeline, you need to add a new step/stage in your project/pipeline. -If one code is shared across multiple applications, `Linked Build Pipeline` can be used, and only one image will be built for multiple applications because if there is only one build, it is not advisable to create multiple CI Pipelines. +Before adding the stage/step, you need to add the API token provided by Devtron as the secret in Jenkins. To do so, follow the steps below: -1. From the **Applications** menu, select your application. -2. On the **App Configuration** page, select **Workflow Editor**. -3. Select **+ New Workflow**. -4. Select **Linked Build Pipeline**. -5. On the **Create linked build pipeline** screen: - * Search for the application in which the source CI pipeline is present. - * Select the source CI pipeline from the application that you selected above. - * Enter a new name for the linked CI pipeline. +1. Go to **Manage Jenkins** → **Credentials**. + + ![Figure 33a: Selecting 'Manage Jenkins'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-manage-jenkins.jpg) - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ca-workflow-linked.jpg) - -6. Click **Create Linked CI Pipeline**. + ![Figure 33b: Selecting 'Credentials'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-credentials.jpg) -Thereafter, the source CI pipeline will indicate the number of Linked CI pipelines. Upon clicking it, it will display the child information as shown below. It reveals the applications and environments where Linked CI is used for deployment. +2. Select **System** under **Stores scoped to Jenkins** → **Global credentials (unrestricted)** → **+Add Credentials**. -![Linked CI with Child Information](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/linkedci.gif) + ![Figure 34a: Selecting 'System' under 'Stores scoped to Jenkins'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-system.jpg) -After creating a linked CI pipeline, you can create a CD pipeline. + ![Figure 34b: Selecting Global credentials (unrestricted)](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-global-cred.jpg) -{% hint style="warning" %} -Linked CI pipelines can't trigger builds. They rely on the source CI pipeline to build images. Trigger a build in the source CI pipeline to see the images available for deployment in the linked CI pipeline's CD stage. -{% endhint %} + ![Figure 34c: Selecting 'Add Credentials'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-add-cred.jpg) +3. Select `Secret text` in the **Kind** field and select the required **Scope** -### 3. Deploy Image from External Service +4. Enter the **API Token** generated from Devtron in the **Secret** field. -For CI pipeline, you can receive container images from an external services via webhook API. +5. Provide an `ID` (devtron-token) in the **ID** field. If left blank, Jenkins will automatically generate an ID for the credential. -You can use Devtron for deployments on Kubernetes while using an external CI tool such as Jenkins or CircleCI. External CI feature can be used when the CI tool is hosted outside the Devtron platform. However, by using an external CI, you will not be able to use some of the Devtron features such as Image scanning and security policies, configuring pre-post CI stages etc. +6. If you want you can also provide an optional description in the **Description** field. +7. Select **Create** to create the secret in Jenkins. -* Create a [new](../../create-application.md) or [clone](../../cloning-application.md) an application. -* To configure `Git Repository`, you can add any Git repository account (e.g., dummy account) and click **Next**. -* To configure the `Container Registry` and `Container Repository`, you can leave the fields blank or simply add any test repository and click **Save & Next**. -* On the `Base Deployment Template` page, select the `Chart type` from the drop-down list and configure as per your [requirements](../../creating-application/deployment-template.md) and click **Save & Next**. -* On the **Workflow Editor** page, click **New Workflow** and select **Deploy image from external service**. + ![Figure 35a: Adding Credential](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-add-secret.jpg) -* On the **Deploy image from external source** page, provide the information in the following fields: + ![Figure 35b: Credential Added](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-secret-added.jpg) -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-from-external-source.jpg) +After adding the API token as a secret, add a new step/stage in your Jenkins project/pipeline. -| Fields | Description | -| --- | --- | -| **Deploy to environment** | | -| **When do you want to deploy** | You can deploy either in one of the following ways: | -| **Deployment Strategy** | Configure the deployment preferences for this pipeline. | +In case your Jenkins project is of type `freestyle`, follow the steps below: -* Click **Create Pipeline**. -A new CI pipeline will be created for the external source. -To get the webhook URL and JSON sample payload to be used in external CI pipeline, click **Show webhook details**. +1. Select the Jenkins project in which you want to integrate the Webhook. -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/webhook-ci/show-webhook-details-v2.jpg) + ![Figure 36: Selecting Jenkins Project](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-select-project.jpg) -* On the **Webhook Details** page, you have to authenticate via `API token` to allow requests from an external service (e.g. Jenkins or CircleCI). +2. Go to the **Configure** → **Environments** and enable the **Use secret text(s) or file(s)** option. -* For authentication, only users with `super-admin` permissions can select or generate an API token: - * You can either use **Select API Token** if you have generated an [API Token](../../global-configurations/authorization/api-tokens.md) under **Global Configurations**. + ![Figure 37: Selecting 'Configure'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-configure.jpg) - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/webhook-ci/select-api-token-webhook-details-v2.jpg) +3. Click **Add** under **Bindings** and select **Secret Text**. - * Or use **Auto-generate token** to generate the API token with the required permissions. Make sure to enter the token name in the **Token name** field. +4. Provide a variable name (eg., `DEVTRON_TOKEN`) for the secret in **Variable** field. This variable name will be used to access the secret. - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/webhook-ci/auto-generate-token-webhook-details-v2.jpg) +5. Select the credential's `ID` in **Specific credentials** under **Credentials**. + + **Note:** In case you have provided a description for your credential, then instead of the credential `ID`, the description will be displayed. -* To allow requests from the external source, you can request the API by using: - * **Webhook URL** - * **cURL Request** + ![Figure 38a: Binding Credential in the project](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-bindings.jpg) -#### Webhook URL + ![Figure 38b: Credential Binding successful](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-secret-binded.jpg) - HTTP Method: `POST` +6. Go to the **Configure** → **Build Steps**, click **Add build step**, and then select **Execute Shell**. - API Endpoint: `https://{domain-name}/orchestrator/webhook/ext-ci/{pipeline-id}` - - JSON Payload: + ![Figure 39: Adding Build Step for Webhook](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-add-build+task.jpg) -```bash - { - "dockerImage": "445808685819.dkr.ecr.us-east-2.amazonaws.com/orch:23907713-2" -} - ``` +7. Enter the cURL request command as shown below. Make sure to enter the `API token` and `dockerImage` in your cURL command and click **Save**. -You can also select metadata to send to Devtron. Sample JSON will be generated accordingly. -You can send the Payload script to your CI tools such as Jenkins and Devtron will receive the build image every time the CI pipeline is triggered or you can use the Webhook URL, which will build an image every time CI pipeline is triggered using Devtron Dashboard. + Note: API Token has been referenced from the secret via **Variable Name** (`DEVTRON_TOKEN`) configured in Jenkins credentials using its `ID` + ![Figure 40: Configuring Webhook Build Step](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-build-task-added.jpg) -#### Sample cURL Request - -```bash -curl --location --request POST \ -'https://{domain-name}/orchestrator/webhook/ext-ci/{pipeline-id}' \ ---header 'Content-Type: application/json' \ ---header 'token: {token}' \ ---data-raw '{ - "dockerImage": "445808685819.dkr.ecr.us-east-2.amazonaws.com/orch:23907713-2" -}' - ``` -#### Response Codes +In case your Jenkins project is of type `pipeline`, `Multibranch Pipeline`, etc., which uses a **Pipeline Script** or **Jenkinsfile**, then you need to add a new stage in the pipeline for configuring the webhook. To do so follow the steps below. + +1. Select the Jenkins project in which you want to integrate the Webhook. + + ![Figure 41: Selecting Jenkins Project](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-select-project-2.jpg) + +2. Go to the **Configure** → **Pipeline**. + + ![Figure 42: Selecting 'Configure'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-configure-2.jpg) + +3. In case you are using **Pipeline Script**, then modify the script to add a new stage as shown below. If you are using **Pipeline script from SCM**, then modify your Jenkinsfile in the same way. + + ![Figure 43: Adding Webhook pipeline stage](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-add-webhook-stage.jpg) + +4. Click **Save**. + +The new images that will be built after adding the webhook will be available to Devtron for deployment. + +Now, you can access the images on the Devtron dashboard and deploy manually. In case, If you select the **Automatic** deployment option, then your application will be deployed automatically everytime a new image is received. + +### Integrate with External Sources - GitHub Actions + +To integrate Webhook with your GitHub Actions workflow, you need to add a new step in your workflow file. + +Before adding the step in the workflow, you need to add the API token provided by Devtron as a secret in your repository. To do so, follow the steps below + +1. Navigate to **Settings** tab of your repository. + + ![Figure 44: Navigating to 'Settings'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-settings.jpg) + +2. Select **Secrets and variables** → **Actions** under **Security**. + + ![Figure 45: Selecting 'Actions'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-secrets.jpg) + +3. Under **Secrets** tab, select **New repository secret**. + + ![Figure 46: Selecting 'New repository secret'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-new-secret.jpg) + +4. Enter a name for your secret in the **Name** field. + +5. Enter the value of the secret in the **Secret** field. + +6. Select **Add Secret** and the secret will be added to your repository. + + ![Figure 47a: Entering secret info](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-add-secret.jpg) + + ![Figure 47b: Secret Added](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-secret-added.jpg) -| Code | Description | -| --- | --- | -| `200` | `app detail page url` | -| `400` | `Bad request` | -| `401` | `Unauthorized` | +After adding the API token as a secret, add a new step in your GitHub Action workflow. To do so, follow the steps below: +1. Navigate to **Actions** tab of your repository. -#### Integrate with External Sources - Jenkins or CircleCI + ![Figure 48: Navigating to Actions](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-actions.jpg) - * On the Jenkins dashboard, select the Jenkins job which you want to integrate with the Devtron dashboard. - * Go to the **Configuration** > **Build Steps**, click **Add build step**, and then click **Execute Shell**. +2. Select your workflow under the **All workflows** section. - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/webhook-ci/add-build-step-jenkins-v2.jpg) + ![Figure 49: Selecting workflow](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-select-workflow.jpg) - * Enter the cURL request command. - * Make sure to enter the `API token` and `dockerImage` in your cURL command and click **Save**. +3. Click the workflow file (`main.yml`) under the workflow name; this will open the workflow file in GitHub. - ![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/webhook-ci/execute-shell-jenkins-v2.jpg) + ![Figure 50: Selecting 'Workflow File'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-select-workflow-file.jpg) - Now, you can access the images on the Devtron dashboard and deploy manually. In case, if you select **Automatic** deployment option, then your application will be deployed automatically everytime a new image is received. +4. Select the edit icon to add the webhook step in the workflow file. - Similarly, you can also integrate with external source such as **CircleCI** by: - - * Select the job on the `CircleCI` dashboard and click `Configuration File`. - * On the respective job, enter the `cURL` command and update the `API token` and `dockerImage` in your cURL command. + ![Figure 51: Selecting 'Edit' Icon](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-edit-icon.jpg) + +5. Add the webhook step in the workflow file and select **Commit changes...** + + ![Figure 52a: Adding Webhook Step](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-step-added.jpg) + + ![Figure 52ab: Committing Changes](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-commit-changes.jpg) + +6. Provide a **Commit message** and a optional description. + +7. Select **Commit changes**, and the workflow file will be updated with the webhook step. + + ![Figure 53: Selecting 'Commit changes'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-commit-changes-2.jpg) + +The new images that will be built after adding the webhook will be available to Devtron for deployment. + +Now, you can access the images on the Devtron dashboard and deploy manually. In case, If you select the **Automatic** deployment option, then your application will be deployed automatically everytime a new image is received. + +--- + +## 4. Sync with Environment + +**Sync with Environment** allows you to reuse the deployed container image from one CD workflow in another CD workflow within the same application. + +It is useful when you want to test a microservice (say, A) in a test environment, and it depends on another microservice (say, B). To ensure accurate testing of **Microservice (A)**, you need a stable version of **Microservice (B)** (the one that is already running in production). However, modifying the production pipeline for testing purposes is not ideal and often not allowed due to stability concerns. + +In such cases, you can use **Sync with Environment** to create a new workflow that uses the deployed image from the existing CD pipeline (of microservice B) from a specific environment. This image then acts as the source CI for the new workflow within the same application. + +This allows the new workflow to use the same image as the stable production environment, thus enabling consistent and reliable testing without impacting production workloads. + +To create a pipeline form **Sync with Environment**, follow the steps below + +1. Navigate to **Configurations** → **Workflow Editor** of your application. + +2. Select **+ New Workflow**; a modal window will appear where you can select the type of pipeline you want to create. + + ![Figure 54: Creating New Workflow](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/sync-env-new-workflow.jpg) + +3. Select **Sync with Environment**, another modal window will appear where you need to select the environment in which the source CD pipeline exists. + + ![Figure 55 : Selecting 'Sync with Environment'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/sync-env.jpg) + +4. Select the environment in which the source CD pipeline exists. You can only select one source CD per workflow. + + **Note:** The CD pipeline used as a source cannot be deleted while it’s linked. + + ![Figure 56: Selecting Source CD Environment](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/sync-env-select-source-cd.jpg) + +5. Select **Deploy to** in the top right corner to select the environment in which you want to deploy the source CD image. + + ![Figure 57a: Selecting 'Deploy To'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/sync-env-deploy-to.jpg) + + ![Figure 57b: Selecting the Deployment Environment](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/sync-env-select-deploy-env.jpg) + +6. Select **Create Pipeline** to create a new workflow. + +You can now configure the deployment pipeline, and if you wish, you can also add more deployment pipelines within the same workflow. + +--- + +## 5. Create a job + +If options like **Build and Deploy from Source Code** do not satisfy your use case, you can use **Create a Job** to define a workflow with a custom CI stage and with deployment capabilities. + +In this workflow, the build stage is replaced by a Job stage, where you can either use preset plugins or define custom tasks to define custom steps to satisfy your use case. For e.g., you can use a preset plugin to pull the container images required for deployment from a container registry (such as ACR or ECR). + +This is useful when the image is built externally (for example, in Jenkins) and needs to be brought into Devtron for further processing and deployment. You can configure tasks like scanning, testing, or notifications using preset plugins either in the Job stage or the pre-CD stage, depending on your use case. + +**Create a Job** differs from **Deploy Image from External Source** by allowing you to define custom tasks after the image is received, using Job tasks. + +To create a workflow using **Create a job**, follow the steps below + +1. Navigate to **Configurations** → **Workflow Editor** of your application. + +2. Select **+ New Workflow**; a modal window will appear where you can select the type of pipeline you want to create. + + ![Figure 58: Creating New Workflow](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/job-ci-new-workflow.jpg) + +3. Select **Create a job**. This opens the **Create job pipeline** Window in which you can create and configure your job. + + ![Figure 59: Selecting 'Create a job'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/job-ci.jpg) + +4. In the **Create job pipeline** window, you can create and configure job pipelines. + + ![Figure 60: Create job pipeline](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/job-ci-create-job-pipeline.jpg) + +It includes 2 stages + +* [**Basic Configurations**](#basic-configurations) + +* [**Tasks to be executed**](#tasks-to-be-executed) + +### Basic Configurations + +This stage allows you to define primary configurations such as Pipeline name, Source Type, Branch Name, and how the job should be triggered. Refer to the following table to configure each field. + + ![Figure 61: Configure Basic Configurations](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/job-ci-basic-config.jpg) + +| Field Name|Description| +| :--- |:--- | +| `Trigger Job Pipeline` |

The job execution may be set to:

| +| `Pipeline Name` | Assign a name to your job pipeline| +| `Source type` | Source type to trigger the job pipeline. Available options: Branch Fixed, Branch Regex, Pull Request, Tag Creation| +| `Branch Name`| Branch that triggers the CI build| +| `Use remote cache`|

Enable this option to use the Docker cache from previous builds. Docker's layer caching mechanism allows unchanged docker images layers to be reused across pipeline runs, thus drastically reducing execution times

The globe toggle, next to Docker Layer Caching means that the configuration is inherited from global

| + +### Tasks to be executed + +The Stage allows you to define tasks for your job. + +You can create one or more tasks. Tasks can be dependent on each other for execution. In other words, the output variable of one task can be used as an input for the next task to execute your job. Tasks will execute in the order they are arranged and can be rearranged by drag-and-drop; however, the order of passing the variables must be followed. + +To create a task: + +1. Navigate to **Tasks to be executed** in the **Create job pipeline** window. + +2. Click **Add Task** to add a task in your job pipeline. + + ![Figure 62: Add task](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/job-ci-task-exec.jpg) + +3. A new task will be added (on the left side of the Create job pipeline window), you can configure the task either by selecting one of the available [preset plugins](#create-task-using-preset-plugins) or by [Executing a custom script](#create-task-using-custom-script) + + ![Figure 63: Type of tasks](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/job-ci-add-task.jpg) + +#### Pulling images through preset plugin + +In Devtron, preset plugins are pre-defined tasks templates that helps you automate and execute common operations such as provisioning infrastructure, taking backups, exporting container images etc., without writing custom scripts. + +Devtron provides a set of built-in preset plugins, and you can also create your own plugins in Devtron according to your specific needs. + +To create a task using the **Pull Images from Container Repository** plugin, follow the steps below: + +**Note:** **Pull Images from Container Repository** plugin only supports ECR and ACR. + +1. After configuring the basic configurations, select the **Tasks to be executed** Tab + +2. Click **+Add Task** from the left side panel. + +3. Search for `Pull Images from Container Repository` in the **Search Plugin** Search bar and select **Pull Images from Container Repository** from the list of plugins. + + ![Figure 64: Search 'Pull Images from Container Repository' plugin](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/job-ci-pull-images.jpg) + + + * The right-side panel will display the fields specific to the **Pull Images from Container Repository** plugin, which are required to be configured. + + * The left-side panel will now shows a task under **Tasks (IN ORDER OF EXECUTION)**, named after the selected plugin(by default), along with its logo.
You can change the task's name using the **Task name** field but plugin's logo will remain indicating that it is a preset plugin. + + ![Figure 65: Gke provisioner plugin](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/job-ci-pull-images-added.jpg) + +4. Refer the [Pull Images from Container Repository](/docs/user-guide/plugins/pull-images-from-container-repository.md) documentation to configure the **Pull Images from Container Repository** fields with appropriate values. + + > Refer to the [Plugins documentation](/docs/user-guide/plugins/README.md) to explore and configure any of the available plugins. + +5. After configuring the fields successfully, your task will be created. If you wish, you can add more tasks by clicking on **+ Add task** in the left-side panel. + +#### Create Task using Custom Script + +In the job stage, you can also define a task using a custom script to meet specific requirements. To create a task a task using a custom script, follow the steps below: + +1. After configuring the basic configurations, select the **Tasks to be executed** Tab. + +2. Click **+Add Task** from the left side panel, and then select **Execute custom task**. + + * The right-side panel will display the fields that are required to be configured in order to create the task. + + * The left-side panel will now display a task under **Tasks (IN ORDER OF EXECUTION)**. + +3. Enter the Task name(required) and Task Description (optional). + +4. Select the **Task type**, it can be `Shell` or `Container Image`. + + * **Shell Tasks**: These execute shell scripts directly within the job runtime environment. In this type of task, you can define inline scripts or use script files from your configured source code. + + * **Container Image Tasks**: These allow you to execute commands and scripts inside a custom Docker container. Instead of using the default environment provided by Devtron, you can specify you own container image with all dependencies and tools required for the tasks. + + These Tasks run using container in container approach, which means the specified image is pulled and run inside the job pod, thus providing a complete isolated environment. + +5. After selecting the **Task type**, you need to configure task-specific fields based on that **Task type**. + +Refer to [Create Task using Custom Script for Devtron Jobs]() to know more. + +After configuring the **Basic Configurations** and adding the tasks, select **Create Pipeline** to create a new workflow with a job stage (instead of a build stage). + +Now, you can add a deployment pipeline to this workflow. The image will be pulled during the execution of the pipeline using the configured plugin, and then passed to the CD stage for deployment. --- ## Updating CI Pipeline You can update the configurations of an existing CI Pipeline except for the pipeline's name. -To update a pipeline, select your CI pipeline. -In the **Edit build pipeline** window, edit the required stages and select **Update Pipeline**. +To update a pipeline, -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/update-pipeline.jpg) +1. Select your CI pipeline. + +2. In the **Edit build pipeline** window, edit the required stages and select **Update Pipeline**. + +![Figure 66: Updating CI pipeline](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/update-pipeline.jpg) --- ## Deleting CI Pipeline -You can only delete a CI pipeline if there is no CD pipeline created in your workflow. +To delete a CI pipeline, make sure that there is no CD pipeline attached to it in your workflow; In that case, you must first delete the CD pipeline, and only then you can delete a CI pipeline. + +To delete a CI pipeline, follow the steps below. + +1. Navigate to **Configurations** → **Workflow Editor** and choose the pipeline you wish to delete. + + ![Figure 67: Selecting Workflow to delete](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/delete-pipeline-select-workflow.jpg) + + +2. Click on the CD Stage, **Edit deployment pipeline** window will open and select **Delete Pipeline**. + + ![Figure 68: Clicking 'Delete Pipeline'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/delete-pipeline-delete-cd.jpg) -To delete a CI pipeline, go to **App Configurations > Workflow Editor** and select **Delete Pipeline**. +3. A pop-up will appear asking you to enter the environment name of the deployment. -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/delete-pipeline.jpg) +4. Enter the environment name and select **Delete**. The CD pipeline will be deleted. + ![Figure 69: Deleting CD Pipeline](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/delete-pipeline-delete-cd-popup.jpg) + In case there are multiple CD pipelines in the workflow, then you need to delete every single one of them in a similar way. + +5. After deleting all CD pipelines, click on the build stage, and the **Edit build pipeline** window will open. + + ![Figure 70: Selecting build stage](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/delete-pipeline-select-build.jpg) + +6. Select **Delete Pipeline** from the bottom left corner of the window button in the CI Stage, a pop-up will appear prompting you to delete the CI pipeline. + + ![Figure 71: Clicking 'Delete Pipeline'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/delete-pipeline-delete-CI.jpg) + + In case your build pipeline is linked to another pipeline through **Linked Build Pipeline**, then you must first delete the linked pipeline in order to delete your pipeline. + +7. Select **Delete** and the CI pipeline will be deleted along with its workflow. + +![Figure 72: Deleting CI pipeline](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/delete-pipeline-delete-ci-popup.jpg) --- ## Extras +### Override Build Configuration + +**Override Options** in **Build Stage** lets you override **Build Configurations** for each workflow of the same application. You can configure overrides in the build stage of each workflow. + +For example, you might want to push development or testing builds to a separate registry from production builds, but in **Build Configurations** you have configured the production container registry to push build images; This means for all the workflows (testing or production), build images will be pushed to the production container registry by default. + +To override this Build configuration for the specific workflow (testing workflow), you can use **Override Options** in the **Build Stage** of that workflow that lets you specify different container registries, how to build container images, and target platforms for different workflows of the same application; This means the images built for testing environment can be included to the testing registry and the images for production environment can be included to the production registry. This helps keep environments isolated, improves deployment safety, and makes managing multiple environments easier. + +#### Creating Build Configuration Override + +To override a container registry, container image, or target platform: + +1. Go to **Applications** and select your application from the **Devtron Apps** tabs. + +2. On the **App Configuration** tab, select **Workflow Editor**. + +3. Select the build pipeline of your application. + + ![Figure 73: Selecting Build Pipeline](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/select-build-override.jpg) + +4. Click **Allow Override** in the **Build Stage**: + + ![Figure 74: Selecting Allow Override](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-allow-override.jpg) + + * Select the new container registry from the drop-down list. + + * [Create and build the new container image](../creating-application/docker-build-configuration.md#build-the-container-image) with different options. + + * Set a [new target platform](../creating-application/docker-build-configuration.md#set-target-platform-for-the-build) from the drop-down list or enter a new target platform. + +5. Select **Update Pipeline**; The override will be effective when the next build is triggered. + +The overridden container registry/container image location/target platform will be reflected on the [Build Configuration](docker-build-configuration.md) page. You can also see the number of build pipelines for which the container registry/container image location/target platform are overridden. + +![Figure 75: Overrides in Build Configuration](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/build-configuration-overridden.jpg) + ### Configuring Webhook {% hint style="info" %} @@ -427,28 +778,96 @@ If you choose [Pull Request](#pull-request) or [Tag Creation](#tag-creation) as #### For GitHub -1. Go to the **Settings** page of your repository and select **Webhooks**. +1. Go to the **Settings** → **Webhooks** of your repository. + + ![Figure 76a: Navigating to Repository Settings](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/deploy-image-ga-settings.jpg) + + ![Figure 76b: Selecting 'Webhooks'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-github-webhooks.jpg) + 2. Select **Add webhook**. + + ![Figure 77: Selecting 'Add webhook'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-github-add-webhook.jpg) + 3. In the **Payload URL** field, enter the Webhook URL that you get on selecting the source type as "Pull Request" or "Tag Creation" in Devtron the dashboard. -4. Change the Content-type to `application/json`. + +4. Select the Content-type as `application/json`. + 5. In the **Secret** field, enter the secret from Devtron the dashboard when you select the source type as "Pull Request" or "Tag Creation". -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-pipeline-4.jpg) +6. Under **Which events would you like to trigger this webhook?**, select **Let me select individual events.** to trigger the webhook for specific events. + + ![Figure 78: Configuring Webhook](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-github-enter-info.jpg) + +7. Select the appropriate triggers + + * For Pull Requests select **Pull Requests**. + + ![Figure 79: Selecting 'Pull Requests'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-github-pull.jpg) + + * For Tag Creation select **Branch or tag creation**. + + ![Figure 80: Selecting 'Branch or tag creation'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-github-branch.jpg) -6. Under **Which events would you like to trigger this webhook?**, select **Let me select individual events.** to trigger the webhook to build CI Pipeline. -7. Select **Branch or tag creation** and **Pull Requests**. 8. Select **Add webhook**. + ![Figure 81: Selecting 'Add webhook'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-github-webhook-added.jpg) + +#### For GitLab + +1. Navigate to **Setting** → **Webhooks** of your repository. + + ![Figure 82: Navigating to Repository Settings](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-gitlab-settings-webhook.jpg) + +2. Select **Add new webhook**. + + ![Figure 83: Selecting 'Add new webhook'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-gitlab-add-new-webhook.jpg) + +3. Enter optional Name and Description. + +4. In the **URL** field, enter the Webhook URL that you get on selecting the source type as "Pull Request" or "Tag Creation" in the Devtron dashboard. + +5. In the **Secret token** field, enter the secret from the Devtron dashboard when you select the source type as "Pull Request" or "Tag Creation". + + ![Figure 84: Configuring Webhook](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-gitlab-enter-info.jpg) + +6. Checkmark the appropriate triggers under the **Trigger** section. + + * For Pull Requests select **Merge request events**. + + ![Figure 85: Selecting 'Merge request events'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-gitlab-merge.jpg) + + * For Tag Creation select **Tag push events** + + ![Figure 86: Selecting 'Tag push events'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-gitlab-tag.jpg) + +7. Select **Add Webhook** and the webhook will be added to your repository. + + ![Figure 87:Selecting 'Add Webhook'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-gitlab-webhook-added.jpg) + #### For Bitbucket Cloud -1. Go to the **Repository settings** page of your Bitbucket repository. -2. Select **Webhooks** and then select **Add webhook**. +1. Navigate to the **Repository settings** page of your Bitbucket repository. + + ![Figure 88: Navigating to Repository Settings](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-bitbucket-settings.jpg) -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-pipeline-6.jpg) +2. Select **Webhooks** under **Workflow** section and then select **Add webhook**. + + ![Figure 89a: Selecting 'Webhooks'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-bitbucket-webhook.jpg) + + ![Figure 89b: Selecting 'Add webhook'](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-bitbucket-add-webhook.jpg) 3. Enter a **Title** for the webhook. + 4. In the **URL** field, enter the Webhook URL that you get on selecting the source type as "Pull Request" or "Tag Creation" in the Devtron dashboard. -5. Select the event triggers for which you want to trigger the webhook. -6. Select **Save** to save your configurations. -![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/ci-pipeline-7.jpg) +5. In the **Secret** field, enter the secret from the Devtron dashboard when you select the source type as "Pull Request" or "Tag Creation". + + ![Figure 90: Configuring Webhook](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-bitbucket-enter-info.jpg) + +6. Select the event triggers for which you want to trigger the webhook under the **Triggers** section. + + ![Figure 91: Selecting triggers](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-bitbucket-trigger.jpg) + +7. Select **Save** to save your configurations. + + ![Figure 92: Saving Configurations](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/creating-application/workflow-ci-pipeline/webhooks-bitbucket-save.jpg)