1.10.9 (core) / 0.26.9 (libraries)

Bugfixes

• [ui] Fix custom time datepicker filter selection for users with custom Dagster timezone settings.

dg & Components (Preview)

• Add support for dg.toml files. dg settings in pyproject.toml are set under tool.dg, but in dg.toml they are set at the top level. A dg.toml may be used in place of pyproject.toml at either the project or workspace level.
• dg-scaffolded workspaces now include a dg.toml instead of pyproject.toml file.
• dg-scaffolded projects now place the root package in an src/ directory and use hatchling as the build backend rather than setuptools.
• Fixed a bug where dg would crash when working with Python packages with an src-based layout.
• dg check yaml now properly validates component files in nested subdirectories of the defs/ folder.

1.10.8 (core) / 0.26.8 (libraries)

New

• [ui] The Dagster UI now allows you to specify extra tags when re-executing runs from failure from the runs feed re-execute dialog, or by holding shift when clicking Re-execute menu items throughout the app.
• [ui] Performance improvements for loading the partitions page for multi-partitioned assets.
• [ui] Fix link in toast messages that appear when launching backfills.
• [ui] Dagster's UI now allows you to copy run tags as a YAML block from the Tags and Configuration modals.
• [ui] The Dagster Run UI now allows you to view the execution plan of a queued run.

Bugfixes

• The AutomationCondition.initial_evaluation condition has been updated to become true for all partitions of an asset whenever the PartitionsDefinition of that asset changes, rather than whenever the structure of the condition changes.
• [dagster-fivetran] Fixed an issue where new runs of code locations using fivetran assets would sometimes raise a "Failure condition: No metadata found for CacheableAssetsDefinition" error if the run was started immediately after a new version of the code location was deployed.
• [dagster-fivetran] Fixed an issue where including multiple sets of assets from build_fivetran_assets_definitions in a single Definitions object would result in "duplicate node" errors when launching a run.
• [ui] Fixed line charts for colorblind themes.
• [ui] Fixed an issue with querystring parsing that can arise when selecting a large number of items in the selection syntax input.
• [ui] Fixed tag filtering on automations list.
• [ui] Fixed hover state on focused inputs.
• [ui] Fixed an issue with the Run step selection input autocomplete where it would suggest key:"*substring*" instead of name:"*substring*".
• [ui] Fixed the "View run" link shown when launching runs

Documentation

• Fix a bug in example code for pyspark.

dg & Components (Preview)

• Added the ability to scaffold Python components.
• The DefsModuleComponent has been renamed to DefsFolderComponent.
• When scaffolding a component, the command is now dg scaffold my_project.ComponentType instead of dg scaffold component my_project.ComponentType.
• [dagster-dg] dagster list defs will now read environment variables from a local .env file if present when constructing the definitions.
• dagster-components has been merged into dagster and use of the dagster-components package has been deprecated.
  dagster-components will remain as a stub package for the next few weeks, but code should be updated to import from dagster.components instead of dagster_components.
• The DbtProjectComponent has been relocated to the dagster-dbt package,
  importable as dagster_dbt.DbtProjectComponent.
• The SlingReplicationCollectionComponent has been relocated to the dagster-sling package,
  importable as dagster_sling.SlingReplicationCollectionComponent.

1.10.7 (core) / 0.26.7 (libraries)

New

• Applying changes from sensor test results now also applies changes from dynamic partition requests.
• When merging assets from multiple code locations, autogenerated specs are now prioritized lower than customized external asset specs.
• [ui] Allowed using command-click to view a run from the toast message that appears when starting a materialization of an asset.
• [ui] Asset graph can now zoom out a lot more.
• [ui] Added a kind tag for dbt Cloud.
• [dagster-dlt] Added backfill policy to dlt_assets, defaulting to single-run. (Thanks @neuromantik33!)

Bugfixes

• Updated AutomationCondition.initial_evaluation condition to become true for all partitions of an asset whenever the PartitionsDefinition of that asset changes, rather than whenever the structure of the condition changes.
• Fixed a bug with several integrations that caused data fetched from external APIs not to be properly cached during code server initialization, leading to unnecessary API calls in run and step worker processes. This affected dagster-airbyte, dagster-dlift, dagster-dbt, dagster-fivetran, dagster-looker, dagster-powerbi, dagster-sigma, and dagster-tableau.
• [ui] Fixed an issue with the Run step selection input autocomplete where it would suggest key:"*substring*" instead of name:"*substring*".
• [ui] Fixed the "View run" link shown when launching runs.
• [ui] Fixed an issue where updating a catalog view caused an infinite loading state.
• Fixed an issue which could cause asset check evaluations emitted from the body of the op to not impact the check status of an asset in the UI.
• Fixed an issue that could cause an asset backfill created by re-executing another backfill from the point of failure to error on the first tick in rare cases.
• Fixed an issue that could cause automation condition evaluations to fail to render in the UI in rare cases.
• [ui] Fixed a regression in the "Cancel Backfill" option for job backfills that have finished queuing runs.
• [ui] Fixed overflow of long runs feed table on backfill page.
• [dagster-dbt] Replaced @validator with @field_validator in dagster_dbt/core/resource.py to prevent Pydantic deprecation warnings. (Thanks @tintamarre!)

Documentation

• Updated the "Asset versioning and caching" guide to reflect the current Dagster UI and "Unsynced" labels.
• Removed a few repeated lines in documentation on customizing automation conditions. (Thanks @zero-stroke!)
• Fixed example in TableRecord documentation to use the new input format.

Configuration

• [dagster-gcp] Updated Dataproc configuration to the latest version. If necessary, consider pinning your dagster-gcp version while you migrate config. Please see the full list of changed fields: https://gist.github.com/deepyaman/b4d562e04fe571e40037a344b7a9937d
• [dagster-aws][dagster-spark] Updated Spark configuration to the latest version (3.5.5). If necessary, consider pinning your dagster-aws and/or dagster-spark version while you migrate config. Please see the full list of changed fields: https://gist.github.com/deepyaman/f358f5a70fea28d5f164aca8da3dee04

Dagster Plus

• [ui] Fixed filtering for multiple tags on list view pages, including Automations.
• [ui] Fixed an issue where the urls generated by catalog filtering would remove all filters if loaded directly.
• [ui] Added a warning on the sign-in page indicating that the sign-in and signup flows will be changing soon.
• [ui] Require confirmation when rolling back a code location to a previous version.

dg & Components (Preview)

• Virtual environment detection settings for projects have changed. Previously, the global settings use_dg_managed_environment and require_local_venv controlled the environment used when launching project subprocesses. This is now configured at the project level. The tool.dg.project.python_environment setting takes a value of either persistent_uv or active. persistent_uv will be used by default in new scaffolded projects and uses a uv-managed .venv in the project root. active is the default if no tool.dg.project.python_environment is set, and just uses the active python environment and opts out of dg management of the environment.
• A new base class, Resolvable, has been added. This can be used to simplify the process of defining a yaml schema for your components. Instead of manually defining a manual ResolvedFrom[...] and ResolvableModel, the framework will automatically derive the model schema for you based off of the annotations of your class.
• Python files with Pythonic Components (i.e. defined with @component) can now contain relative imports.
• The dg init command now accepts optional --workspace-name and --project-name options to allow scaffolding an initial workspace and project via CLI options instead of prompts.
• Added a new dagster_components.dagster.DefsFolderComponent that can be used at any level of your defs/ folder to apply asset attributes to the definitions at or below that level. This was previously named dagster_components.dagster.DefsModuleComponent.

1.10.6 (core) / 0.26.6 (libraries)

New

• Added a new AutomationCondition.executed_with_tags() condition that makes it possible to filter for updates from runs with particular tags.
• AssetCheckEvaluation can now be yielded from Dagster ops to log an evaluation of an asset check outside of an asset context.
• Added the kinds argument to dagster.AssetOut, allowing kinds to be specified in @multi_asset.
• [dagster-dbt] AssetCheckEvaluations are now yielded from ops leveraging DbtCliResource.cli(...) when asset checks are included in the dbt asset lineage.
• [dagster-sling] The SlingResource.replicate() method now takes an option stream parameter, which allows events to be streamed as the command executes, instead of waiting until it completes (thanks, @natpatchara-w!).
• [dagster-graphql] The DagsterGraphQLClient now supports an auth keyword argument, which is passed to the underlying RequestsHTTPTransport constructor.
• [ui] The asset selection syntax input now allows slashes "/" in the freeform search.
• [ui] The backfill pages now show summary information on all tabs for easier backfill monitoring.

Bugfixes

• Fixed issue with AutomationCondition.newly_requested() which could cause it to fail when nested within AutomationCondition.any_deps_match() or AutomationCondition.all_deps_match().
• Fixed a bug with AutomationCondition.replace() that would cause it to not effect AutomationCondition.since() conditions.
• Fixed a bug with several integrations that caused data fetched from external APIs not to be properly cached during code server initialization, leading to unnecessary API calls in run and step worker processes. This affected: dagster-airbyte, dagster-dlift, dagster-dbt, dagster-fivetran, dagster-looker, dagster-powerbi, dagster-sigma, and dagster-tableau.
• Fixed a bug that could cause invalid circular dependency errors when using asset checks with additional dependencies.
• [dagster-fivetran] Loading assets for a Fivetran workspace containing incomplete and broken connectors now no longer raises an exception.
• [ui] Fixed the colorblind (no red/green) theme behavior when in dark mode.
• [ui] The Asset > Partitions page no longer displays an error in some cases when creating dynamic partitions.
• [ui] The Launch and Report Events buttons no longer error if you click it immediately after creating a new dynamic partition.

dg & Components (Preview)

• __pycache__ files are no longer included in the output of dg list component (thanks @stevenayers!)
• When resolving the deps of an AssetSpec from yaml, multi-part asset keys are now correctly parsed (thanks @stevenayers!)
• The entrypoint group for dg projects has been renamed from dagster.components to dagster_dg.library
• dg check yaml is now run by default before dg dev and dg check defs

1.10.5 (core) / 0.26.5 (libraries)

New

• async def yield_for_execution is now supported on ConfigurableResource. An event_loop argument has been added to context builders to support direct execution.
• dagster dev deduplicates stacktraces when code locations fail to load, and will by default truncate them to highlight only user code frames.
• Improved error message experience for resources expecting an env var which was not provided.
• [ui] An updated asset selection syntax is now available in the asset graph, insights, and alerts. The new syntax allows combining logical operators, lineage operators, and attribute filters.
• [dagster-polars] The minimal compatible deltalake version has been bumped to 0.25.0; the PolarsDeltaIOManager is now using the rust engine for writing DeltaLake tables by default.

Bugfixes

• Fixed a bug with AutomationCondition.replace() that would cause it to not effect AutomationCondition.since() conditions.
• Fixed a bug that could cause invalid circular dependency errors when using asset checks with additional dependencies.
• Fixed an issue in Dagster OSS where failed runs of multiple partitions didn't update those partitions as failed in the Dagster UI or trigger failure automation conditions.
• Fixed an issue where dagster dev would fail to load code that took more than 45 seconds to import unless the --use-legacy-code-server-behavior flag was used.
• [dagster-airbyte] Fixed an issue that caused the group name of assets created using build_airbyte_assets_definitions function to error when attempting to modify the default group name.
• [dagster-fivetran] Fixed an issue that caused the group name of assets created using build_fivetran_assets_definitions function to error when attempting to modify the default group name.

1.10.4 (core) / 0.26.4 (libraries)

New

• [ui] The asset overview tab for a partitioned asset now shows metadata and schema of the most recent materialization, not today's partition.
• [ui] In run logs, asset materialization and observation events now show the output partition as well as the asset key.
• [ui] The backfills view has moved to Runs > Backfills and is no longer available on the Overview tab.
• [ui] Pool event information from a run now links to the pool configuration page.
• Added support for passing tags to the created RunRequest when using build_sensor_for_freshness_checks().
• [dagster-gcp] The PickledObjectGCSIOManager now replaces the underlying blob when the same asset is materialized multiple times, instead of deleting and then re-uploading the blob.
• [docs] Added docs covering run-scoped op concurrency.
• [dagster-fivetran] Fivetran connectors fetched in Dagster can now be filtered and selected using the ConnectorSelectorFn.

Bugfixes

• Fixed a bug where if a run was deleted while the re-execution system was determining whether the run should be retried an error was raised. Now, if a run is deleted while the re-execution system is determining whether the run should be retried, the run will not be retried.
• [ui] Fixed an issue where assets with automation conditions wouldn't show the jobs/sensors/schedules targeting them.
• [ui] Steps properly transition to failed in the Run gantt chart when resource initialization fails.

1.10.3 (core) / 0.26.3 (libraries)

New

• Added links from pool info in run event logs to the respective pool configuration pages.
• Added queued run information on the pool info page, even if the pool granularity is set to op.
• [ui] Added information about asset partitions that fail to materialize due to run cancellations to the asset partition detail page.
• [ui] Added two new themes for users with reduced sensitivity to red and green light.
• [ui] Added Not Diamond icon for asset kinds tag. (Thanks @dragos-pop!)
• [ui] Added Weaviate icon for asset kinds tag. (Thanks @jjyeo!)
• [ui] Made Alerts page visible to users with Viewer roles.
• [dagster-postgres] Removed the cap on PostgresEventLogStorage QueuePool by setting max_overflow to -1. (Thanks @axelwas!)

Bugfixes

• Fixed a bug where a sensor emitting multiple RunRequests with the same run key within a single tick would cause two runs with the same key to be executed. Now, only the first run will be executed. (Thanks @Gw1p!)
• Fixed a bug where run step selections were not affecting which pools limit a given run.
• Fixed an issue where seeding the random number generator during code import or when initializing a resource could cause every step to write to the same stdout or stderr key.
• [ui] Fixed an issue where certain jobs weren't showing the assets they targeted.
• Asset backfills will now move into a CANCELED state instead of a FAILURE state when not every requested partition has been marked as materialized or failed by the backfill.
• [dagster-dbt] Fixed a bug breaking packaged_project_dir since supporting profiles_dir in DbtProject.
• Fixed an issue with DbIOManagers being unable to process subclasses of handled types.
• [ui] Preserved asset selection filters when navigating folders in the asset catalog.
• [ui] Corrected PostgreSQL SVG icon for asset kinds tag. (Thanks @dragos-pop!)
• [ui] Fixed an issue that caused Markdown with code blocks in languages not supported for syntax highlighting to crash the page.
• Fixed an issue where asset backfills included failed partitions in the in-progress list in logging output.

Documentation

• Fixed broken image links in quickstart examples. (Thanks @stevenayers!)
• [dagster-dbt] Made several fixes to the "Using dbt with Dagster" page. (Thanks @jjyeo!)
• Fixed broken link in defining-assets.md. (Thanks @Exlll!)
• Fixed link in CONTRIBUTING.md leading to a 404. (Thanks @Exlll!)
• Fixed typo in managing-code-locations-with-definitions.md. (Thanks @kgeis!)
• Fixed typo in asset-versioning-and-caching.md. (Thanks @petrusek!)

Dagster Plus

• [ui] Enabled setting long-running job alerts in minutes instead of hours.
• [dagster-insights] Fix links to branch deployments in the deployment list UI.
• [dagster-insights] Adjusted the way batching runs from the create_snowflake_insights_asset_and_schedule sensor using the schedule_batch_size_hrs parameter works to yield a single partition range run instead of individual runs per partition.

1.10.2 (core) / 0.26.2 (libraries)

New

• Performance improvements for backfills of large partition sets.
• The prefix of temporary directories created when running a temporary Dagster instance (as with dagster dev) has been changed from tmp to .tmp_dagster_home_. Thanks @chazmo03!
• [dagster-aws] Added sample Terraform modules for Dagster deployment on AWS ECS.
• [dagster-dbt] Added pool support for dbt integrations.
• [dagster-dlt] Added pool support for dlt integrations.
• [dagster-sling] Added pool support for sling integrations.
• [dagster-aws] Added AWS RDSResource. Thanks @shimon-cherrypick!
• [dagster-mysql] Added MySQLResource. Thanks @shimon-cherrypick!
• [dagster-azure] Added Azure Blob Storage Resource. Thanks @shimon-cherrypick!
• [ui] Expanding/collapsing groups in the Asset Graph will no longer reset your zoom.
• [ui] Changed the queue criteria dialog to reference pools instead of concurrency keys.
• [ui] The Instance Backfills page is being removed in the upcoming March 6 release in favor of the new Runs > Backfills view.
• [ui] When re-executing a run that is part of a backfill that has completed, Dagster UI notifies you that the re-execution will not update the backfill stats.
• [ui] The backfill actions menu now includes "Re-execute" and "Re-execute from failure", which create new backfills targeting the same partitions, and the partitions which failed to materialize, respectively.
• [ui] The latest asset check evaluation is shown in the Evaluation History tab, and AssetCheckResult descriptions are rendered in the table making it easier to publish a summary of check evaluation.
• [ui] The Materialize button appears more quickly on asset pages in the Dagster UI.
• [ui] The queue details modal for a run no longer closes as new runs arrive and links to the correct concurrency page.

Bugfixes

• Fixed an issue where if two separate code locations defined the same asset key with an automation condition, duplicate runs could be created by Declarative Automation.
• Fixed the psycopg2.errors.UndefinedColumn database error when trying to set a concurrency key without first having run dagster instance migrate.
• Fixed an issue where Declarative Automation sensors in code locations that included source assets referencing assets with automation conditions in other code locations would sometimes cause duplicate runs to be created.
• Fixed a bug in the enforcement of global op concurrency limits.
• Fixed an issue where when using dagster dev, some changes were not reflected in the UI after pressing the "Reload Definitions" button.
• Fixed the issue where a resource initialization error within a sensor definition test incorrectly recommended using build_schedule_context instead of build_sensor_context.
• Fixed migration issue where dagster instance migrate was failing for instances with non-empty concurrency limits tables.
• [ui] Fixed an issue where a "Message: Cannot return null for non-nullable field PartitionKeys.partitionKeys." error was raised in the launchpad for jobs with unpartitioned assets.
• [ui] Fixed concurrency link escaping in the View queue criteria dialog.
• [ui] Fixed an issue where the deployment switcher can become permanently "unset" when navigating from Org Settings back to a deployment.
• [ui] Fixed an issue with the traversal operators on the asset graph (asset++) not including assets connected to the target asset by paths of varying distance.

Dagster Plus

• A setting is available in agent configuration direct_snapshot_uploads (directSnapshotUploads in helm) which opts in to a new more efficient scheme for how definitions are handled during code location updates.
• Introduced new test utilities event_log and dagster_event in dagster-cloud-test-infra to facilitate the creation of test data with sensible defaults for EventLogEntry and DagsterEvent objects.
• [bigquery-insights][bugfix] Support querying for insights from the configured execution_project if defined.
• [bigquery-insights][bugfix] When execution_project is defined in the dbt profile, fall back to fetching the dataset from the dbt profile's project if the dataset cannot be found in the execution_project.

1.10.1 (core) / 0.26.1 (libraries)

Bugfixes

• Fixed an issue where runs containing pool-assigned ops without limits set got stuck in the run queue.
• Fixed an issue where a "Message: Cannot return null for non-nullable field PartitionKeys.partitionKeys." error was raised in the launchpad for jobs with unpartitioned assets.
• [ui] Updated "Queue criteria" modal to reference and link to pool concurrency settings pages.
• [ui] The "Queue criteria" modal for a run no longer closes as new runs arrive.

1.10.0 (core) / 0.26.0 (libraries)

New

• Added a new AutomationCondition.data_version_changed() condition.
• [dagster-msteams] Added support for sending messages to PowerAutomate flows using AdaptiveCard formatting.
• dagster definitions validate is now less verbose, primarily highlighting load errors.
• [ui] Made defunct code locations removable when editing environment variables.
• [ui] Added a warning icon to the Agents item in Deployment settings, indicating when there are no active agents.
• [dagster-tableau] Changed logic to show embedded data sources in case published data sources are not present. Also, pulled more metadata from Tableau. (Thanks @VenkyRules!)
• Added new decorators to reflect our new API lifecycle: @preview, @beta and @superseded. Also added new annotations and warnings to match these new decorators.

Bugfixes

• [ui] Fixed persistence of the group-by setting in the run timeline view.
• [ui] Fixed timestamped links to asset pages from asset check evaluations in run logs.
• [ui] Fixed excessive rendering and querying on the Concurrency configuration page.
• Fixed the step stats calculations for steps that fail and request a retry before the step starts. This happened if a failure occurred in the step worker before the compute function began execution. This should help with sporadic hanging of step retries.
• Fixed an issue where the Concurrency UI was broken for keys with slashes.
• Fixed an issue with emitting AssetResult with ops or multi-assets that are triggered multiple times in the same run.
• [dagster-dbt] Fixed a bug introduced in dagster-dbt 0.25.7 that would cause execution to fail when using the @dbt_assets decorator with an io_manager_key specified.
• [dagster-dbt] Refactored UnitTestDefinition instantiation to address failure to initialize dbt models with unit tests. (Thanks @kang8!)
• Fixed issue where dagster instance migrate was failing for instances with tables having non-empty concurrency limits.
• Fixed an issue where Declarative Automation sensors in code locations that included source assets referencing assets with automation conditions in other code locations would sometimes cause duplicate runs to be created.
• Turned on run blocking for concurrency keys/pools by default. For op granularity, runs are dequeued if there exists at least one op that can execute once the run has started. For run granularity, runs are dequeued if all pools have available slots.
• [dagster-dbt] Added pool support.
• [dagster-dlt] Added pool support.
• [dagster-sling] Added pool support.

Documentation

• Corrected docs on managing concurrency.
• Fixed a Markdown link to "assets metadata." (Thanks @rchrand!)
• Fixed a pip install command for Zsh. (Thanks @aimeecodes!)

Breaking Changes

• The include_sources param on all AssetSelection APIs has been renamed to include_external_assets.

• Disallowed invalid characters (i.e. anything other than letters, numbers, dashes, and underscores) in pool names.

• Changed the default run coordinator to be the queued run coordinator. This requires the Dagster daemon to be running for runs to be launched. To restore the previous behavior, you can add the following configuration block to your dagster.yaml:

  run_coordinator:
    module: dagster.core.run_coordinator.sync_in_memory_run_coordinator
    class: SyncInMemoryRunCoordinator
  

Deprecations

• [dagster-sdf] Moved the dagster-sdf library to the community-supported repo.
• [dagster-blueprints] Removed the dagster-blueprints package. We are actively developing a project, currently named Components, that has similar goals to Blueprints of increasing the accessibility of Dagster.
• Removed the @experimental decorator in favor of the @preview and @beta decorators. Also removed annotations and warnings related to the @experimental decorator.

Dagster Plus

• Shipped a range of improvements to alerts in Dagster+, including more granular targeting, streamlined UIs, and more helpful content. Stay tuned for some final changes and a full announcement in the coming weeks!