Add SQL, schema, metadata, and DAG for modeling prospects table from …

[chelseatroy]

…snowflake in BigQuery

Description

Related Tickets & Documents

• DENG-XXXX
• DSRE-XXXX

Reviewer, please follow this checklist

[scholtzan]

These new lines look slightly odd. They should be safe to delete?

[scholtzan]

Suggested change

	contexts_com_pocket_prospect_1[0].language AS LANGUAGE,
	contexts_com_pocket_prospect_1[0].language AS language,

just for consistency

[dataops-ci-bot]

Integration report for "Merge branch 'main' into snowfake-migration-stg-prospects"

sql.diff

Click to expand!

Only in /tmp/workspace/generated-sql/dags/: bqetl_content_ml_daily.py
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_content_ml_daily.py /tmp/workspace/generated-sql/dags/bqetl_content_ml_daily.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_content_ml_daily.py	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_content_ml_daily.py	2025-05-21 22:39:32.000000000 +0000
@@ -0,0 +1,66 @@
+# Generated via https://github.com/mozilla/bigquery-etl/blob/main/bigquery_etl/query_scheduling/generate_airflow_dags.py
+
+from airflow import DAG
+from airflow.sensors.external_task import ExternalTaskMarker
+from airflow.sensors.external_task import ExternalTaskSensor
+from airflow.utils.task_group import TaskGroup
+import datetime
+from operators.gcp_container_operator import GKEPodOperator
+from utils.constants import ALLOWED_STATES, FAILED_STATES
+from utils.gcp import bigquery_etl_query, bigquery_dq_check, bigquery_bigeye_check
+
+docs = """
+### bqetl_content_ml_daily
+
+Built from bigquery-etl repo, [`dags/bqetl_content_ml_daily.py`](https://github.com/mozilla/bigquery-etl/blob/generated-sql/dags/bqetl_content_ml_daily.py)
+
+#### Description
+
+Daily extracts for corpus item data to be evaluated for new tab presentation.
+
+#### Owner
+
+[email protected]
+
+#### Tags
+
+* impact/tier_2
+* repo/bigquery-etl
+"""
+
+
+default_args = {
+    "owner": "[email protected]",
+    "start_date": datetime.datetime(2025, 5, 5, 0, 0),
+    "end_date": None,
+    "email": ["[email protected]", "[email protected]"],
+    "depends_on_past": False,
+    "retry_delay": datetime.timedelta(seconds=1800),
+    "email_on_failure": True,
+    "email_on_retry": True,
+    "retries": 2,
+    "max_active_tis_per_dag": None,
+}
+
+tags = ["impact/tier_2", "repo/bigquery-etl"]
+
+with DAG(
+    "bqetl_content_ml_daily",
+    default_args=default_args,
+    schedule_interval="0 4 * * *",
+    doc_md=docs,
+    tags=tags,
+    catchup=False,
+) as dag:
+
+    snowflake_migration__prospects__v1 = bigquery_etl_query(
+        task_id="snowflake_migration__prospects__v1",
+        destination_table="prospects_v1",
+        dataset_id="snowflake_migration",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        task_concurrency=1,
+    )
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: snowflake_migration
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/dataset_metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/dataset_metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/dataset_metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/dataset_metadata.yaml	2025-05-21 22:29:53.000000000 +0000
@@ -0,0 +1,12 @@
+friendly_name: Snowflake Migration
+description: |-
+  Queries for modeling raw snowplow data about prospects
+  (candidate links to be curated by ML models and the editorial team for New Tab).
+
+user_facing: false
+dataset_base_acl: derived
+labels: {}
+workgroup_access:
+  - role: roles/bigquery.dataViewer
+    members:
+      - workgroup:mozilla-confidential
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/metadata.yaml	2025-05-21 22:33:40.000000000 +0000
@@ -0,0 +1,36 @@
+friendly_name: Prospects v1
+description: |-
+  Model prospective article options for the Fx New Tab from raw Snowplow data
+owners:
+- [email protected]
+- [email protected]
+labels:
+  schedule: daily
+  incremental: true
+  owner1: skamath
+  owner2: mlcooper
+  dag: bqetl_content_ml_daily
+scheduling:
+  dag_name: bqetl_content_ml_daily
+  date_partition_parameter: null
+bigquery:
+  time_partitioning:
+    type: day
+    field: happened_at
+    require_partition_filter: true
+    expiration_days: 775.0
+  range_partitioning: null
+  clustering:
+    fields:
+    - prospect_source
+    - language
+    - topic
+    - publisher
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references:
+  query.sql:
+  - moz-fx-data-shared-prod.snowplow_external.events
+require_column_descriptions: false
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/query.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/query.sql	2025-05-21 22:29:53.000000000 +0000
@@ -0,0 +1,81 @@
+WITH stg_prospects AS (
+  SELECT
+    event_id,
+    unstruct_event_com_pocket_object_update_1.trigger AS object_update_trigger,
+  -- prospect info
+    contexts_com_pocket_prospect_1[0].prospect_id AS prospect_id,
+    contexts_com_pocket_prospect_1[0].url AS url,
+    contexts_com_pocket_prospect_1[0].scheduled_surface_id AS scheduled_surface_id,
+    contexts_com_pocket_prospect_1[0].prospect_source AS prospect_source,
+    TIMESTAMP_SECONDS(contexts_com_pocket_prospect_1[0].created_at) AS created_at,
+    TIMESTAMP_SECONDS(DIV(contexts_com_pocket_prospect_1[0].reviewed_at, 1000)) AS reviewed_at,
+    contexts_com_pocket_prospect_1[0].prospect_review_status AS prospect_review_status,
+    ARRAY_TO_STRING(contexts_com_pocket_prospect_1[0].status_reasons, ",") AS status_reasons,
+    contexts_com_pocket_prospect_1[0].status_reason_comment AS status_reason_comment,
+    contexts_com_pocket_prospect_1[0].reviewed_by AS reviewed_by,
+    contexts_com_pocket_prospect_1[0].title AS title,
+    contexts_com_pocket_prospect_1[0].excerpt AS excerpt,
+    contexts_com_pocket_prospect_1[0].image_url AS image_url,
+    contexts_com_pocket_prospect_1[0].language AS LANGUAGE,
+    contexts_com_pocket_prospect_1[0].topic AS topic,
+    contexts_com_pocket_prospect_1[0].is_collection AS is_collection,
+    contexts_com_pocket_prospect_1[0].is_syndicated AS is_syndicated,
+    ARRAY_TO_STRING(contexts_com_pocket_prospect_1[0].authors, ",") AS authors,
+    contexts_com_pocket_prospect_1[0].publisher AS publisher,
+    contexts_com_pocket_prospect_1[0].domain AS domain,
+    TO_JSON(contexts_com_pocket_prospect_1[0].features) AS features,
+    TO_JSON(contexts_com_pocket_prospect_1[0].run_details) AS run_details,
+    contexts_com_pocket_prospect_1[0]._schema_version AS schema_version,
+  -- event info
+    derived_tstamp AS happened_at,
+  FROM
+    `moz-fx-data-shared-prod.snowplow_external.events`
+  WHERE
+    event_name = 'object_update'
+    AND unstruct_event_com_pocket_object_update_1.object = 'prospect'
+    AND SAFE_CAST(contexts_com_pocket_prospect_1[0].created_at AS INT64) IS NOT NULL
+    AND SAFE_CAST(contexts_com_pocket_prospect_1[0].created_at AS INT64)
+    BETWEEN 946684800
+    AND UNIX_MILLIS(CURRENT_TIMESTAMP())
+    AND SAFE_CAST(contexts_com_pocket_prospect_1[0].reviewed_at AS INT64) IS NOT NULL
+    -- reviewed_at is in miliseconds (for some reason), so we need to divide
+    AND SAFE_CAST(DIV(contexts_com_pocket_prospect_1[0].reviewed_at, 1000) AS INT64)
+    BETWEEN 946684800
+    AND UNIX_MILLIS(CURRENT_TIMESTAMP())
+  -- This ensures recommended_at is between Jan 1, 2000, and the current time to remain within BQ limits for dates
+  QUALIFY
+    ROW_NUMBER() OVER (PARTITION BY happened_at ORDER BY happened_at) = 1
+)
+SELECT
+  p.prospect_id,
+  p.object_update_trigger,
+  p.url,
+  p.scheduled_surface_id,
+  p.prospect_source,
+  p.created_at,
+  p.reviewed_at,
+  p.prospect_review_status,
+  p.status_reasons,
+  p.status_reason_comment,
+  p.reviewed_by,
+  p.title,
+  p.excerpt,
+  p.image_url,
+  p.language,
+  p.topic,
+  p.authors,
+  p.publisher,
+  p.domain,
+  p.is_collection,
+  p.is_syndicated,
+  p.happened_at,
+  p.features,
+  p.run_details,
+  p.schema_version,
+  TO_BASE64(
+    SHA256(CONCAT(p.prospect_id, p.object_update_trigger))
+  ) AS prospect_id_object_update_trigger_key
+FROM
+  stg_prospects p
+QUALIFY
+  ROW_NUMBER() OVER (PARTITION BY prospect_id, object_update_trigger ORDER BY happened_at DESC) = 1;
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/schema.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/snowflake_migration/prospects_v1/schema.yaml	2025-05-21 22:29:53.000000000 +0000
@@ -0,0 +1,116 @@
+fields:
+- mode: NULLABLE
+  type: STRING
+  name: prospect_id
+  description: Unique identifier for prospects
+- mode: NULLABLE
+  type: STRING
+  name: url
+  description: The url of the prospects item
+- mode: NULLABLE
+  type: STRING
+  name: scheduled_surface_id
+  description: Recommended destination where the prospect item is expected to appear
+    (NEW_TAB_EN_INTL, NEW_TAB_EN_US, NEW_TAB_DE_DE, NEW_TAB_EN_GB).
+- mode: NULLABLE
+  type: STRING
+  name: prospect_source
+  description: Source identified by the ML process for the prospect (SYNDICATED, ORGANIC_TIMESPENT,
+    GLOBAL).
+- mode: NULLABLE
+  type: TIMESTAMP
+  name: created_at
+  description: timestamp when the prospect was first created
+- mode: NULLABLE
+  type: TIMESTAMP
+  name: reviewed_at
+  description: timestamp when the prospect was reviewed by the curator. It is also
+    the timestamp that caused this event action
+- mode: NULLABLE
+  type: STRING
+  name: prospect_review_status
+  description: The curator's review status for the prospect
+- mode: NULLABLE
+  type: STRING
+  name: status_reasons
+  description: The list of curator review status reasons
+- mode: NULLABLE
+  type: STRING
+  name: status_reason_comment
+  description: Curator review status reason comment
+- mode: NULLABLE
+  type: STRING
+  name: reviewed_by
+  description: The curator who reviewed the prospect
+- mode: NULLABLE
+  type: STRING
+  name: title
+  description: The title of the reviewed corpus item
+- mode: NULLABLE
+  type: STRING
+  name: excerpt
+  description: The excerpt for the reviewed corpus item
+- mode: NULLABLE
+  type: STRING
+  name: image_url
+  description: The url of the main image of the reviewed corpus item
+- mode: NULLABLE
+  type: STRING
+  name: language
+  description: The language of the reviewed_corpus_item
+- mode: NULLABLE
+  type: STRING
+  name: topic
+  description: The topic of the reviewed_corpus_item
+- mode: NULLABLE
+  type: STRING
+  name: authors
+  description: The list of authors of the reviewed_corpus_item
+- mode: NULLABLE
+  type: STRING
+  name: publisher
+  description: The name of the online publication that published this story.
+- mode: NULLABLE
+  type: BOOLEAN
+  name: is_collection
+  description: Indicates whether the reviewed_corpus_item is a collection
+- mode: NULLABLE
+  type: BOOLEAN
+  name: is_syndicated
+  description: Indicates whether the reviewed_corpus_item is a syndicated article
+- mode: NULLABLE
+  type: JSON
+  name: features
+  description: 'ML features for the prospect item. Note: These features are being
+    passed through for ML use and will not be engineered in the warehouse
+
+    '
+- mode: NULLABLE
+  type: JSON
+  name: run_details
+  description: 'Details about the run, including the flow name and run ID. Note: The
+    ML team could include additional debug properties about the run, but these properties
+    will not be engineered in the warehouse
+
+    '
+- mode: NULLABLE
+  type: STRING
+  name: schema_version
+  description: Snowplow version identifier.
+- mode: NULLABLE
+  type: TIMESTAMP
+  name: happened_at
+  description: Event creation timestamp
+- mode: NULLABLE
+  type: STRING
+  name: prospect_id_object_update_trigger_key
+  description: 'A combination of the prospect_id and object_update_trigger columns
+    to create a unique event identifier
+
+    '
+- mode: NULLABLE
+  type: STRING
+  name: object_update_trigger
+- mode: NULLABLE
+  type: STRING
+  name: domain

Link to full diff