Use --no-deps when installing compiled requirements files

[relud]

to work around pypa/pip#9644

see also mozilla/gcp-ingestion#2001

Checklist for reviewer:

• Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
• If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
  in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
• If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated

For modifications to schemas in restricted namespaces (see CODEOWNERS):

• Follow the change control procedure

[relud]

remove previous workaround from #2064 which was having no impact

[dataops-ci-bot]

Integration report for "Use --no-deps when installing compiled requirements files"

sql.diff

Click to expand!

diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/clients_daily/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_daily/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/clients_daily/schema.yaml	2022-02-24 18:52:00.418582582 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_daily/schema.yaml	2022-02-24 18:45:22.000000000 +0000
@@ -2,97 +2,74 @@
 - mode: NULLABLE
   name: submission_date_s3
   type: DATE
-- description: null
-  mode: NULLABLE
+- mode: NULLABLE
   name: submission_date
   type: DATE
 - mode: NULLABLE
   name: sample_id
   type: INTEGER
-- description: null
-  mode: NULLABLE
+- mode: NULLABLE
   name: client_id
   type: STRING
-- description: null
-  mode: NULLABLE
+- mode: NULLABLE
   name: aborts_content_sum
   type: INTEGER
-- description: null
-  mode: NULLABLE
+- mode: NULLABLE
   name: aborts_gmplugin_sum
   type: INTEGER
-- description: null
-  mode: NULLABLE
+- mode: NULLABLE
   name: aborts_plugin_sum
   type: INTEGER
-- description: null
-  mode: NULLABLE
+- mode: NULLABLE
   name: active_addons_count_mean
   type: FLOAT
-- description: null
-  fields:
-  - description: null
-    mode: NULLABLE
+- fields:
+  - mode: NULLABLE
     name: addon_id
     type: STRING
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: blocklisted
     type: BOOLEAN
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: name
     type: STRING
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: user_disabled
     type: BOOLEAN
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: app_disabled
     type: BOOLEAN
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: version
     type: STRING
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: scope
     type: INTEGER
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: type
     type: STRING
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: foreign_install
     type: BOOLEAN
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: has_binary_components
     type: BOOLEAN
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: install_day
     type: INTEGER
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: update_day
     type: INTEGER
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: signed_state
     type: INTEGER
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: is_system
     type: BOOLEAN
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: is_web_extension
     type: BOOLEAN
-  - description: null
-    mode: NULLABLE
+  - mode: NULLABLE
     name: multiprocess_compatible
     type: BOOLEAN
   mode: REPEATED

[sean-rose]

You read my mind! (thanks for tackling this)

• There are some pip install commands in .circleci/config.yml which also need to be updated.
• A similar change needs to be made wherever pip-sync is used (e.g. --pip-args "--no-deps"), or pip-sync could be swapped out for pip install.
  • On a related note, in the readme's "Installing bqetl" section, running pip-sync in step 4 ends up uninstalling bqetl from the venv because bqetl isn't in the requirements files, which would be a reason to use pip install instead.

[relud]

You read my mind! (thanks for tackling this)

* There are some `pip install` commands in `.circleci/config.yml` which also need to be updated.

👍

* A similar change needs to be made wherever pip-sync is used (e.g. `--pip-args "--no-deps"`), or pip-sync could be swapped out for `pip install`.

CI caches the pip venv, and needs to use pip-sync to uninstall removed dependencies to ensure reproducibility.

  * On a related note, in the readme's "Installing bqetl" section, running pip-sync in step 4 ends up uninstalling bqetl from the venv because bqetl isn't in the requirements files, which would be a reason to use `pip install` instead.

not using pip-sync is outside the scope of this PR

[relud]

  * On a related note, in the readme's "Installing bqetl" section, running pip-sync in step 4 ends up uninstalling bqetl from the venv because bqetl isn't in the requirements files, which would be a reason to use `pip install` instead.

not using pip-sync is outside the scope of this PR

but we can retain bqetl with <(echo mozilla-bigquery-etl), which i've added.

[dataops-ci-bot]

Integration report for "don't erase bqetl with pip-sync"

sql.diff

Click to expand!

Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/experiment_cumulative_search_count: schema.yaml
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/main: schema.yaml

[sean-rose]

By default I think pip-sync only looks for requirements.txt, so should this specify both requirements.txt and java-requirements.txt like is done elsewhere?

Suggested change

	venv/bin/pip-sync --pip-args=--no-deps
	venv/bin/pip-sync --pip-args=--no-deps requirements.txt \
	java-requirements.txt

[relud]

java requirements are intentionally left out here, as they are optional and should not be required for unit tests.

[dataops-ci-bot]

Integration report for "Merge branch 'main' into pip-no-deps"

sql.diff

No content detected.