Enable FCM Integration Test (firebase#3145)

* Enable the FCM integration test (IT) for Chrome.

Context: the tests were switched off a couple of years ago due to flakiness. It has not been maintained since then. During the time, Some Selenium API methods used were deprecated and removed; the Firebase projects used are no longer managed or owned by the FCM team. Consequently, the test became unfunctional. In the effort of providing safety for the upcoming FCM releases, this PR is created to fix, deflake, refactor and improve the old tests.

This PR did the following:
- Enabled comprehensive IT for chrome (ver.80). Now we are covering send&foreground recevie for FCM messages (messages with {notification} payload, {data} payload and {notification, data} payload), delete/update token, use default/customized ServiceWorker.

- Defalaked test. The IT is now reasonably stable without retry. Previously we are retrying 1 or 3 times.

- Optimized test. Previously we create a webDriver for each test, which is slow and annoying. Because a window is created and brought to focus and killed frequently, it makes working on other tasks and testing nearly impossible (Probably using a headless browser would work but I haven't found a satisfying solution to have the app in the state of foreground and background which is a requirement for FCM functions). With the way the tests are organized, the IT only spin up a new web driver when necessary. Some data on performance: (old) 'test-send' take 74 seconds (only measured 'test-send' because the other test suites were not functional at the time); (now) 'test-send', 'test-deleteToken', 'test-updateToken', 'test-useDefaultServiceWorker', 'test-useValidManifest' takes in total 33s (10 run average).

- General refactoring. Including refactors on expect blocks, createWebDriver, use const for constants usage, etc. The code should be much easier to understand and maintain.

Future work:
- Enable test on firefox once I get the notification permission working.
- Run the IC against some milestone chrome/firefox version (if it makes sense) to ensure backward compatibility. We should try to avoid firebase#2712 . :)

* [AUTOMATED]: License Headers

* Enable integration test (IT) for FCM.

Context: FCM IT were turned off a couple years ago due to flakiness. It became mostly unfunctional as repo structure change overtime. The goal is to fix and enable IT for more confident developement flow and safer releases.

This CL does the following:
- Fix the IT to be functional again. The IT is derteminated from my experiements (no longer flaky). Therefore, The CL removed the retry mechasim (previously retry 3 times) which makes running IT cheaper and more enjoyable.

- Include IT for test:change for FCM package: the entire IT test suites is resoanblly fast (from my exeperience 1-3 miutes to complete. As it grows larger, maybe it makes run tests in parallel in Saucelab)

Futhure work:
- Enable testing for firefox

* Correct int syntax

js doesn't allow underscore for int

* This file wasn't auto-saved

* Trigger FCM IT

why dot.env(FCM_SECRETS) are not included in env?

* Test Secrets can be accessed w/o dotenv

* Add fcm sercret to workflow

* Update test-changed.yml

* test send (background only)

Because headless chrome doesn't have the concept of foreground and background

* remove dotenv

* feed secrest into test:all workflow

* Update test-all.yml

* background messaging checking

* [AUTOMATED]: License Headers

* rerun

* added waiting

* wait

* Update test-send.js

* Update test-send.js

* Examine wrong sercret

* Update sendMessage.js

* Update sendMessage.js

* Update sendMessage.js

* Update sendMessage.js

* Update sendMessage.js

* update fcm project

* Update package.json

* Update test-send.js

* open new tab for backgournd receive

* removed  test-send

somehow not workingin github workflow?

* Adding Reties

* Change timeout limit

* retry 3 times

* adjust mocha setting

* update

Co-authored-by: Kai Wu <[email protected]>

Author: zwu52

integration/messaging/download-browsers.js

@@ -1,6 +1,6 @@
 /**
  * @license
- * Copyright 2017 Google Inc.
+ * Copyright 2017 Google LLC
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -18,14 +18,9 @@
 const seleniumAssistant = require('selenium-assistant');
 
 console.log('Starting browser download - this may take some time.');
-Promise.all([
-  seleniumAssistant.downloadLocalBrowser('chrome', 'stable', 48),
-  seleniumAssistant.downloadLocalBrowser('chrome', 'beta', 48),
-  seleniumAssistant.downloadLocalBrowser('chrome', 'unstable', 48),
-  seleniumAssistant.downloadLocalBrowser('firefox', 'stable', 48),
-  seleniumAssistant.downloadLocalBrowser('firefox', 'beta', 48),
-  seleniumAssistant.downloadLocalBrowser('firefox', 'unstable', 48)
-])
+// TODO: enable firefox testing once figure out how to give notification permission with SE webdriver.
+// TODO: Run the integration test against multiple major chrome versions to ensure backward compatibility
+Promise.all([seleniumAssistant.downloadLocalBrowser('chrome', 'stable', 80)])
   .then(() => {
     console.log('Browser download complete.');
   })

integration/messaging/package.json

@@ -4,15 +4,15 @@
   "version": "0.2.1",
   "scripts": {
     "pretest:manual": "node ./download-browsers.js",
-    "test": "echo 'Tests disabled due to flakiness'",
+    "test": "mocha --exit",
     "test:manual": "mocha --exit"
   },
   "dependencies": {
     "firebase": "7.15.1"
   },
   "devDependencies": {
     "chai": "4.2.0",
-    "chromedriver": "80.0.2",
+    "chromedriver": "^83.0.0",
     "express": "4.17.1",
     "geckodriver": "1.19.1",
     "mocha": "7.1.2",

integration/messaging/test/static/app.js

@@ -1,6 +1,6 @@
 /**
  * @license
- * Copyright 2017 Google Inc.
+ * Copyright 2017 Google LLC
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -15,8 +15,6 @@
  * limitations under the License.
  */
 
-const EIGHT_DAYS_IN_MS = 8 * 86400000;
-
 /**
  * This class just wraps the expected behavior or a demo app that will
  * be orchestrated by selenium tests (Although manual use of the demo will
@@ -36,40 +34,49 @@ class DemoApp {
     this._triggerDeleteToken = this.triggerDeleteToken;
     this._triggerGetToken = this.triggerGetToken;
     this._triggerTimeForward = this.triggerTimeForward;
+    this._clearInstanceForTest = this.clearInstanceForTest;
+    this.appendMessage = this.appendMessage;
 
     // Initialize Firebase
     firebase.initializeApp(firebaseConfig);
 
     this._messaging = firebase.messaging();
 
-    if (options.applicationKey) {
-      this._messaging.usePublicVapidKey(options.applicationKey);
+    if (options.vapidKey) {
+      console.debug('VapidKey is provided to the test app. ');
+      this._messaging.usePublicVapidKey(options.vapidKey);
+    } else {
+      console.debug('VapidKey is not specified. Skip setting it');
     }
 
     if (options.swReg) {
+      console.debug('ServiceWorker is provided to the test app');
       this._messaging.useServiceWorker(options.swReg);
+    } else {
+      console.debug(
+        'ServiceWorker is not specified. The default ServiceWorker will be used.'
+      );
     }
 
-    this._messaging.onMessage(payload => {
-      console.log(`Message received: `, payload);
-      this._messages.push(payload);
+    this._messaging.onMessage(message => {
+      this.appendMessage(message);
     });
 
-    // Initializa state of token
-    this._messaging
-      .requestPermission()
-      .then(() => this._messaging.getToken())
-      .then(
-        token => {
-          console.log('getToken() worked: ', token);
-          this._token = token;
-        },
-        err => {
-          console.log('getToken() failed: ', err.message, err.stack);
-          this._errors.push(err);
-          this._token = null;
-        }
-      );
+    this._messaging.getToken().then(
+      token => {
+        console.log('Test app getToken() succeed. Token: ', token);
+        this._token = token;
+      },
+      err => {
+        console.log('Test app getToken() failed: ', err.message, err.stack);
+        this._errors.push(err);
+        this._token = null;
+      }
+    );
+  }
+
+  appendMessage(payload) {
+    this._messages.push(payload);
   }
 
   async triggerDeleteToken(token) {
@@ -95,10 +102,15 @@ class DemoApp {
     return this._token;
   }
 
-  async triggerTimeForward() {
+  triggerTimeForward() {
     this._clock.tick(EIGHT_DAYS_IN_MS);
   }
 
+  clearInstanceForTest() {
+    this._errors = [];
+    this._messages = [];
+  }
+
   get token() {
     return this._token;
   }

integration/messaging/test/static/constants.js

@@ -0,0 +1,35 @@
+/**
+ * @license
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Test project managed by the FCM team
+const PUBLIC_VAPID_KEY =
+  'BNjjus3nz38aYtbDLVfunY3VULg0Yq5T4GXWd7iDDmeqWCUNqfrK1eiKVdoT0VncLuCjfJ1GmdfmNZz-AjHfkxM';
+const FIREBASE_CONFIG = {
+  apiKey: 'AIzaSyBpIe0xyUNHOwtE_go32NmUJF4acsc6S6c',
+  projectId: 'fcm-web-sdk-test',
+  messagingSenderId: '750970317741',
+  appId: '1:750970317741:web:f382be3155e250906a4f24'
+};
+
+const TAG = 'FCM_INTEGRATION_TEST: ';
+const EIGHT_DAYS_IN_MS = 8 * 86400000;
+const TEST_DB = 'FCM_INTEGRATION_TEST_DB';
+const TEST_DB_VERSION = 1;
+const BACKGROUND_MESSAGES_OBJECT_STORE = 'background_messages';
+// indexDb object store creation require a "primary key", "ndx" is used.
+const BACKGROUND_MESSAGES_OBJECT_STORE_PRIMARY_KEY = 'ndx';
+const BACKGROUND_MESSAGES_OBJECT_STORE_DEFAULT_NDX = 'default_ndx';

integration/messaging/test/static/default-sw/index.html

@@ -1,17 +1,19 @@
 <html>
-    <head>
-      <title>FCM Demo</title>
-      <meta name="viewport" content="width=device-width,initial-scale=1">
-    </head>
-    <body>
-      <h1>Default SW</h1>
-      <script src="/firebase/firebase-app.js"></script>
-      <script src="/firebase/firebase-messaging.js"></script>
-      <script src="/app.js"></script>
-      <script src="../valid-no-vapid-key/firebaseConfig.js"></script>
-      <script src="https://cdnjs.cloudflare.com/ajax/libs/sinon.js/4.1.3/sinon.min.js"></script>
-      <script>
-        window.__test = new window.DemoApp(window.firebaseConfig);
-      </script>
-    </body>
-    </html>
+  <head>
+    <title>FCM Demo</title>
+    <meta name="viewport" content="width=device-width,initial-scale=1" />
+  </head>
+  <body>
+    <h1>Default SW</h1>
+    <script src="/firebase-app.js"></script>
+    <script src="/firebase-messaging.js"></script>
+    <script src="../app.js"></script>
+    <script src="../constants.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/sinon.js/4.1.3/sinon.min.js"></script>
+    <script>
+      window.__test = new window.DemoApp(FIREBASE_CONFIG, {
+        vapidKey: PUBLIC_VAPID_KEY
+      });
+    </script>
+  </body>
+</html>

integration/messaging/test/static/firebase-messaging-sw.js

@@ -1,6 +1,6 @@
 /**
  * @license
- * Copyright 2017 Google Inc.
+ * Copyright 2017 Google LLC
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -15,13 +15,4 @@
  * limitations under the License.
  */
 
-importScripts('/sw-shared.js');
-importScripts('/valid-no-vapid-key/firebaseConfig.js');
-
-firebase.initializeApp(self.firebaseConfig);
-
-const messaging = firebase.messaging();
-messaging.setBackgroundMessageHandler(data => {
-  const title = 'Background Notification';
-  return self.registration.showNotification(title, {});
-});
+importScripts('./sw-base.js');

integration/messaging/test/static/invalid-manifest/index.html

@@ -0,0 +1,71 @@
+/**
+ * @license
+ * Copyright 2017 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+importScripts('../constants.js');
+
+// HEAD targets served through express
+importScripts('/firebase-app.js');
+importScripts('/firebase-messaging.js');
+
+firebase.initializeApp(FIREBASE_CONFIG);
+const messaging = firebase.messaging();
+
+messaging.setBackgroundMessageHandler(payload => {
+  console.log(
+    TAG +
+      'a background message is received: ' +
+      JSON.stringify(payload) +
+      '. Storing it into idb for tests to read...'
+  );
+
+  addPayloadToDb(payload);
+});
+
+async function addPayloadToDb(payload) {
+  const dbOpenReq = indexedDB.open(TEST_DB);
+
+  dbOpenReq.onupgradeneeded = () => {
+    const db = dbOpenReq.result;
+
+    // store creation is a synchronized call
+    console.log('creating object store...');
+    db.createObjectStore(BACKGROUND_MESSAGES_OBJECT_STORE, {
+      keyPath: BACKGROUND_MESSAGES_OBJECT_STORE_PRIMARY_KEY
+    });
+  };
+
+  dbOpenReq.onsuccess = () => {
+    const db = dbOpenReq.result;
+
+    addPayloadToDbInternal(db, {
+      ...payload,
+      // ndx is required as the primary key of the store. It doesn't have any other testing purpose
+      ndx: BACKGROUND_MESSAGES_OBJECT_STORE_DEFAULT_NDX
+    });
+  };
+}
+
+async function addPayloadToDbInternal(db, payload) {
+  // onsuccess might race with onupgradeneeded. Consequently causing " object stores was not found" error. Therefore, wait briefly for db.createObjectStore to complete
+  const delay = ms => new Promise(res => setTimeout(res, ms));
+  await delay(/* milliseconds= */ 30000);
+
+  tx = db.transaction(BACKGROUND_MESSAGES_OBJECT_STORE, 'readwrite');
+
+  console.log('adding message payload to db: ' + JSON.stringify(payload));
+  addReq = tx.objectStore(BACKGROUND_MESSAGES_OBJECT_STORE).add(payload);
+}

integration/messaging/test/static/invalid-manifest/manifest.json

@@ -1,18 +1,18 @@
 <html>
-    <head>
-      <title>FCM Demo</title>
-      <link rel="manifest" href="./manifest.json">
-      <meta name="viewport" content="width=device-width,initial-scale=1">
-    </head>
-    <body>
-      <h1>Valid Manifest</h1>
-      <script src="/firebase/firebase-app.js"></script>
-      <script src="/firebase/firebase-messaging.js"></script>
-      <script src="/app.js"></script>
-      <script src="../valid-no-vapid-key/firebaseConfig.js"></script>
-      <script src="https://cdnjs.cloudflare.com/ajax/libs/sinon.js/4.1.3/sinon.min.js"></script>
-      <script>
-        window.__test = new window.DemoApp(window.firebaseConfig);
-      </script>
-    </body>
-    </html>
+  <head>
+    <title>FCM Demo</title>
+    <link rel="manifest" href="./manifest.json" />
+    <meta name="viewport" content="width=device-width,initial-scale=1" />
+  </head>
+  <body>
+    <h1>Valid Manifest</h1>
+    <script src="/firebase-app.js"></script>
+    <script src="/firebase-messaging.js"></script>
+    <script src="../app.js"></script>
+    <script src="../constants.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/sinon.js/4.1.3/sinon.min.js"></script>
+    <script>
+      window.__test = new window.DemoApp(FIREBASE_CONFIG, {});
+    </script>
+  </body>
+</html>

integration/messaging/test/static/sw-base.js

@@ -1,3 +1,3 @@
 {
-  "gcm_sender_id": "103953800507"
+  "gcm_sender_id": "750970317741"
 }