Skip to content

Commit 02d6960

Browse files
committed
Initial import
0 parents  commit 02d6960

File tree

3 files changed

+265
-0
lines changed

3 files changed

+265
-0
lines changed

LICENSE

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
BSD 2-Clause License
2+
3+
Copyright (c) 2021, Matt Turner
4+
All rights reserved.
5+
6+
Redistribution and use in source and binary forms, with or without
7+
modification, are permitted provided that the following conditions are met:
8+
9+
1. Redistributions of source code must retain the above copyright notice, this
10+
list of conditions and the following disclaimer.
11+
12+
2. Redistributions in binary form must reproduce the above copyright notice,
13+
this list of conditions and the following disclaimer in the documentation
14+
and/or other materials provided with the distribution.
15+
16+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
17+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
19+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
20+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
21+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
22+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
23+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
24+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
25+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

+89
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# gmail-gitlab-filtering
2+
`gmail-gitlab-filtering.gs` is a [Google Apps Script](https://developers.google.com/apps-script) for Gmail to sort and filter email from [GitLab](https://about.gitlab.com/).
3+
4+
[![BSD-2 license](https://img.shields.io/badge/License-BSD2-blue.svg)](LICENSE)
5+
6+
# The Problem
7+
Receiving all messages from a mailing list and filtering by whether I am a direct recipient (in `To:` or `Cc:`) has been an effective strategy for me to both
8+
9+
1. follow development at large
10+
2. not miss requests for my input
11+
12+
With mailing lists, I accomplish this with a combination of Gmail filters:
13+
14+
````
15+
Matches: to:([email protected])
16+
Do this: Apply label "mattst88"
17+
````
18+
````
19+
Matches: list:mesa-dev.lists.freedesktop.org
20+
Do this: Apply label "mesa-dev"
21+
````
22+
````
23+
Matches: -{to:[email protected]}
24+
Do this: Skip Inbox
25+
````
26+
27+
Thus, mail from mailing lists is labeled appropriately and doesn't clutter my inbox. Any messages with me in `To:` or `Cc:` cause the thread to be labeled with a personal `mattst88` label and appear in the inbox.
28+
29+
I have not been able to replicate this with GitLab and Gmail, given the limitations of each.
30+
31+
Emails from GitLab contain [many headers](https://docs.gitlab.com/ee/user/profile/notifications.html#email-headers-you-can-use-to-filter-email) that can be used to filter the message. Of the headers GitLab uses, Gmail's filtering system can only filter on `List-Id`.
32+
33+
GitLab can be configured to email updates under different circumstances: any activity, only for threads you've participated in, only for comments that mention you, etc.
34+
35+
This leaves me with a choice of receiving notifications only for threads I'm involved in or for all threads but without the ability to easily find requests directed to me.
36+
37+
# The Solution
38+
Google Apps Script provides a method of automating many operations on a Gmail account (sending email, searching, labeling, etc.) via the [GmailApp](https://developers.google.com/apps-script/reference/gmail) class. The `gmail-gitlab-filtering.gs` script is run every 10 minutes via a trigger on scripts.google.com and performs filtering and labeling based on the `X-GitLab` headers.
39+
40+
This allows me to appropriately label notifications that are directed to me, as well as dynamically create labels for notifications received from new projects.
41+
42+
## How
43+
Email from GitLab is labeled using Gmail's default filtering with a top-level label and skips the inbox. In my case, all mail from `[email protected]` is given the `freedesktop` label:
44+
45+
````
46+
Matches: from:([email protected])
47+
Do this: Skip Inbox, Apply label "freedesktop"
48+
````
49+
50+
Threads with this label are considered unprocessed.
51+
52+
The `gmail-gitlab-filtering.gs` script searches for threads with that label, and inspects the headers of each message using the [GmailMessage.getHeader()](https://developers.google.com/apps-script/reference/gmail/gmail-message#getHeader(String)) function.
53+
54+
The script records the value of all `X-GitLab-Project-Path` headers and whether any message in the thread contained a `X-GitLab-NotificationReason` header.
55+
56+
Threads containing a `X-GitLab-NotificationReason` header retain the personal label and are moved to the inbox. Threads without `X-GitLab-NotificationReason` header remain archived and the personal label is removed. All threads are labeled with `${unprocessedLabel}/${X-GitLab-Project-Path}`, and the label is dynamically created if needed. The unprocessed label is always removed as the final step, in case the script's runtime exceeds the allowed timelimit (see below).
57+
58+
For example, if I were to receive a notification of a merge request in the [mesa/shader-db](https://gitlab.freedesktop.org/mesa/shader-db/) project that mentioned me, the script would move the thread to the inbox, leave the `mattst88` label intact, label the mail `freedesktop/mesa/shader-db` (creating the label if needed), and remove the `freedesktop` label.
59+
60+
## Implementation notes and limitations
61+
None of the limitations cause problems for my usage. I receive a few hundred GitLab notifications per day.
62+
63+
### Google Apps Scripts are [subject to quotas](https://developers.google.com/apps-script/guides/services/quotas).
64+
While developing this script, I hit two quotas:
65+
66+
- Script runtime: 6 min / execution
67+
- Email read/write (excluding send): 20,000 / day
68+
69+
To fit into the 6 minutes / execution limit, the script operates on a maximum of 240 threads per execution. See the `maxThreads` variable. I arrived at the 240 number empirically; it's not definitive.
70+
71+
I reached the 20,000 / day email read/write quota during development and expect to never reach it again now that the script functions.
72+
73+
To improve efficiency (and perhaps avoid hitting quotas), the script creates lists of [GmailThread](https://developers.google.com/apps-script/reference/gmail/gmail-thread)s so they can be passed in batches to [GmailApp.moveThreadsToInbox](https://developers.google.com/apps-script/reference/gmail/gmail-app#moveThreadsToInbox(GmailThread)), [GmailLabel.addToThreads](addToThreads), and [GmailLabel.removeFromThreads](https://developers.google.com/apps-script/reference/gmail/gmail-label#removeFromThreads(GmailThread)) †. It's unclear to me which functions use quota and how much they use.
74+
75+
† These functions accept only 100 threads per call, so the script calls them multiple times on `.slice()`s of the lists.
76+
77+
### Google Apps Scripts don't support async/await
78+
I initially though I would be able to improve performance of the script by using [async/await](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous/Async_await). While the V8 Runtime [recognizes the keywords](https://developers.google.com/apps-script/guides/v8-runtime?hl=en#improved_function_detection) it does not make use of them. An upstream bug is filed [here](https://issuetracker.google.com/issues/149937257).
79+
80+
I found an interesting work-around: [Async.gs](https://gist.github.com/sdesalas/2972f8647897d5481fd8e01f03122805). It works by scheduling a script execution in the future. I did not experiment with it.
81+
82+
# Setup
83+
1. Create a project on [script.google.com](https://script.google.com/)
84+
2. Paste `gmail-gitlab-filtering.gs` into the default `Code.gs` file
85+
3. Modify as needed for your particular filtering rules
86+
4. Set up a trigger to run the script periodically. I followed the instructions from [Marcin Jasion](https://mjasion.pl/)'s [How to label GitLab notification in Gmail by headers?](https://mjasion.pl/label-gitlab-notifications/) article.
87+
88+
# I'm not a JavaScript developer
89+
I muddled through writing the script with the help of the [Apps Script documentation](https://developers.google.com/apps-script/overview) and the Mozilla Web Docs [JavaScript Reference](https://developer.mozilla.org/en-US/docs/Web/JavaScript). Improvements to the code are welcome.

gmail-gitlab-filtering.gs

+151
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
/* SPDX-License-Identifier: BSD-2-Clause */
2+
3+
const personalLabelName = "mattst88"
4+
const unprocessedLabels = ["freedesktop"];
5+
const maxThreads = 240;
6+
7+
let personalLabel;
8+
function processInbox() {
9+
updateUserLabels();
10+
personalLabel = GmailApp.getUserLabelByName(personalLabelName);
11+
12+
for (const label of unprocessedLabels) {
13+
processLabel(userLabels[label]);
14+
}
15+
}
16+
17+
let toInboxThreads = [];
18+
let toRemoveThreads = new Map();
19+
let toAddThreads = new Map();
20+
21+
function processLabel(unprocessedLabel) {
22+
let threads = GmailApp.search("label:" + unprocessedLabel.getName(), 0, maxThreads);
23+
if (threads.length < 1) {
24+
Logger.log("No threads to process with label:" + unprocessedLabel.getName());
25+
return;
26+
} else {
27+
Logger.log("Processing threads with label:" + unprocessedLabel.getName());
28+
}
29+
30+
for (const i in threads) {
31+
Logger.log("Processing thread " + i);
32+
33+
processThread(unprocessedLabel, threads[i]);
34+
}
35+
36+
/* the GmailLabel.addToThreads/removeFromThreads functions
37+
* only process 100 threads at a time */
38+
const chunk_size = 100;
39+
40+
/* Apply labels to threads */
41+
for (const [label, threads] of toAddThreads) {
42+
Logger.log("Adding " + label.getName() + " label to " + threads.length + " threads");
43+
44+
for (let i = 0; i < threads.length; i += chunk_size) {
45+
const end = Math.min(i+chunk_size, threads.length);
46+
label.addToThreads(threads.slice(i, end));
47+
Logger.log("\t... " + end + " done");
48+
}
49+
}
50+
51+
/* Move threads to Inbox */
52+
Logger.log("Moving " + toInboxThreads.length + " to Inbox");
53+
for (let i = 0; i < toInboxThreads.length; i += chunk_size) {
54+
const end = Math.min(i+chunk_size, toInboxThreads.length);
55+
GmailApp.moveThreadsToInbox(toInboxThreads.slice(i, end));
56+
Logger.log("\t... " + end + " done")
57+
}
58+
59+
/* Remove labels from threads */
60+
for (const label of [personalLabel, unprocessedLabel]) {
61+
const threads = toRemoveThreads.get(label);
62+
Logger.log("Removing " + label.getName() + " label from " + threads.length + " threads");
63+
64+
for (let i = 0; i < threads.length; i += chunk_size) {
65+
const end = Math.min(i+chunk_size, threads.length);
66+
label.removeFromThreads(threads.slice(i, end));
67+
Logger.log("\t... " + end + " done");
68+
}
69+
}
70+
}
71+
72+
function processThread(unprocessedLabel, thread) {
73+
let moveToInbox = false;
74+
let labelNames = [];
75+
76+
let messages = thread.getMessages();
77+
for (const message of messages) {
78+
/* If the X-GitLab-NotificationReason header exists in any message
79+
* in the thread, it was sent to us because we were mentioned, we participated, etc.
80+
* We want to move those threads to the Inbox. */
81+
let notificationReason = message.getHeader("X-GitLab-NotificationReason")
82+
if (notificationReason) {
83+
moveToInbox = true;
84+
}
85+
86+
/* Push the project path to a list. We'll deduplicate later. */
87+
const projectPath = message.getHeader("X-GitLab-Project-Path");
88+
if (projectPath) {
89+
labelNames.push(projectPath);
90+
}
91+
}
92+
93+
/* Deduplicate labels list */
94+
labelNames = labelNames.filter(onlyUnique);
95+
96+
for (const labelName of labelNames) {
97+
/* Get/create a label nested under our unprocessed label */
98+
const label = getLabel(unprocessedLabel.getName() + "/" + labelName);
99+
100+
if (!toAddThreads.has(label)) {
101+
toAddThreads.set(label, []);
102+
}
103+
toAddThreads.get(label).push(thread);
104+
}
105+
106+
if (moveToInbox) {
107+
toInboxThreads.push(thread);
108+
} else {
109+
if (!toRemoveThreads.has(personalLabel)) {
110+
toRemoveThreads.set(personalLabel, []);
111+
}
112+
toRemoveThreads.get(personalLabel).push(thread);
113+
}
114+
115+
if (!toRemoveThreads.has(unprocessedLabel)) {
116+
toRemoveThreads.set(unprocessedLabel, []);
117+
}
118+
toRemoveThreads.get(unprocessedLabel).push(thread);
119+
}
120+
121+
/* Makes a hash table of "name" -> label */
122+
function makeNameToLabelTbl(labels) {
123+
let table = [];
124+
for (const label of labels) {
125+
table[label.getName()] = label;
126+
}
127+
return table;
128+
}
129+
130+
/* Cache of user labels, indexed by name string */
131+
let userLabels = [];
132+
function updateUserLabels() {
133+
userLabels = makeNameToLabelTbl(GmailApp.getUserLabels());
134+
}
135+
136+
/* Returns a GmailLabel given a name string.
137+
* If it doesn't exist, it creates it. */
138+
function getLabel(name) {
139+
let label;
140+
if (!userLabels[name]) {
141+
label = GmailApp.createLabel(name);
142+
updateUserLabels();
143+
} else {
144+
label = userLabels[name];
145+
}
146+
return label;
147+
}
148+
149+
function onlyUnique(value, index, self) {
150+
return self.indexOf(value) === index;
151+
}

0 commit comments

Comments
 (0)