Create Exercise11_RumbleDB_2024_Moodle_Solutions.ipynb

Author: LudoCatt

Big_Data/exercise11/Exercise11_RumbleDB_2024_Moodle_Solutions.ipynb

@@ -0,0 +1,194 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Big Data – Exercises - Solutions\n",
+    "\n",
+    "# Fall 2024 -  Week 11 - RumbleDB"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Moodle quiz (11.2): querying a bigger git-archive dataset\n",
+    "\n",
+    "You will have to submit the results of this exercise to Moodle to obtain the weekly bonus. You will need these things:\n",
+    "- Something related to the query output (we will grade this)\n",
+    "- The query you wrote (ungraded)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This exercise was designed to run on the exam magic box (and tested there too). It should work on all systems, but if you run into issues there you can look at the tutorial on how to run docker on [GitHub codespaces](https://github.com/RumbleDB/bigdata-exercises/blob/master/Big_Data/exercise05/GitHub_Codespaces.pdf), or the alternative instructions in [last year's exercises](https://github.com/RumbleDB/bigdata-exercises/tree/08ba6ba6222d96003ad7bd895a71ab6c32bcc872/Big_Data/exercise11).\n",
+    "\n",
+    "To get started, run the cell below to properly connect jupyter with rumbleDB (don't worry about the cell, we don't expect you to know what this does)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install rumbledb\n",
+    "%load_ext rumbledb\n",
+    "%env RUMBLEDB_SERVER=http://rumble:9090/jsoniq"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Check the data\n",
+    "We provide you with a bigger git-archive dataset [git-archive-big.json](https://www.rumbledb.org/samples/git-archive-big.json), you can already check that you get the correct number of records. The dataset should contain 206978 records. You can either use `wget` to download and read the dataset locally or simply read with `json-file` from the URI.\n",
+    "\n",
+    "We recommend running the cell below to download the data (reading it directly from the URL is slow and hard to debug using the notebook interface)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!wget https://www.rumbledb.org/samples/git-archive-big.json"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%jsoniq\n",
+    "count(for $i in json-file(\"git-archive-big.json\")\n",
+    "return $i)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# json-file(\"https://www.rumbledb.org/samples/git-archive-big.json\") # to read it from the URL"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%jsoniq\n",
+    "distinct-values(json-file(\"git-archive-big.json\").type)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Question 1: Give the login name of the two actors that committed to master the most in PushEvent events.\n",
+    "\n",
+    "Write the two names, separated by a comma with no space in between them.\n",
+    "\n",
+    "Hint: Note that all commits in a push event are stored in a list (you should count those as distinct commits)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%jsoniq\n",
+    "for $i in json-file(\"git-archive-big.json\")\n",
+    "where $i.type eq \"PushEvent\" and $i.payload.ref eq \"refs/heads/master\"\n",
+    "group by $ac := $i.actor.login\n",
+    "let $cnt := count($i.payload.commits[])\n",
+    "order by $cnt descending\n",
+    "count $c\n",
+    "where $c le 3\n",
+    "return [$cnt, $ac]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Question 2: For how many repos do we have both a creation and deletion event in the data?\n",
+    "\n",
+    "Write the number and nothing else."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%jsoniq\n",
+    "count(for $i in json-file(\"git-archive-big.json\")\n",
+    "where $i.type = \"CreateEvent\" or $i.type = \"DeleteEvent\"\n",
+    "group by $repo := $i.repo.id\n",
+    "where count(distinct-values($i.type)) eq 2\n",
+    "return $repo)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Question 3: Which repository has the highest number of people pushing to it?\n",
+    "\n",
+    "Give both the repository id and the number of people, separated by a comma with spaces in between.\n",
+    "\n",
+    "Hint: Differentiate users (_actors_) using their actor id."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%jsoniq\n",
+    "for $i in json-file(\"git-archive-big.json\")\n",
+    "where $i.type = \"PushEvent\"\n",
+    "group by $repo := $i.repo.id\n",
+    "let $people := count(distinct-values($i.actor.id))\n",
+    "order by $people descending\n",
+    "count $c\n",
+    "where $c le 10\n",
+    "return [$repo, $people]"
+   ]
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}