OpenDirectoryDownloader.ipynb

{
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "OpenDirectoryDownloader.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0,
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NTJ-Y9RF-KFs"
      },
      "source": [
        "# OpenDirectoryDownloader\n",
        "\n",
        "<https://github.com/KoalaBear84/OpenDirectoryDownloader>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ca6j0GrJiwpa"
      },
      "source": [
        "#@markdown <h2>Setup and install dependencies</h2>\n",
        "#@markdown Run this only once!<br>\n",
        "#@markdown Automatically uses the latest version\n",
        "%cd /content/\n",
        "!wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb\n",
        "!sudo dpkg -i packages-microsoft-prod.deb\n",
        "!sudo apt-get update;\n",
        "!sudo apt-get install -y apt-transport-https\n",
        "!sudo apt-get install -y jq\n",
        "LatestVersionUrl=!curl https://api.github.com/repos/KoalaBear84/OpenDirectoryDownloader/releases/latest -s | jq -r '.assets[] | select(.name | endswith(\"linux-x64-self-contained.zip\")) | .browser_download_url | tostring'\n",
        "LatestVersionUrl=LatestVersionUrl[0]\n",
        "!wget $LatestVersionUrl -O OpenDirectoryDownloader.zip\n",
        "!unzip -o /content/OpenDirectoryDownloader.zip -d /content/Indexer/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Jl35t4l-KFv"
      },
      "source": [
        "## Command line parameters:\n",
        "\n",
        "Always up-to-date reference: <https://github.com/KoalaBear84/OpenDirectoryDownloader#usage>\n",
        "\n",
        "| Short | Long                 | Description                                                                                                                                                                                                                   |\n",
        "| ----- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n",
        "| `-u`  | `--url`              | Url to scan                                                                                                                                                                                                                   |\n",
        "| `-t`  | `--threads`          | Number of threads (default 5)                                                                                                                                                                                                 |\n",
        "| `-o`  | `--timeout`          | Number of seconds for timeout                                                                                                                                                                                                 |\n",
        "| `-w`  | `--wait`             | Number of seconds to wait between calls (when single threaded is too fast..)                                                                                                                                                  |\n",
        "| `-q`  | `--quit`             | Quit after scanning (No \"Press a key\")                                                                                                                                                                                        |\n",
        "| `-c`  | `--clipboard`        | Automatically copy the Reddits stats once the scan is done                                                                                                                                                                    |\n",
        "| `-j`  | `--json`             | Save JSON file                                                                                                                                                                                                                |\n",
        "| `-f`  | `--no-urls`          | Do not save URLs file                                                                                                                                                                                                         |\n",
        "| `-r`  | `--no-reddit`        | Do not show Reddit stats markdown                                                                                                                                                                                             |\n",
        "| `-l`  | `--upload-urls`      | Uploads urls file                                                                                                                                                                                                             |\n",
        "| `-e`  | `--exact-file-sizes` | Exact file sizes (WARNING: Uses HEAD requests which takes more time and is heavier for server)                                                                                                                                |\n",
        "|       | `--fast-scan`        | Only use sizes from HTML, no HEAD requests, even if the approx. size cannot be extracted from the HTML                                                                                                                         |\n",
        "| `-s`  | `--speedtest`        | Does a speed test after indexing                                                                                                                                                                                              |\n",
        "| `-a`  | `--user-agent`       | Use custom default User Agent                                                                                                                                                                                                 |\n",
        "|       | `--username`         | Username                                                                                                                                                                                                                      |\n",
        "|       | `--password`         | Password                                                                                                                                                                                                                      |\n",
        "|       | `--github-token`     | GitHub Token                                                                                                                                                                                                                  |\n",
        "| `-H`  | `--header`           | Supply a custom header to use for each HTTP request. Can be used multiple times for multiple headers. See below for more info.                                                                                                |\n",
        "|       | `--output-file`      | Output file to use for urls file                                                                                                                                                                                              |\n",
        "|       | `--proxy-address`    | Proxy address, like \"socks5://127.0.0.1:9050\" (needed for .onion)                                                                                                                                                             |\n",
        "|       | `--proxy-username`   | Proxy username                                                                                                                                                                                                                |\n",
        "|       | `--proxy-password`   | Proxy password                                                                                                                                                                                                                |\n",
        "|       | `--no-browser`       | Disallow starting Chromium browser (for Cloudflare)                                                                                                                                                                           |"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uUNjNySGnB90"
      },
      "source": [
        "#@markdown <h2>Index Open Directory</h2>\n",
        "#@markdown After scanning the urls file can be found in Scans folder (see Folder icon on the left sidebar)\n",
        "\n",
        "Url = \"\" #@param {type:\"string\"}\n",
        "Threads = 5 #@param {type:\"slider\", min:1, max:50, step:1}\n",
        "Timeout = 100 #@param {type:\"slider\", min:1, max:300, step:1}\n",
        "Wait = 0 #@param {type:\"slider\", min:0, max:300, step:1}\n",
        "ExactFileSizes = False #@param {type:\"boolean\"}\n",
        "FastScan = False #@param {type:\"boolean\"}\n",
        "UploadUrlsFile = False #@param {type:\"boolean\"}\n",
        "Speedtest = True #@param {type:\"boolean\"}\n",
        "UserAgent = \"\" #@param {type:\"string\"}\n",
        "Username = \"\" #@param {type:\"string\"}\n",
        "Password = \"\" #@param {type:\"string\"}\n",
        "GithubToken = \"\" #@param {type:\"string\"}\n",
        "ProxyAddress = \"\" #@param {type:\"string\"}\n",
        "ProxyUsername = \"\" #@param {type:\"string\"}\n",
        "ProxyPassword = \"\" #@param {type:\"string\"}\n",
        "AdditionalParameters = \"\" #@param {type:\"string\"}\n",
        "#@markdown (Any additional parameters, like headers. You'll need to add the actual flags, e.g. `--header 'Referer: https://example.com'`)\n",
        "\n",
        "StringParameters = {\n",
        "    \"UserAgent\": '--user-agent',\n",
        "    \"Username\": '--username',\n",
        "    \"Password\": '--password',\n",
        "    \"GithubToken\": '--github-token',\n",
        "    \"ProxyAddress\": '--proxy-address',\n",
        "    \"ProxyUsername\": '--proxy-username',\n",
        "    \"ProxyPassword\": '--proxy-password',\n",
        "}\n",
        "NumberParameters = {\n",
        "    \"Threads\": '--threads',\n",
        "    \"Timeout\": '--timeout',\n",
        "    \"Wait\": '--wait',\n",
        "}\n",
        "BooleanParameters = {\n",
        "    \"ExactFileSizes\": '--exact-file-sizes',\n",
        "    \"FastScan\": '--fast-scan',\n",
        "    \"UploadUrlsFile\": '--upload-urls',\n",
        "    \"Speedtest\": '--speedtest',\n",
        "}\n",
        "\n",
        "Parameters = []\n",
        "\n",
        "for key, value in StringParameters.items():\n",
        "    if eval(key) != \"\":\n",
        "        Parameters.append(value + ' \"' + eval(key) + '\"')\n",
        "\n",
        "for key, value in NumberParameters.items():\n",
        "    if eval(key) != \"\":\n",
        "        Parameters.append(value + ' ' + str(eval(key)))\n",
        "\n",
        "for key, value in BooleanParameters.items():\n",
        "    if eval(key):\n",
        "        Parameters.append(value)\n",
        "\n",
        "parameterString = \" \".join(Parameters)\n",
        "\n",
        "!/content/Indexer/OpenDirectoryDownloader --json --no-browser --quit $parameterString $AdditionalParameters --url $Url"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "xF58wCj2_bo0"
      }
    }
  ]
}