Tips/Secret Manager.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "84bc7b9a",
   "metadata": {},
   "source": [
    "![tracker](https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?path=statmike%2Fvertex-ai-mlops%2FTips&file=Secret+Manager.ipynb)\n",
    "<!--- header table --->\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://colab.research.google.com/github/statmike/vertex-ai-mlops/blob/main/Tips/Secret%20Manager.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\">\n",
    "      <br>Run in<br>Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fstatmike%2Fvertex-ai-mlops%2Fmain%2FTips%2FSecret%2520Manager.ipynb\">\n",
    "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\">\n",
    "      <br>Run in<br>Colab Enterprise\n",
    "    </a>\n",
    "  </td>      \n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://github.com/statmike/vertex-ai-mlops/blob/main/Tips/Secret%20Manager.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      <br>View on<br>GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/statmike/vertex-ai-mlops/main/Tips/Secret%20Manager.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      <br>Open in<br>Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee29e18e-a926-4ed1-8d97-7b7c0f6a288e",
   "metadata": {},
   "source": [
    "# Secret Manager\n",
    "\n",
    "Sometimes your code needs to send or retrieve data to another source that requires credentials.  Don't hardcode these!\n",
    "\n",
    "With [Secret Manager](https://cloud.google.com/secret-manager) you can safely store these credentials with encryption (always).  Then, retrieve values at the point where they are needed without exposing them in your code or passing them around as arguments to code.\n",
    "\n",
    "Secret Manager has [client librariers](https://cloud.google.com/secret-manager/docs/reference/libraries) in multiple languages, including the `gcloud` CLI.\n",
    "\n",
    "Since this repository primarily uses Python, the examples below will make use of the [Python Client for Secret Manager API](https://cloud.google.com/python/docs/reference/secretmanager/latest).\n",
    "\n",
    "Installation:\n",
    "```\n",
    "pip install google-cloud-secret-manager\n",
    "```\n",
    "\n",
    "Importing:\n",
    "```\n",
    "from google.cloud import secretmanager\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "588f3449-5edd-4fd0-9d0e-474941cf34a0",
   "metadata": {},
   "source": [
    "---\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4070e3d3-dd1b-432e-bed7-8b4a91f405f1",
   "metadata": {},
   "source": [
    "### Package Installs (if needed)\n",
    "\n",
    "This notebook uses the Python Clients for\n",
    "- Google Service Usage\n",
    "    - to enable APIs (BigQuery Connection)\n",
    "\n",
    "The cells below check to see if the required Python libraries are installed.  If any are not it will print a message to do the install with the associated pip command to use.  These installs must be completed before continuing this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "309bce75-8ea3-4365-aab6-2a15850795ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import google.cloud.service_usage_v1\n",
    "except ImportError:\n",
    "    print('You need to pip install google-cloud-service-usage')\n",
    "    !pip install google-cloud-service-usage -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "008da4a8-e9c0-47a7-bf88-7f09fd028b1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import google.cloud.secretmanager\n",
    "except ImportError:\n",
    "    print('You need to pip install google-cloud-secret-manager')\n",
    "    !pip install google-cloud-secret-manager -q"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41233d8e-31b7-4f3a-8812-dafefe390504",
   "metadata": {},
   "source": [
    "### Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f7c6ac2-c59a-4908-a30d-c0fb30f799dd",
   "metadata": {},
   "source": [
    "inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "136560cb-f0c7-40a9-aa40-fab8431b3a69",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'statmike-mlops-349915'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project = !gcloud config get-value project\n",
    "PROJECT_ID = project[0]\n",
    "PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7796ea73-54fd-4bcb-bf7d-6f4b4d2b8058",
   "metadata": {},
   "source": [
    "packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f03103e9-58f1-4716-bf60-df3ea445a12c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.cloud import service_usage_v1\n",
    "from google.cloud import secretmanager"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e168a62d-3458-4650-9998-931c882af5f8",
   "metadata": {},
   "source": [
    "clients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ca00251f-5fb2-4442-8f20-72061e9c93ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "secret_client = secretmanager.SecretManagerServiceClient()\n",
    "su_client = service_usage_v1.ServiceUsageClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b7ab790-7963-4a21-ad55-11d0f5d7ff39",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Enable APIs\n",
    "\n",
    "Secret Manager uses the Secret Manager API which needs to be enabled. \n",
    "\n",
    "Options for enabeling:  In this notebook option 2 is used.\n",
    " 1. Use the APIs & Services page in the console: https://console.cloud.google.com/apis\n",
    "     - `+ Enable APIs and Services`\n",
    "     - Search for Secret Manager\n",
    "     - Or, [direct link](http://console.cloud.google.com/apis/library/secretmanager.googleapis.com)\n",
    " 2. Use [Google Service Usage](https://cloud.google.com/service-usage/docs) API from Python\n",
    "     - [Python Client For Service Usage](https://github.com/googleapis/python-service-usage)\n",
    "     - [Python Client Library Documentation](https://cloud.google.com/python/docs/reference/serviceusage/latest)\n",
    "     \n",
    "The following code cells use the Service Usage Client to:\n",
    "- get the state of the service\n",
    "- if 'DISABLED':\n",
    "    - Try enabling the service and return the state after trying\n",
    "- if 'ENABLED' print the state for confirmation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c16382d-6a25-4334-915e-88161a188139",
   "metadata": {},
   "source": [
    "#### Secret Manager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "82a69371-3b5e-4bff-8393-6b9375545bf4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "API already enabled for project: statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "secretmanager = su_client.get_service(\n",
    "    request = service_usage_v1.GetServiceRequest(\n",
    "        name = f'projects/{PROJECT_ID}/services/secretmanager.googleapis.com'\n",
    "    )\n",
    ").state.name\n",
    "\n",
    "\n",
    "if secretmanager == 'DISABLED':\n",
    "    print(f'API is currently {secretmanager} for project: {PROJECT_ID}')\n",
    "    print(f'Trying to Enable...')\n",
    "    operation = su_client.enable_service(\n",
    "        request = service_usage_v1.EnableServiceRequest(\n",
    "            name = f'projects/{PROJECT_ID}/services/secretmanager.googleapis.com'\n",
    "        )\n",
    "    )\n",
    "    response = operation.result()\n",
    "    if response.service.state.name == 'ENABLED':\n",
    "        print(f'API is now enabled for project: {PROJECT_ID}')\n",
    "    else:\n",
    "        print(response)\n",
    "else:\n",
    "    print(f'API already enabled for project: {PROJECT_ID}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a186c7f7-9646-46d9-99f4-e290e601cb1c",
   "metadata": {},
   "source": [
    "---\n",
    "## Using Secret Manager\n",
    "\n",
    "- List Secrets\n",
    "- Create Secret\n",
    "- Retrieve Secret\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67eeaf12-d708-47f0-b204-793cb460c9ff",
   "metadata": {},
   "source": [
    "### List Secrets\n",
    "\n",
    "The code below will list all the secrets in the current project. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "ed3ffc24-9636-4618-bef7-d29cc2441b24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/1026793852137/secrets/github_api\n",
      "projects/1026793852137/secrets/openAI\n"
     ]
    }
   ],
   "source": [
    "for s in secret_client.list_secrets(parent = f'projects/{PROJECT_ID}'):\n",
    "    print(s.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bba6835f-c13d-479a-b19a-f5295bd530cc",
   "metadata": {},
   "source": [
    "You can also go to the console page for this project to review the secrets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "2dbfe541-2df8-4143-be49-37b9139f5c0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review a list of secrets using the console:\n",
      "https://console.cloud.google.com/security/secret-manager?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review a list of secrets using the console:\\nhttps://console.cloud.google.com/security/secret-manager?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79c6c2cd-ec26-4255-b9b3-b47a327c9996",
   "metadata": {},
   "source": [
    "### Create Secret\n",
    "\n",
    "The code below will create a new secret in Secret Manager.  **This can also be done directly in the console.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "2c512034-982b-4af8-a8d6-e4b047be92ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "secret = secret_client.create_secret(\n",
    "    parent = f\"projects/{PROJECT_ID}\",\n",
    "    secret_id = 'fake_secret',\n",
    "    secret = {'replication': {'automatic': {}}}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "b02322e2-aa40-43a5-9370-f4acb90de6ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/secrets/fake_secret'"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "secret.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "8c954ace-007e-4763-972a-068fc81bd886",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/1026793852137/secrets/fake_secret\n",
      "projects/1026793852137/secrets/github_api\n",
      "projects/1026793852137/secrets/openAI\n"
     ]
    }
   ],
   "source": [
    "for s in secret_client.list_secrets(parent = f'projects/{PROJECT_ID}'):\n",
    "    print(s.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "176a77ac-8cd8-4bed-867e-1acce150e272",
   "metadata": {},
   "source": [
    "### Create Secret Version (a new value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "1a2e248f-d232-43f3-af75-694b717c7887",
   "metadata": {},
   "outputs": [],
   "source": [
    "version = secret_client.add_secret_version(\n",
    "    parent = secret.name,\n",
    "    payload = {'data': b'Not a real secret!'}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "808ea804-f9de-479d-8311-42d6808bd92e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/secrets/fake_secret/versions/1'"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "version.name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "419f5e7a-a5e2-44e3-b4e5-e7c95f00bea7",
   "metadata": {},
   "source": [
    "### List Secret Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "13fcc416-3321-457f-a442-452a581a002f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/1026793852137/secrets/fake_secret/versions/1\n"
     ]
    }
   ],
   "source": [
    "for v in secret_client.list_secret_versions(parent = secret.name):\n",
    "    print(v.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f46cb32-b45f-41ad-8b3d-19b62e0348ef",
   "metadata": {},
   "source": [
    "### Retrieve Secret Version (value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "dc3484d9-76fa-427d-94b0-cb6343ce42fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"projects/1026793852137/secrets/fake_secret/versions/1\"\n",
       "payload {\n",
       "  data: \"Not a real secret!\"\n",
       "  data_crc32c: 3349683616\n",
       "}"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve = secret_client.access_secret_version(name = version.name)\n",
    "retrieve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "f204ef45-fe4e-4613-9f92-cd00a8a92e31",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'Not a real secret!'"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve.payload.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "30f0a259-8633-410b-bd05-2c1bf0c920e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Not a real secret!'"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve.payload.data.decode('UTF-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "498995b4-f3cb-4344-87f1-83535c2f37d3",
   "metadata": {},
   "source": [
    "---\n",
    "## Using In Your Code\n",
    "\n",
    "This notebook went through the one time setup process.  Now, let's look at what is need at a minimun to use secret manager in our ML code!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "8ed669d6-cb13-47df-8580-cd01bb3e4630",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.cloud import secretmanager\n",
    "secret_client = secretmanager.SecretManagerServiceClient()\n",
    "\n",
    "secret_value = secret_client.access_secret_version(name = f'projects/{PROJECT_ID}/secrets/fake_secret/versions/latest').payload.data.decode('utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "2bbe207f-24e2-4c57-89da-2eddb33c14a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not a real secret!\n"
     ]
    }
   ],
   "source": [
    "print(secret_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f60dc5b9-7375-4a43-8a9e-64918db82fb8",
   "metadata": {},
   "source": [
    "---\n",
    "## Deleting a Secret\n",
    "\n",
    "This notebook created a fake secret for demonstration.  This example shows how to delete a secret version, and the secret."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "1310027e-7b65-4da6-961e-a2bfb7b28af6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/1026793852137/secrets/fake_secret\n",
      "projects/1026793852137/secrets/github_api\n",
      "projects/1026793852137/secrets/openAI\n"
     ]
    }
   ],
   "source": [
    "# list secrets\n",
    "for s in secret_client.list_secrets(parent = f'projects/{PROJECT_ID}'):\n",
    "    print(s.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "5e817c7f-d915-4ed1-95cd-726b7312a108",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/1026793852137/secrets/fake_secret/versions/1\n"
     ]
    }
   ],
   "source": [
    "# list version for current secret\n",
    "for v in secret_client.list_secret_versions(parent = secret.name):\n",
    "    print(v.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "69105552-56a1-4d0f-ac09-32b4d810db25",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"projects/1026793852137/secrets/fake_secret/versions/1\"\n",
       "create_time {\n",
       "  seconds: 1677340260\n",
       "  nanos: 51590000\n",
       "}\n",
       "destroy_time {\n",
       "  seconds: 1677340270\n",
       "  nanos: 984645217\n",
       "}\n",
       "state: DESTROYED\n",
       "replication_status {\n",
       "  automatic {\n",
       "  }\n",
       "}\n",
       "etag: \"\\\"15f5883705065f\\\"\""
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# delete version for current secret\n",
    "result = secret_client.destroy_secret_version(name = version.name)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "e7ea44de-e58b-4c85-a42f-4721b91b8022",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<State.DESTROYED: 3>"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "b6eded92-60ff-41e4-8c63-5e3be0ffe3dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete secret\n",
    "secret_client.delete_secret(name = secret.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "f73c86a5-a511-47fd-9129-8a22083ed6dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/1026793852137/secrets/github_api\n",
      "projects/1026793852137/secrets/openAI\n"
     ]
    }
   ],
   "source": [
    "# list secrets\n",
    "for s in secret_client.list_secrets(parent = f'projects/{PROJECT_ID}'):\n",
    "    print(s.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1ce135f-52e0-4c08-96d9-d0d5628fa018",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-3.m94",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m94"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}