Skip to content

Commit 2484581

Browse files
authored
Update S3 API key steps and expand limitations section (#293)
1 parent 3930b6c commit 2484581

File tree

1 file changed

+27
-19
lines changed

1 file changed

+27
-19
lines changed

serverless/storage/s3-api.mdx

Lines changed: 27 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -44,11 +44,11 @@ Create a network volume in one of the following datacenters to use the S3-compat
4444
1. In the Runpod console, navigate to the [Settings page](https://www.runpod.io/console/user/settings).
4545
2. Expand the **S3 API Keys** section and select **Create an S3 API key**.
4646
3. Give your key a name and select **Create**.
47-
4. Make a note of your access key ID and secret access key to use in the next step.
47+
4. Make a note of the **access key** (e.g., `user_***...`) and **secret** (e.g., `rps_***...`) to use in the next step.
4848

4949
<Warning>
5050

51-
For security, Runpod will show your secret access key only once, so you may wish to save it elsewhere (e.g., in your password manager, or in a GitHub secret). Treat your secret access key like a password and don't share it with anyone.
51+
For security, Runpod will show your API key secret only once, so you may wish to save it elsewhere (e.g., in your password manager, or in a GitHub secret). Treat your API key secret like a password and don't share it with anyone.
5252

5353
</Warning>
5454

@@ -61,8 +61,8 @@ Create a network volume in one of the following datacenters to use the S3-compat
6161
1. If you haven't already, [install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) on your local machine.
6262
2. Run the command `aws configure` in your terminal.
6363
3. Provide the following when prompted:
64-
* **AWS Access Key ID**: Enter your Runpod user ID. You can find this in the [Secrets section](https://www.runpod.io/console/user/secrets) of the Runpod console, in the description of your S3 API key. By default, the description will look similar to: `Shared Secret for user_2f21CfO73Mm2Uq2lEGFiEF24IPw 1749176107073`. `user_2f21CfO73Mm2Uq2lEGFiEF24IPw` is the user ID (yours will be different).
65-
* **AWS Secret Access Key**: Enter your Runpod S3 API key's secret access key.
64+
* **AWS Access Key ID**: Enter your **access key** (e.g., `user_***...`) from the previous step.
65+
* **AWS Secret Access Key**: Enter your **secret** (e.g., `rps_***...`) from the previous step.
6666
* **Default Region name**: You can leave this blank.
6767
* **Default output format**: You can leave this blank or set it to `json`.
6868

@@ -164,10 +164,10 @@ You can also use the Boto3 library to interact with the S3-compatible API, using
164164

165165
The script below demonstrates how to upload a file to a Runpod network volume using the Boto3 library. It takes command-line arguments for the network volume ID (as an S3 bucket), the datacenter-specific S3 endpoint URL, the local file path, the desired object (file path on the network volume), and the AWS Region (which corresponds to the Runpod datacenter ID).
166166

167-
Your Runpod S3 API key credentials must be set as environment variables:
167+
Your Runpod S3 API key credentials must be set as environment variables using the values from the [Setup and authentication](#setup-and-authentication) step:
168168

169-
* `RUNPOD_USER_EMAIL`: Should be set to your Runpod account email.
170-
* `RUNPOD_S3_API_KEY`: Should be set to your Runpod S3 API key's secret access key.
169+
* `AWS_ACCESS_KEY_ID`: Should be set to your Runpod S3 API key **access key** (e.g., `user_***...`).
170+
* `AWS_SECRET_ACCESS_KEY`: Should be set to your Runpod S3 API key's **secret** (e.g., `rps_***...`).
171171

172172
```python
173173
#!/usr/bin/env python3
@@ -177,7 +177,7 @@ import argparse
177177
import boto3 # AWS SDK for Python, used to interact with Runpod S3-compatible APIs
178178

179179
def create_s3_client(region: str, endpoint_url: str):
180-
180+
181181
# Creates and returns an S3 client configured for Runpod network volume S3-compatible API.
182182
#
183183
# Args:
@@ -190,14 +190,14 @@ def create_s3_client(region: str, endpoint_url: str):
190190
# boto3.client: An S3 client object, configured for the Runpod S3 API.
191191

192192
# Retrieve Runpod S3 API key credentials from environment variables.
193-
aws_access_key_id = os.environ.get("RUNPOD_USER_EMAIL")
194-
aws_secret_access_key = os.environ.get("RUNPOD_S3_API_KEY")
193+
aws_access_key_id = os.environ.get("AWS_ACCESS_KEY_ID")
194+
aws_secret_access_key = os.environ.get("AWS_SECRET_ACCESS_KEY")
195195

196196
# Ensure necessary S3 API key credentials are set in the environment
197197
if not aws_access_key_id or not aws_secret_access_key:
198198
raise EnvironmentError(
199-
"Please set RUNPOD_USER_EMAIL (with S3 API Key Access Key) and "
200-
"RUNPOD_S3_API_KEY (with S3 API Key Secret Access Key) environment variables. "
199+
"Please set AWS_ACCESS_KEY_ID (with S3 API Key Access Key) and "
200+
"AWS_SECRET_ACCESS_KEY (with S3 API Key Secret Access Key) environment variables. "
201201
"These are obtained from 'S3 API Keys' in the Runpod console settings."
202202
)
203203

@@ -211,7 +211,7 @@ def create_s3_client(region: str, endpoint_url: str):
211211
)
212212

213213
def put_object(s3_client, bucket_name: str, object_name: str, file_path: str):
214-
214+
215215
# Uploads a local file to the specified Runpod network volume.
216216
#
217217
# Args:
@@ -230,14 +230,14 @@ def put_object(s3_client, bucket_name: str, object_name: str, file_path: str):
230230
raise
231231

232232
def main():
233-
233+
234234
# Parses command-line arguments and orchestrates the file upload process
235235
# to a Runpod network volume.
236-
236+
237237
# Set up command-line argument parsing
238238
parser = argparse.ArgumentParser(
239239
description="Upload a file to a Runpod Network Volume using its S3-compatible API. "
240-
"Requires RUNPOD_USER_EMAIL and RUNPOD_S3_API_KEY env vars to be set "
240+
"Requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars to be set "
241241
"with your Runpod S3 API key credentials."
242242
)
243243
parser.add_argument(
@@ -311,19 +311,27 @@ The S3-compatible API supports the following operations. For detailed informatio
311311
Large file handling is supported through multipart uploads, allowing you to transfer files larger than 5GB.
312312

313313
## Limitations
314-
314+
- **Storage capacity**: Network volumes have a fixed storage capacity, unlike the virtually unlimited storage of standard S3 buckets. The `CopyObject` and `UploadPart` actions do not check for available free space beforehand and may fail if the volume runs out of space. This behavior is similar to applying a size quota in S3.
315+
- **Maximum file size:** 4TB (the maximum size of a network volume).
315316
- **Multipart uploads**:
316317
- Parts from multipart uploads are stored on disk until either `CompleteMultipartUpload` or `AbortMultipartUpload` is called.
317318
- The S3-compatible API enforces the 5GB maximum single file part upload size, but not the 5TB maximum file size.
318319
- The 5MB minimum part size for multipart uploads is not enforced.
319-
- **Storage capacity**: Network volumes have a fixed storage capacity, unlike the virtually unlimited storage of standard S3 buckets. The `CopyObject` and `UploadPart` actions do not check for available free space beforehand and may fail if the volume runs out of space. This behavior is similar to applying a size quota in S3.
320320
- **Object names**: Unlike traditional S3 key-value stores, object names in the Runpod S3-compatible API correspond to actual file paths on your network volume. Object names containing special characters (e.g., `#`) may need to be URL encoded to ensure proper processing.
321321
- **Time synchronization**: Requests that are out of time sync by 1 hour will be rejected. This is more lenient than the 15-minute window specified by the AWS SigV4 authentication specification.
322+
- **Unsupported S3 features:**
323+
- Object versioning.
324+
- Object encryption.
325+
- Object tagging.
326+
- Object ACLs.
327+
- Object locking.
328+
- Website redirects.
329+
- Storage classes besides `STANDARD` (all network volume objects use the `STANDARD` storage class).
322330

323331
## Reference documentation
324332

325333
For comprehensive documentation on AWS S3 commands and libraries, refer to:
326334

327335
- [AWS CLI S3 reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html).
328336
- [AWS S3 API reference](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html).
329-
- [Boto3 S3 reference](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html).
337+
- [Boto3 S3 reference](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html).

0 commit comments

Comments
 (0)