You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: serverless/storage/s3-api.mdx
+27-19Lines changed: 27 additions & 19 deletions
Original file line number
Diff line number
Diff line change
@@ -44,11 +44,11 @@ Create a network volume in one of the following datacenters to use the S3-compat
44
44
1. In the Runpod console, navigate to the [Settings page](https://www.runpod.io/console/user/settings).
45
45
2. Expand the **S3 API Keys** section and select **Create an S3 API key**.
46
46
3. Give your key a name and select **Create**.
47
-
4. Make a note of your access key ID and secret access key to use in the next step.
47
+
4. Make a note of the **access key** (e.g., `user_***...`) and **secret** (e.g., `rps_***...`) to use in the next step.
48
48
49
49
<Warning>
50
50
51
-
For security, Runpod will show your secret access key only once, so you may wish to save it elsewhere (e.g., in your password manager, or in a GitHub secret). Treat your secret access key like a password and don't share it with anyone.
51
+
For security, Runpod will show your API key secret only once, so you may wish to save it elsewhere (e.g., in your password manager, or in a GitHub secret). Treat your API key secret like a password and don't share it with anyone.
52
52
53
53
</Warning>
54
54
@@ -61,8 +61,8 @@ Create a network volume in one of the following datacenters to use the S3-compat
61
61
1. If you haven't already, [install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) on your local machine.
62
62
2. Run the command `aws configure` in your terminal.
63
63
3. Provide the following when prompted:
64
-
***AWS Access Key ID**: Enter your Runpod user ID. You can find this in the [Secrets section](https://www.runpod.io/console/user/secrets) of the Runpod console, in the description of your S3 API key. By default, the description will look similar to: `Shared Secret for user_2f21CfO73Mm2Uq2lEGFiEF24IPw 1749176107073`. `user_2f21CfO73Mm2Uq2lEGFiEF24IPw` is the user ID (yours will be different).
65
-
***AWS Secret Access Key**: Enter your Runpod S3 API key's secret access key.
64
+
***AWS Access Key ID**: Enter your **access key** (e.g., `user_***...`) from the previous step.
65
+
***AWS Secret Access Key**: Enter your **secret** (e.g., `rps_***...`) from the previous step.
66
66
***Default Region name**: You can leave this blank.
67
67
***Default output format**: You can leave this blank or set it to `json`.
68
68
@@ -164,10 +164,10 @@ You can also use the Boto3 library to interact with the S3-compatible API, using
164
164
165
165
The script below demonstrates how to upload a file to a Runpod network volume using the Boto3 library. It takes command-line arguments for the network volume ID (as an S3 bucket), the datacenter-specific S3 endpoint URL, the local file path, the desired object (file path on the network volume), and the AWS Region (which corresponds to the Runpod datacenter ID).
166
166
167
-
Your Runpod S3 API key credentials must be set as environment variables:
167
+
Your Runpod S3 API key credentials must be set as environment variables using the values from the [Setup and authentication](#setup-and-authentication) step:
168
168
169
-
*`RUNPOD_USER_EMAIL`: Should be set to your Runpod account email.
170
-
*`RUNPOD_S3_API_KEY`: Should be set to your Runpod S3 API key's secret access key.
169
+
*`AWS_ACCESS_KEY_ID`: Should be set to your Runpod S3 API key **access key** (e.g., `user_***...`).
170
+
*`AWS_SECRET_ACCESS_KEY`: Should be set to your Runpod S3 API key's **secret** (e.g., `rps_***...`).
171
171
172
172
```python
173
173
#!/usr/bin/env python3
@@ -177,7 +177,7 @@ import argparse
177
177
import boto3 # AWS SDK for Python, used to interact with Runpod S3-compatible APIs
# Parses command-line arguments and orchestrates the file upload process
235
235
# to a Runpod network volume.
236
-
236
+
237
237
# Set up command-line argument parsing
238
238
parser = argparse.ArgumentParser(
239
239
description="Upload a file to a Runpod Network Volume using its S3-compatible API. "
240
-
"Requires RUNPOD_USER_EMAIL and RUNPOD_S3_API_KEY env vars to be set "
240
+
"Requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars to be set "
241
241
"with your Runpod S3 API key credentials."
242
242
)
243
243
parser.add_argument(
@@ -311,19 +311,27 @@ The S3-compatible API supports the following operations. For detailed informatio
311
311
Large file handling is supported through multipart uploads, allowing you to transfer files larger than 5GB.
312
312
313
313
## Limitations
314
-
314
+
-**Storage capacity**: Network volumes have a fixed storage capacity, unlike the virtually unlimited storage of standard S3 buckets. The `CopyObject` and `UploadPart` actions do not check for available free space beforehand and may fail if the volume runs out of space. This behavior is similar to applying a size quota in S3.
315
+
-**Maximum file size:** 4TB (the maximum size of a network volume).
315
316
-**Multipart uploads**:
316
317
- Parts from multipart uploads are stored on disk until either `CompleteMultipartUpload` or `AbortMultipartUpload` is called.
317
318
- The S3-compatible API enforces the 5GB maximum single file part upload size, but not the 5TB maximum file size.
318
319
- The 5MB minimum part size for multipart uploads is not enforced.
319
-
-**Storage capacity**: Network volumes have a fixed storage capacity, unlike the virtually unlimited storage of standard S3 buckets. The `CopyObject` and `UploadPart` actions do not check for available free space beforehand and may fail if the volume runs out of space. This behavior is similar to applying a size quota in S3.
320
320
-**Object names**: Unlike traditional S3 key-value stores, object names in the Runpod S3-compatible API correspond to actual file paths on your network volume. Object names containing special characters (e.g., `#`) may need to be URL encoded to ensure proper processing.
321
321
-**Time synchronization**: Requests that are out of time sync by 1 hour will be rejected. This is more lenient than the 15-minute window specified by the AWS SigV4 authentication specification.
322
+
-**Unsupported S3 features:**
323
+
- Object versioning.
324
+
- Object encryption.
325
+
- Object tagging.
326
+
- Object ACLs.
327
+
- Object locking.
328
+
- Website redirects.
329
+
- Storage classes besides `STANDARD` (all network volume objects use the `STANDARD` storage class).
322
330
323
331
## Reference documentation
324
332
325
333
For comprehensive documentation on AWS S3 commands and libraries, refer to:
0 commit comments