Skip to content

Commit e3b44be

Browse files
qiuosierqiuosier
committed
Update readme files.
1 parent 9f5be6a commit e3b44be

File tree

4 files changed

+220
-1
lines changed

4 files changed

+220
-1
lines changed

‎LLM/deployment/Readme.md

+206
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Deploy LLM Apps and Agents with OCI Data Science
2+
3+
The integration of Large Language Models (LLMs) into various applications and agents has been a transformative step in AI. With the ability to process and understand vast amounts of data from APIs, LLMs are revolutionizing the way we interact with technology. Oracle Cloud Infrastructure (OCI) Data Science provides a robust platform for deploying these sophisticated models, making it easier for developers and data scientists to bring their LLM-powered applications to life.
4+
5+
The process of deploying LLM apps and agents involves:
6+
1. Prepare your applications as model artifact
7+
2. Register the model artifact with OCI Data Science Model Catalog
8+
3. Build container image with dependencies, and push the image to OCI Container Registry
9+
4. Deploy the model artifact using the container image with OCI Data Science Model Deployment
10+
11+
![Workflow of LLM Apps and Agent Deployment](images/workflow.png)
12+
13+
## IAM Policies
14+
15+
Make sure you have the [Model Deployment Policies](https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-policies-auth.htm) configured. In addition, you may need the following policies:
16+
* `manage generative-ai-family` for using the LLM from generative AI service.
17+
* `manage objects` for saving results to object storage.
18+
19+
## Prepare Model Artifact
20+
21+
You can use Oracle ADS to prepare the model artifact.
22+
23+
First, create a template folder locally with the [`score.py`](model_artifacts/score.py) file. For example, we can call it `llm_apps_template`.
24+
```
25+
llm_apps_template
26+
├── score.py
27+
```
28+
The `score.py` serves as an agent for invoking your application with JSON payload.
29+
30+
Next, you can use ADS to create a [`generic model`](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/model_registration/frameworks/genericmodel.html) and save a copy of the template to `my_apps` folder:
31+
```python
32+
from ads.model.generic_model import GenericModel
33+
34+
generic_model = GenericModel.from_model_artifact(
35+
uri="llm_apps_template", # Contains the model artifact templates
36+
artifact_dir="my_apps", # Location for the new model artifacts
37+
model_input_serializer="cloudpickle"
38+
)
39+
generic_model.reload_runtime_info()
40+
```
41+
42+
Then, you can add your own applications to the `my_apps` folder. Here are some requirements:
43+
* Each application should be a Python module.
44+
* Each module should have an `invoke()` function as the entrypoint.
45+
* The `invoke()` function should take a dictionary and return another dictionary.
46+
47+
You can find a few example applications in the [model_artifacts](model_artifacts).
48+
49+
Once you added your application, you can call the `verify()` function to test it locally:
50+
```python
51+
generic_model.verify({
52+
"inputs": "How much is $80 USD in yen?",
53+
"module": "exchange_rate.py"
54+
})
55+
```
56+
57+
Note that with the default `score.py` template, you will invoke your application with two keys:
58+
* `module`: The module in the model artifact (`my_apps` folder) containing the application to be invoked. Here we are using the [exchange_rate.py](model_artifacts/exchange_rate.py) example. You can specify a default module using the `DEFAULT_MODULE` environment variables.
59+
* `inputs`: the value should be the payload for your application module. This example uses a string. However, you can use list or other JSON payload for your application.
60+
61+
The response will have the following format:
62+
```json
63+
{
64+
"outputs": "The outputs returned by invoking your app/agent",
65+
"error": "Error message, if any.",
66+
"traceback": "Traceback, if any.",
67+
"id": "The ID for identifying the request.",
68+
}
69+
```
70+
71+
If there is an error when invoking your app/agent, the error message along with the traceback will be returned in the response.
72+
73+
## Register the Model Artifact
74+
75+
Once your apps and agents are ready, you need save it to OCI Data Science Model Catalog before deployment:
76+
```python
77+
generic_model.save(display_name="LLM Apps", ignore_introspection=True)
78+
```
79+
80+
## Build Container Image
81+
82+
Before deploying the model, you will need to build a container image with the dependencies for your apps and agents.
83+
84+
To configure your environment for pushing image to OCI container registry (OCIR). Please refer to the OCIR documentation for [Pushing Images Using the Docker CLI](https://docs.oracle.com/en-us/iaas/Content/Registry/Tasks/registrypushingimagesusingthedockercli.htm).
85+
86+
The [container](container) directory contains files for building a container image for OCI Data Model Deployment service. You can add your dependencies into the [`requirement.txt`](container/requirements.txt) file. You may also modify the [`Dockerfile`](container/Dockerfile) if you need to add system libraries.
87+
88+
```bash
89+
docker build -t <image-name:tag> .
90+
```
91+
92+
Once the image is built, you can push it to OCI container registry.
93+
```bash
94+
docker push <image-name:tag>
95+
```
96+
97+
## Deploy as Model Deployment
98+
99+
To deploy the model, simply call the `deploy()` function with your settings:
100+
* For most application, a CPU shape would be sufficient.
101+
* Specify log group and log OCID to enable logging for the deployment.
102+
* Custom networking with internet access is required for accessing external APIs or OCI Generative AI APIs in a different region.
103+
* Add environments variables as needed by your application, including any API keys or endpoints.
104+
* You may set the `DEFAULT_MODULE` for invoking the default app
105+
106+
```python
107+
import os
108+
109+
generic_model.deploy(
110+
display_name="LLM Apps",
111+
deployment_instance_shape="VM.Standard.E4.Flex",
112+
deployment_log_group_id="<log_group_ocid>",
113+
deployment_predict_log_id="<log_ocid>",
114+
deployment_access_log_id="<log_ocid>",
115+
deployment_image="<image-name:tag>",
116+
# Custom networking with internet access is needed for external API calls.
117+
deployment_instance_subnet_id="<subnet_ocid>",
118+
# Add environments variables as needed by your application.
119+
# Following are just examples
120+
environment_variables={
121+
"TAVILY_API_KEY": os.environ["TAVILY_API_KEY"],
122+
"PROJECT_COMPARTMENT_OCID": os.environ["PROJECT_COMPARTMENT_OCID"],
123+
"LLM_ENDPOINT": os.environ["LLM_ENDPOINT"],
124+
"DEFAULT_MODULE": "app.py",
125+
}
126+
)
127+
```
128+
129+
## Invoking the Model Deployment
130+
131+
Once the deployment is active, you can invoke the application with HTTP requests. For example, with the [exchange_rate.py](model_artifacts/exchange_rate.py) agent:
132+
```python
133+
import oci
134+
import requests
135+
136+
response = requests.post(
137+
endpoint,
138+
json={
139+
"inputs": "How much is $50 USD in yen?.",
140+
"module": "exchange_rate.py"
141+
},
142+
auth=oci.auth.signers.get_resource_principals_signer()
143+
)
144+
response.json()
145+
```
146+
147+
The response will be similar to the following:
148+
```python
149+
{
150+
'error': None,
151+
'id': 'fa3d7111-326f-4736-a8f4-ed5b21654534',
152+
'outputs': {
153+
'input': 'How much is $50 USD in yen?.',
154+
'output': ' The exchange rate for USD to JPY is 151.000203. So, $50 USD is approximately 7550.01 JPY.'
155+
},
156+
'traceback': None
157+
}
158+
```
159+
160+
Note that the model deployment has a timeout limit of 1 minute for the HTTP requests.
161+
The [`score.py`](model_artifacts/score.py) allows you to specify an `async` argument to save the results to an OCI object storage location.
162+
For example:
163+
```python
164+
import oci
165+
import requests
166+
167+
response = requests.post(
168+
endpoint,
169+
json={
170+
"inputs": "",
171+
"module": "long_running.py",
172+
# Use async argument to specify a location for saving the response as JSON
173+
"async": "oci://bucket@namespace/prefix"
174+
},
175+
auth=oci.auth.signers.get_resource_principals_signer()
176+
)
177+
# The response here will have an ID and a URI for the output JSON file.
178+
async_data = response.json()
179+
async_data
180+
```
181+
182+
When the `async` argument is specified, the endpoint will return a response without waiting for the app/agent to complete running the task.
183+
```python
184+
{
185+
'id': 'bd67c258-69d0-4857-a3aa-6bc2836ba99d',
186+
'outputs': 'oci://bucket@namespace/prefix/bd67c258-69d0-4857-a3aa-6bc2836ba99d.json'
187+
}
188+
```
189+
The app/agent will continue work until it finishes, and then save the response as a JSON file into the URI in the `outputs`.
190+
191+
You can use [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) with [ocifs](https://ocifs.readthedocs.io/en/latest/) to check the object storage location and load it once the file is ready.
192+
```python
193+
import json
194+
import time
195+
import fsspec
196+
197+
fs = fsspec.filesystem("oci")
198+
while not fs.exists(async_data.get("outputs")):
199+
time.sleep(10)
200+
201+
with fsspec.open(async_data.get("outputs")) as f:
202+
results = json.load(f)
203+
204+
# results will contain the final response.
205+
results
206+
```

‎LLM/deployment/container/Readme.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Container for serving LLM apps with Flask and uWSGI
1+
# Container for Serving LLM Apps and Agents with Flask and uWSGI
22

33
This directory contains files for building a container image for serving LLM applications on OCI Data Science model deployment service.
44
* `Dockerfile`: The Dockerfile for building the image based on Oracle Linux 9.

‎LLM/deployment/images/workflow.png

621 KB
Loading

‎LLM/deployment/model_artifacts/Readme.md

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Artifacts for Deploying LLM Apps and Agents
2+
3+
This directory contains model artifacts and examples for deploying LLM apps and agents with OCI Data Science Model Deployment Service.
4+
* `score.py`. This is the module handling the requests and invoking the user modules.
5+
* `runtime.yaml`. This file is currently not used but reserved for the future developments.
6+
7+
To add your application, simply add a Python module containing your application with an `invoke()` function.
8+
9+
The following files are example applications:
10+
* `app.py`, a bare minimum app.
11+
* `translate.py`, a LangChain app translating English to French.
12+
* `exchange_rate.py`, a LangChain agent which can answer question with real time exchange rate.
13+
* `graph.py`, a LangGraph multi-agent example.

0 commit comments

Comments
 (0)