diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/capabilities/auto-processing.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/capabilities/auto-processing.mdx new file mode 100644 index 00000000000..0743f8ccb5a --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/capabilities/auto-processing.mdx @@ -0,0 +1,138 @@ +--- +title: "Pipeline Auto-Processing" +navTitle: "Auto-Processing" +description: "Pipeline Auto-Processing" +--- + +## Overview +Pipeline Auto-Processing is designed to keep source data and pipeline output in sync. Without this capability, users would have to +manually trigger processing or provide external scripts, schedulers or triggers: +- **Full sync:** Insert/delete/update are all handled automatically. No lost updates, missing data or stale records. +- **Change detection:** Only new or changed records are processed. No unnecessary re-processing of known records. +- **Batch processing:** Records are grouped into batches to be processed concurrently. Reducing overhead and achieving optimal performance e.g. with GPU-based AI model inference tasks. +- **Background processing:** When enabled, the pipeline runs in a background worker process so that it doesn't block or delay other DB operations. Ideal for processing huge datasets. +- **Live processing for Postgres tables:** When the data source is a Postgres table, live trigger-based auto processing can be enabled so that pipeline results are always guaranteed up to date. +- **Quick turnaround:** Once a batch has finished processing, the results are immediately available. No full listing of the source data is needed to start processing. This is important for large external volumes where a full listing can take a long time. + +### Example for Knowledge Base Pipeline +A knowledge base is created for a Postgres table containing products with product descriptions. +The user configures background auto-processing to always keep embeddings in sync without blocking or delaying any operations on the products table. + +The pipeline will process any pre-existing product records in the background, the user can query the statistics table to see the progress. + +The background process will run when new data is inserted, existing data modified or deleted. + +Queries on the Knowledge Base (i.e. retrieval operations) will always return accurate results within a small background processing delay. + + +### Supported pipelines +The Knowledge Base pipeline supports all capabilities listed here. + +The Preparer pipeline does not yet support batch processing and background auto-processing + +## Auto-Processing modes +We offer the following Auto-Processing modes to suit different requirements and use-cases. + +!!! Note +Live auto-processing is only available for table sources. Not for volume sources. +!!! + +### Live +We set up Postgres Triggers on the source table to immediately process any changes. Processing happens within the trigger function. +This means it happens within the same transaction that modifies the data, guaranteeing up to date results. + +#### Considerations +- Transactional guarantee / immediate results. Pipeline results are always up to date with the source data. +- Blocks / delays operations on the source data. Modifying transactions on the source data are delayed until processing is complete. + +### Background +We start a Postgres background worker for each pipeline TODO + + +!!! Note +Make sure Postgres allows running enough background workers for the number of pipelines where you wish to use this processing mode. +!!! +#### Considerations +TODO + +### Disabled +TODO + + + +## Observability +We provide detailed status and progress output for all auto-processing modes. + +A good place to get an overview is the statistics table. +Look up the view `aidb.knowledge_base_stats` or use its short alias `aidb.kbstat`. The table shows all configured knowledge base pipelines, +which processing mode is set, and statistics about the processed records: +```sql +SELECT * from aidb.kbstat; +__OUTPUT__ + knowledge base | auto processing | table: unprocessed rows | volume: scans completed | count(source records) | count(embeddings) +------------------------+-----------------+-------------------------+-------------------------+-----------------------+------------------- + kb_table_text_bg | Background | 0 | | 15 | 15 + kb_table_text_manual | Disabled | 0 | | 15 | 15 + kb_table_image_manual | Disabled | 0 | | 3 | 3 + kb_table_text_live | Live | 0 | | 15 | 15 + kb_table_image_bg | Background | 0 | | 3 | 3 + kb_volume_text_bg | Background | | 6 | 7 | 7 + kb_volume_text_manual | Disabled | | 0 | 0 | 0 + kb_volume_image_bg | Background | | 4 | 177 | 6 + kb_volume_image_manual | Disabled | | 1 | 177 | 6 +(9 rows) +``` + +The [change detection](#change-detection) mechanism is central to how auto-processing works. It is different for volume and table sources. +For this reason, the stats table has different columns for these two source types. + +* `table: unprocessed rows`: How many unique rows are listed in the backlog of change events. + * If auto-processing is disabled, no (new) change events are captured. +* `volume: scans completed`: How many full listings of the source have been completed so far. +* `count(source records)`: How many records exist in the source for this pipeline. + * for table sources, this number is always accurate. + * for volume sources, we can only update this number after a full scan has completed. +* `count(embeddings)`: How many embeddings exist in the vector destination table for this pipeline. + + + +## Configuration +Auto-processing can be configured at creation time: +- With [`aidb.create_table_knowledge_base`](../reference/knowledge_bases#aidbcreate_table_knowledge_base) +- With [`aidb.create_volume_knowledge_base`](../reference/knowledge_bases#aidbcreate_volume_knowledge_base) + +As well as for existing pipelines: +- With [`aidb.set_auto_knowledge_base`](../reference/knowledge_bases#aidbset_auto_knowledge_base) + +## Batch processing +In Background and Disabled modes, (auto) processing happens in batches of configurable size. Within each batch, + +## Change detection +AIDB auto-processing is designed around change detection mechanisms for table and volume data sources. This allows it to only +process data when necessary. + +### Table sources +When background auto-processing is configured, Postgres triggers are set up on the source table to detect changes. These triggers are very lightweight. +They only record change events and insert them into a "change events" table. No actual processing happens in the trigger function. + +The background worker will then process these events asynchronously. + +!!! Notice +When auto-processing is disabled, no change detection on tables is possible. This means the manual `bulk_embedding()` operation has to process +all source records. +!!! + + +### Volume sources +This source type provides a `last_modified` timestamp for each source record. The system keeps track of those timestamps in a "state" table. +In each pipeline execution, the system lists the contents of the volume and compares it to the timestamps to see whether any records have changed or were added. + +This mechanism works in disabled and in background auto-processing. + +The system detects deleted objects after a full listing is complete. Only then can it be certain that a previously processed record is no longer present in the source. + +Unfortunately, object stores (and other external storage locations supported by our volumes) have limited query capabilities. This means: +!!! Note +Change detection for volumes is based on polling i.e., repeated listing. This might be an expensive operation when using cloud object stores like AWS S3. +You can use a long `background_sync_interval` (like one per day) on pipelines with volume sources to control this cost. +!!! diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/capabilities.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/capabilities/index.mdx similarity index 100% rename from advocacy_docs/edb-postgres-ai/ai-accelerator/capabilities.mdx rename to advocacy_docs/edb-postgres-ai/ai-accelerator/capabilities/index.mdx diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/index.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/index.mdx index bfc9ad4eca8..cad84d61a54 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/index.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/index.mdx @@ -20,6 +20,7 @@ navigation: - preparers - knowledge_base - pgfs +- volumes - "#Pipelines resources" - reference - rel_notes diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/knowledge_base/usage.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/knowledge_base/usage.mdx index c9c34f4821f..e41cd7204e5 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/knowledge_base/usage.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/knowledge_base/usage.mdx @@ -28,7 +28,7 @@ create_table_knowledge_base( topk INTEGER DEFAULT 1, distance_operator aidb.distanceoperator DEFAULT 'L2', options JSONB DEFAULT '{}'::JSONB, - processing_mode aidb.PipelineProcessingMode DEFAULT 'Disabled' + auto_processing aidb.pipelineautoprocessingmode DEFAULT 'Disabled' ) ``` @@ -96,7 +96,7 @@ aidb.create_volume_knowledge_base( topk INTEGER DEFAULT 1, distance_operator aidb.distanceoperator DEFAULT 'L2', options JSONB DEFAULT '{}'::JSONB - processing_mode aidb.PipelineProcessingMode DEFAULT 'Disabled' + auto_processing aidb.pipelineautoprocessingmode DEFAULT 'Disabled' ) ``` diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/models/index.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/models/index.mdx index 00acbc80534..9aef445c313 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/models/index.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/models/index.mdx @@ -19,4 +19,4 @@ Pipelines has a model registry that manages configured instances of models. Any ## Next steps -Once you are familiar with models, you can learn how to use those models with [knowledge bases](../knowledge_bases). +Once you are familiar with models, you can learn how to use those models with [knowledge bases](../knowledge_base). diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/models/primitives.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/models/primitives.mdx index 5058f6f0b8e..6a0cb8ec6f5 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/models/primitives.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/models/primitives.mdx @@ -4,7 +4,7 @@ navTitle: "Primitives" description: "The model primitives available in EDB Postgres AI - AI Accelerator Pipelines." --- -For most use cases, we recommend that you use the aidb [knowledge bases](../knowledge_bases) to interact with models. They can manage creating embeddings and retrieving matching data for many applications. +For most use cases, we recommend that you use the aidb [knowledge bases](../knowledge_base) to interact with models. They can manage creating embeddings and retrieving matching data for many applications. However, if you need to interact with models directly, you can use the following primitives. The encode functions generate embeddings for text and images, and the decode functions generate text from embeddings. diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/models/supported-models/nim_reranking.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/models/supported-models/nim_reranking.mdx index 686e5fef9bd..a67068dd58e 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/models/supported-models/nim_reranking.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/models/supported-models/nim_reranking.mdx @@ -21,24 +21,39 @@ Reranking is a method in text search that sorts results by relevance to make the * nvidia/llama-3.2-nv-rerankqa-1b-v2 (default) +## Example +The function accepts a string as the "rerank query" and an array of texts to rerank. +The `id` column in the output refers to the index of the text in the input array. + +```sql +aidb=# SELECT * from aidb.rerank_text('my_nim_reranker', 'how can I open a door?', '{Ask for help, Push the handle, Lie down and wait, Shout at it}'::text[]) ORDER BY logit_score DESC; + text | logit_score | id +-------------------+--------------+---- + Push the handle | -3.697265625 | 1 + Ask for help | -6.2578125 | 0 + Shout at it | -7.39453125 | 3 + Lie down and wait | -11.375 | 2 +(4 rows) +``` + ## Creating the default model ```sql SELECT aidb.create_model( - 'my_nim_reranker', + 'my_nim_reranker', 'nim_reranking', - credentials=>'{"api_key": ""'::JSONB -); + '{"url":"http://nim-nv-rerankqa-llama-l-1xgpu-g6-predictor.default.svc.cluster.local/v1/ranking", "model": "nvidia/llama-3.2-nv-rerankqa-1b-v2"}' + ); ``` -There's only one model, the default `nvidia/nvclip`, so you don't need to specify the model in the configuration. +This example uses a locally deployed NIM model that does not require credentials. Credentials and other configuration can be provided as described in [using models](../using-models). ## Model configuration settings The following configuration settings are available for NIM models: -* `model` — The NIM model to use. The default is `nvidia/llama-3.2-nv-rerankqa-1b-v2` and is the only model available. +* `model` — The NIM model to use. The default is `nvidia/llama-3.2-nv-rerankqa-1b-v2` and is the only supported model. * `url` — The URL of the model to use. This setting is optional and can be used to specify a custom model URL. The default is `https://ai.api.nvidia.com/v1/retrieval`. ## Model credentials diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions/index.mdx similarity index 50% rename from advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions.mdx rename to advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions/index.mdx index 078afe4cc2c..66b6d41f7a1 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions/index.mdx @@ -1,50 +1,36 @@ --- title: "PGFS functions for Pipelines" navTitle: "PGFS functions" -description: "How to use PGFS functions to access external storage in Pipelines." +description: "How to use PGFS functions to manage external storage in Pipelines." --- -## Using the PGFS functions - -The PGFS extension provides a set of functions to create and manage storage locations. - -### Creating a storage location +## Creating a storage location Start with creating a storage location. A storage location is a reference to a location in an external file system. You can create a storage location with the `pgfs.create_storage_location` function: ```sql -select pgfs.create_storage_location('my_storage', 's3://my_bucket','','{}'::JSONB,'{}'::JSONB); +select pgfs.create_storage_location('storage_location_name', 'protocol://path', options => '{}'::JSONB, credentials => '{}'::JSONB); ``` -The `create_strorage_location` function takes a name for the storage location and then a URL for the location. Prefix the URL with `s3:` for an S3-compatible bucket or `file:` for a local file system. - - -```sql -select pgfs.create_storage_location('my_file_storage', 'file:///tmp/my_path', NULL, '{}'::json, '{}'::json ); -``` +### Storage provider types +Detailed instructions for the supported storage providers can be found here: -When using the `file:` schema, provide an absolute path, one that starts with `/`, for example `/tmp/my_path`). Together with the schema indicator `file://`, there are then three slashes at the beginning of the path. - -The function also takes an optional `msl_id` parameter, which isn't used. It also requires `options` and `credentials` parameters. If those are unused, you must pass them as empty JSON objects. +- [S3-compatible storage](s3) +- [Local file system](local) ### Creating a storage location with options and credentials -Using the `options` and `credentials` parameters allows a range of other settings to be passed. +Using the `options` and `credentials` parameters allows a range of settings to be passed. The `options` parameter is a JSON object that can be used to pass additional options to the storage location. The `credentials` parameter is a JSON object that can be used to pass credentials to the storage location. The difference between `options` and `credentials` is that options remain visible to users querying the extension while credentials are hidden to all users except superusers and the user that creates the storage location. -For example, you can create a storage location with options and credentials like this: - -```sql -select pgfs.create_storage_location('my_storage', 's3://my_private_bucket', null, '{"region": "eu-west"}'::JSONB, '{"access_key_id": "youraccesskeyid", "secret_access_key":"yoursecretaccesskey"}'::JSONB); -``` - -Once you've created a storage location, you can use it to create foreign tables and access files in the external file system. To use it with aidb, you need to create a volume from the storage location. To do that, see [Creating a volume](../knowledge_bases/usage#creating-a-new-volume). +### Testing storage locations and using them with AIDB +To use a storage location with aidb, you need to create a volume from the storage location. To do that, see [Creating a volume](../../volumes). -### Listing storage locations +## Listing storage locations You can list all storage locations with the `pgfs.list_storage_locations` function: @@ -54,7 +40,7 @@ select * from pgfs.list_storage_locations(); This command returns a table of currently defined storage locations. Credentials are shown only if the user has the necessary permissions. Otherwise the column is NULL. -### Getting a storage location +## Getting a storage location You can get the details of a specific storage location with the `pgfs.get_storage_location` function: @@ -64,7 +50,7 @@ select * from pgfs.get_storage_location('my_storage'); This command returns the details of the storage location named `my_storage`. -### Updating a storage location +## Updating a storage location You can update a storage location with the `pgfs.update_storage_location` function: @@ -72,7 +58,7 @@ You can update a storage location with the `pgfs.update_storage_location` functi select pgfs.update_storage_location('my_storage', 's3://my_bucket', null, '{"region": "eu-west"}' ``` -### Deleting a storage location +## Deleting a storage location You can delete a storage location with the `pgfs.delete_storage_location` function: diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions/local.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions/local.mdx new file mode 100644 index 00000000000..a30f25ed606 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions/local.mdx @@ -0,0 +1,26 @@ +--- +title: "Pipelines PGFS with local file storage" +navTitle: "Local file storage" +description: "How to use Pipelines PGFS with local file storage." +--- + + +## Overview: local file systems +PGFS uses the `file:` prefix to indicate a local file system. + +The general syntax for using local file systems is this: +```sql +select pgfs.create_storage_location( + 'local_images', + 'file:///var/lib/edb/pipelines/images' + ); +``` + +!!! Note +Paths must always be absolute; e.g., start at the root `/`. Together with the protocol prefix `file://`, this means your path will have three slashes as in the example above. +!!! + +!!! Note +Any local path that you want to access must be allowlisted in the [PGFS settings](../settings). +!!! + diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions/s3.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions/s3.mdx new file mode 100644 index 00000000000..a97bd57cc69 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions/s3.mdx @@ -0,0 +1,66 @@ +--- +title: "Pipelines PGFS with S3-compatible storage" +navTitle: "S3 storage" +description: "PGFS options and credentials with S3-compatible storage." +--- + + +## Overview: S3-compatible storage +PGFS uses the `s3:` prefix to indicate an S3-compatible storage system. + +The general syntax for using S3 is this: +```sql +select pgfs.create_storage_location( + 'storage_location_name', + 's3://bucket_name', + options => '{}'::JSONB, + credentials => '{}'::JSONB + ); +``` + +### The `options` argument in JSON format offers the following settings: +| Option | Description | +|------------------|--------------------------------------------------------------------------------------------------------------------------------| +| `bucket` | The bucket name. Used to explicitly provide the bucket name if it can't be passed in the url. | +| `skip_signature` | Disable HMAC authentication (set this to "true" when you're not providing access_key_id/secret_access_key in the credentials). | +| `region` | The region of the S3-compatible storage system. If the region is not specified, the client will attempt auto-discovery. | +| `endpoint` | The endpoint of the S3-compatible storage system. | + +### The `credentials` argument in JSON format offers the following settings: +| Option | Description | +|---------------------|-------------------------------------------------------------------------| +| `access_key_id` | HMAC credentials (this is often the "username" in non-AWS S3 providers) | +| `secret_access_key` | HMAC credentials (this is often the "password" in non-AWS S3 providers) | +| `session_token` | A session token can be used instead of HMAC credentials | + + +## Example: AWS S3 public bucket +This is an example of using a public bucket on AWS S3. Public buckets do not require authentication. + +```sql +SELECT pgfs.create_storage_location('edb_ai_example_images', 's3://public-ai-team', + options => '{"region": "eu-central-1", "skip_signature": "true"}' + ); +``` + +## Example: AWS S3 private bucket +This is an example of using a private bucket on AWS S3. Private buckets require authentication. Here, we're using HMAC credentials. + +```sql +SELECT pgfs.create_storage_location('internal_ai_project', 's3://my-company-ai-images', + options => '{"region": "eu-central-1"}', + credentials => '{"access_key_id": "secret", "secret_access_key":"secret!"}' + ); +``` + +## Example: non-AWS S3 / S3-compatible +This is an example of using an S3-compatible system like minIO. The `endpoint` must be provided in this case; it can only be omitted when using AWS S3. + +```sql +SELECT pgfs.create_storage_location('ai_images_local_minio', 's3://my-ai-images', + options => '{"endpoint": "https://minio-api.apps.local"}', + credentials => '{"access_key_id": "my_username", "secret_access_key":"my_password"}' + ); +``` + + diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/index.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/index.mdx index 1f0b20c6251..e5d0fe29093 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/index.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/index.mdx @@ -10,22 +10,18 @@ PGFS is a key component of Pipelines, allowing you to work with data that isn't PGFS is implemented as a foreign data wrapper (FDW) and installed as an extension. The extension provides a set of functions to interact with the file system from the database. + +## PGFS storage locations and AIDB volumes +To use a PGFS storage location with AIDB, a `volume` must be created in AIDB: +* **PGFS storage location:** Represents an external storage provider with all necessary configuration like a path and credentials. +* **AIDB volume:** Connects a sotrage location to AIDB. Multiple volumes for the same storage location are possible. In a volume, you configure the data format in the destination and optionally, a sub-path. + ## Installing the PGFS extension See [Installing](../installing) to learn how to install the PGFS extension as part of the Pipelines installation. ## Using PGFS -There are two approaches to using PGFS: - -* (Recommended) Use the [PGFS functions](functions) to create and delete storage locations - -* Use SQL commands such as the `CREATE SERVER` command to create a server object that creates the storage location. - -### Using the PGFS functions - -See the [PGFS functions](functions) reference for details on the functions available in the PGFS extension. - -## Using PGFS with SQL +See the [PGFS management functions](functions) reference for details on the functions available in the PGFS extension. +With these functions you can create and manage storage locations, to be later used through AIDB volumes. -See the [PGFS SQL](sql) reference for details on how to use PGFS with SQL commands. diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/sql/index.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/sql/index.mdx deleted file mode 100644 index 2f7fd630568..00000000000 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/sql/index.mdx +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: "PGFS SQL in Pipelines" -navTitle: "PGFS SQL" -description: "How to work with the Postgres File System (PGFS) in Pipelines." ---- - -!!! Note - This feature is available in EDB Postgres AI - AI Accelerator Pipelines. We recommend that you - use the [PGFS functions](../functions) to configure a storage location. - -## Using PGFS with SQL - -### Using the `CREATE SERVER` command - -You can use the `CREATE SERVER` command to create a server object that connects to the storage location. You can then create a foreign table that's associated with the server. - -This example creates a server object that connects to an S3-compatible bucket: - -```sql -CREATE SERVER my_server FOREIGN DATA WRAPPER pgfs OPTIONS (url 's3://my_bucket'); -``` - -### Options and credentials - -You can pass options and credentials to the storage locations. -The options and credentials are specific to the type of storage location you're using. - -- [S3-compatible storage](s3) -- [Local file system](local) diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/sql/local.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/sql/local.mdx deleted file mode 100644 index 9d4f9f1e5c8..00000000000 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/sql/local.mdx +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: "Pipelines PGFS with local file storage" -navTitle: "Local file storage" -description: "How to use Pipelines PGFS with local file storage." ---- - -!!! Note - This feature is available in EDB Postgres AI - AI Accelerator Pipelines. We recommend that you - use the [PGFS functions](../functions) to configure a storage location. - -PGFS uses the `file:` prefix to indicate a local file system. You can use the `file:` prefix to access files on the local file system. - -## Using local file storage through PGFS and SQL - -This example uses local file storage: - -```sql -CREATE SERVER images_local FOREIGN DATA WRAPPER pgfs OPTIONS (url 'file:///var/lib/edb/pipelines/images'); -``` - -This command creates a server object called `images_local` that connects to the local file system at `/var/lib/edb/pipelines/images`. - -You can then create a foreign table that's associated with the server: - -```sql -CREATE FOREIGN TABLE images (id INT, name TEXT) SERVER images_local; -``` - -This foreign table is associated with the `images_local` server object and can be used to access the files in the `/var/lib/edb/pipelines/images` directory. - -You can also use the `file:` prefix to access files on the local file system when you create a foreign table. For example: - -```sql -CREATE FOREIGN TABLE images (id INT, name TEXT) SERVER images_local OPTIONS (filename 'file:///var/lib/edb/pipelines/images/image1.jpg'); -``` - -This foreign table is associated with the `images_local` server object and can be used to access the file `/var/lib/edb/pipelines/images/image1.jpg`. diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/sql/s3.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/sql/s3.mdx deleted file mode 100644 index 2aab557c2c4..00000000000 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/sql/s3.mdx +++ /dev/null @@ -1,48 +0,0 @@ ---- -title: "Pipelines PGFS with S3-compatible storage" -navTitle: "S3 storage" -description: "PGFS options and credentials with S3-compatible storage." ---- - -!!! Note - This feature is available in EDB Postgres AI - AI Accelerator Pipelines. We recommend that you - use the [PGFS functions](../functions) to configure a storage location. - -PGFS uses the `s3:` prefix to indicate an S3-compatible storage system. You can use the `s3:` prefix to access files on an S3-compatible storage system. - -## Using S3-compatible storage through PGFS with SQL - -This example uses S3-compatible storage: - -```sql -CREATE SERVER images_s3 FOREIGN DATA WRAPPER pgfs OPTIONS (url 's3://my-bucket', , skip_signature 'true'); -``` - -This command creates a server object called `images_s3` that connects to the S3-compatible storage system at `my-bucket`. With no credentials provided, the server assumes the S3 storage system is open to the public. For a public bucket, ensure that you include the `skip_signature` option and set it to `true`. - -You can then create a foreign table that's associated with the server: - -```sql -CREATE FOREIGN TABLE images (id INT, name TEXT) SERVER images_s3; -``` - -This foreign table is associated with the `images_s3` server object and can be used to access the files in the `my-bucket` directory. - -You can also use the `s3:` prefix to access files on the S3-compatible storage system when you create a foreign table. For example: - -```sql -CREATE FOREIGN TABLE images (id INT, name TEXT) SERVER images_s3 OPTIONS (filename 's3://my-bucket/image1.jpg'); -``` - -This foreign table is associated with the `images_s3` server object and can be used to access the file `my-bucket/image1.jpg`. - -## Supported options with S3-compatible storage - -The following options are supported when you create a storage location with S3-compatible storage. - -| Option | Description | -|-----------------|-----------------------------------------------------------------------------| -| `skip_signature` | Skip the signature when connecting to the S3-compatible storage system. | -| `region` | The region of the S3-compatible storage system. | -| `endpoint` | The endpoint of the S3-compatible storage system. | -| `credentials` | The credentials for the S3-compatible storage system. | diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/knowledge_bases.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/knowledge_bases.mdx index 738c2facf89..b3a57104036 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/knowledge_bases.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/knowledge_bases.mdx @@ -5,7 +5,7 @@ description: "Reference documentation for EDB Postgres AI - AI Accelerator Pipel deepToC: true --- -This reference documentation for Pipelines knowledge bases includes information on the functions and views available in the aidb extension related to [knowledge bases](../knowledge_bases/). +This reference documentation for Pipelines knowledge bases includes information on the functions and views available in the aidb extension related to [knowledge bases](../knowledge_base/). ## Views @@ -13,24 +13,45 @@ This reference documentation for Pipelines knowledge bases includes information Also referenceable as `aidb.kbs`, the `aidb.knowledge_bases` view provides a list of all knowledge bases in the database. It includes information about the knowledge base name, the model used, and the source data type. -| Column | Type | Description | -|-------------------------------|------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| id | integer | | -| name | text | Name of the knowledge base. | -| vector_table_name | text | Name of the table where the embeddings are stored. Gets newly created if it doesn’t exist. Managed by aidb. | -| vector_table_key_column | text | Column to use to store the key that references the key in source data when computing embeddings. We recommend using the default and letting aidb manage this table. | -| vector_table_vector_column | text | Column to store embeddings in. We recommend using the default and letting aidb manage this table. | -| model_name | text | Name of the model to use for embedding computation and retrievals. | -| topk | integer | Default number of results to return during a retrieve. Similar to LIMIT in SQL. | -| distance_operator | [aidb.DistanceOperator](#aidbdistanceoperator) | During retrieval, the vector operation to use to compare the vectors. | -| options | jsonb | Currently unused. | -| processing_mode | aidb.pipelineprocessingmode | Automated processing mode | -| source_type | text | Type of source data the knowledge base is working with. Can be 'Table' or 'Volume'. | -| source_table_name | regclass | Name of the table that has the source data Pipelines computes embeddings for and retrieves from. Only applicable to knowledge bases configured with aidb.create_table_knowledge_base(). | -| source_table_data_column | text | Column name in the source table that Pipelines computes embeddings for. This is also the column that's returned in retrieve operations. | -| source_table_data_column_type | [aidb.PipelineDataFormat](#aidbpipelinedataformat) | Type of data the knowledge base is working with. Uses type [`aidb.PipelineDataFormat`](#aidbpipelinedataformat). Only relevant for table-based knowledge bases. In the case of a volume-based knowledge base, the format/type information is discovered from the volume. | -| source_table_key_column | text | Column to use as key for storing the embedding in the vector table. This provides a reference from the embedding to the source data. | -| source_volume_name | text | Name of the volume to use as a data source. Only applicable to knowledge bases configured with aidb.create_volume_knowledge_base(). | +| Column | Type | Description | +|--------------------------|--------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| id | integer | | +| name | text | Name of the knowledge base. | +| vector_schema | text | Schema for vector_table. | +| vector_table | text | Name of the table where the embeddings are stored. Gets newly created if it doesn’t exist. Managed by aidb. | +| vector_key_column | text | Column to use to store the key that references the key in source data when computing embeddings. We recommend using the default and letting aidb manage this table. | +| vector_data_column | text | Column to store embeddings in. We recommend using the default and letting aidb manage this table. | +| model_name | text | Name of the model to use for embedding computation and retrievals. | +| topk | integer | Default number of results to return during a retrieve. Similar to LIMIT in SQL. | +| distance_operator | [aidb.DistanceOperator](#aidbdistanceoperator) | During retrieval, the vector operation to use to compare the vectors. | +| options | jsonb | Unused. | +| auto_processing | [aidb.PipelineAutoProcessingMode](#aidbpipelineautoprocessingmode) | Auto-processing mode. | +| owner_role | text | The Postgres role who created this pipeline. Background auto-processing will run the pipeline as this role. | +| batch_size | integer | How many records to process concurrently when running this pipeline. | +| background_sync_interval | interval | Used for background auto-processing. This is the interval between pipeline executions. | +| source_type | text | Indicates whether this pipeline uses a table or volume as data source. | +| source_schema | text | Schema for source_table. | +| source_table | text | Name of the table used as input for the pipeline. Unused if the knowledge base uses a volume as source. | +| source_data_column | text | Column name in the source table that Pipelines computes embeddings for. This is also the column that's returned in retrieve operations. | +| source_data_format | [aidb.PipelineDataFormat](#aidbpipelinedataformat) | Format of the data the knowledge base is working with. Uses type [`aidb.PipelineDataFormat`](#aidbpipelinedataformat). | +| source_key_column | text | Column to use as key for storing the embedding in the vector table. This provides a reference from the embedding to the source data. | +| source_volume_name | text | Name of the volume to use as a data source. Only applicable to knowledge bases configured with aidb.create_volume_knowledge_base(). | + + + +### `aidb.knowledge_base_stats` + +Also referenceable as `aidb.kbstat`, the `aidb.knowledge_base_stats` view provides current statistics about auto processing for knowledge base pipelines. + +| Column | Type | Description | +|-------------------------|--------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------| +| knowledge base | text | Name of the knowledge base. | +| auto processing | [aidb.PipelineAutoProcessingMode](#aidbpipelineautoprocessingmode) | Auto-processing mode. | +| table: unprocessed rows | bigint | For table knowledge bases: The number of unprocessed rows that are new or might have changed since the last execution. | +| volume: scans completed | bigint | For volume knowledge bases: The number of full listings of the source volume that were performed so far. | +| count(source records) | bigint | The number of records in the source of the knowledge base. For volume knowledge bases, this is number of records seen during the last full scan. | +| count(embeddings) | bigint | The number of embeddings stored in the destination table. | + ## Types @@ -38,8 +59,8 @@ Also referenceable as `aidb.kbs`, the `aidb.knowledge_bases` view provides a lis The `aidb.DistanceOperator` type is an enum that represents the distance operators that can be used during retrieval. -| Value | Description | -|---------|---------------------| +| Value | Description | +|---------|--------------------| | L2 | Euclidean distance | | Inner | Inner product | | Cosine | Cosine similarity | @@ -80,18 +101,18 @@ CREATE TYPE PipelineDataFormat AS ENUM ( ); ``` -### `aidb.PipelineProcessingMode` +### `aidb.PipelineAutoProcessingMode` -The `aidb.PipelineProcessingMode` type is an enum used to define which processing mode is configured for a Pipeline (e.g. a Knowledge Base or a Preparer). +The `aidb.PipelineAutoProcessingMode` type is an enum used to define how auto processing for a Pipeline shoud behave (e.g. a Knowledge Base or a Preparer). -| Value | Description | -|-------------|---------------------------------------------------------------------------------| -| Live | New data is processed immediately while being added (using Postgres Triggers) | -| Background | Currently unused | -| Disabled | No automated processing | +| Value | Description | +|------------|-------------------------------------------------------------------------------| +| Live | New data is processed immediately while being added (using Postgres Triggers) | +| Background | Continuous processing in the background (using Postgres background workers) | +| Disabled | No automated processing | ```sql -CREATE TYPE PipelineProcessingMode AS ENUM ( +CREATE TYPE PipelineAutoProcessingMode AS ENUM ( 'Live', 'Background', 'Disabled' @@ -107,21 +128,24 @@ Creates a knowledge base for a given table. #### Parameters -| Parameter | Type | Default | Description | -|--------------------|------------------------------------------------------------------|--------------|----------------------------------------------------| -| name | TEXT | Required | Name of the knowledge base | -| model_name | TEXT | Required | Name of the model to use | -| source_table | regclass | Required | Name of the table to use as source | -| source_data_column | TEXT | Required | Column name in source table to use | -| source_data_format | [aidb.PipelineDataFormat](#aidbpipelinedataformat) | Required | Type of data in that column ("Text"."Image","PDF") | -| source_key_column | TEXT | 'id' | Column to use as key to reference the rows | -| vector_table | TEXT | NULL | | -| vector_data_column | TEXT | 'embeddings' | | -| vector_key_column | TEXT | 'id' | | -| topk | INTEGER | 1 | | -| distance_operator | [aidb.distanceoperator](#aidbdistanceoperator) | 'L2' | | -| options | JSONB | '{}'::JSONB | Options | -| index_type | TEXT | 'vector' | Type of index to use for the vector table. | +| Parameter | Type | Default | Description | +|--------------------------|--------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------| +| name | TEXT | Required | Name of the knowledge base. | +| model_name | TEXT | Required | Name of the model to use. | +| source_table | regclass | Required | Name of the table to use as source. | +| source_data_column | TEXT | Required | Column name in source table to use. | +| source_data_format | [aidb.PipelineDataFormat](#aidbpipelinedataformat) | Required | Format of data in that column ("Text", "Image", "PDF"). | +| source_key_column | TEXT | 'id' | Column to use as key to reference the rows. | +| vector_table | TEXT | NULL | | +| vector_data_column | TEXT | 'embeddings' | | +| vector_key_column | TEXT | 'id' | | +| topk | INTEGER | 1 | | +| distance_operator | [aidb.distanceoperator](#aidbdistanceoperator) | 'L2' | | +| options | JSONB | '{}'::JSONB | Unused | +| auto_processing | [aidb.PipelineAutoProcessingMode](#aidbpipelineautoprocessingmode) | 'Disabled' | Configure auto-processing for this pipeline. | +| index_type | TEXT | 'vector' | Type of index to use for the vector table. | +| batch_size | int | 100 | How many records to process concurrently. | +| background_sync_interval | interval | '30 seconds' | Interval between pipeline executions if background auto-processing is configured. | !!! note "Index_types" * If `index_type` is set to `vector`, the system will automatically create a [hnsw](https://github.com/pgvector/pgvector?tab=readme-ov-file#hnsw) index on the vector table based on the distance operator used in the knowledge base. @@ -143,10 +167,10 @@ Creates a knowledge base for a given table. ```sql SELECT aidb.create_table_knowledge_base( name => 'test_knowledge_base', - model_name => 'simple_model', + model_name => 'bert', source_table => 'test_source_table', source_data_column => 'content', - source_data_type => 'Text', + source_data_type => 'Text' ); ``` @@ -156,18 +180,21 @@ Creates a knowledge base for a given PGFS volume. #### Parameters -| Parameter | Type | Default | Description | -|--------------------|-----------------------|--------------|--------------------------------------------| -| name | TEXT | Required | Name of the knowledge base | -| model_name | TEXT | Required | Name of the model | -| source_volume_name | TEXT | Required | Name of the volume | -| vector_table | TEXT | NULL | Name of the vector table | -| vector_data_column | TEXT | 'embeddings' | Name of the vector column | -| vector_key_column | TEXT | 'id' | Name of the key column | -| topk | INTEGER | 1 | Number of results to return | -| distance_operator | aidb.distanceoperator | 'L2' | Distance operator | -| options | JSONB | '{}'::JSONB | Options | -| index_type | TEXT | 'vector' | Type of index to use for the vector table. | +| Parameter | Type | Default | Description | +|--------------------------|--------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------| +| name | TEXT | Required | Name of the knowledge base | +| model_name | TEXT | Required | Name of the model | +| source_volume_name | TEXT | Required | Name of the volume | +| vector_table | TEXT | NULL | Name of the vector table | +| vector_data_column | TEXT | 'embeddings' | Name of the vector column | +| vector_key_column | TEXT | 'id' | Name of the key column | +| topk | INTEGER | 1 | Number of results to return | +| distance_operator | aidb.distanceoperator | 'L2' | Distance operator | +| options | JSONB | '{}'::JSONB | Options | +| auto_processing | [aidb.PipelineAutoProcessingMode](#aidbpipelineautoprocessingmode) | 'Disabled' | Configure auto-processing for this pipeline. | +| index_type | TEXT | 'vector' | Type of index to use for the vector table. | +| batch_size | int | 100 | How many records to process concurrently. | +| background_sync_interval | interval | '30 seconds' | Interval between pipeline executions if background auto-processing is configured. | !!! note "Index_types" * If `index_type` is set to `vector`, the system will automatically create a [hnsw](https://github.com/pgvector/pgvector?tab=readme-ov-file#hnsw) index on the vector table based on the distance operator used in the knowledge base. @@ -196,14 +223,14 @@ SELECT aidb.create_volume_knowledge_base( ### `aidb.set_auto_knowledge_base` -Sets the processing mode for this Knowledge Base. This function is used to enable and disable auto-embedding: `Live` mode enables auto-embedding and `Disabled` disables it. +Sets the auto-processing mode for this Knowledge Base. #### Parameters -| Parameter | Type | Default | Description | -|---------------------------------|--------------------------------|--------------|------------------------------------------------------------| -| knowledge_base_name | TEXT | | Name of knowledge base for which to enable auto-embedding | -| mode | aidb.PipelineProcessingMode | | Desired processing mode | +| Parameter | Type | Default | Description | +|---------------------|-------------------------------------------------------------------------------------|---------|-----------------------------------------------------------| +| knowledge_base_name | TEXT | | Name of knowledge base for which to enable auto-embedding | +| mode | [aidb.PipelineAutoProcessingMode](./knowledge_bases#aidbpipelineautoprocessingmode) | | Desired auto-processing mode | #### Example @@ -214,25 +241,30 @@ SELECT aidb.set_auto_knowledge_base('test_knowledge_base', 'Disabled'); ### `aidb.bulk_embedding` +Compute the embeddings for the source records in this pipeline and store them in vector destination. -Generates embeddings for all data in a given table if there's existing data in the table. +The behavior of this function depends on the configured auto_processing mode: +* **Live and Disabled auto-processing:** Directly process all source records. +* **Background auto-processing:** Mark all source records for processing but don't perform the operation. The background worker will perform it. #### Parameters -| Parameter | Type | Default | Description | -|---------------------------------|--------------------------------|--------------|-----------------------------------------------------------------| -| knowledge_base_name | TEXT | | Name of knowledge base for which to generate embeddings| +| Parameter | Type | Default | Description | +|---------------------|------|---------|----------------------------------------------------------| +| knowledge_base_name | TEXT | | Name of knowledge base for which to generate embeddings. | +| silent | BOOL | false | Disable printing status and progress logs. | #### Example ``` -edb=# select aidb.bulk_embedding('test_knowledge_base'); +aidb=# SELECT aidb.bulk_embedding('kb_volume_image_manual'); __OUTPUT__ -INFO: bulk_embedding_text found 3 rows in knowledge base test_knowledge_base - bulk_embedding ----------------- - -(1 row) +INFO: kb_volume_image_manual: (re)setting state table to process all data... +INFO: kb_volume_image_manual: Starting... Batch size 10, count(known source records): 0, scans completed: 0, count(embeddings): 0 +INFO: kb_volume_image_manual: Batch iteration finished, count(known source records): 10, scans completed: 0, count(embeddings): 10 +[...] +INFO: kb_volume_image_manual: Batch iteration finished, count(known source records): 177, scans completed: 0, count(embeddings): 177 +INFO: kb_volume_image_manual: finished, count(known source records): 177, scans completed: 1, count(embeddings): 177 ``` ### `aidb.retrieve_key` diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/models.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/models.mdx index 267ab1ede7b..f6723dc2e88 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/models.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/models.mdx @@ -185,8 +185,8 @@ Decodes text using a model, generating a vector representation of a given text i #### Returns -| Column | Type | Description | -|--------------|------|-------------------| +| Column | Type | Description | +|---------------|------|------------------| | `decode_text` | text | The decoded text | ### `aidb.decode_text_batch` @@ -195,15 +195,15 @@ Decodes a batch of text using a model, generating a representation of a given te #### Parameters -| Parameter | Type | Default | Description | -|--------------|------|---------|-----------------------------------| -| `model_name` | text | | Name of the model to decode with | -| `text` | text\[\] | | Array of text to decode | +| Parameter | Type | Default | Description | +|--------------|----------|---------|----------------------------------| +| `model_name` | text | | Name of the model to decode with | +| `text` | text\[\] | | Array of text to decode | #### Returns -| Column | Type | Description | -|--------------|------|-------------------| +| Column | Type | Description | +|---------------|------|------------------| | `decode_text` | text | The decoded text | ### `aidb.encode_image` @@ -212,15 +212,15 @@ Encodes an image using a model, generating a vector representation of a given im #### Parameters -| Parameter | Type | Default | Description | -|--------------|------|---------|-----------------------------------| -| `model_name` | text | | Name of the model to encode with | +| Parameter | Type | Default | Description | +|--------------|-------|---------|----------------------------------| +| `model_name` | text | | Name of the model to encode with | | `image` | bytea | | Image to encode | #### Returns -| Column | Type | Description | -|--------------|------|-------------------| +| Column | Type | Description | +|----------------|-------|-------------------| | `encode_image` | bytea | The encoded image | ### `aidb.rerank_text` @@ -229,15 +229,18 @@ Reranks text using a model, generating a vector representation of a given text i #### Parameters -| Parameter | Type | Default | Description | -|--------------|------------------|---------|-----------------------------------| -| `model_name` | text | | Name of the model to rerank with | -| `text` | text | | Text to rerank | -| `inputs` | strings\[\] | \[\] | Inputs for the model provider | +| Parameter | Type | Default | Description | +|--------------|----------|---------|-----------------------------------------------| +| `model_name` | text | | Name of the model to rerank with | +| `query` | text | | Query based on which the input will be ranked | +| `input` | text\[\] | \[\] | Inputs to be ranked | #### Returns -| Column | Type | Description | -|---------------|----------------------|----------------------------| -| `rerank_text` | double precision\[\] | Array of reranking logits | +| Column | Type | Description | +|---------------|------------------|--------------------------------------------| +| `text` | text | The text from "input" | +| `logit_score` | double precision | Score/rank of this text | +| `id` | int | index that the text had in the input array | + diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/preparers.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/preparers.mdx index a66ad598aa4..8b639f08f2e 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/preparers.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/preparers.mdx @@ -101,10 +101,10 @@ Sets the automatic processing mode for this preparer. This function is used to e #### Parameters -| Parameter | Type | Default | Description | -|---------------------------------|-------------------------------------------------------------------------|--------------|--------------------------| -| preparer_name | TEXT | | Name of preparer | -| mode | [aidb.PipelineProcessingMode](./knowledge_bases#aidbpipelineprocessingmode) | | Desired processing mode | +| Parameter | Type | Default | Description | +|---------------|-------------------------------------------------------------------------------------|---------|-------------------------| +| preparer_name | TEXT | | Name of preparer | +| mode | [aidb.PipelineAutoProcessingMode](./knowledge_bases#aidbpipelineautoprocessingmode) | | Desired processing mode | #### Example diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/retrievers.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/retrievers.mdx index b7ce5c36dcc..b4a6041a4f8 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/retrievers.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/reference/retrievers.mdx @@ -36,7 +36,7 @@ Creates a retriever for a given table. | vector_data_column | TEXT | 'embeddings' | | | vector_key_column | TEXT | 'id' | | | topk | INTEGER | 1 | | -| distance_operator | [aidb.distanceoperator](#aidbdistanceoperator) | 'L2' | | +| distance_operator | [aidb.distanceoperator](knowledge_bases##aidbdistanceoperator) | 'L2' | | | options | JSONB | '{}'::JSONB | Options | | index_type | TEXT | 'vector' | Type of index to use for the vector table. | @@ -117,14 +117,14 @@ SELECT aidb.create_retriever_for_volume( **Replaced by** [`aidb.set_auto_knowledge_base`](knowledge_bases#aidbset_auto_knowledge_base) -Sets the processing mode for this Retriever. This function is used to enable and disable auto-embedding: `Live` mode enables auto-embedding and `Disabled` disables it. +Sets the auto-processing mode for this knowledge base. #### Parameters | Parameter | Type | Default | Description | |---------------------------------|--------------------------------|--------------|------------------------------------------------------------| | retriever_name | TEXT | | Name of retriever for which to enable auto-embedding | -| mode | aidb.PipelineProcessingMode | | Desired processing mode | +| mode | aidb.PipelineAutoProcessingMode | | Desired auto-processing mode | #### Example diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/ai-accelerator_4.0.0_rel_notes.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/ai-accelerator_4.0.0_rel_notes.mdx index b6ffe1b8170..b24688434f8 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/ai-accelerator_4.0.0_rel_notes.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/ai-accelerator_4.0.0_rel_notes.mdx @@ -7,12 +7,16 @@ editTarget: originalFilePath Released: 1 May 2025 -In this release, we are renaming retrievers to knowledge bases and renaming preparer functions for consistency. +In this release, we introduce a completely re-written processing back-end for the knowledge base pipeline. +TODO +we are renaming retrievers to knowledge bases and renaming preparer functions for consistency. ## Highlights +- New processing back-end for the knowledge base pipeline supporting: background processing, batch processing, full iterative sync for tables and volumes. - Renaming of retrievers to knowledge bases. - Renaming of preparer functions for consistency. +- Renaming of "processing mode" in pipelines config to "auto processing" for consistency. - Added arm64 support for the AI Accelerator on Debian 12 and RHEL 9. ## Enhancements @@ -22,11 +26,23 @@ In this release, we are renaming retrievers to knowledge bases and renaming prep

The functions aidb.create_retriever_for_table() and aidb.create_retriever_for_volume() have been renamed to aidb.create_knowledge_base_for_table() and aidb.create_knowledge_base_for_volume(), respectively.

Also renamed is the aidb.delete_retriever function (to aidb.delete_knowledge_base()) and the aidb.RetrieverSourceDataFormat enum has been renamed to aidb.PipelineDataFormat.

For consistency, the preparer pipeline functions have also been renamed. The aidb.create_preparer_for_table() function has been renamed to aidb.create_table_preparer() and aidb.create_preparer_for_volume() has been renamed to aidb.create_volume_preparer().

-

The aidb.retrivers view is now the aidb.knowledge_bases view.Aliases for the aidb.knowledge_bases and aidb.preparers views have also been added as aidb.kbs and aidb.preps respectively.

+

The aidb.retrievers view is now the aidb.knowledge_bases view. Aliases for the aidb.knowledge_bases and aidb.preparers views have also been added as aidb.kbs and aidb.preps respectively.

The older functions and view names are still available but marked as deprecated. They will be removed in a future release.

+
Renamed "processing mode" in pipelines config to "auto processing" for consistency

The setting for "auto processing" in pipeline creation, config and overview functions has been re-named from processing_mode to auto_processing. +We did this to have consistency with the name of the feature itself which is called auto-processing.

+
Added arm64 support for the AI Accelerator on Debian 12 and RHEL 9.

The AI Accelerator now supports arm64 architecture on Debian 12 and RHEL 9. This includes support for the arm64 architecture in the aidb packages.

+
The aidb.rerank_text() function now returns results as a rich table

The aidb.rerank_text() function used to rerank a list of texts based on a ranking query, now returns its results in a table, one row per text. The table includes columns for the raw logit_score, the text itself and the index from the input. +This makes it much easier to use the result in larger SQL query e.g. by joining with other data or sorting the rows by logit_score.

+
+
Improved debugging capabilities for AI models accessed over HTTP APIs

AIDB now prints rich debugging output for any errors encountered when using external models via API. This includes NIM models and OpenAI API compatible models. +The HTTP status code, error response content, and request content are logged to help users debug issues like incorrect endpoints, incorrect model configuration or incorrect credentials, among others.

+
+
Improved debugging capabilities for AI models accessed over HTTP APIs

AIDB now prints rich debugging output for any errors encountered when using external models via API. This includes NIM models and OpenAI API compatible models. +The HTTP status code, error response content, and request content are logged to help users debug issues like incorrect endpoints, incorrect model configuration or incorrect credentials, among others.

+
diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/src/rel_notes_4.0.0.yml b/advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/src/rel_notes_4.0.0.yml index 1fc11265c03..b5ba9ce5aae 100644 --- a/advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/src/rel_notes_4.0.0.yml +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/src/rel_notes_4.0.0.yml @@ -3,29 +3,41 @@ product: AI Accelerator - Pipelines version: 4.0.0 date: 1 May 2025 intro: | - In this release, we are renaming retrievers to knowledge bases and renaming preparer functions for consistency. + In this release, we introduce a completely re-written processing back-end for the knowledge base pipeline. + TODO + we are renaming retrievers to knowledge bases and renaming preparer functions for consistency. highlights: | + - New processing back-end for the knowledge base pipeline supporting: background processing, batch processing, full iterative sync for tables and volumes. - Renaming of retrievers to knowledge bases. - Renaming of preparer functions for consistency. + - Renaming of "processing mode" in pipelines config to "auto processing" for consistency. - Added arm64 support for the AI Accelerator on Debian 12 and RHEL 9. relnotes: - relnote: Renamed retrievers to knowledge bases and renamed preparer functions. details: | - The retriever pipeline has been renamed to the knowledge base pipeline. - - The functions `aidb.create_retriever_for_table()` and `aidb.create_retriever_for_volume()` have been renamed to `aidb.create_knowledge_base_for_table()` and `aidb.create_knowledge_base_for_volume()`, respectively. - + The retriever pipeline has been renamed to the knowledge base pipeline. + + The functions `aidb.create_retriever_for_table()` and `aidb.create_retriever_for_volume()` have been renamed to `aidb.create_knowledge_base_for_table()` and `aidb.create_knowledge_base_for_volume()`, respectively. + Also renamed is the `aidb.delete_retriever` function (to `aidb.delete_knowledge_base()`) and the `aidb.RetrieverSourceDataFormat` enum has been renamed to `aidb.PipelineDataFormat`. For consistency, the preparer pipeline functions have also been renamed. The `aidb.create_preparer_for_table()` function has been renamed to `aidb.create_table_preparer()` and `aidb.create_preparer_for_volume()` has been renamed to `aidb.create_volume_preparer()`. - - The `aidb.retrivers` view is now the `aidb.knowledge_bases` view.Aliases for the `aidb.knowledge_bases` and `aidb.preparers` views have also been added as `aidb.kbs` and `aidb.preps` respectively. - + + The `aidb.retrievers` view is now the `aidb.knowledge_bases` view. Aliases for the `aidb.knowledge_bases` and `aidb.preparers` views have also been added as `aidb.kbs` and `aidb.preps` respectively. + The older functions and view names are still available but marked as deprecated. They will be removed in a future release. jira: "AID-377" addresses: "" type: Enhancement impact: High +- relnote: Renamed "processing mode" in pipelines config to "auto processing" for consistency + details: | + The setting for "auto processing" in pipeline creation, config and overview functions has been re-named from `processing_mode` to `auto_processing`. + We did this to have consistency with the name of the feature itself which is called auto-processing. + jira: "" + addresses: "" + type: Enhancement + impact: High - relnote: Added arm64 support for the AI Accelerator on Debian 12 and RHEL 9. details: | The AI Accelerator now supports arm64 architecture on Debian 12 and RHEL 9. This includes support for the arm64 architecture in the `aidb` packages. @@ -33,3 +45,32 @@ relnotes: addresses: "" type: Enhancement impact: Medium + +- relnote: The aidb.rerank_text() function now returns results as a rich table + details: | + The `aidb.rerank_text()` function used to rerank a list of texts based on a ranking query, now returns its results in a table, one row per text. The table includes columns for the raw logit_score, the text itself and the index from the input. + This makes it much easier to use the result in larger SQL query e.g. by joining with other data or sorting the rows by logit_score. + jira: "AID-404" + addresses: "" + type: Enhancement + impact: Medium + +- relnote: Improved debugging capabilities for AI models accessed over HTTP APIs + details: | + AIDB now prints rich debugging output for any errors encountered when using external models via API. This includes NIM models and OpenAI API compatible models. + The HTTP status code, error response content, and request content are logged to help users debug issues like incorrect endpoints, incorrect model configuration or incorrect credentials, among others. + jira: "AID-322" + addresses: "" + type: Enhancement + impact: Medium + + +- relnote: Improved debugging capabilities for AI models accessed over HTTP APIs + details: | + AIDB now prints rich debugging output for any errors encountered when using external models via API. This includes NIM models and OpenAI API compatible models. + The HTTP status code, error response content, and request content are logged to help users debug issues like incorrect endpoints, incorrect model configuration or incorrect credentials, among others. + jira: "AID-322" + addresses: "" + type: Enhancement + impact: Medium + diff --git a/advocacy_docs/edb-postgres-ai/ai-accelerator/volumes.mdx b/advocacy_docs/edb-postgres-ai/ai-accelerator/volumes.mdx new file mode 100644 index 00000000000..c4cb002fbd4 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/ai-accelerator/volumes.mdx @@ -0,0 +1,68 @@ +--- +title: AIDB volumes for accessing PGFS storage locations +navTitle: Volumes +description: AIDB volumes for accessing PGFS storage locations. +--- + +## Overview +Volumes in AIDB allow you to use external storage, instead of a postgres table, as the data source for pipelines. + +Unstructured data like PDFs or images is often stored in object stores like AWS S3. With AIDB volumes, you can integrate such data into your AI pipelines. + +### PGFS storage locations and AIDB volumes +AIDB volumes allow access to PGFS storage locations. This means a storage location in PGFS must be created first. See [PGFS](pgfs). +* **PGFS storage location:** Represents an external storage provider with all necessary configuration like a path and credentials. +* **AIDB volume:** Connects a storage location to AIDB. Multiple volumes for the same storage location are possible. In a volume, you configure the data format in the destination and optionally, a sub-path. + +## Creating a volume +Assign a name to the new volume and specify which PGFS storage location it should point to. + +The third argument is an optional sub-path in the storage location. This allows you to create multiple volumes in the same storage location, pointing to different paths within a bucket. + +The last argument is the data format, the supported types are [`aidb.PipelineDataFormat`](reference/knowledge_bases#aidbpipelinedataformat). This is metadata used by pipelines to decide how to treat the objects in the volume. The data format is not a filter. You should ensure that the volume only contains objects of the specified format. + +```sql +SELECT aidb.create_volume('volume_name', 'pgfs_storage_location_name', '/', 'Image'); +``` + +## Other management functions +The following management functions allow you to list and delete volumes: + +```sql +SELECT aidb.list_volumes(); +``` + +```sql +SELECT aidb.delete_volume('volume_name'); +``` + +_Note: deleting a PGFS storage location will delete all volumes created on top of it._ + + + +## Accessing volumes +Volumes can be accessed in two different ways: +1. By using them within a pipeline as the data source +2. With direct access SQL functions + +The direct access SQL functions are useful to test newly created storage locations and volumes before configuring them in a pipeline. +They can also be used to build custom SQL queries that work with external storage. + +### Listing content +This command will return a table listing all the objects present in the volume. + +```sql +SELECT * FROM aidb.list_volume_content('volume_name'); +``` + +### Reading an object +This command will read the contents of an object and return them as a BYTEA type. +```sql +SELECT aidb.read_volume_file('volume_name', 'object_name.txt'); +``` + +As an example, if your objects contain plain text, they can be converted like this: + +```sql +SELECT convert_from(aidb.read_volume_file('my_text_files', 'hello_world.txt'), 'utf8'); +```