diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 580fc92..50b78f0 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -1,8 +1,8 @@ site_name: LanceDB Documentation -site_url: https://lancedb.github.io/docs -repo_url: https://github.com/lancedb/documentation -edit_uri: https://github.com/lancedb/documentation -repo_name: lancedb/documentation +site_url: https://lancedb.com/documentation +repo_url: https://github.com/lancedb/lancedb +edit_uri: https://github.com/lancedb/lancedb +repo_name: lancedb/lancedb docs_dir: src watch: - src @@ -64,7 +64,7 @@ plugins: 'guides/scalar_index.md': 'guides/indexing/scalar-index.md' 'cloud/rest.md': 'api/cloud.md' 'api_reference.md': 'api/index.md' - 'migration.md': 'api/api-changes.md' + 'migration.md': 'api/cloud.md' 'faq.md': 'overview/faq.md' 'cloud/cloud_faq.md': 'cloud/cloud-faq.md' 'ann_indexes.md': 'guides/indexing/vector-index.md' @@ -353,16 +353,30 @@ nav: - HyDE: rag/advanced_techniques/hyde.md - FLARE: rag/advanced_techniques/flare.md - - API & SDK Reference: + - API Reference: - Overview: api/index.md - - API Changes: api/api-changes.md - LanceDB Cloud REST API: api/cloud.md - - Client SDKs: - - Python SDK: python/python.md - - JavaScript SDK: - - Current SDK (@lancedb/lancedb): js/globals.md - - Legacy SDK (vectordb): javascript/modules.md - - Rust SDK: https://docs.rs/lancedb/latest/lancedb/ + - Python SDK: + - Overview: python/index.md + - Connections: + - Sync: python/sync/connections.md + - Async: python/async/connections.md + - Tables & Data: + - Sync: python/sync/tables.md + - Async: python/async/tables.md + - Pydantic Data Models: python/integrations/pydantic.md + - Search & Querying: + - Sync: python/sync/querying.md + - Async: python/async/querying.md + - Indexes: python/features/indices.md + - AI & ML Features: + - Embeddings: python/integrations/embeddings.md + - Reranking: python/integrations/reranking.md + - Context: python/features/context.md + - LanceDBCloud: python/saas/index.md + - JavaScript SDK: + - Overview: js/index.md + - Rust SDK: https://docs.rs/lancedb/latest/lancedb/ - Changelog: - LanceDB Cloud & Enterprise: changelog/lancedb-changelog.md - ↗LanceDB Open Source: https://github.com/lancedb/lancedb/releases diff --git a/docs/overrides/main.html b/docs/overrides/main.html index 17a50df..df0ee0b 100644 --- a/docs/overrides/main.html +++ b/docs/overrides/main.html @@ -1,5 +1,5 @@ {% extends "base.html" %} {% block announce %} -
June 1st, 2025: We just launched LanceDB Cloud - Sign up and access LanceDB for free!
+
We just launched LanceDB Cloud Beta - Sign up for early access!
{% endblock %} \ No newline at end of file diff --git a/docs/src/api/api-changes.md b/docs/src/api/api-changes.md deleted file mode 100644 index 8f2dd7b..0000000 --- a/docs/src/api/api-changes.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: Migrating to LanceDB | Data Migration Guide -description: Learn how to migrate your data to LanceDB from other vector databases and formats. Includes step-by-step guides, compatibility considerations, and best practices. ---- - -# **Rust-Backed Client Migration Guide** - -In an effort to ensure all clients have the same set of capabilities we have -migrated the Python and Node clients onto a common Rust base library. In Python, -both the synchronous and asynchronous clients are based on this implementation. -In Node, the new client is available as `@lancedb/lancedb`, which replaces -the existing `vectordb` package. - -This guide describes the differences between the two Node APIs and will hopefully assist users -that would like to migrate to the new API. - -## TypeScript/JavaScript - -For JS/TS users, we offer a brand new SDK [@lancedb/lancedb](https://www.npmjs.com/package/@lancedb/lancedb) - -We tried to keep the API as similar as possible to the previous version, but there are a few small changes. Here are the most important ones: - -### Creating Tables - -[CreateTableOptions.writeOptions.writeMode](./javascript/interfaces/WriteOptions.md#writemode) has been replaced with [CreateTableOptions.mode](./js/interfaces/CreateTableOptions.md#mode) - -=== "vectordb (deprecated)" - - ```ts - db.createTable(tableName, data, { writeMode: lancedb.WriteMode.Overwrite }); - ``` - -=== "@lancedb/lancedb" - - ```ts - db.createTable(tableName, data, { mode: "overwrite" }) - ``` - -### Changes to Table APIs - -Previously `Table.schema` was a property. Now it is an async method. - -#### Creating Indices - -The `Table.createIndex` method is now used for creating both vector indices -and scalar indices. It currently requires a column name to be specified (the -column to index). Vector index defaults are now smarter and scale better with -the size of the data. - -=== "vectordb (deprecated)" - - ```ts - await tbl.createIndex({ - column: "vector", // default - type: "ivf_pq", - num_partitions: 2, - num_sub_vectors: 2, - }); - ``` - -=== "@lancedb/lancedb" - - ```ts - await table.createIndex("vector", { - config: lancedb.Index.ivfPq({ - numPartitions: 2, - numSubVectors: 2, - }), - }); - ``` - -### Embedding Functions - -The embedding API has been completely reworked, and it now more closely resembles the Python API, including the new [embedding registry](./js/classes/embedding.EmbeddingFunctionRegistry.md): - -=== "vectordb (deprecated)" - - ```ts - - const embeddingFunction = new lancedb.OpenAIEmbeddingFunction('text', API_KEY) - const data = [ - { id: 1, text: 'Black T-Shirt', price: 10 }, - { id: 2, text: 'Leather Jacket', price: 50 } - ] - const table = await db.createTable('vectors', data, embeddingFunction) - ``` - -=== "@lancedb/lancedb" - - ```ts - import * as lancedb from "@lancedb/lancedb"; - import * as arrow from "apache-arrow"; - import { LanceSchema, getRegistry } from "@lancedb/lancedb/embedding"; - - const func = getRegistry().get("openai").create({apiKey: API_KEY}); - - const data = [ - { id: 1, text: 'Black T-Shirt', price: 10 }, - { id: 2, text: 'Leather Jacket', price: 50 } - ] - - const table = await db.createTable('vectors', data, { - embeddingFunction: { - sourceColumn: "text", - function: func, - } - }) - - ``` - -You can also use a schema driven approach, which parallels the Pydantic integration in our Python SDK: - -```ts -const func = getRegistry().get("openai").create({apiKey: API_KEY}); - -const data = [ - { id: 1, text: 'Black T-Shirt', price: 10 }, - { id: 2, text: 'Leather Jacket', price: 50 } -] -const schema = LanceSchema({ - id: new arrow.Int32(), - text: func.sourceField(new arrow.Utf8()), - price: new arrow.Float64(), - vector: func.vectorField() -}) - -const table = await db.createTable('vectors', data, {schema}) - -``` diff --git a/docs/src/api/cloud.md b/docs/src/api/cloud.md index dd55184..9cba12a 100644 --- a/docs/src/api/cloud.md +++ b/docs/src/api/cloud.md @@ -1,11 +1,7 @@ --- title: LanceDB Cloud REST API Specification -hide: - - navigation --- -[Return to the previous page.](index.md) - [OAD(../docs/openapi.yml)] diff --git a/docs/src/js/globals.md b/docs/src/js/index.md similarity index 100% rename from docs/src/js/globals.md rename to docs/src/js/index.md diff --git a/docs/src/python/async/connections.md b/docs/src/python/async/connections.md new file mode 100644 index 0000000..867df4b --- /dev/null +++ b/docs/src/python/async/connections.md @@ -0,0 +1,7 @@ +# Connections (Asynchronous) + +Connections represent a connection to a LanceDb database and can be used to create, list, or open tables. + +::: lancedb.connect_async + +::: lancedb.db.AsyncConnection diff --git a/docs/src/python/async/querying.md b/docs/src/python/async/querying.md new file mode 100644 index 0000000..5e06dd4 --- /dev/null +++ b/docs/src/python/async/querying.md @@ -0,0 +1,15 @@ +# Querying (Asynchronous) + +Queries allow you to return data from your database. Basic queries can be created with the [AsyncTable.query][lancedb.table.AsyncTable.query] method to return the entire (typically filtered) table. Vector searches return the rows nearest to a query vector and can be created with the [AsyncTable.vector_search][lancedb.table.AsyncTable.vector_search] method. + +::: lancedb.query.AsyncQuery + options: + inherited_members: true + +::: lancedb.query.AsyncVectorQuery + options: + inherited_members: true + +::: lancedb.query.AsyncHybridQuery + options: + inherited_members: true diff --git a/docs/src/python/async/tables.md b/docs/src/python/async/tables.md new file mode 100644 index 0000000..70f9ff9 --- /dev/null +++ b/docs/src/python/async/tables.md @@ -0,0 +1,5 @@ +# Tables (Asynchronous) + +Tables hold your actual data as a collection of records / rows. + +::: lancedb.table.AsyncTable diff --git a/docs/src/python/features/context.md b/docs/src/python/features/context.md new file mode 100644 index 0000000..d1c3e5b --- /dev/null +++ b/docs/src/python/features/context.md @@ -0,0 +1,5 @@ +# Context + +::: lancedb.context.contextualize + +::: lancedb.context.Contextualizer diff --git a/docs/src/python/features/fts.md b/docs/src/python/features/fts.md new file mode 100644 index 0000000..91dea52 --- /dev/null +++ b/docs/src/python/features/fts.md @@ -0,0 +1,13 @@ +# Full Text Search + +This section covers the core Full Text Search (FTS) functionality in LanceDB. For building FTS queries in your application, see the [Synchronous FTS Query Builder](../sync/querying.md#lancedb.query.LanceFtsQueryBuilder) or [Asynchronous FTS Query Builder](../async/querying.md). + +## Core FTS Functions + +These functions provide the fundamental FTS capabilities: + +::: lancedb.fts.create_index + +::: lancedb.fts.populate_index + +::: lancedb.fts.search_index diff --git a/docs/src/python/features/indices.md b/docs/src/python/features/indices.md new file mode 100644 index 0000000..196b6c9 --- /dev/null +++ b/docs/src/python/features/indices.md @@ -0,0 +1,19 @@ +# Indexes + +Indices can be created on a table to speed up queries. This section lists the indices that LanceDb supports. + +::: lancedb.index.BTree + +::: lancedb.index.Bitmap + +::: lancedb.index.LabelList + +::: lancedb.index.FTS + +::: lancedb.index.IvfPq + +::: lancedb.index.HnswPq + +::: lancedb.index.HnswSq + +::: lancedb.index.IvfFlat diff --git a/docs/src/python/index.md b/docs/src/python/index.md new file mode 100644 index 0000000..6e33cab --- /dev/null +++ b/docs/src/python/index.md @@ -0,0 +1,47 @@ +# Python API Reference + +This section contains the API reference for the Python API. LanceDB provides both synchronous and asynchronous API clients. + +## Installation + +```shell +pip install lancedb +``` + +## Quick Start + +The general flow of using the API is: + +1. Use [lancedb.connect][] or [lancedb.connect_async][] to connect to a database. +2. Use the returned [lancedb.DBConnection][] or [lancedb.AsyncConnection][] to create or open tables. +3. Use the returned [lancedb.table.Table][] or [lancedb.AsyncTable][] to query or modify tables. + +## API Reference + +### Synchronous API +- [Connections](sync/connections.md) +- [Tables](sync/tables.md) +- [Querying](sync/querying.md) + +### Asynchronous API +- [Connections](async/connections.md) +- [Tables](async/tables.md) +- [Querying](async/querying.md) + +### Features +- [Embeddings](features/embeddings.md) +- [Context](features/context.md) +- [Full Text Search](features/fts.md) +- [Indices](features/indices.md) + +### Integrations +- [Pydantic](integrations/pydantic.md) +- [Reranking](integrations/reranking.md) + +### SaaS Integration +For information about using LanceDB with SaaS services, see the [SaaS Integration Guide](../saas/index.md). + +## Common Issues + +Multiprocessing with `fork` is not supported. You should use `spawn` instead. + diff --git a/docs/src/python/integrations/embeddings.md b/docs/src/python/integrations/embeddings.md new file mode 100644 index 0000000..93dcd45 --- /dev/null +++ b/docs/src/python/integrations/embeddings.md @@ -0,0 +1,15 @@ +# Embeddings + +::: lancedb.embeddings.registry.EmbeddingFunctionRegistry + +::: lancedb.embeddings.base.EmbeddingFunctionConfig + +::: lancedb.embeddings.base.EmbeddingFunction + +::: lancedb.embeddings.base.TextEmbeddingFunction + +::: lancedb.embeddings.sentence_transformers.SentenceTransformerEmbeddings + +::: lancedb.embeddings.openai.OpenAIEmbeddings + +::: lancedb.embeddings.open_clip.OpenClipEmbeddings diff --git a/docs/src/python/integrations/pydantic.md b/docs/src/python/integrations/pydantic.md new file mode 100644 index 0000000..86ce320 --- /dev/null +++ b/docs/src/python/integrations/pydantic.md @@ -0,0 +1,7 @@ +# Pydantic Integration + +::: lancedb.pydantic.pydantic_to_schema + +::: lancedb.pydantic.vector + +::: lancedb.pydantic.LanceModel diff --git a/docs/src/python/integrations/reranking.md b/docs/src/python/integrations/reranking.md new file mode 100644 index 0000000..c9bf139 --- /dev/null +++ b/docs/src/python/integrations/reranking.md @@ -0,0 +1,11 @@ +# Reranking + +::: lancedb.rerankers.linear_combination.LinearCombinationReranker + +::: lancedb.rerankers.cohere.CohereReranker + +::: lancedb.rerankers.colbert.ColbertReranker + +::: lancedb.rerankers.cross_encoder.CrossEncoderReranker + +::: lancedb.rerankers.openai.OpenaiReranker diff --git a/docs/src/python/python.md b/docs/src/python/python.md deleted file mode 100644 index 8af43f2..0000000 --- a/docs/src/python/python.md +++ /dev/null @@ -1,168 +0,0 @@ -# Python API Reference - -This section contains the API reference for the Python API. There is a -synchronous and an asynchronous API client. - -The general flow of using the API is: - -1. Use [lancedb.connect][] or [lancedb.connect_async][] to connect to a database. -2. Use the returned [lancedb.DBConnection][] or [lancedb.AsyncConnection][] to - create or open tables. -3. Use the returned [lancedb.table.Table][] or [lancedb.AsyncTable][] to query - or modify tables. - - -## Installation - -```shell -pip install lancedb -``` - -The following methods describe the synchronous API client. There -is also an [asynchronous API client](#connections-asynchronous). - -## Connections (Synchronous) - -::: lancedb.connect - -::: lancedb.db.DBConnection - -## Tables (Synchronous) - -::: lancedb.table.Table - -## Querying (Synchronous) - -::: lancedb.query.Query - -::: lancedb.query.LanceQueryBuilder - -::: lancedb.query.LanceVectorQueryBuilder - -::: lancedb.query.LanceFtsQueryBuilder - -::: lancedb.query.LanceHybridQueryBuilder - -## Embeddings - -::: lancedb.embeddings.registry.EmbeddingFunctionRegistry - -::: lancedb.embeddings.base.EmbeddingFunctionConfig - -::: lancedb.embeddings.base.EmbeddingFunction - -::: lancedb.embeddings.base.TextEmbeddingFunction - -::: lancedb.embeddings.sentence_transformers.SentenceTransformerEmbeddings - -::: lancedb.embeddings.openai.OpenAIEmbeddings - -::: lancedb.embeddings.open_clip.OpenClipEmbeddings - -## Context - -::: lancedb.context.contextualize - -::: lancedb.context.Contextualizer - -## Full text search - -::: lancedb.fts.create_index - -::: lancedb.fts.populate_index - -::: lancedb.fts.search_index - -## Utilities - -::: lancedb.schema.vector - -::: lancedb.merge.LanceMergeInsertBuilder - -## Integrations - -## Pydantic - -::: lancedb.pydantic.pydantic_to_schema - -::: lancedb.pydantic.vector - -::: lancedb.pydantic.LanceModel - -## Reranking - -::: lancedb.rerankers.linear_combination.LinearCombinationReranker - -::: lancedb.rerankers.cohere.CohereReranker - -::: lancedb.rerankers.colbert.ColbertReranker - -::: lancedb.rerankers.cross_encoder.CrossEncoderReranker - -::: lancedb.rerankers.openai.OpenaiReranker - -## Connections (Asynchronous) - -Connections represent a connection to a LanceDb database and -can be used to create, list, or open tables. - -::: lancedb.connect_async - -::: lancedb.db.AsyncConnection - -## Tables (Asynchronous) - -Table hold your actual data as a collection of records / rows. - -::: lancedb.table.AsyncTable - -## Indices (Asynchronous) - -Indices can be created on a table to speed up queries. This section -lists the indices that LanceDb supports. - -::: lancedb.index.BTree - -::: lancedb.index.Bitmap - -::: lancedb.index.LabelList - -::: lancedb.index.FTS - -::: lancedb.index.IvfPq - -::: lancedb.index.HnswPq - -::: lancedb.index.HnswSq - -::: lancedb.index.IvfFlat - -## Querying (Asynchronous) - -Queries allow you to return data from your database. Basic queries can be -created with the [AsyncTable.query][lancedb.table.AsyncTable.query] method -to return the entire (typically filtered) table. Vector searches return the -rows nearest to a query vector and can be created with the -[AsyncTable.vector_search][lancedb.table.AsyncTable.vector_search] method. - - -::: lancedb.query.AsyncQuery - options: - inherited_members: true - -::: lancedb.query.AsyncVectorQuery - options: - inherited_members: true - -::: lancedb.query.AsyncFTSQuery - options: - inherited_members: true - -::: lancedb.query.AsyncHybridQuery - options: - inherited_members: true - -## Common issues - -Multiprocessing with `fork` is not supported. You should use `spawn` instead. - diff --git a/docs/src/python/saas-python.md b/docs/src/python/saas/index.md similarity index 100% rename from docs/src/python/saas-python.md rename to docs/src/python/saas/index.md diff --git a/docs/src/python/sync/connections.md b/docs/src/python/sync/connections.md new file mode 100644 index 0000000..651ac43 --- /dev/null +++ b/docs/src/python/sync/connections.md @@ -0,0 +1,7 @@ +# Connections (Synchronous) + +Connections represent a connection to a LanceDb database and can be used to create, list, or open tables. + +::: lancedb.connect + +::: lancedb.db.DBConnection diff --git a/docs/src/python/sync/querying.md b/docs/src/python/sync/querying.md new file mode 100644 index 0000000..7506c46 --- /dev/null +++ b/docs/src/python/sync/querying.md @@ -0,0 +1,18 @@ +# Querying (Synchronous) + +Queries allow you to return data from your database. Basic queries can be created with the [Table.query][lancedb.table.Table.query] method to return the entire (typically filtered) table. Vector searches return the rows nearest to a query vector and can be created with the [Table.vector_search][lancedb.table.Table.vector_search] method. + +::: lancedb.query.Query + +::: lancedb.query.LanceQueryBuilder + +::: lancedb.query.LanceVectorQueryBuilder + +::: lancedb.query.LanceFtsQueryBuilder + options: + inherited_members: true + description: | + A query builder for Full Text Search operations. This provides a fluent interface for building FTS queries. + For the core FTS functionality (creating indices, etc.), see the [Full Text Search](../features/fts.md) section. + +::: lancedb.query.LanceHybridQueryBuilder diff --git a/docs/src/python/sync/tables.md b/docs/src/python/sync/tables.md new file mode 100644 index 0000000..5c1793d --- /dev/null +++ b/docs/src/python/sync/tables.md @@ -0,0 +1,5 @@ +# Tables (Synchronous) + +Tables hold your actual data as a collection of records / rows. + +::: lancedb.table.Table