feat: pgvector - make error messages more informative

[superkelvint]

The pgvector document store does not currently report SQL exceptions, which makes debugging difficult.

Using hayhooks, for example, with this patch, the error goes from this:

│ Server error: Pipeline execution failed: The following component failed to run: │
│ Component name: 'indexing:writer'                                               │
│ Component type: 'DocumentWriter'                                                │
│ Error: Could not write documents to PgvectorDocumentStore.                      │
│ You can find the SQL query and the parameters in the debug logs.                │

to this:

│ Server error: Pipeline execution failed: The following component failed to run:
│ Component name: 'indexing:writer'
│ Component type: 'DocumentWriter'
│ Error: Could not write documents to PgvectorDocumentStore:         │
│ expected 768 dimensions, not 384 
│You can find the SQL query and the parameters in the debug logs."}                   │

Proposed Changes:

Include the SQL exception in the reporting message.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[anakin87]

Ok. I see the issue but I would not overcrowd the error message if possible.

With Haystack

import os
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore
from haystack import Document

os.environ["PG_CONN_STR"] = "postgresql://postgres:postgres@localhost:5432/postgres"

document_store = PgvectorDocumentStore(
    embedding_dimension=5,
    vector_function="cosine_similarity",
    recreate_table=True,
    search_strategy="hnsw",
)

document_store.write_documents([
    Document(content="This is first", embedding=[0.1]*2),
    Document(content="This is second", embedding=[0.3]*2)
    ])
print(document_store.count_documents())

Error:

Traceback (most recent call last):
  File "/Users/stefano.fiorucci/dev/haystack-core-integrations/integrations/pgvector/src/haystack_integrations/document_stores/pgvector/document_store.py", line 791, in write_documents
    self._cursor.executemany(sql_insert, db_documents, returning=True)
  File "/Users/stefano.fiorucci/Library/Application Support/hatch/env/virtual/pgvector-haystack/tK4vRGuL/pgvector-haystack/lib/python3.12/site-packages/psycopg/cursor.py", line 128, in executemany
    raise ex.with_traceback(None)
psycopg.errors.DataException: expected 5 dimensions, not 2

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/stefano.fiorucci/dev/haystack-core-integrations/integrations/pgvector/try.py", line 14, in <module>
    document_store.write_documents([
  File "/Users/stefano.fiorucci/dev/haystack-core-integrations/integrations/pgvector/src/haystack_integrations/document_stores/pgvector/document_store.py", line 801, in write_documents
    raise DocumentStoreError(error_msg) from e
haystack.document_stores.errors.errors.DocumentStoreError: Could not write documents to PgvectorDocumentStore. 
You can find the SQL query and the parameters in the debug logs.

In this case, the cause of the error is easily understandable, by inspecting the stacktrace.

In Hayhooks
Similar setting

from haystack import Pipeline
from hayhooks import BasePipelineWrapper
import os
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore
from haystack import Document
from haystack.components.writers import DocumentWriter

os.environ["PG_CONN_STR"] = "postgresql://postgres:postgres@localhost:5432/postgres"

class PipelineWrapper(BasePipelineWrapper):
    def setup(self) -> None:
        document_store = PgvectorDocumentStore(
        embedding_dimension=5,
        vector_function="cosine_similarity",
        recreate_table=True,
        search_strategy="hnsw",
        )
        
        pipe = Pipeline()
        document_writer = DocumentWriter(document_store)
        pipe.add_component("document_writer", document_writer)
        self.pipeline = pipe

    def run_api(self, text: str) -> str:
        document = Document(content=text, embedding=[0.1]*2)
        result = self.pipeline.run({"document_writer": {"documents": [document]}})
        return result

Error

Server error
Pipeline execution failed: The following component failed to run:
Component name: 'document_writer'
Component type: 'DocumentWriter'
Error: Could not write documents to PgvectorDocumentStore. 
You can find the SQL query and the parameters in the debug logs.

@mpangrazzi WDYT? Would it be possible/appropriate to show the entire stacktrace in Hayhooks?

[mpangrazzi]

@anakin87 @superkelvint yes it's definitely possible to show full stacktraces on Hayhooks! You simply need to set HAYHOOKS_SHOW_TRACEBACKS env variable to a truthy value (e.g. 1 or true) and (re)launch the server. That variable is documented in Configuration section.

[superkelvint]

Thanks for the feedback! I understand the concern about overcrowding error messages.

That said, I'd like to emphasize how including the SQL exception (even in a truncated form) could actually improve the developer experience. In environments like Hayhooks, where the full stack trace is hidden by default unless HAYHOOKS_SHOW_TRACEBACKS is set, the exception is unhelpful and the user is left with a generic error like:

Could not write documents to PgvectorDocumentStore.

Even outside of Hayhooks, just using plain Haystack, the raised exception adds an extra layer that can get in the way. It hides the real error behind a generic message, which makes debugging slower and less streamlined.

Would you be open to conditionally including the SQL error? For example:

raise DocumentStoreError(f"{error_msg}: {type(e).__name__}: {e}") from e

or even

raise DocumentStoreError(f"{error_msg}: {str(e)[:200]}") from e

This would keep the message compact while still providing valuable context that can immediately point to the issue. It helps streamline the debugging process by providing developers with actionable information right at the point of failure.

Let me know if you'd prefer this be gated behind a flag or if there's another compromise you'd suggest—happy to adapt!

[CLAassistant]

All committers have signed the CLA.

[anakin87]

Hey @superkelvint, I understand your point.
I'm OK with the idea and I pushed some small improvements.

Could you please sign the CLA?

[superkelvint]

Thanks @anakin87! I have just signed the CLA.

[anakin87]

the improvement is available in the new package release: https://pypi.org/project/pgvector-haystack/3.3.0/