ibis-server Connection Refused: Startup Sequence Dependency Issue Causing SQL Query Failures

[i-am-alvin]

ibis-server Connection Refused: Startup Sequence Dependency Issue Causing SQL Query Failures

Description

When attempting to execute SQL queries through wren-ui, a connect ECONNREFUSED 172.18.0.2:8000 error occurs, preventing connections to the ibis-server. After investigation, this appears to be an issue with the Docker container startup sequence, where service dependencies are not properly handled. Even though the ibis-server's HTTP service is running, its API endpoints remain unresponsive until all related services are restarted in a specific order.

Steps to Reproduce

1. Start WrenAI Docker containers: docker compose up -d
2. Try to execute an SQL query similar to:

SELECT COUNT("dud"."user_id") FROM "dim_user_data" AS "dud" 
WHERE CAST("dud"."joined_at_tw" AS TIMESTAMP WITH TIME ZONE) >= CAST('2025-03-30 00:00:00' AS TIMESTAMP WITH TIME ZONE) 
AND CAST("dud"."joined_at_tw" AS TIMESTAMP WITH TIME ZONE) < CAST('2025-03-31 00:00:00' AS TIMESTAMP WITH TIME ZONE)

3. Observe the error: connect ECONNREFUSED 172.18.0.2:8000

Environment

• Docker version: Docker version 24.0.6 (or higher)
• WrenAI version: 0.18.0
• wren-engine version: 0.14.8
• wren-ai-service version: 0.18.2
• wren-ui version: 0.23.4
• OS: macOS 14.3.0

Expected Behavior

SQL queries should execute normally without connection errors.

Actual Behavior

Encountering Invalid SQL syntax connect ECONNREFUSED 172.18.0.2:8000 error, unable to connect to ibis-server.

Investigation

1. Checked ibis-server container status and logs, confirmed the service is started and listening on port 8000
2. Verified network configuration, confirming wrenai-ibis-server-1's IP address is 172.18.0.2
3. Tested ibis-server's /health endpoint directly from the wren-ui container using curl, found it can connect normally
4. Analyzed wren-ui logs, discovered that IbisAdaptor's connection attempts to ibis-server were being refused
5. Examined ibis-server's JavaEngineConnector, found it depends on the wren-engine service

Solution

Restarting services in the following order resolved the issue:

1. First restart wren-engine: docker restart wrenai-wren-engine-1
2. Wait approximately 10 seconds, then restart ibis-server: docker restart wrenai-ibis-server-1
3. Wait another 10 seconds, finally restart wren-ui: docker restart wrenai-wren-ui-1

Improvement Suggestions

1. Add more explicit service dependencies and health checks in docker-compose.yaml
2. Increase ibis-server startup delay to ensure wren-engine is fully initialized
3. Improve error handling to provide more explicit error messages rather than just connection errors
4. Consider implementing retry mechanisms to automatically retry when services are temporarily unavailable

Additional Notes

This issue may be more likely to reproduce in environments with limited system resources or slower networks, as service initialization times would be longer.

[wwwy3y3]

@i-am-alvin

Wow. This issue is well written! Thanks a lot!

Add more explicit service dependencies and health checks in docker-compose.yaml

Yeah, I think adding healthcheck and condition: service_healthy to the depends_on configs we already have in docker-compose might be the easiest way to solve it.

Feel free to contribute if you'd like to, btw.