[Feature] Implement Health Check Endpoint for Delayed Service Startup

[isaacncz]

OS type
Ubuntu

Description
When running the example Translation using Docker Compose, one of the images takes additional time to pull a model from the Huggingface upon startup. During this period, the service is unresponsive to HTTP requests, resulting in HTTP 500 errors.

To improve reliability, would like to propose adding a health check endpoint that can verify when the service is ready to handle requests. This will allow other services and users to know when the service is up and running, avoiding unnecessary errors and improving the user experience.

Expected Behavior:

Docker Compose starts all services.
The service in question takes some time to pull the model.
A health check endpoint will be available to verify when the model has finished loading and the service is ready.
Proposed Solution:

Add a /health endpoint that returns:
200 OK when the service is fully operational.
503 Service Unavailable or similar status when the service is still initializing or loading the model.
Optionally, provide a message or a status code that indicates the estimated time remaining for startup.

[louie-tsai]

v1/health_check endpoint might work for you.
https://github.com/opea-project/GenAIComps/tree/main/comps/embeddings/tei/langchain#3-consume-embedding-service
https://github.com/opea-project/GenAIComps/tree/main/comps/llms/text-generation#31-check-service-status

it should be implemented as a cores feature.
https://github.com/opea-project/GenAIComps/blob/main/comps/cores/mega/http_service.py#L68

could you try whether v1/health_check work for your case?

[louie-tsai]

@isaacncz
I put one of the example for health check below.
health check
curl http://localhost:3007/v1/health_check -X GET -H 'Content-Type: application/json'
response from microservice
{"Service Title":"opea_service@llm_tgi/MicroService","Service Description":"OPEA Microservice Infrastructure"}

[louie-tsai]

@isaacncz
Do you have further questions?
if not, we will close the ticket.

[isaacncz]

@louie-tsai i have tested the health check, it worked. However, for llm microservice, i will not be able to check whether the model is already downloaded completely.

[louie-tsai]

@isaacncz
no check for model download completion yet indeed.
@kevinintel
There is need to show the LLM model download completion status via health_check or statistics.
please help to evaluate the feature.

[kevinintel]

It depends on serving framework, we only know the service ready or not

[louie-tsai]

@kevinintel
will let you handle this feature request which is asking about serving framework readiness.