Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable delayed microservices spawning based on dependencies #772

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

Khangf
Copy link

@Khangf Khangf commented Oct 9, 2024

Description

This PR enables delayed spawning of the micrservices and megaservices from OPEA to wait for the dependencies (LLMs/microservices) to be up.

Issues

#764

Type of change

List the type of change like below. Please delete options that are not relevant.

  • New feature (non-breaking change which adds new functionality)

Dependencies

N/A

Tests

We tested the code changes with the Translation example.

…pea-project#764

Signed-off-by: Foong, Khang Sheong <khang.sheong.foong@intel.com>

# Make sure LLM/Microservice endpoint are ready
logger.info(f"Init microservice - {name}")
if llm_endpoint is not None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of just waiting for LLM microservice. Should we make it more general to wait for all dependency? maybe just traverse through DAG and make sure next node available first?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact it does wait for the other dependency if they are not LLM related. This is what we observed when we tested this:

  1. Microservice will wait for LLMs to be up
  2. Megaservice will wait for the microservice to be up

else:
url = f"{host}:{port}{endpoint}"
if protocol.upper() == "HTTP":
url = f"http://{url}"
elif protocol.upper() == "HTTPS":
url = f"https://{url}"
# enable future support for gRPC?
logger.info(f"Endpoint requested for microservice @ {url}")
success = False
while success is False:
try:
for item in methods:
# Do we want to cater for GET request?
# if yes, how to make it clean
if item.upper() == "POST":
response = requests.post(url)
if response.status_code == 200:
logger.info(f"Microservice endpoint is ready - {url}")
else:
logger.info(f"Microservice endpoint is ready - but error status code - {url}")
success = True
else:
logger.info(f"Microservice endpoint method not supported - Skipping - {url}")
success = True
except requests.exceptions.RequestException as e:
logger.info(f"Error: {e} - {url}")
time.sleep(2.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants