[SkyServe] : API Authentication Options, HTTPS, More Stable Web Server that http serve

[jithinsarath]

I've combed thru the docs to try and find an answer, without any luck.

1. Do we have the ability to use a production grade HTTP Server instead of python's in-built one?
2. Can we implement HTTPS?
3. When exposing APIs while serving, can we implement authentication?

If a rogue actor gets hold of the IP:Port of a running instance, the costs could go up significantly if there's no auth

I do not expect the project team to solve these, but of there are some directions given, I could take a shot at it.

[concretevitamin]

Thanks for raising this @jithinsarath.

1. SkyServe uses Uvicorn at the moment for the controller / load balancer servers. It has not shown up as bottlenecks as of yet due to the compute-heavy nature of GenAI models. Do you foresee for your use cases this may become a bottleneck?
2. We'd love to get community's help on this! Encrypting the requests/responses will be great to see in SkyServe.
3. Yes, it should be possible. There are two prototypes of this based on Nginx, from @cblmemo:

• [Serve][Example] Authentication example #3251 user-side: user simply sets up the auth in YAML; during curl, pass curl --location-trusted .. -H "Authorization: Bearer static_secret_token" ..
• [Serve] nginx authentication #3250 server-side: changes SkyServe to support nginx auth by default

We'd like to know more about your requirements to figure out the best ways forward :) Happy to follow up here or on Slack.

[cblmemo]

For 3, we recently added an example on authorization leveraging the underlying serving engine; pls check here :)) Does that suit your need?

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[cblmemo]

Quick update: for 2, we have an ongoing PR #3380 and we plan to merge it soon ;) for 1, recently we are exploring the possibility to adopy envoy proxy as our load balancer. Stay tuned!