Architecture#

High-Level View#

Serve runs on Ray and utilizes Ray actors.

There are three kinds of actors that are created to make up a Serve instance:

• Controller: A global actor unique to each Serve instance that manages the control plane. The Controller is responsible for creating, updating, and destroying other actors. Serve API calls like creating or getting a deployment make remote calls to the Controller.

• HTTP Proxy: By default there is one HTTP proxy actor on the head node. This actor runs a Uvicorn HTTP server that accepts incoming requests, forwards them to replicas, and responds once they are completed. For scalability and high availability, you can also run a proxy on each node in the cluster via the proxy_location field inside serve.start() or the config file.

• gRPC Proxy: If Serve is started with valid port and grpc_servicer_functions, then the gRPC proxy is started alongside with the HTTP proxy. This Actor runs a grpcio server. The gRPC server that accepts incoming requests, forwards them to replicas, and responds once they are completed.

• Replicas: Actors that actually execute the code in response to a request. For example, they may contain an instantiation of an ML model. Each replica processes individual requests from the proxy. The replica may batch the requests using @serve.batch. See the batching docs.

Lifetime of a request#

When an HTTP or gRPC request is sent to the corresponding HTTP or gRPC proxy, the following happens:

1. The request is received and parsed.

2. Ray Serve looks up the correct deployment associated with the HTTP URL path or application name metadata. Serve places the request in a queue.

3. For each request in a deployment’s queue, an available replica is looked up and the request is sent to it. If no replicas are available (that is, more than max_ongoing_requests requests are outstanding at each replica), the request is left in the queue until a replica becomes available.

Each replica maintains a queue of requests and executes requests one at a time, possibly using asyncio to process them concurrently. If the handler (the deployment function or the __call__ method of the deployment class) is declared with async def, the replica will not wait for the handler to run. Otherwise, the replica blocks until the handler returns.

When making a request via a DeploymentHandle instead of HTTP or gRPC for model composition, the request is placed on a queue in the DeploymentHandle, and we skip to step 3 above.

Fault tolerance#

Application errors like exceptions in your model evaluation code are caught and wrapped. A 500 status code will be returned with the traceback information. The replica will be able to continue to handle requests.

Machine errors and faults are handled by Ray Serve as follows:

• When replica Actors fail, the Controller Actor replaces them with new ones.

• When the proxy Actor fails, the Controller Actor restarts it.

• When the Controller Actor fails, Ray restarts it.

• When using the KubeRay RayService, KubeRay recovers crashed nodes or a crashed cluster. You can avoid cluster crashes by using the GCS FT feature.

• If you aren’t using KubeRay, when the Ray cluster fails, Ray Serve cannot recover.

When a machine hosting any of the actors crashes, those actors are automatically restarted on another available machine. All data in the Controller (routing policies, deployment configurations, etc) is checkpointed to the Ray Global Control Store (GCS) on the head node. Transient data in the router and the replica (like network connections and internal request queues) will be lost for this kind of failure. See the end-to-end fault tolerance guide for more details on how actor crashes are detected.

Ray Serve Autoscaling#

Ray Serve’s autoscaling feature automatically increases or decreases a deployment’s number of replicas based on its load.

• The Serve Autoscaler runs in the Serve Controller actor.

• Each DeploymentHandle and each replica periodically pushes its metrics to the autoscaler.

• For each deployment, the autoscaler periodically checks DeploymentHandle queues and in-flight queries on replicas to decide whether or not to scale the number of replicas.

• Each DeploymentHandle continuously polls the controller to check for new deployment replicas. Whenever new replicas are discovered, it sends any buffered or new queries to the replica until max_ongoing_requests is reached. Queries are sent to replicas in round-robin fashion, subject to the constraint that no replica is handling more than max_ongoing_requests requests at a time.

Ray Serve API Server#

Ray Serve provides a CLI for managing your Ray Serve instance, as well as a REST API. Each node in your Ray cluster provides a Serve REST API server that can connect to Serve and respond to Serve REST requests.

FAQ#