Agent anatomy four building blocks
The agentic runtime is composed of four key building blocks. The LLM drives the execution lifecycle of a request within the agentic runtime. It controls the different phases of orchestration such as planning running different tool calls summarizing and even running guardrails. Tools give the agent its capabilities such as accessing resources taking actions and running tools. Memory personalizes user experience and seeds knowledge. Prompt controls behavior giving the agent a persona providing user intent to the model and directing orchestration with guidelines and rules.
When you run all of this at scale orchestrating the end-to-end execution observability becomes a very important aspect. You need to know how the reasoning logic is actually performing at scale. You have to inspect the traces to understand the reasoning loop the controls and make sure that the agent is behaving as desired. Safety is paramount. You need safety guardrails in different stages of the execution starting with receiving requests from the users planning orchestration and also summarizing results.
The trip-planner example
For the rest of the session the presenters used a trip-planner as a running example. The agent would plan a three-day winter trip. They assumed a capable LLM wired with the trip-planner and gave it three capabilities. A search tool that can query and find destinations based on seasonal parameters. A weather tool to look up conditions for a given city. A synthesizer that can compose everything together giving a day-to-day itinerary.
When a request arrives the LLM does not just reply. It plans. It looks at its available tools and issues an execution. A system runs the tool call and results are sent back to the agent. The model looks at the results the tools at disposal and may take more actions such as fetching weather synthesizing results and sending the response back to the user. Orchestration can be controlled in different ways. Frameworks help build static graphs which are more deterministic. Or you can let the LLM decide the plan and orchestrate it which is more non-deterministic. They have their own pros and cons.
Vanilla approach hits scaling ceiling
Popular agent dev frameworks help compose graphs let LLMs decide certain things by themselves and control certain flow. These frameworks are pretty good at executing tools at scale handling multi-agent interactions and handling tricky problems like memory management stateful checkpoints and stream processing. Async flows are really important when you have a lot of tool calls happening in parallel and when you multiply that with the magnitude of agents running at scale in production.
The presenters built a baseline using standard Python tools and frameworks. They composed the tools stitched the graph connecting the LLM the tools and a checkpointer to manage memory. With this compiled graph they had an agent. They translated it to a FastAPI application giving them a fully functioning agent they could query. The standard testing approach was to start a server and send a RESTful request to query the agent. Inspecting trails as requests are processed gives insights on the behavior of an agent which is important for course-correcting the agent eventually making it efficient.
The next natural approach is to push this to a remote environment. Typically you would containerize this and turn it into a Kubernetes deployment using a standard spec. This works well if you think of it as a classic RESTful service. When you take the same paradigm and apply it to a multi-turn agent working statefully at scale orchestrating many different tool calls it quickly hits the ceiling. There are a lot of scaling limitations.
Production requirements checklist
Bhumik Vinodkumar Thakkar outlined a long list of requirements before the trip-planner can accept real user queries. Capacity and scheduling: you need options to decide replicas minimum replica maximum replica to handle dynamic traffic requirements provision different types of resources to handle production load. You want control of scheduling decisions such as preemption policies and no-affinity rules. Agents have dynamic traffic patterns and you need enough controls to handle such patterns without affecting user experience while keeping costs low.
Agents' workloads are unique because they make a lot of network calls. You need to look at how many requests are currently going on in parallel to see if you want to add one more replica or reduce one more replica. You need an intelligent way of applying autoscaling configurations. As any distributed system agentic workloads need fault tolerance baked into the frameworks. Faults can occur in any layer such as infra layer where a node can start malfunctioning or application layer problems such as memory leaks bringing down the application or crash loops.
Protocol support goes beyond basic HTTP REST. You may need server-sent events and streaming because agentic workloads may run longer than a regular HTTP request doing multiple operations and sending incremental responses back to users. Agents can also do long-running research tasks in the background. You need APIs to submit a background task for deep research operations and other APIs to query updates on that task and query the status of that task.
On the operational side agent observability is interesting because there is a dynamic chain of events. For the same user query you might see a different orchestration because LLMs may decide things a little differently. You need visibility into LLM token usage how many tool calls are being performed for a given user query and so on. Agents deal directly with natural language both in the user query and the response. You need guardrails baked into the whole agentic framework so anything unintentional is not getting leaked and user experience is not compromised.
In one user query many things happen. The trip-planner exposes three tools but in the real world there can be hundreds of tool calls available. Some agent may call into the trip-planner agent and so on. The call graph gets very complex. You need a way of tracing the whole requirement lifecycle across all the tools being invoked. If there is anything upstream you want to stitch that information so there is a nice consistent tracing story available.
How Ray Serve addresses the gaps
Ray Serve addresses a lot of these requirements right out of the box. Ray Serve has something called RayCluster which is the underlying compute on which Ray Serve applications are deployed. A RayCluster can be provisioned to have a heterogeneous workload. If you have a requirement of running both CPU and GPU workloads under one umbrella Ray Serve gives you that opportunity. RayClusters directly work with the Kubernetes autoscaler using KubeRay operator. KubeRay operators manage Ray-specific custom resource definitions giving a lot of control on different scheduling decisions and they do reconciliations to make updates pass through.
While there is high-level Kubernetes-level autoscaling which the KubeRay operator takes care of Ray also provides application-specific autoscaling configurations. You can scale up and scale down based on the number of parallel requests and number of queued requests. You can also manage the min replica max replica and so on.
One of the best features of Ray Serve is native support for both FastAPI and gRPC. All the capabilities available in FastAPI and gRPC can be directly made available inside Ray Serve applications. All the streaming server-sent events HTTP workloads are right there. On top of that there is detailed documentation on how to use Ray Actors which are background asynchronous tasks you can submit on a RayCluster. These can be leveraged in addition to Celery to perform async task orchestration useful for deep research agentic operations.
The Ray dashboard is feature-rich with panels for underlying compute metrics for CPU memory utilization request level metrics number of ongoing requests QPS status codes performance metrics such as latencies Ray Core logs for debugging and application logs emitted from the applications themselves.
What this means for teams shipping agents
Teams building agents face a gap between prototype and production. The vanilla path of containerizing a FastAPI app and deploying to Kubernetes works for simple services but breaks down for multi-turn stateful agents with dynamic tool orchestration. You need intelligent autoscaling that understands parallel request patterns not just CPU memory. You need fault tolerance at both infra and application layers. You need streaming and background task support for long-running operations. You need observability that captures token usage tool call counts and traces across complex call graphs. You need guardrails for natural language inputs and outputs.
Ray Serve packages these capabilities into a framework-agnostic layer. It runs on top of Kubernetes via KubeRay but adds agent-specific autoscaling protocols and observability. Teams can keep their existing agent frameworks and run them on Ray infrastructure. The dashboard gives a single pane of glass for compute metrics request metrics and logs. Ray Actors provide a native way to handle deep research background tasks without bolting on Celery.
For teams already on Ray the path is straightforward provision a RayCluster configure autoscaling policies and deploy. For teams on other frameworks Ray Serve offers a migration target that solves the scaling problems that appear when agents go from demo to production traffic.