Presentation: Beyond the Prototype: Scaling Framework Agnostic AI Agent Infrastructure with Ray

Transcript

Deepak Mohanakumar Chandramouli: Before we kick off, our focus today is basically building an agent and running it at scale with Ray. How many of you are familiar with Ray? Ray is an open-source distributed computing framework. It's mainly specializing in machine learning workloads, such as training, fine-tuning, running inference. Today what we are taking you through is also running agents on Ray at scale. That's the focus. This is the lineup for our session today. We are going to start with a quick refresher of an agent's building block. I think we've had a lot of sessions talking about prompts, context, memory, and so on. We'll take a quick refresher of that. Followed by that, we will walk through the developer lifecycle of building an agent and running it at scale, locally and in a remote environment. We'll do the same by running it with Ray. Finally, we want to call out a few things and share some of the learnings that we have from production.

Agent Anatomy

This may be familiar grounds. The agentic runtime generally is composed of a few key building blocks. First one is LLM. LLMs drive the execution lifecycle of a request within the agentic runtime. It controls the different phases of orchestration, such as planning, running different tool calls, summarizing, and even running guardrails. Next is tools. It is what gives the agent its capabilities, such as accessing resources, sometimes taking actions, and most often running tools and taking actions. Memory is very important in personalizing user experience and also seeding knowledge. We've had a few sessions where we had speakers talk about manning memory at scale and also knowledge base. All of those come in play. Last but not least, prompt is very important. We had a couple of sessions, very interesting ones talking about context, how it controls the behavior of the model execution. There are various aspects to prompts, very interesting ones.

They give the agent a persona. We're all used to saying, you're Claude, do this. That's one aspect. Second one is providing users intent to the model, the LLMs. The third one is the directives. These are the ones that really control the orchestration. It has many different aspects such as guidelines, rules, what to do and what not to do. When you run all of this at scale, basically orchestrating the end-to-end execution, observability becomes a very important aspect. We need to know how the reasoning logic is actually performing at scale. We have to inspect the traces to understand the reasoning loop, the controls, and make sure that the agent is behaving as desired. Safety is paramount. We need safety guardrails in different stages of the execution, starting with receiving requests from the users, planning, orchestration, and also summarizing results.

Meet the Trip-Planner

For the rest of the session, we're going to take a simple example, a trip-planner, build the agent, run through all the stages of deployment. Let's assume we have a very nice LLM that we're able to wire with this trip-planner. We're going to give it a few other capabilities as well. Let us say we want to give it a search capability that can query and find destinations based on seasonal parameters. We'll wire this up with a weather tool, so you can look up the weather conditions for a given city, and also a synthesizer that can compose everything together, giving us an itinerary, basically a day-to-day plan. Let's visualize how this agent may act. It's basically our desired behavior. When I send a request to this agent, I would want this to plan a three-day winter trip. The LLM does not just reply. It plans. It looks at its tools from the disposal, and then it would issue an execution.

This might be a system that might be running the tool call, and the results are sent back to the agent again. The model looks at the results, the tools at disposal, and it may take more actions such as fetching weather, synthesizing the results, sending the response back to the user. Now, we can control this orchestration in a few different ways. There are frameworks out there that help us build static graphs, which is more deterministic, but also, we can let the LLM decide the plan and orchestrate it, which is more non-deterministic. They have their own pros and cons.

Agent DevX and Ops - Vanilla

We have a few different popular agent dev frameworks. I think all of us who have been developing agents may have used them. They help us compose the graphs, let the LLMs decide certain things by itself, but also control certain flow. These frameworks are pretty good at executing tools at scale, handling multi-agent interactions. They're also pretty good at handling tricky problems like memory management, stateful checkpoints, and stream processing, which is very table stakes when you think about multi-turn, long-context agent interactions. Async flows are really important when you have a lot of tool calls happening in parallel and when you multiply that with the magnitude of agents running at scale in production. Let's just take an example of developer experience. We're going to build a baseline here. Let's use the standard Python tools and frameworks to build this agent. Here is one way to implement this. We've seen the tools that we are going to give the agent a capability.

We'll use this implementation, just compose the tools. Then we're going to stitch the graph here, connecting the dots between the LLM, the tools, and also maybe wire a checkpointer to manage memory. With this compiled graph, we have an agent. We can translate this to a FastAPI application. This gives us the capability to have a fully functioning agent that we are able to query. So far, we have just used standard Python approach. If you want to test this, the standard approach is you start a server and then you send a RESTful request and you query the agent. We would want to inspect the trails as these requests are being processed, the tools are being orchestrated, and this gives us insights on the behavior of an agent. It's very important because we want to course-correct the agent eventually, making it efficient. The next natural approach is we want to push this to a remote environment.

Typically, we would want to containerize this and then turn this into a Kubernetes deployment using a standard spec. This is a pretty good approach. It works well if you think of it as a classic RESTful service. When you take the same paradigm and then apply to maybe a multi-turn agent working statefully at scale, orchestrating many different tool calls, it quickly hits the ceiling. There are a lot of scaling limitations.

Bhumik Vinodkumar Thakkar: That was a quick ride to prod for the agent, but we are not done yet. We have a bunch of things which are still missing, which we need to account for while doing this agent deployment in production. Before the trip-planner can actually start accepting the real user queries, we need to make sure that we are provisioning enough capacity. We have some scheduling decisions we are making for this particular agent. We are also thinking about all sorts of scaling challenges, as Deepak was pointing out, that it can have dynamic traffic loads, dynamic traffic patterns. The tool calls may have different types of branches and orchestration which may go on here. To make sure that we are able to understand issues, diagnose issues, and detect issues effectively in production, it is necessary for us to have an end-to-end observability story. This is not it. We have a lot of other things we want to make sure that is in place before we can actually deploy the agent in production for user queries.

Let's dive deeper into all of these requirements. First let's look at some core requirements, such as capacity and scheduling. Capacity is, as a user, as an agent admin, I want some options to decide what are the replicas, what are the minimum replica, maximum replica the agents are going to be running to handle these dynamic traffic requirements, provision different types of resources to handle the production load. Also, I want to be in control of the scheduling decisions such as what are the preemption policies, what are the no-definity rules, and so on. Also, since agents can have this dynamic traffic pattern, we really want to make sure that there are enough controls available to the agent administrators to make sure that they are able to handle such patterns. The goal here is to make sure that we are able to handle all this dynamic traffic pattern without really affecting the user experience while keeping the costs low.

Another interesting callout here is that it is not just CPU or memory which we care about here. Agents' workloads are very unique in a sense that they make a lot of network calls. We also need to look at how many requests are currently going on in parallel right now to see if we want to add one more replica or reduce one more replica. We need some intelligent way of applying these autoscaling configurations. Last, as any distributed system, agentic workloads are no different. They also need to bake fault tolerance right into the frameworks. For example, the faults can occur in any layer such as infra layer where a node can start malfunctioning for whatever reason. Then there can be an application layer problem as well. For example, there could be some memory leak which is just bringing down the application or having some sort of crash loop. We need to account for all of those things when we are deploying the agent at scale.

Next, we already mentioned that in the trip-planner agent we want to make a REST API call to just query the agent. That is table stakes, that is support for HTTP REST protocol. However, we may also need some sort of server-sent event and streaming type of protocol support as well. Because there could be agentic workloads which are running a bit longer than a regular HTTP request. They are doing multiple operations and they are sending some incremental responses back to the users. In such scenarios, server-sent events and streaming are really useful. Last, the agents can also do a long-running research task in the background. In such cases, we may need some APIs to just submit a background task which the agents can perform as a deep research operation. There should be some other API which can let us query the updates on this particular task, query the status of this task, and so on.

Next, the operational side. The moment we talk about taking the user queries, we need to make sure that we have enough operational excellence built into the frameworks as well. Agent observability is an interesting area, mainly because there is a dynamic chain of events which can occur. In a lot of cases, for the same user query, you might see a different orchestration because the LLMs may decide things a little differently for the same user query as well. We really need some more visibility into what is the LLM token usage, how many tool calls are being performed for a given user query, and so on. Next, a lot of agents, for example, our trip-planner agent is dealing directly with natural language, both in terms of the user query as well as in the response which we are going to be sending back to the users. In those things, it is very important to make sure that we have enough guardrails baked into the whole agentic framework so that anything unintentional is not getting leaked and the user experience is not compromised.

Last, there are many things which are happening in one user query. We talked about three tools which we are exposing as part of this trip-planner agent, but imagine in the real world, there can be hundreds of tool calls available. Some agent may be calling into the trip-planner agent and so on. The call graph gets very complex. In such cases, we really need to have some way of tracing the whole requirement lifecycle across all the tools which are being invoked. If there is anything upstream, then we also want to stitch that information so that there is a nice, consistent tracing story available to us.

Agent DevX and Ops - With Ray

That was a really long list of requirements, and hopefully we'll try to address most of that. The good thing is Ray Serve addresses a lot of these requirements right out of the box. Let's dive deeper into that. Ray Serve has something called RayCluster. That is the underlying compute on which the Ray Serve applications are getting deployed. A RayCluster can be provisioned to have a heterogeneous workload. For example, if you have a requirement of running both CPU and GPU workloads under one umbrella, Ray Serve gives you that opportunity. Then RayClusters directly work with the Kubernetes autoscaler using KubeRay operator. KubeRay operators are used to manage the Ray-specific custom resource definitions. That gives us a lot of control on the different scheduling decisions, and they will do all sorts of reconciliations to make the updates pass through and so on. While there is a high-level Kubernetes-level autoscaling, which the KubeRay operator takes care of using the KubeRay operator and the Kubernetes-level autoscaler, Ray also provides application-specific autoscaling configurations to address the requirements which we talked about, that we want to scale up and scale down based on the number of parallel requests, number of queued requests.

We can also manage the min replica, max replica, and so on. One of the best features of Ray Serve is that it comes with the native support for both FastAPI and gRPC. All the good capabilities which are available in FastAPI and gRPC can be directly made available inside Ray Serve applications as well. All the streaming, server-sent event, HTTP, all of those workloads are right there. On top of that, there is a very nice, detailed documentation on the open-source Ray documentation about how to use the Ray Actors. Those are the background asynchronous tasks which one can submit on a RayCluster. How can those be leveraged in addition to Celery to perform an async task orchestration? It can be useful for those deep research agentic operations which we just talked about. Last, the most important aspect is that all the Ray Serve applications and the RayClusters come with a very feature-rich Ray dashboard.

This dashboard has a lot of useful panels such as, you can look at the underlying compute metrics for CPU, memory utilization, you can look at the request level metrics, the number of ongoing requests, the QPS, the status codes, and so on. You can look at some performance metrics such as latencies. There are a lot of Ray Core logs which are available if you need to debug certain issues. You can also look at the application logs which we are emitting out from the applications itself. With this setup, let me now hand this off again to Deepak to see how our trip-planner agent can be converted into a Ray Serve application.

Deepak Mohanakumar Chandramouli: This is a really good set of capabilities, but we now need to leverage this. I'm going to bring back where we left off before. We built a trip-planner agent, we wrapped it up with a classic FastAPI, we tested it locally. This is a classic JSON response that we're getting back from the agent. The good news is Ray Serve supports FastAPI out of the box. If you have a FastAPI application, you can directly port it over to Ray Serve and run it on Ray. We'll see how. Those working with different environments, gRPC is also supported out of the box. Bhumik mentioned a lot about streaming and SSE. I think this is table stakes for multi-turn operations, especially long context and streaming operations where there is an incremental response that the agent is sending back to the caller, be it another agent or user. This is very important when we want to run this at scale. One thing I want to call out is Ray supports any sort of async generator implementations that sends incremental chunks. This gets us very close to porting this application over to Ray.

So far, what we have done is we've implemented the agent using standard open-source approach. Here is how we can transition that application into Ray. If you look at these three blocks of code, this is the place where the magic is happening. Here we take our trip-planner FastAPI app and wrap it up with the serve ingress. This binds our agent to a Ray Serve deployment. Along with that, we add a bunch of knobs, which takes care of autoscaling. Bhumik talked about adding replica level configurations where you can specify min and max replicas that define the elasticity of the agent at scale. Knobs such as target_ongoing_requests tell us when to trigger autoscaling. When a request in flight on an agent replica goes beyond five, for this example, it triggers adding a new replica. It also supports specifying CPU and GPU resources and also memory resources that are required to specify the cluster spec.

Last, when you're running agent at scale, backpressure is also an area that you need to deal with. If you have too many requests coming in, there are a couple of ways to deal with. One is we can queue them up. The other one is we can drop requests. There are a few knobs that we can use that can tell the agent replica to either queue up the request or drop it, maybe returning 429s to our users. Going back to our test lifecycle, we now converted our FastAPI app to Ray Serve app. We want to make sure that the validation of the reasoning loop is still the same. Basically, verify everything is working as intended. Ray supports running something locally using the serve CLI. It starts the server just the way we had with Uvicorn, and we can query the agent using the RESTful protocols, and we validate this.

With this, we've transitioned our agent from FastAPI to Ray Serve. The bonus here is the dashboards. When you run a Ray application, it ships a dashboard regardless of whether the environment is local or remote. It comes with a few levels of observability stack, starting with hardware resource utilization, resource consumption such as CPU and memory, and then application-level health: logs, metrics, events. Also, autoscaling information. When you have traffic coming in, the dashboard informs us the number of in-flight requests, the number of requests in queue, the queue depth, scheduler depth, and also how many requests are being dropped. This is really useful when you are testing your application locally with a small subset of traffic, and then you can extrapolate that to production.

Speaking of production, when we want to think of now taking this over to Kubernetes, it adds a bunch of other requirements, auxiliary requirements such as handling AuthN, AuthZ, shipping logs, collecting metrics, wiring up secrets that may be required at runtime. Beyond that, once we deploy an agent, we also need to deal with in-place updates, a small change to a configuration may need a restart for agents, or maybe zero downtime releases. The RayService CRD, which is our custom resource definition, is the bridge to Kubernetes. Like Bhumik mentioned, we have two parts here. The RayService CRD is the YAML declarative spec for the developer, and the KubeRay operator is the controller that is responsible to reconcile the cluster to the desired state, and also the application to the desired state. There are two parts here. There is a Ray Serve cluster, and then there is the Ray Serve application, and we have other toolings like init containers and sidecars.

We'll see how that works. In this example, we are writing a Ray Serve CRD, and you can see there is two blocks of code here. There is a headGroupSpec, and there is a workerGroupSpec. The headGroupSpec helps us define the brain of the RayCluster. It is the one that maintains the state of the cluster, observes the traffic, and also makes decisions on autoscaling. The workerGroupSpec defines the workhorses, which are basically the agent replicas. Here we can specify the min and max, defining the elasticity, and also other logical resource requirements. Once we've established the cluster topology, the next step is the application configs. If you remember, we wrote a classic Ray Serve Python code, which had a deployment binding and also a few knobs for autoscaling. The same can be overlaid here using the configuration that we can write in the CRD. It also provides a few other options to give it a name, like a trip-planner, and then also the entry point, which is the module that we wrote when we bind the application.

Last but not least, like any other Kubernetes deployment calls, we need to mount secrets, maybe from a remote vault, or ship logs, collect metrics. We can use the sidecar patterns and the init container patterns to add these snippets into the same service CRD, which will help us deploy these components. With this, we've been able to turn the code into a Kubernetes deployment, and we can ship this to a Kubernetes cluster.

Trip-Planner: The Topology

This would be the end manifestation of the collected components. If you recollect, we wrote a RayCluster spec, which defines the topology. This box is the RayCluster. It composes the Ray Head and the Ray Worker PODs. These are the workhorses. For a user sending a request to the agent, the gateway is the entryway with Kubernetes ingress. From there on, the RayCluster takes over. The request enters the HTTP proxy, which is responsible for routing the request into specific replicas. This happens across all the worker PODs, which are horizontal scale. These agent replicas in turn send heartbeat back to the Ray Head POD. The Ray Head POD is also able to take that information and make decisions on autoscaling. With that, we have established a full cluster topology and an agentic deployment in Ray. This is not the end of it. We also have to think about fault tolerance across the regions, across the globe, if you have a global user base.

Agent Control Plane

Bhumik Vinodkumar Thakkar: So far, we talked about the RayService CRD. It may scale well for just the trip-planner agent, but the moment we have more than one agent, maintaining those individual CRDs may not really work. That is where we might need something like an agent control plane, which can help us perform some agent lifecycle related operations to manage hundreds or thousands of agentic workloads in the organization. Let's see what are some of the key capabilities we can expect from such an agentic control plane. First thing is the lifecycle operation. That is, it may expose certain APIs to create the agent, deploy the agent, delete the agent, get agent-specific details, and so on. Then we also need to make sure that there are permissions set for each agent for the right users for the right type of access. One can imagine that there is a group of administrators who can perform those lifecycle operations, such as changing the agent, deploying those agents, and so on.

There could be a set of just the agent users who are able to make the query, but are not able to change the agent in any way. Then there can be a lot of metadata which can be tagged for this agentic application. We want to maintain all sorts of versions, the configurations, who made the deployments, if there are any more tags, org level tags or business level tags, we want to add to these agentic workloads, we can do that here as well. Last, we want to capture all the deployment specs, the compute specs, the replica counts, the queues, if we are maintaining any queues behind the scenes for the resource quota. The control plane is responsible for making sure that the agentic workloads are getting deployed to the right queues, getting the right resources, getting deployed in those right regions.

Once we have many agents running in production, we may also want to perform some common operation for each request which lands on this agent. That is where we may need something like a centralized data plane, which can perform all these operations before the request actually gets processed by the underlying RayService. Let's look at some of these expectations. First is authentication. Any request which lands on the data plane, we want to first perform an authentication to make sure that the user is actually what the user is claiming to be. Then the authorization. This is where it might refer back to those control plane operations which were set for this agentic workload that for this agent, these users are only permitted to access the agent. Then rate limiting. The moment we have multiple users accessing the agent, we also need to make sure that there are enough rate limiting practices put in here so that each data plane request will be looked at for the rate limiting enforcement and will only be let in if the rate limiting is still not breached.

Next, the data plane also plays an important role in stitching the request for any upstream trace IDs. As I mentioned earlier, that some other agent or some other services may be calling this agent and they may also come with their own trace IDs. It is the job of this data plane, which will stitch those request IDs and also will hydrate the request IDs for the subsequent operations which will take place inside the agentic workload. This will actually help make sure that we have unified tracing available for all user requests. The most important operation is that this data plane is going to find the right RayCluster which belongs to this particular agent. Our trip-planner agent may have its own RayCluster where it is hosted. There could be other agents which are developed by different teams and they might have their own RayClusters as well. The data plane's job is after performing all of these validations, if the request is still valid, then it will route this request to the right RayCluster so that the actual application code execution can take place.

Architecture - Deep Dive

So far, this is what we talked about. Let's take a couple of use cases of the user queries. If I want to just take a control plane operation, so imagine somebody has deployed an agent already like a trip-planner agent, and we want to just update the number of minimum replicas because we are getting too much user traffic. Here is how the journey will look like. User will call the endpoint service or the control plane. All sorts of validations will happen here. As I mentioned, the authentication, authorization, that will happen here. If the request is still valid, it will be sent to the control plane. Here in the control plane, there will be some more validations. We can also think of some history service which will keep track of all the updates and operations which are taking place on this agent so that if administrators or some other users who want to understand what's going on for this particular agent, they can see, these are all different operations which happened for a trip-planner agent, for example.

Then once everything looks good at the control plane layer, it is now going to update the actual RayService CRD, which will be reconciled by the KubeRay operator. That is how a regular update request will look like. Now let's look at how a data plane request may look like. A data plane request here is somebody is trying to query the agent. Let's say I want to build an itinerary for three days for a winter trip. In that case, as a user, my request will first go to the gateway service. Here, the first set of validation is, again, authentication, authorization. It will make sure that I have access to the trip-planner agent to query this particular agent. If the access is there, the rate limiting is also within the bounds. Then the gateway service will figure out, it might look up some databases or caches to figure out what is the right RayCluster to send this request to. Once it finds out that RayCluster, the request lands on the RayCluster where the trip-planner agent is hosted. That is a typical data plane lifecycle.

Scaling Out to Multi-Region

So far, we have been talking about just single region. Now let's imagine that the trip-planner agent, which we are discussing, is needed across different regions. The reasons could be that the users are across the globe. We want to have some fault tolerance as well, and as well as we want to have a better user experience, that users sitting in Asia should not be talking to an agent deployed in the U.S., for example, because of latency reasons and so on. In such a case, we might need to think about how we can extend that single region architecture into a multi-region architecture. Here, what we have done is that we have just extended the previous architecture and added a few more components to make it more scalable across the globe. The main component we have added is this GSLB, the Global Server Load Balancer for all our data plane requests.

The primary function of this GSLB is to intercept all the requests. It is going to be the entry point for all the requests which is coming to the agent. It is then going to decide that given the geolocation and given the health of different regions, this is the right region in which this particular request should be routed to. Once the region is located, then it is the region-specific data plane which will perform its own authentication and all the steps which we talked about. Similarly, for the data plane, if an admin wants to update the agent sitting in multiple regions, they can directly interact with the region-specific control planes. This will also allow them to have different configurations and different setup for agents, the same agent running in different regions. I might want to have a different type of prompt. I might want to have different configurations for agents running in different regions. This is what can be accomplished by the region-specific control planes as well.

Callouts For Agentic Workloads

So far, we have been able to deploy our trip-planner agent across the globe with all of the steps. However, we would still like to call out a few things. We are only going to cover very briefly about this callout. Safety first, so agents deal with natural language use cases. It is very essential to make sure that all the user queries are validated. It's like avoiding the SQL injection attacks. Here also in agentic workloads, we need to also make sure that the user queries are validated, and make sure that it is not trying to do something unexpected or not trying to retrieve something unexpected from our agentic workloads. Then the workloads are dynamic in nature because LLMs are probabilistic. We also want to have some guardrails to make sure that the agent orchestration doesn't go out of bounds. It's similar to avoiding DDoS attack. I can probably ask this trip-planner agent to generate an itinerary for 365 days.

It might just go into the loop and just create a very long itinerary, and that might just kill a lot of resources, and it might starve a lot of users as well and might just burn a lot of LLM tokens. We need to make sure that those kinds of guardrails are also put in place right from the beginning. Then, once we give the answer back to the user, we want to make sure that the responses adhere to the instructions which we are providing to the agent. We want to make sure that nothing unexpected goes out of the agent because it can have a lot of implications if the agents are not behaving as we are imagining. There are a lot of techniques to do that, and there are dedicated sessions as well. I'll highly recommend doing that.

The next thing is the prompt engineering and versioning. Here is a simple prompt for the trip-planner agent, which we start with initially. It is just going to do a bunch of tool calling and things like that. However, when we iterate on the agentic workloads, the prompts may also need some iterations. To make the output a bit more comprehensive, we have updated the prompt to say that for any results which you give back to the user in the trip-planner agent, add a reasoning why you chose this particular location and so on. The output becomes more compelling. This was just one variation one can try to improve their trip-planner agent, for example. In production, there could be many such iterations the users may make on their agents until they reach the expected levels. The most important thing here is that we should really version these prompts, such that in case of issues, we know that this particular version of the prompt was deployed.

It is very much easy to track, roll back, roll forward, and things like that. Next, eval. This is a very unique step for any agentic lifecycle. The normal CI/CD steps for doing unit test, functional test, integration test, they're necessary because we are developing a Pythonic app here. However, the agentic workloads behave so dynamically that we also want to make sure that we are evaluating the agent outputs. Whenever we are making any fundamental change, if I'm making a change in a prompt, we should be running an eval. If I'm changing the LLM, which is used behind the scenes, we should be running an eval. Any change can have a very big impact on the agent's outcome, even though all the functional tests, the unit test may seem ok, the behavior might have changed. For example, here is what I would generally recommend. Prepare a golden dataset.

In our case for trip-planner, I might have some positive use cases, such as create a three-day winter trip or create a five-day summer trip, five-day trip to Boston, for example. I might also give some negative prompts, which the agent should be able to handle fully, and behave in a certain expected manner. For example, if I ask the agent to just say, reverse a linked list, it should politely say that that is not expected. These are the kinds of things we want to feed into the golden set and make sure that we run those evals whenever we make any change into the application. We look at different scores, and there is a lot of literature about this, like what are different things one cares about. For example, here we care about the safety, and the instruction adherence. If I'm saying that only focus on trip planning related queries, that is the instruction I want the agent to adhere to.

All of those things can be evaluated, and we can use some popular techniques like LLM-as-a-judge to do these evaluations, and then gate all the releases based on the scores so that nothing unexpected gets out. Last, this is how the production pipeline looks like. We all are familiar with the CI/CD build process. We run unit test. Eval is where the new component gets into the agentic lifecycle. We run the evals. If everything looks good, then we run some more smoke tests, and then we take it to the first region. The first region looks good. Let it bake there for a few hours, days, whatever is the comfort level, and then take it out to the whole production. These are our callouts.

Wrap-Up

So far, we talked about a lot of requirements, which we wanted to cover in the agentic workload. We looked at how the vanilla DevX for trip-planner agent was developed using the traditional Pythonic tools and APIs, and then we looked at how Ray Serve can solve a lot of those things right out of the box. Then we saw a bunch of callouts, which are very much specific to the agentic workloads.

Questions and Answers

Participant 1: You guys decided to use Ray. Were there other frameworks that went into considerations as part of that decision?

Deepak Mohanakumar Chandramouli: I think the first iteration of production deployment is what I showed before, taking it through the standard Kubernetes deployment. Then, we are part of the Ray team, so we maintain and develop and deploy Ray across the company. We see a lot of those problems that can be solved well with Ray. That led us to the natural transition from taking from a Kubernetes deployment to Ray. A lot of other reasons as well. There's a lot of gravity on machine learning lifecycles, LLMs. LLMs run on Ray. It made natural sense for us to co-host agents with Ray, because the other parts of the machine learning lifecycle ecosystem run on Ray.

Bhumik Vinodkumar Thakkar: Another interesting thing we saw on the official Ray documentation is that MCPs can also be hosted using Ray Serve. End of the day, those are also FastAPIs. We saw that relevance as well that MCP servers are also suited for Ray Serve applications. We are also thinking that we want to use Ray Serve for our agentic workload. It gave us a validation as well that we are thinking in those right directions. There is actually a really good documentation from Ray about how to deploy MCPs using Ray Serve.

Participant 1: You definitely get a lot of good stuff out of the box with Ray.

Participant 2: The trip-planner example is a toy example, obviously. My question is, so the usage of this infrastructure approach, is this something that's already running in production for some of the real problems that you're working on, or is this a prototype that you are trying to make happen?

Bhumik Vinodkumar Thakkar: Trip-planner was definitely a prototype which we made up for this presentation.

Participant 2: The infra solution you're talking about, is this the real production solution or is this the prototype which you explained as on the example of a trip-planner?

Bhumik Vinodkumar Thakkar: We think that there is a lot of potential for this to be a production solution. I might not be able to comment definitively on what is production, what is not inside what we are doing. Definitely, this has a lot of potential to help handle the real-time use cases. Again, as I mentioned that the official guidance from Ray Serve about MCPs is validating this a lot as well.

Participant 3: You mentioned there would be later talks on this. You mentioned in-graph classifiers. I was wondering if you can give just like a one-minute overview of what that is and what technologies you use to accomplish that.

Bhumik Vinodkumar Thakkar: On a high level, it is making sure that every turn inside the agentic workloads, I'm making an MCP call, for example, to a search tool. Probably if it is coming back with some sort of natural language response, then I can do some sort of classifier on top of that. The graph here is the orchestration graph which gets built out here for an agentic workload. Here for three tools, it's a very small graph. Imagine that when you have some real use cases where there are at least 15 to 20 tools, and some of those tools are also natural language in nature, then that is where classifying those things will actually be very useful.

Participant 3: I was thinking more about if the classification is different, then just the policies on tool call is doing something different?

Bhumik Vinodkumar Thakkar: It is something like that. End of the day, we want to make sure that there are guardrails in place so that the agents don't misbehave. They don't do something unexpected. Since it is taking the user query, the users can ask all sorts of things for the agents to do. That is just to make sure that at every step, we are catching things before it is too late.

Participant 4: I was wondering if you could talk a little bit about how you're addressing orchestration of long-lived agents versus ephemeral agents. When I think of like a Ray workflow, I tend to think of something like ephemeral tests that are connected in a workflow. It sounds like from what you're saying that you're also connecting, say like turn-based long-lasting agents.

Bhumik Vinodkumar Thakkar: This is mostly about the ephemeral tasks that we want to submit, some ephemeral job or operation and spins up some resources and then it kills those resources.

Participant 4: It seems like it's fairly easy to handle ephemeral agents or ephemeral tasks. It sounds like you also have multi-turn agents that you're orchestrating as well. How are you performing that? How are you doing both that turn-based orchestration along with ephemeral organization that might occur in the same workflow?

Deepak Mohanakumar Chandramouli: There are two parts. In terms of running an agent offline, like an ephemeral agent, what we do is we take advantage of another construct called Ray Jobs. Within a RayCluster, you can do RayService applications which are online, Ray Jobs which are batch. Those are like parallel async jobs that can be run. The other way is what Bhumik touched upon a little bit. There is something called actors within Ray, which enables you to run async tasks, stateful tasks, or stateless tasks. They can run in parallel threads and then complete their tasks, and then come back to give the control back to the main thread. There are a couple of ways to achieve this. If you're talking about two different agents deployed at production grade and then talking to each other, that's a different architecture altogether. That's basically synchronous calls being made.

See more presentations with transcripts