This article is sponsored by BigPanda and was written, edited, and published in alignment with our Emerj sponsored content guidelines. Learn more about our thought leadership and content creation services on our Emerj Media Services page.
Every business-critical service failure starts the clock on lost revenue and customer trust. Enterprise systems now run at machine speed, generating alerts faster than humans can respond to.
After a global IT outage in July 2024, triggered by a single faulty software update, one major U.S. airline reported to the SEC about $380 million in lost revenue and $170 million in added costs from roughly 7,000 flight cancellations over five days.
A study of a large commercial bank, presented at the 2020 International Conference on Software Engineering, found alert storms of hundreds to thousands of alerts a minute occurring about weekly, each taking six engineers 55 minutes on average to handle. Of 44 engineers surveyed, 77% said there were too many alerts to identify problems manually. The researchers also observed that most engineers tended to ignore the storms until customer complaints exposed the failure.
Emerj recently hosted a series on the AI in Business Podcast examining how enterprises can modernize IT operations by preventing recurring failures, correlating alert noise into actionable incidents, and automating repetitive L1 work, including guests Assaf Resnick, Founder & CEO at BigPanda; Luke Rotta, Director of Site Reliability Engineering at Charles Schwab; Dave Glick, Senior Vice President of Growth Technology at Walmart; and Dhrubojyoti Das Deb, Senior Vice President of Technology at JPMorganChase.
This article examines the key insights enterprise leaders can apply as the economics of IT operations shift from scaling teams to scaling intelligence:
- Enterprise-owned knowledge graph for faster incident resolution: Capture incident history, live system data, and operators’ undocumented knowledge so agents can pinpoint root causes and recommend fixes in minutes.
- Cross-application alert correlation for faster root cause analysis: Group thousands of related alerts into single incidents and trace them across connected applications so teams stop investigating symptoms and find the originating fault sooner.
- Volume-first automation priorities to recover team capacity: Automate high-volume, repetitive L1 tasks before high-risk ones and reinvest the hours saved into further automation.
- Ticket elimination before added support headcount: Fix the causes of recurring tickets instead of adding L1 and L2 staff, freeing engineers for cost-saving and revenue-driving work.
Listen to the full episodes below:
Episode 1: Inside the Shift to Agentic IT Ops – with Assaf Resnick of BigPanda
Guest: Assaf Resnick, Founder & CEO at BigPanda
Expertise: AI in IT Operations, Enterprise Automation, Venture Capital, Enterprise Software
Brief Recognition: Assaf Resnick is Founder & CEO of BigPanda, a company he has led since 2012. Before founding BigPanda, he was a Principal at Sequoia Capital, where he invested in early- and growth-stage software and internet companies. He holds a Bachelor of Science in Business Administration from the University of California, Berkeley’s Haas School of Business.
Episode 2: How to Reset Your IT Operations and Build for Intelligent Response – with Luke Rotta of Charles Schwab
Guest: Luke Rotta, Director of Site Reliability Engineering at Charles Schwab
Expertise: Enterprise AI Strategy, IT Strategy, Site Reliability Engineering, Digital Transformation
Brief Recognition: Luke Rotta is Director of Site Reliability Engineering at Charles Schwab, with 25 years of financial services technology leadership. He leads reliability engineering, automation, and operational improvements across Schwab’s wealth technology platform. He also taught DevOps through the University of Chicago’s professional education program and served on an advisory board at Seton Hall University. Rotta holds an M.S. in Business Information Technology and a B.S. in Computer Science from DePaul University.
Episode 3: Intelligent Operations at Enterprise Scale – with Dave Glick of Walmart
Guest: Dave Glick, SVP of Enterprise Business Services at Walmart
Expertise: Generative AI, Distributed Systems, E-commerce Technology, Technology Leadership
Brief Recognition: Dave Glick is SVP of Enterprise Business Services at Walmart. Previously, he served as CTO of Flexe, a logistics technology company, and spent 20 years at Amazon building core retail and operations systems, including the company’s original automated pricing system, warehouse management capabilities, and Amazon Flex. He also served on the boards of Crowd Cow and Businesses Ending Slavery and Trafficking (BEST). Glick holds a Ph.D. in Physics from the University of North Carolina at Chapel Hill and a B.S. in Physics from the University of Michigan.
Episode 4: Reducing Alert Noise with AI in Enterprise IT – with Dhrubojyoti Das Deb of JPMorganChase
Guest: Dhrubojyoti Das Deb, Senior Vice President of Technology at JPMorganChase
Expertise: Enterprise Architecture, GenAI, Cloud Modernization, Banking and Payments Technology
Brief Recognition: Dhrubojyoti Das Deb is Senior Vice President of Technology at JPMorganChase, where he leads engineering for a major e-commerce credit and debit card processing platform handling nearly 70% of U.S. merchant traffic. He leads more than 100 software engineers across the U.S., India, and Singapore, overseeing architecture and delivery for systems designed for high availability and fault tolerance. Previously, he spent more than 12 years at Cognizant in technical architecture, engineering, and delivery leadership roles. He holds a B.Tech in Applied Electronics and Instrumentation Engineering from Haldia Institute of Technology.
Enterprise-Owned Knowledge Graph for Faster Incident Resolution
Assaf Resnick opens the series by arguing that enterprises are treating IT operations as a staffing problem when it has become a data problem. Hybrid cloud, microservices, and CI/CD (Continuous Integration and Continuous Delivery) mean systems change and break faster than ever, generating, by his estimate, a thousand times more data at a thousand times the speed.
Outsourcing to global systems integrators, the cost-cutting trend of the past three decades, has not solved the problem. With an average tenure of about seven months, engineers often leave soon after they learn the environment. Round-the-clock coverage adds three shift handoffs a day, each a point where context can be lost. Dhrubojyoti Das Deb raises the same risk later in the series: knowledge of legacy applications leaves with the subject matter experts who hold it.
Resnick’s alternative is to onboard agents the way an enterprise would onboard a new employee, through an IT knowledge graph that holds three kinds of context:
- Live awareness: What is happening across the environment right now, connected at a scale no human operator can track.
- Operational history: Everything that has happened in the environment, including transcripts from previous bridge calls.
- Tribal knowledge: The undocumented expertise of long-tenured operators, plus what agents learn by observing how the strongest operators resolve problems.
He illustrates the impact with a hypothetical network issue affecting a large bank’s ATMs. Today, thousands of alerts reach an L1 team that may be new to the environment. A 15-minute triage target stretches to two hours, and a bridge call runs six more hours before someone traces the cause to a security update made three weeks earlier.
With the knowledge graph in place, an agent spots the problem in about 30 seconds, matches it to a similar incident from seven months earlier, and recommends rolling back the change.
Luke Rotta notes that while incidents share patterns, in more than two decades supporting systems he has found each one has some piece that is slightly different. Where an agent’s confidence falls below its threshold, Resnick says that it hands the issue to a human. It routes the issue to an L1 operator or directly to the specialist who owns that domain, rather than convening a full bridge call. In Rotta’s experience, getting the right people involved is where much of the slowness in resolving incidents comes from.
Resnick illustrates the shift in cost structure as a move from roughly 300 outsourced operators to an agentic layer managed by two or three site reliability engineers. He also points to a Fortune 500 healthcare customer that reduced incidents and self-inflicted outages by an amount he valued at half a billion dollars.
He advises against changing tooling, processes, outsourcing relationships, and org charts all at once, comparing the shift to how CIOs moved to public cloud, one workflow at a time:
- Add an agentic coworker: Bring the agent into bridge calls and give it access to operational data so it learns how teams resolve incidents.
- Start with low-barrier workflows: Point it at real, recurring problems where a quick win is likely.
- Promote early wins internally: Use visible results to build confidence for broader rollout.
- Expand toward full coverage: Once the agent handles around 30% of the work, set a roadmap toward the rest.
Resnick is equally clear about who should own that knowledge, regardless of which vendors or consultants help build it:
“I think the important thing is that you build an IT knowledge graph that encapsulates all of the knowledge and data and history within your enterprise. And that should be yours. It shouldn’t sit inside of the walled garden of a specific vendor.”
— Assaf Resnick, Founder & CEO at BigPanda
Cross-Application Alert Correlation for Faster Root Cause Analysis
The bottleneck in incident response, according to Dhrubojyoti Das Deb, is no longer a lack of data but knowing where a problem started. Logging, tracing, and telemetry have grown with the business, but each application’s data still sits in its own silo.
In a chain of five connected applications, he explains, the fourth may show the error while the problem started in the first, which raised no alerts and passed bad data through the second and third. Teams lose time investigating the application that shows the error, and the issue persists until they trace the fault back to its source.
Luke Rotta reframes the cost of that delay as opportunity cost rather than downtime alone. While a service is unavailable, the business cannot capture new sales or transactions, and the impact varies with how busy that time of day is.
Dave Glick sees the same drag in engineering. When a ticket arrives, engineers act as “digital detectives,” working out what happened and trying to reproduce it, toil he wants removed from engineers and operators alike.
Alongside the loss of expert knowledge, Das Deb identifies three frictions that make it hard to separate signal from noise:
- Alert overload: He estimates that 99% of incoming data is noise, such as debug and informational messages.
- Complex dependency stacks: Applications built on different languages and platforms, from Java and Python to cloud and AI services, create dependencies that are harder to trace.
- Constant change: Multiple CI/CD releases a week change an application’s state, and new data may not reach traceability and telemetry.
Rotta argues that unintegrated tools keep multiplying without solving the real problem: getting data contextualized fast enough for responders. Das Deb’s teams aim to keep analysts in one to three tools, using a single tool across as many applications as possible so data can be correlated.
Rotta defines what that context must answer during an incident: what changed, what is broken, why it isn’t working, who to contact, and what happened last time.
Das Deb sees AI handling much of that work in two stages:
- Intelligent grouping: Combine alerts with incident history, deployment history, and application growth to collapse thousands of alerts into a few unique incidents.
- Root cause analysis: Correlate logs, traces, metrics, deployments, and code changes to identify the likely cause.
Rotta stresses that this takes multiple agents and a context layer rather than one large agent, which cannot process that volume of data. Assaf Resnick describes what that looks like: specialist agents for databases, networks, and cloud investigating an incident together, rather than a bridge call pulling in the organization’s most experienced people.
Das Deb describes how the analyst’s work changes:
“What it can do is go through those 5,000 messages and say, all of them are variations of two incidents. So that way, 5,000 incidents are getting reduced to two, and then somebody can go and check these two. That’s where the human in the loop comes in. AI does the smart work for us, and we take the judgment and decision.”
— Dhrubojyoti Das Deb, Senior Vice President of Technology at JPMorganChase.
In practice, the workflow shifts:
Before: Analysts work through thousands of alerts and investigate the application that raised them, which may not be the source.
After: AI groups alerts into a handful of incidents and correlates them with logs, deployments, and history, so analysts validate a likely cause instead of searching for one.
Das Deb describes the process as a cycle: a person validates the output, AI refines it, the fix is deployed, and new observability data feeds back into the model. He sees automated remediation and self-healing as a later stage, since AI cannot be trusted to solve everything from day one and needs to be trained step by step.
He also stresses structured data. If a log lacks a timestamp, the sub-module name, the application name, or context on what the error relates to, AI cannot tell where an issue came from. Many legacy production environments, he adds, need to be made ready before they can supply that data.
Volume-First Automation Priorities to Recover Team Capacity
Luke Rotta recommends automating volume before risk. High-risk failures have usually happened before and probably already have systems in place to catch them. The bigger drain is small, repetitive work. Rotta notes that many financial firms still have people manually fixing servers that run out of disk space, a three-decade-old problem. Automation, increasingly AI-driven, can resolve known patterns like this, report how often it acted, and escalate only what it cannot fix:
“I think the highest volume, because a lot of times operational teams, it’s a death by a thousand cuts. That’s what really wears on the mindset, the culture, the capacity of the team, all these little things every day. So I think if you go after volume, you can find the themes and you can certainly use automation, AI to start shifting that workload over to a machine versus a human.”
— Luke Rotta, Director of Site Reliability Engineering at Charles Schwab
Assaf Resnick describes the manual L1 effort agentic operations remove. Before an incident is even confirmed, operators phone network, cloud, and development teams to ask whether anyone sees a problem. With agents in place, he says, that runaround no longer happens.
At Dave Glick’s weekly AI office hours, about three quarters of demos reduce toil, such as change requests and release management, rather than build production features. A weekly ticket summary that once took a dedicated team now takes AI minutes, and it surfaces the patterns.
In a previous environment, Rotta’s team got hours back daily and reinvested them in more automation. The obstacle is what he calls the “hamster wheel”: teams have no time to automate the volume that consumes them, so turning a few things off may be one way to win back the first hours. Glick adds that freed capacity does not mean less work. AI lets non-engineers handle requests that never made the annual plan, so “the pie is getting bigger.”
Asked where leaders should start, Rotta describes once shutting off all alerting for a day. Nothing broke, the business was unaffected, and the few issues that did surface showed which alerts were worth keeping. He acknowledges heavily regulated firms may lack the appetite for that step, so his practical starting point is to keep reporting on raw alerts but react only to SLO (service level objective) and SLI (service level indicator) breaches.
For choosing what to automate, Dhrubojyoti Das Deb recommends listing every operational friction, prioritizing by risk or by ease, and automating the easiest pieces first. Two working out of ten builds the confidence to tackle the rest.
Across the series, the guests offer three practices for scaling automation safely:
- Prove it first: Rotta suggests running incidents the usual human-led way while AI simulates its actions, then comparing the results. Testing against a simulated application is faster than a full proof of concept.
- Keep accountability with people: Das Deb is clear that no company can tell a customer “AI did this.” Product and business teams, which sell to customers, need to understand the automation and own its risk first.
- Govern through existing controls: Glick’s teams apply the data privacy and governance policies they already had, with InfoSec and compliance teams using the same coding agent. A registry shows whether an agent already exists, a launch pad moves agents into production, and his aim is one release management agent and a few testing agents for the organization, not a hundred personal versions.
Ticket Elimination Before Added Support Headcount
Adding more L1 and L2 support to absorb rising ticket volume, Dave Glick contends, treats the symptom, not the cause. In engineering, as he prefers to frame it, teams are always balancing features that reduce costs and drive revenue against keeping existing systems running, and every recurring ticket takes time away from building those features:
“My mantra is eliminate, automate, optimize. The best ticket is no ticket. If we get out of a whole class of tickets, we can spend our time building features that affect customers and sellers.”
— Dave Glick, SVP of Enterprise Business Services at Walmart
Glick focuses on what he calls paper cuts, the small, long-standing issues that frustrate customers. In one case, sellers kept hitting errors because of an image upload limit no one could explain. Raising it took a one-line code change, aimed at reducing errors and seller frustration. Left alone, he warns, these distractions only get bigger.
His paper-cuts program runs on three practices:
- Keep a running list: His current list holds around a hundred minor fixes that teams otherwise never get to.
- Fix now, not next quarter: When the team proposed scheduling the work for Q3, his response was “How about tonight?” A few fixes each week add up over six months.
- Let AI take a first pass: He is testing whether a coding agent can work through the list and draft fixes for engineers to review, though he doesn’t think it can clear all hundred yet.
He also had a coding agent write the test suite for a careers site launch in about a week, work that would have taken months by hand. His team was closing about 15 bugs a day, but the product team kept raising edge cases no one had tested, and without the agent the launch date would have been missed. The lasting change is the default question. His teams no longer wait for something to go wrong before asking how AI can help.
Assaf Resnick positions prevention, rather than faster reaction, as the starting line for agentic operations, using past incidents to reduce future mistakes. One of his examples is change-management agents that review the changes an enterprise is about to make and stop risky ones from going through. In his ATM example, that kind of change caused an eight-hour incident.
Dhrubojyoti Das Deb points to a simple signal for where to look. When the same incident keeps repeating, the repetition is itself a failure to address.
Taken together, the guests point to three places to cut ticket volume before adding headcount:
- Start with repeats: Any ticket or incident that keeps coming back is a candidate for permanent elimination.
- Clear the paper cuts: Fix small, long-standing issues every week, with a coding agent drafting fixes where possible.
- Gate risky changes: Review planned changes before they ship, so risky ones are stopped before they cause incidents.