Optimizing Software Workflow: Exploring Fundamental Impacts in Technology

Software only has value when shipped; faster release cadence matters. Development, deployment, and operations require different thinking, metrics, and investments, Titus Winters explained in his talk Software Workflow Optimization at Craft Conference. He mentioned four impacts in tech: product success, compute capacity, engineering capacity, and strategic innovation. The biggest expense is people, not compute, which goes mostly unmeasured. AI shifts engineers toward deeper problem understanding, not fewer engineers, he argued.

Developing software with or without AI, we have to carefully reason about the goals, inputs, and outputs of each phase of the software workflow, Winters mentioned. He referred to the paper Develop, Deploy, Operate - a holistic model for understanding the costs and value of software development that he wrote with Leah Rivers and Salim Virji, which provides guidance for reasoning about impact and efficiency in the software process, whether AI-assisted or not.

Software only has value when you have shipped it to customers, Winters argued. When bugs happen in production, you are risking actual bottom line losses, he added. There is value in being able to remedy an issue by shipping a well-tested update quickly and reliably. Faster release cadence is almost always better, and probably better than your leadership recognizes.

Development is a creative process; this is what we want humans to spend time on, Winters argued. Deployment is a factory process, and operations is an automation process:

These three processes aren’t just different activities; they require different thinking, different metrics, and different investments.

Winters mentioned four fundamental impacts in tech:

• Product success: Revenue, adoption, customer satisfaction, NPS, etc
• Compute capacity: Cloud costs, storage, networking, etc
• Engineering capacity: "Productivity", human efficiency
• Strategic innovation: Make new good things cheap/possible, make existing bad things rare/impossible

We have to be looking at all four of these things and balancing the impact mixture, Winters mentioned:

Running fewer tests might save you on compute costs, but the resulting quality issues might cause actual bottom-line losses or engineering burnout when you try to get the product to stability.

The biggest expense is rarely compute; it’s people, Winters said. We cannot directly measure what those people are producing or what the production of it actually costs. We’ve been measuring everything except the biggest line item, which hardly seems rigorous, he added.

Software engineering is not just coding; it’s also people and time, and we cannot measure the people part. Winters mentioned that with the shift to AI for coding tasks, we might have the potential to build experiments using AI as a proxy for the human parts of development.

We want our engineers to be solving problems primarily, Winters argued. With AI, developers are going to remain in the loop, but no longer free of guardrails. They work within a system that has encoded expert knowledge and organizational policies, so they don’t have to internalize all of the details and can focus on the unique problem at hand. The shift that we are looking towards is not a reduction in the number of engineers; it will be a shift away from focusing only on raw software expertise, in favor of a deeper understanding of the actual problem that is being solved, he concluded.

InfoQ interviewed Titus Winters after his talk.

InfoQ: What are the challenges of managing the impact of engineering capacity, and how can we deal with those challenges?

Titus Winters: Compared to tracking token costs or cloud spend, it’s a lot harder to track where engineering time is going. Even for companies that do, we need enough understanding of the engineering process to decide whether a given task is wasteful, and that requires some subtlety.

We were making good inroads on explaining that developer experience and productivity are important related concerns, but AI seems to have sucked the air out of that. Not because the necessary investments changed, but because the discourse is so messy. The same lessons are still true post-AI, just accelerated: we need clear processes, clean code, good tools, fast feedback signals, good tests, etc.

InfoQ: How can we build experiments with AI to evaluate the human parts of development?

Winters: If we accept that token consumption is a tolerable proxy for human effort in software tasks, we can start to build experiments to put a quantitative cost or benefit on various aspects of software.

What we need is a robust set of evals for software agents: across a variety of software tasks (new feature, bugfix, refactoring), across the platforms we care about (frontend, backend, mobile, etc), across the languages we care about, and at a range of difficulties. Each software task in this eval suite needs a set of functional and quality acceptance criteria, hidden from the AI. And we need the ability to measure token costs for each eval run.

With that setup, we can now run empirical tests. Do you want to measure whether good unit test coverage is actually a good thing? Get two agents, one that is prompted to follow all of our practices in unit testing, one that actively doesn’t write unit tests. Run the eval suite, probably repeatedly. Measure token consumption. Important practices should likely reduce token cost across the board. Specialized practices may only apply in some regions of the eval space.

The same thing applies to tools. Would a semantic index help humans navigate a large codebase? Or more specifically, how much? Give the experimental agent access to a good indexer, leave the control agent with its standard out-of-the-box approach. We spend a few more tokens explaining how to use the tools, but if the tool pays off in net token costs, we should invest in that tooling.