The BA in The Loop: Business Analysis in the Age of AI

A recently-promoted senior business analyst has done a great job over the past few years with requirements elicitation and management, and then seeing those requirements through to delivery (since this can be a messy process, check out a guide to requirements elicitation and management). Recently, they have also become proficient in using a company-approved AI tool for business analysts and for accelerating their BA work. They even built their own agents to make it all work more smoothly and more efficiently.

Now, that amazing AI-first BA has transitioned to a new project where they will be working primarily in the discovery phase, something they've never done before. Knowing that they're already a strong BA, you provide them with some resources they can use to conduct discovery and encourage them to use AI to assist them.

Risks of AI-Generated Deliverables

What happened next? That BA produced a beautiful Business Model Canvas, followed by an equally lovely Product Vision Board. They even went so far as to produce a Radical Vision Statement. All of these deliverables, which were created with a heavy reliance on AI, can be a useful part of the discovery process (side note: I consider nothing to be mandatory in discovery except an open mind because different projects will require different processes and deliverables).

So, what's wrong with all that?

The BA had no prior experience with any of those deliverables and didn't really know the proper way to collect the information for them and what the information might be used for down the line. The BA created these deliverables with AI and did not have the knowledge or background to assess the quality of the outputs. In essence, the human-in-the-loop safeguard failed because the human didn't know enough to check the quality of the AI output.

That leads to the big question:

How do we know when a person has the correct knowledge and decision-making capability to be an effective human in the loop?

Defining the Effective "Human in the Loop" in Business Analysis

The above example is meant to illustrate that this is not a question about seniority, but rather about experience, knowledge, and decision-making ability. But if it were simply a question about seniority, we would discuss solutions such as never letting junior BAs use AI. Instead, we should break this question down further:

Four Questions That Separate Knowledge From Guesswork

• Does the BA have the relevant foundational knowledge? It's not necessary for them to have created a specific deliverable manually a certain number of times, but they should understand the problem that a deliverable is intended to solve, what the appropriate inputs are, and how the outputs might influence later decisions.
• Can the BA recognize uncertainty? Having a low level of knowledge about a certain deliverable or process may not be a problem on its own, but not being able to recognize a lack of knowledge is certainly problematic. In other words, a BA should be able to look at an AI output and recognize when they don't know whether an output is correct or not.
• Does the BA have the ability to validate output? And, building on that, does the BA actually validate output? Everyone who uses AI should know that the output can be confidently incorrect, so it is important to compare the output to sources, including stakeholders and external sources.
• Does the BA understand the consequences of being wrong? The more AI is adopted and incorporated into our work, the greater the chance of it being used in a way that can have significant impacts. It's one thing to use AI to reword an email or make a slide deck prettier, but it's quite another when it is being used to prototype a solution in the heavily regulated healthcare industry where people's lives might literally be at risk.

These are all heavy questions. And we should be seeking the answers to these questions about ourselves and our fellow business analysts when we are hiring, training, onboarding, and promoting them.

What to Ask When Hiring, Training, and Promoting BAs

All of these processes require us to ask questions such as the following:

• What information did you provide to the AI? What information did you exclude?
• What assumptions did the AI make?
• How did you validate the output?
• What parts of the output did you change? Why?
• What would make you unwilling to use the output?
• Who else should review or approve the output?
• What could happen if the output is wrong?

Seeking the answers to these questions should become a standard part of the hiring, training, onboarding, promotion, and performance review processes for business analysts. It's not enough to just grant access to AI tools and pair them with pre-AI processes and deliverables.

P.S. from Georgi: for another viewpoint on this, a bit earlier in the AI-adoption curve, see my own The Future-Ready Business Analyst: Why Generative AI Matters, worth reading alongside David’s take here.