What Happens When Rafiki Cannot Find The Answer?

Introduction

In my previous article, I explored how our SharePoint agent, Rafiki, complements the human support provided by a workplace Buddy. The idea is straightforward: our Buddy helps new colleagues understand the organisation and their role, while Rafiki helps them find information held in SharePoint.

But there’s another side to this.

What happens when someone asks Rafiki a question and it can’t find a useful answer?

It’s tempting to treat this as a problem with the AI. Perhaps the prompt needs adjusting, the agent needs refining, or something needs changing in its configuration. Sometimes that might be true. But I think unanswered questions raise a more interesting possibility: what if the problem isn’t Rafiki at all?

The information might exist, but can we find it?

Organisations accumulate documentation over time. Policies are published, guidance is written, procedures are updated, and SharePoint sites grow to accommodate new teams and requirements.

Yet having information somewhere in Microsoft 365 doesn’t necessarily mean someone can find it when they need it.

A document might sit in an unexpected location, use terminology employees wouldn’t think to search for, or be buried among several similar documents. The information may be perfectly valid, but its usefulness is limited if people struggle to discover it.

An AI agent provides another way to search for that knowledge, but it cannot guarantee that every relevant piece of information will be found or interpreted correctly.

When Rafiki can’t answer a question, it gives us a reason to investigate. Is the content missing? Is it difficult to discover? Or is there another explanation

Sometimes the guidance isn’t clear

There is another possibility worth considering: the information exists, but it doesn’t provide a clear answer.

Perhaps a policy describes the general principle without explaining how it applies in practice. Maybe two documents contain slightly different guidance, or a procedure assumes the reader already understands an internal process.

These situations can be difficult for employees, particularly new starters who haven’t yet learned the organisation’s unwritten rules.

They can also be difficult for an AI agent. If the underlying documentation is ambiguous or contradictory, improving the agent’s instructions won’t necessarily resolve the uncertainty.

This is where unanswered questions become useful feedback. They can highlight areas where our documentation needs clarification, ownership, or review.

Knowing when to involve a person

Of course, not every question should have a definitive answer in a document.

Some situations require context, professional judgement, approval, or a conversation with someone who understands the circumstances. Finding a policy is not the same as deciding how it applies to an individual case.

That’s why I don’t see human support as a fallback that AI should eliminate. It remains an important part of the experience.

If Rafiki cannot provide a reliable answer, the next step should be helping the colleague understand where to go next, whether that means consulting their Buddy, contacting the relevant team, or speaking to the appropriate subject matter expert.

The important distinction is between an agent that admits it cannot help and an experience that leaves someone stranded.

Turning unanswered questions into something useful

There is an opportunity here to look beyond individual responses.

If we record and review unanswered questions appropriately, recurring themes might reveal missing documentation, confusing procedures, outdated guidance, or information that employees struggle to locate.

That doesn’t mean every unanswered question indicates a content problem. Some may be too specific, outside the agent’s intended scope, or dependent on circumstances that require human input.

However, patterns could help us decide where improvements would make the biggest difference.

It also creates a useful feedback loop: employees ask questions, Rafiki helps where it can, and the questions that remain unanswered give us clues about what to improve.

For me, that’s an interesting part of introducing AI into an existing Microsoft 365 environment. The value isn’t limited to the answers an agent provides. There may also be value in understanding the questions it cannot answer.

What happens when your AI assistant can’t find the answer? Do you treat it primarily as a technical issue, or do you use unanswered questions to identify gaps in your organisational knowledge?

I’d be interested to hear how others are approaching this, particularly when the answer involves not just improving the AI, but improving the information and processes behind it.