Case 01
Critical Meeting Preparation
Scenario
You are presenting a proposal to the Operations Leadership Forum at Northstar Financial Services. The proposal recommends launching a small pilot for an internal knowledge assistant that helps staff find policy guidance, process steps, and reusable templates faster.
Draft Content to Review
We are proposing a 6-week pilot of an internal knowledge assistant for operations staff. The assistant will help users ask questions in natural language and receive answers based on approved internal documents. The pilot will focus on three areas: account maintenance, service request handling, and common exception workflows.
The current process requires staff to search across multiple SharePoint folders, older email threads, process PDFs, and team FAQs. This creates inconsistent answers, repeated questions to subject matter experts, and time wasted searching for information. The new assistant will reduce search time and improve consistency.
The pilot will include 30 users from Operations, Client Service, and Business Support. The tool will not make decisions, approve cases, or send client communications. It will only retrieve and summarise internal reference material. Users will be reminded to verify answers against source links.
Expected benefits include faster onboarding, fewer repeated questions, and improved consistency in how staff follow process guidance. We believe the pilot can reduce search time by 30 percent, but this estimate is based on informal feedback and not on a measured baseline.
The pilot will use existing approved documents. Access will follow existing permissions. We will capture feedback from users every week and review sample answers for quality. At the end of the pilot, we will recommend whether to stop, extend, or scale the solution.
Areas still being finalised include the exact document set, ownership of answer quality, success metrics, and the ongoing operating model if the pilot is successful.
Known Tensions
- Compliance wants clear ownership for incorrect or outdated responses.
- Operations leaders want evidence of productivity improvement, not just user satisfaction.
- Technology wants to limit scope to avoid integration complexity during the pilot.
- Some team managers worry staff may over-rely on generated answers.
Prompt
Act as my critical colleague before an important meeting.
Review the content below and help me prepare properly. Identify what is unclear, what may be challenged, what is missing, and where the logic is weak.
Then give me:
- Key talking points
- Likely questions or objections
- Strong responses I can give
- Risks or sensitive areas to handle carefully
- A sharper version of the content that is clearer, stronger, and easier to present
Focus on making me prepared, confident, and hard to challenge.
Example output from ChatGPT
Operations Leadership Forum: Critical Review & Meeting Preparation
Overall assessment: Promising proposal, not yet approval-ready
Your proposal has the right foundations: a real operational problem, a limited pilot, defined use cases, and boundaries around what the assistant can do.
But the proposal is vulnerable to one fundamental challenge:
You are asking leadership to approve a pilot before defining how success will be measured, who owns the risks, and what safeguards will protect operations.
That is likely to become the centre of the discussion.
The stronger approach is to stop positioning this as an AI assistant experiment and start positioning it as a controlled operational improvement with a measurable business case.
1. Where the proposal is weak
1. The 30% productivity claim is not defensible
Informal feedback cannot establish expected savings. Leaders may challenge the baseline, the sample, and whether search-time savings translate into productive capacity.
Fix: Treat 30% as a test hypothesis, not a benefit commitment. Measure actual retrieval time before and during the pilot.
2. Accountability is unresolved
Saying users must verify responses shifts responsibility onto staff without explaining who owns document accuracy, assistant quality, incidents, or correction.
Fix: Assign named business, knowledge, technology, and control owners before launch.
3. "Approved internal documents" is too vague
Approved by whom? Are they current? What if the assistant finds two conflicting procedures? What if an old FAQ contradicts a newly updated policy?
Fix: Create a small, version-controlled document inventory with named owners, precedence rules, and an exclusion policy.
4. Existing permissions do not automatically mean safe retrieval
Search indexes, cached content, service accounts, and generated summaries can introduce access risks even when source repositories have permission controls.
Fix: Verify source-level authorisation at retrieval, including negative tests across different user roles.
5. The pilot lacks a credible experimental design
Thirty users for six weeks sounds manageable, but what is the baseline, control group, task mix, expected usage, or threshold for passing?
Fix: Compare matched tasks using current search methods against assistant-assisted retrieval, while accounting for accuracy and verification time.
6. Six weeks may be too short for the stated scope
Finalising documents, clearing controls, configuring permissions, onboarding users, and collecting credible evidence could consume most of the pilot period.
Fix: Separate readiness from live evaluation. Start the six-week clock only when launch gates are satisfied.
2. What leadership will want answered
The forum represents four different decision-making perspectives. You need to address all four before they raise their concerns.
| Stakeholder | What they really want to know | What you must demonstrate |
|---|---|---|
| Operations leadership | Is this worth investing time and money in? | Measured net time savings and a credible path to value |
| Compliance and Risk | Who is accountable when something goes wrong? | Clear ownership, traceability, escalation, and stop controls |
| Technology | Can we deliver this without creating a platform project? | Fixed scope, approved components, simple architecture, support responsibilities |
| Team managers | Will staff trust the wrong answer or stop thinking? | Training, verification expectations, quality testing, feedback mechanisms |
Important distinction: A search assistant does not become low-risk simply because it cannot execute transactions. Incorrect policy guidance can still cause financial, regulatory, operational, or client harm when a person acts on it.
3. The pilot I would take to the forum
Northstar Knowledge Assistant
Proposed pilot
6
Live weeks
30
Users
3
Workflow areas
Recommended decision criteria
Proposed thresholds for discussion, not validated targets or commitments.
| Net retrieval-time improvement | ≥20% |
|---|---|
| Grounded, correct responses on evaluation set | ≥95% |
| Unauthorised information disclosures | 0 |
| Unresolved high-severity defects at launch | 0 |
| Mandatory source evidence | 100% of substantive answers |
These should be agreed with Operations, Compliance, and Technology. Aggregate accuracy alone is not sufficient: high-consequence guidance needs separate evaluation and escalation controls.
Recommended pilot boundaries
- In scope: Finding approved process guidance, summarising documented steps, locating controlled templates, and identifying the relevant policy source.
- Out of scope: Case-specific approval, policy interpretation where guidance is ambiguous, external client communications, transaction execution, and recommendations that override an authorised decision-maker.
- Source hierarchy: Approved policies first, followed by controlled procedures and current templates. Unmanaged email threads and informal FAQs should not become authoritative sources.
- Failure behaviour: When sources are missing, contradictory, outdated, or access-restricted, the assistant should say it cannot establish a reliable answer and direct users to the appropriate team.
And one design principle should be non-negotiable:
The assistant must be able to decline to answer. A confident answer is not necessarily a correct or authorised answer.
4. Key talking points for the meeting
I would organise your presentation around five messages, in this order.
-
We are solving a retrieval problem, not looking for an AI use case.
Staff already have the information, but finding the correct version across fragmented repositories takes time and creates friction.
-
We are testing operational value, not claiming it in advance.
The 30% saving is an initial hypothesis. The pilot will measure time, correctness, and verification effort against the current process.
-
The system will assist staff, not replace their judgement or authority.
No approvals, decisions, client communications, or transactional actions. Staff remain responsible for following established procedures.
-
Governance is part of the pilot design, not a later enhancement.
We will define source ownership, permissions, answer testing, incident handling, and change management before live use.
-
We are asking for permission to learn within boundaries, not permission to scale.
The forum will receive evidence at the end and make a separate decision to stop, extend, or consider scaling.
5. Likely objections and strong responses
These are the questions I would expect from a challenging leadership forum.
Where does the 30% saving come from?
It is a working hypothesis based on informal operational feedback, not a validated estimate. We will establish a baseline using representative tasks and measure the total time required to find, verify, and use guidance. I would rather return with a defensible 15% improvement than claim 30% we cannot substantiate.
What happens if the assistant gives incorrect policy guidance?
We will have a named business owner for each source, a designated owner for assistant quality, and a clear reporting and correction process. Every substantive answer will link to its evidence. Where evidence is ambiguous or conflicting, the assistant should not fabricate a resolution. High-severity failures will trigger containment and a documented review.
Why do we need AI? Why not just clean up SharePoint?
We should improve the underlying knowledge base regardless. The pilot tests whether natural-language retrieval adds measurable value on top of organised, controlled content. We should benchmark the assistant against the current search method, not assume AI is superior. If better document organisation solves the problem at lower cost and risk, that is a valid outcome.
Doesn't asking users to verify sources remove the productivity benefit?
That is exactly why we will measure the full task, including verification time. The proposition is not that verification disappears; it is that finding the right source and relevant section becomes faster. If the total task time does not improve, we should not claim a productivity benefit.
How can existing access permissions be trusted in the AI layer?
We will use an approved retrieval architecture, enforce access checks for the requesting user, and test restricted-document scenarios before launch. We will not rely on a generic statement that existing permissions carry through automatically. Technology and Information Security must confirm this control.
Are 30 users enough to justify scaling?
Thirty users can provide meaningful operational feedback and task-level measurements for a limited pilot, but they are not sufficient by themselves to prove enterprise-wide benefits. We will sample across roles and workflow types, document limitations, and use results to decide whether a larger controlled evaluation is justified.
What is the total cost and who will support it?
We will define the pilot cost envelope before approval, including licensing or usage, configuration, testing, training, document preparation, and support time. A scaling recommendation would include recurring costs, operational ownership, maintenance, and a business case based on measured rather than assumed benefit.
What if the assistant is successful but employees become dependent on it?
We will train users on what the tool can and cannot do, make the evidence visible, require normal approvals, and test whether users recognise incomplete or conflicting information. We will also evaluate incorrect reliance, not just answer satisfaction. The goal is faster access to guidance, not delegated policy judgement.
6. How to prove productivity improvement
This will be one of the most scrutinised parts of your business case.
Measure total successful task time, not simply how fast the assistant produces an answer.
Illustrative productivity model
Current task time (minutes)
8
Pilot task time (minutes)
6
Tasks per user per week
10
Active users
30
Retrieval-time change
25.0%
Gross hours saved per week
10.0 h
Illustrative inputs only, not Northstar measurements. Results exclude implementation, maintenance, training, error remediation and other overheads. Hours saved are not automatically cash savings.
For the actual evaluation, I recommend three layers:
| Layer | Measurement | Why it matters |
|---|---|---|
| Efficiency | Median and percentile task-completion time, including verification | Establishes net task improvement |
| Quality | Correctness, citation support, completeness, current-policy alignment, abstention | Tests whether faster is also reliable |
| Operational value | Repeated SME enquiries, rework, task success, onboarding effort | Shows whether improvements extend beyond search |
Use a pre-agreed evaluation set that includes straightforward, ambiguous, conflicting, outdated, and restricted-information scenarios. Assess both the existing method and the assistant against that set.
For better causal evidence, randomly alternate methods on comparable tasks where feasible. Otherwise, differences in task difficulty, participant experience, or workflow changes could make the apparent savings misleading.
7. Risks and sensitive areas to handle carefully
| Risk | Required control or response |
|---|---|
| Outdated procedures | Document owner, validity review, version tracking, refresh and removal process |
| Conflicting sources | Explicit precedence rules and escalation instead of guessing |
| Permission leakage | User-specific authorisation tests, including restricted and cross-team documents |
| Hallucinated guidance | Answer-level grounding, systematic evaluation and appropriate refusal |
| Sensitive information exposure | Data classification, minimisation, approved processing environment, logging restrictions and retention controls |
| Prompt injection in retrieved documents | Treat retrieved text as untrusted instructions; test malicious document content |
| Excessive human reliance | Training, clear authority boundaries, verification and escalation requirements |
| Model or document changes | Version control, regression testing and rollback procedures |
| Weak adoption evidence | Usage analytics plus interviews and representative task testing |
| Hidden operating cost | Full cost inventory including licences, SME time, controls and ongoing support |
One especially sensitive issue: Avoid describing the system as providing "approved answers." The documents may be approved, but the generated summaries themselves have not necessarily been approved.
A more accurate phrase is:
"The assistant generates source-linked responses from a controlled collection of approved documents. Those responses remain subject to established policy authority and human verification."
8. Sharper version of your proposal
This is the version I would present to the Operations Leadership Forum. It is more explicit about the business problem, controls, evidence, and decision being requested.
Northstar Knowledge Assistant
Proposal for a Six-Week Controlled Operations Pilot
Executive recommendation
We seek approval to prepare and launch a six-week, controlled pilot of an internal knowledge assistant for 30 employees across Operations, Client Service, and Business Support, subject to agreed readiness criteria.
The objective is to determine whether natural-language access to approved internal knowledge can improve operational efficiency and consistency without introducing unacceptable information, compliance, or operational risks.
This is a proposal to test a measurable operational improvement, not a request to approve enterprise-wide AI deployment.
1. The problem
Operations employees rely on policies, standard operating procedures, process documents, and templates distributed across multiple repositories.
Finding the correct guidance often requires navigating SharePoint folders, searching PDF documents, reviewing team FAQs, or contacting subject matter experts.
This creates three potential operational inefficiencies:
- Time spent finding and validating the correct information.
- Repeated reliance on subject matter experts for routine questions.
- Inconsistent interpretation or application of documented guidance.
We have qualitative indications of these issues, but have not yet established their frequency, financial impact, or measured baseline.
2. The proposed solution
We propose a retrieval-based internal assistant that allows authorised employees to ask natural-language questions and receive concise, source-linked responses from a controlled collection of approved internal documents.
The assistant will initially support:
- Account maintenance.
- Service request handling.
- Common exception workflows.
The assistant will help employees locate relevant policies, understand documented process steps, and find controlled templates.
It will not approve requests, determine policy exceptions, execute transactions, modify business records, or communicate with clients.
Existing decision-making authority and normal control procedures will remain unchanged.
3. Controlled scope and governance
The pilot will use a limited, explicitly approved document inventory with accountable business owners.
Before launch, we will establish:
- Document ownership, validity checks and source precedence.
- Role-based retrieval permissions and negative access testing.
- Named accountability for answer quality, operational usage and incident management.
- Quality evaluation against representative operational questions.
- Clear handling of missing, outdated, contradictory or insufficient evidence.
- Logging, feedback, correction, escalation, suspension and rollback procedures.
- Mandatory training on limitations and appropriate verification.
The assistant will provide source citations for substantive answers and should decline to provide guidance where reliable supporting evidence is unavailable.
4. Measuring success
The primary measure will be improvement in successful task completion time, including the effort required to verify the answer.
We will establish a baseline against existing retrieval methods and compare results using representative operational tasks.
Additional measures will include:
- Accuracy and completeness against approved source material.
- Rate of valid, supported answers and appropriate refusals.
- Retrieval permission failures and control incidents.
- Reduction in repeat subject matter expert enquiries.
- User adoption and observed reliance behaviours.
- Pilot implementation and operating costs.
The previously suggested 30% reduction in search time will be treated as an unvalidated hypothesis, not a committed outcome.
We propose an initial 20% net task-time improvement threshold for discussion, subject to agreement on measurement methodology and minimum quality and control requirements.
5. Delivery approach
Pre-pilot readiness: Finalise the document inventory, business ownership, technical approach, evaluation design, risk controls, and cost envelope.
Weeks 1–2: Onboard participants, establish usage patterns, validate task measurement, and closely monitor initial responses.
Weeks 3–4: Conduct controlled evaluation, review quality and risk findings, and address issues within the approved scope.
Weeks 5–6: Complete comparative measurements, validate findings, review operating costs and prepare the decision recommendation.
The six-week live evaluation will begin only once readiness criteria have been satisfied.
6. Exit and decision criteria
At the conclusion, the team will recommend one of three outcomes:
Stop: Measurable value is insufficient, critical controls fail, or risks cannot be reduced to an acceptable level.
Extend: Results are promising but evidence, quality, adoption, or the operating model requires additional validation.
Consider scaling: Defined performance, control, and cost criteria are met, subject to a separate approval for broader deployment.
Successful completion of a pilot will not itself constitute approval to scale.
7. Decision requested
We request the forum's agreement to:
- Endorse the controlled pilot approach and initial scope.
- Nominate an accountable Operations sponsor and knowledge owners.
- Authorise readiness work, subject to a defined resource and cost envelope.
- Agree the evaluation principles and risk acceptance process.
- Delegate the go/no-go launch decision to the designated accountable authorities once readiness conditions are met.
The intended outcome is a clear, evidence-based decision on whether the assistant merits further investment, not a predetermined commitment to deploy AI.
9. Ownership and execution plan
This is the part I would not leave unresolved when walking into the meeting.
| Accountability | Proposed owner role | Responsibility |
|---|---|---|
| Business sponsor | Head of Operations or delegate | Business outcome, funding, scope |
| Knowledge authority | Designated process/policy owners | Source validity and authoritative guidance |
| Product owner | Operations transformation lead | Requirements, user experience, adoption |
| Technology owner | Enterprise Technology | Architecture, security implementation, operations |
| Quality owner | Named business SME lead supported by Technology | Evaluation, defect triage, quality reporting |
| Risk oversight | Compliance and Information Security | Independent challenge, controls, issue escalation |
| Evaluation lead | Operations performance or analytics | Baseline, measurements, benefit validation |
These are proposed roles, not confirmed Northstar appointments.
Readiness checklist
Pre-launch approval gates
0 of 7
Executive sponsor and accountable owners nominated
Document inventory and source precedence approved
Architecture and access-control testing signed off
Data security, retention and incident controls agreed
Representative evaluation set and metrics agreed
Cost envelope and support arrangements approved
User training and high-severity stop criteria confirmed
10. Your 90-second opening statement
This is how I would open the forum presentation.
Every day, our operations teams need to find the right policy, procedure, or template to complete their work.
The information generally exists. The challenge is finding the right version, understanding what applies, and doing that consistently across teams.
Today, that can mean searching several SharePoint folders, opening PDFs, or contacting a colleague who knows where the answer lives.
We think there's an opportunity to improve this. But rather than assume AI is the answer, we're proposing a small, controlled experiment to establish whether it actually helps.
The proposal is simple: 30 users, three operational workflow areas, and six weeks of live evaluation once the necessary controls are in place.
The assistant will retrieve and summarise approved internal knowledge, with links to the sources. It won't approve cases, make operational decisions, or communicate with clients.
And we will measure what matters: whether people complete tasks faster, whether the guidance is accurate, whether the controls work, and whether the benefit justifies the cost.
We've heard suggestions that this could reduce search time by 30%. We haven't validated that number, and I don't want us making an investment decision based on an assumption.
Instead, I want us to establish the baseline, run the evaluation, and return with evidence.
We're not asking for approval to scale an AI solution. We're asking for approval to test whether a controlled knowledge assistant can solve a real operational problem, and to make the next decision based on what we learn.
11. Final challenge: prepare yourself for the hardest questions
Use these questions to rehearse. Select a response mentally or speak it aloud, then reveal my recommended response.
LEADERSHIP CHALLENGE 1 OF 5
If the underlying policies are outdated, doesn't AI just make the wrong information easier to find?
Existing approval means we can trust all documents.
We will first clean the entire bank's knowledge base.
We need a controlled inventory, authoritative owners, and a rule to withhold uncertain guidance.
PreviousCheck response
12. Three recommendations before entering the room
First, remove the unqualified 30% benefit claim from your headline. Replace it with a measurable hypothesis and an evaluation methodology. Your credibility will improve when you openly distinguish what you know from what you intend to test.
Second, close the ownership gap before requesting live deployment approval. Compliance is right to challenge this. Named accountability, current source material, and enforceable access controls are necessary design conditions.
Third, make the decision requested precise. Are you seeking sponsorship for readiness work, authorisation to spend a defined amount, or permission to launch with real users? These are different approvals. I recommend requesting conditional pilot endorsement with separate readiness sign-off.
Your strongest closing line would be:
"The question today is not whether AI can answer operational questions. It is whether we can demonstrate that it helps our people find the right guidance faster, with the controls and accountability our business requires. That's what this pilot is designed to establish."
That framing makes the business problem central, treats the risks seriously, and gives leadership a defensible reason to support the next step without committing prematurely to scale.
