Why These 9 AI Vendor Questions, Specifically
Every AI vendor pitch sounds great. The differentiation is in how the team handles the awkward questions — the ones that make a polished sales engineer go quiet and look at the slide deck. These nine are the ones we wish every buyer asked before signing.
Each question is here because it caught a real failure mode in a real engagement. If your AI vendor cannot give you a confident, specific answer to all nine, the bad fit is not subjective — it is structural.
- • Vendors filter on capability. You should filter on candor.
- • Ask each question with no follow-ups — see how they fill the silence.
- • The right answer is often "here's how we've gotten that wrong before."
- • A vendor without a rollback story is a vendor without scars.
The 9 Questions
1. Tell me about a deployment that didn't work and what you learned.
A vendor who cannot describe a failed deployment has either not done enough of them or is not honest. Both are bad. Listen for specifics: the failure mode, the recovery, the changed process.
2. Walk me through your data layer for a customer like us.
Most vendors can show their model layer. Half can show their interface. The best ones can show you their data layer — how context is retrieved, how permissions work, how data stays fresh. If they handwave this, the data layer is held together with cron jobs.
3. What happens when the AI is uncertain?
The correct answer involves human-in-the-loop, confidence scores, and rollback. If the answer is "the AI is very accurate" — wrong answer. Production systems have to handle uncertainty as a first-class case.
4. Show me the rollback plan.
Specifically: if the AI does something embarrassing tomorrow, what is the ten-minute procedure to disable it? If they have to think about it, they haven't shipped to anyone real.
5. How do you handle model updates?
Foundation models change every quarter. Behavior shifts. The vendor should have a regression-testing process and a model-update protocol. If they don't, your system will silently break.
6. Who on your team will I actually work with after kickoff?
The sales team is rarely the delivery team. Get names. Get LinkedIn profiles. Get the seniority level. Many AI vendors send senior engineers to demos and junior engineers to production.
7. What is your 36-month total cost of ownership for this project?
A vendor who has only a year-one quote has not been around long enough to have a 36-month dataset. See our hidden costs guide for the line items that should appear.
8. How do you handle PII and sensitive data?
Specifically: where does data live, how long is it retained, what gets sent to the model provider, what is your subprocessor list? A vague answer is a regulatory risk. See our privacy checklist.
9. Who else has done a project like ours, and can I talk to them?
Two references on live calls, same five questions for each. If references will only do email or written quotes, you are buying marketing material.
Scoring the Answers
The point isn't whether the vendor says the "right" words. It's whether they can be specific, whether they can be self-critical, and whether they can handle silence after a question without filling it with marketing.
Bad fit is rarely about capability. It's about candor. Calibrated questions surface candor faster than capability assessments do.
Score each answer 0–3 where 3 means "specific, scarred, transparent." Below 18/27 total, the vendor is probably not ready for you. Want a second pair of eyes on a proposal? Send it our way at flowtix.ai/contact.
FAQ
Will vendors be offended by these questions? Good vendors welcome them. Bad ones bristle. Both reactions are useful signals.
Can I ask these on a first call? Yes, but a 60-minute deep dive is the right setting. Don't try to sneak nine questions into a 20-minute demo.
What if the vendor refuses to answer #1? End the process. Lack of failure narrative is the strongest disqualifier.