Table of contents
Share This Article
- Why Agentic AI Vendor Evaluation Is Failing Enterprise Buyers
- What an Agentic AI Vendor Evaluation Actually Tests
- The 30-Question Agentic AI Vendor Evaluation Checklist
- Scoring the Responses: A Weighted Decision Matrix
Most agentic AI purchases fail before the contract is signed, because the evaluation tested the demo instead of the product. This agentic AI vendor evaluation checklist gives you 30 questions, grouped into six categories with a weighted scoring matrix, that separate the vendors who can run agents in your production environment from the ones who can only run them on stage.
The stakes are unusual for an enterprise software category this young. Gartner predicts that over 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. The same research estimates that of the thousands of vendors marketing “agentic AI,” only around 130 offer genuinely agentic capability. The rest are engaging in what Gartner calls agent washing: rebranding chatbots, RPA tools, and AI assistants without the autonomy, tool use, or reasoning loops that define an agent. Your evaluation process is the only thing standing between your 2027 budget and that 40% statistic.
Why Agentic AI Vendor Evaluation Is Failing Enterprise Buyers
The standard enterprise software evaluation playbook assumes the product behaves the same way in the demo as it does in production. Agentic systems break that assumption in three ways.
First, agents are non-deterministic. A demo shows you one trajectory through one scripted scenario. Production exposes the agent to thousands of trajectories, including the ones where it loops, stalls, calls the wrong tool, or confidently completes the wrong goal. LangChain’s State of Agent Engineering survey found that 70% of engineering leaders name non-deterministic outputs as their number one production-readiness barrier, and that evaluation gaps are the single largest blocker keeping agent pilots out of production.
Second, agents take actions with side effects. A chatbot that answers badly embarrasses you. An agent that acts badly updates the wrong record, issues the wrong refund, or emails the wrong customer. The evaluation bar for a system with write access to your CRM, ERP, or payment stack is closer to the bar you apply to a new employee with system credentials than to a SaaS reporting tool. We covered the downstream consequences of skipping this discipline in our analysis of AI agent sprawl as an enterprise security problem ā most sprawl starts with a procurement decision that never asked who the agent acts as.
Third, the vendor landscape itself is unstable. The category is consolidating fast: platform vendors are bundling agents into existing licences, model providers are moving up the stack, and a long tail of thin wrappers will not survive contact with 2027 renewal cycles. A vendor’s answers about viability, exit, and data portability matter as much as their answers about capability.
The checklist below is built around those three failure modes. It is designed to be copied into your RFP or used live in vendor sessions. Every question has a purpose noted, so your team knows what a good answer sounds like.
What an Agentic AI Vendor Evaluation Actually Tests
Before the questions, a frame. A complete agentic AI vendor evaluation tests six dimensions, and weak evaluations almost always over-index on the first one.
The table below summarises the six categories, what each one predicts, and how much weight we recommend assigning in the scoring matrix.
| Category | Questions | What It Predicts | Suggested Weight |
|---|---|---|---|
| Capability and architecture | 1ā6 | Whether the product is genuinely agentic or agent-washed | 20% |
| Security and the tool supply chain | 7ā12 | Breach and data-exfiltration exposure | 20% |
| Governance, auditability, compliance | 13ā18 | Regulatory survivability in your jurisdictions | 20% |
| Reliability and evaluation | 19ā23 | Whether the agent works at month six, not just week one | 15% |
| Commercial terms and pricing mechanics | 24ā27 | Three-year TCO and budget predictability | 15% |
| Vendor viability and exit | 28ā30 | What happens when the vendor pivots, is acquired, or fails | 10% |
If your current evaluation template only covers the first row, you are evaluating a demo, not a vendor.
The 30-Question Agentic AI Vendor Evaluation Checklist
1. Capability and Architecture (Questions 1ā6)
Q1. Show us the agent completing a multi-step task in our environment, on our data, with the scripted demo disabled. The purpose is to detect agent washing in the first hour. A genuine agent plans, calls tools, observes results, and adapts. A rebranded chatbot or RPA flow breaks the moment the input deviates from the script.
Q2. Which decisions does the agent make autonomously, and which are deterministic workflow steps? Vendors blur this line deliberately. You want a precise map of where the LLM reasons and where hard-coded logic runs, because the autonomous portion is where your risk and your evaluation effort concentrate.
Q3. Which foundation models does the platform support today, and what is the process for swapping or adding models? Model lock-in is the quietest form of platform lock-in. If the platform only runs the vendor’s own model, price rises and capability gaps are yours to absorb.
Q4. How does the agent connect to external tools and systems ā proprietary connectors, Model Context Protocol, OpenAPI, or custom code? Integration quality is the strongest single predictor of agent success in production. Ask for the connector list in writing and test the two systems that matter most to your workflow during the proof of concept.
Q5. What happens when the agent cannot complete a goal ā what are the retry, timeout, escalation, and fallback behaviours, and are they configurable? Every production agent fails sometimes. Vendors who cannot describe their failure paths in detail have not run at scale.
Q6. What loop-depth, token-budget, and runaway-cost controls exist per agent and per task? An agent that loops is not just a quality problem; it is a billing problem. This question bridges into the commercial section, and the answer should be a product feature, not a dashboard you check after the invoice arrives.
2. Security and the Tool Supply Chain (Questions 7ā12)
The tool layer is where agentic AI security incidents actually happened in the last year. Researchers have documented systemic vulnerabilities in Model Context Protocol implementations affecting thousands of publicly accessible servers, and 2026 has already produced incidents including an unauthenticated Azure DevOps MCP package (CVE-2026-32211, CVSS 9.1) and malicious MCP servers exfiltrating email at scale. Your vendor’s tool supply chain is your supply chain.
Q7. What identity does the agent act under when it calls our systems ā a service account, the end user’s delegated identity, or a per-agent identity? Per-agent identity with scoped permissions is the defensible answer. A shared service account with broad access is how audit findings begin.
Q8. How are tools allow-listed, and can we enforce a deny-by-default tool policy per agent and per environment? You want deny-by-default as a platform primitive, not a consulting engagement.
Q9. Which MCP servers or third-party connectors does the platform bundle, who maintains them, and how are they vetted, versioned, and patched? Ask for the software bill of materials for the agent’s tool layer. If the vendor cannot produce one, every community-built connector in their marketplace is inside your perimeter.
Q10. How is prompt injection through retrieved content and tool outputs mitigated, and has the platform been red-teamed for it? Share the report. Indirect prompt injection is the canonical agentic attack path: hostile instructions hidden in a document, web page, or email the agent reads. A serious vendor has commissioned testing and will share at least a summary under NDA.
Q11. Where do our prompts, retrieved documents, tool outputs, and agent traces physically reside, and are they used for training? Data residency matters doubly for agents because traces capture your internal systems’ responses, not just user inputs. Get training-use exclusions in the contract, not the FAQ.
Q12. What were your last two security incidents and how were customers notified? Every vendor at scale has incidents. The vendors to avoid are the ones who claim zero.
3. Governance, Auditability, and Compliance (Questions 13ā18)
Q13. What does the audit log capture per agent action ā model version, prompt, retrieved context, tool called, parameters, result, and human approvals? The full chain is the requirement. An audit log that records “agent completed task” satisfies no regulator in any of your jurisdictions.
Q14. Can we replay a specific agent decision from six months ago with the exact model version and context that produced it? Decision replay is what incident investigation, customer dispute resolution, and regulatory examination all require. Most platforms cannot do this. The ones that can are taking governance seriously.
Q15. How is human-in-the-loop approval tiered by risk ā can we require approval for specific tools, transaction values, or customer segments? Blanket HITL destroys the ROI case; zero HITL destroys the risk case. Tiered, policy-driven approval is the production pattern. Our Singapore agentic AI governance implementation guide covers how regulators in APAC expect this to be structured.
Q16. Which compliance attestations does the platform hold ā SOC 2 Type II, ISO 27001, ISO 42001 ā and which specifically cover the agentic components, not just the hosting infrastructure? Vendors routinely present infrastructure certifications as product certifications. Read the scope statement.
Q17. How does the platform support our obligations under the regulations that bind us ā and can the vendor speak to them specifically? For a Singapore bank that means MAS expectations including the AI Risk Management Guidelines now in their transition window; for Hong Kong, HKMA’s supervisory expectations on AI; for the UK, FCA operational resilience and SS1/23 outsourcing guidance; for the EU, the AI Act’s high-risk obligations on their revised timeline; for the US, sectoral rules from the SEC, FINRA, or HIPAA depending on your industry. A vendor who answers with a generic “we are compliant” has not deployed in your sector.
Q18. Who is accountable when the agent gets it wrong ā what do the contract’s liability, indemnity, and error-correction clauses actually say about autonomous actions? This is the question that exposes how confident the vendor is in their own product. Most contracts currently push all liability for agent actions onto the customer. That is negotiable, but only before signature.
4. Reliability and Evaluation (Questions 19ā23)
The industry’s own numbers explain why this section exists: observability adoption among teams running agents has reached roughly 89%, while systematic evals adoption sits near 52%. Teams can watch their agents fail in beautiful dashboards but cannot prevent the failures. Your vendor should be ahead of that curve, not part of it.
Q19. What goal-completion rate does the platform achieve on workflows comparable to ours, and how is goal completion defined and measured? Task-level success metrics (the API call succeeded) are not workflow-level success (the customer’s problem was resolved). Ask for the latter. Treat refusal to share any number as signal.
Q20. What evaluation harness ships with the platform ā can we define our own eval sets, run them on every model or prompt change, and gate deployments on the results? Evals-as-deployment-gate is the practice that separates mature platforms from demos with APIs.
Q21. What observability is built in ā traces, token costs, latency distributions, tool-error rates, per-agent and per-workflow ā and does it export to our existing stack? The export question matters: agent telemetry locked inside the vendor’s console becomes another silo.
Q22. What SLAs cover the platform, and what remedies exist when agent quality degrades after a model update we did not initiate? Silent model updates that change agent behaviour are among the most common production complaints of the past year. You want contractual notice periods and version pinning.
Q23. Give us three reference customers in our industry and size band who have run agents in production for over six months. Then ask those references about implementation surprises, real uptime, and what they would do differently. Six months is the threshold; everyone’s agent works in week one.
5. Commercial Terms and Pricing Mechanics (Questions 24ā27)
Agentic pricing models are unstable and frequently redesigned. Salesforce launched Agentforce at US$2 per conversation, then introduced per-action Flex Credits after customer pushback; Microsoft prices Copilot Studio in consumption credits whose burn rate depends on agent design choices your team controls. None of this maps cleanly to seats, and the variance between a well-designed and badly-designed agent on the same platform can be the entire business case ā a dynamic we quantified in our guide to LLM cost optimisation in production.
Q24. Walk us through the complete pricing model ā per conversation, per action, per credit, per token, per seat ā and model our top three workflows at 10x current volume. Insist the vendor builds the model with your numbers in the room. Pricing surprises at scale are the most cited driver of the cancellations Gartner forecasts.
Q25. What is included in the platform fee versus billed as consumption, and what has your average customer’s consumption bill done over their first 12 months? The second half of that question is the real one.
Q26. What are the full implementation costs ā integration, data preparation, prompt and eval engineering, change management ā based on actual customer projects, not list estimates? Industry analyses consistently find hidden implementation costs adding 60ā120% to stated platform pricing. Get the vendor’s professional-services day-rate card and a fixed-scope estimate in the proposal.
Q27. What happens to pricing at renewal ā what caps exist on increases, and what happened to your existing customers’ pricing at their first renewal? A category this young has no stable pricing norms. Caps belong in the contract.
6. Vendor Viability and Exit (Questions 28ā30)
Q28. Show us your funding runway, customer count in production, and product roadmap commitments in writing. In a market where analysts expect a minority of current vendors to exist in their present form by 2028, viability diligence is not rude; it is fiduciary.
Q29. If we terminate, what do we get back ā agent definitions, prompts, eval sets, traces, fine-tuned artefacts ā in what format, and what does the platform’s export actually produce today? Run the export during the proof of concept. The difference between the contractual answer and the actual file is the size of your lock-in.
Q30. What is the migration path off your platform ā and can you name a customer who has left and describe how it went? A vendor confident in retention answers this calmly. A vendor who bristles is telling you the exit door is painted on.
Scoring the Responses: A Weighted Decision Matrix
Questions without scoring produce opinions, not decisions. Score each question 0ā3: zero for no credible answer, one for a roadmap answer, two for a partial production answer, three for a demonstrated, referenceable answer. Multiply category totals by the weights from the table above and you get a comparable 0ā100 score per vendor.
The thresholds below reflect what we have seen work in enterprise selections.
| Weighted Score | Interpretation | Recommended Action |
|---|---|---|
| 80ā100 | Production-credible platform | Proceed to paid proof of concept |
| 65ā79 | Strong product, specific gaps | Proceed only with contractual remediation of gaps |
| 50ā64 | Demo-stage maturity | Revisit in two quarters; do not sign multi-year terms |
| Below 50 | Likely agent-washed | Disqualify |
Two hard rules override the arithmetic. Any vendor scoring zero on Q7 (agent identity), Q13 (audit log), or Q29 (export) is disqualified regardless of total, because those three failures are unfixable from the customer side. And no vendor should pass on the strength of capability scores alone: a 95% capability score with a 30% governance score describes a system you cannot deploy in a regulated environment, which for most readers of this checklist means a system you cannot deploy.
Common Vendor Evaluation Mistakes That the Checklist Prevents
A regional bank in Southeast Asia we advised ran a six-week evaluation of three agent platforms in late 2025. Two vendors aced every scripted demo. The selection flipped when the team ran Q1 ā same task, vendor demo environment disabled, bank’s sandbox data ā and one platform’s “autonomous” loan-document agent turned out to be a fixed eight-step RPA flow that failed on any document variant it had not been configured for. The capability questions exist because agent washing is not detectable from a stage seat.
A UK insurer evaluating customer-service agents in early 2026 scored their preferred vendor highest on capability and price, signed, and then spent four months in remediation when internal audit found the platform’s logs could not reconstruct which policy wording the agent had retrieved before advising a customer. The audit log question (Q13) and the replay question (Q14) would have surfaced that in week one, before signature, when it was the vendor’s problem to fix rather than the insurer’s.
The pattern across both: evaluations fail when they test what the vendor wants to show rather than what production will demand. The fix is procedural, not heroic ā run the questions, in your environment, against your data, and score in writing. If you are still deciding whether to evaluate vendors at all or build in-house, our guide to building an AI agent for the enterprise covers that upstream decision; this checklist assumes you have decided to buy and now need to buy well.
Running the Evaluation: A Four-Week Process
Week 1 ā Paper screen. Issue the 30 questions as a written RFP supplement to your longlist. Disqualify on the three hard-rule questions. A longlist of eight typically becomes a shortlist of three on written answers alone.
Week 2 ā Live sessions. Half-day session per shortlisted vendor: Q1 live in your sandbox, security and governance deep-dives with your CISO and compliance lead in the room, and the pricing model built against your workflow volumes.
Weeks 3ā4 ā Paid proof of concept. One real workflow, your data, your systems, your eval set, with goal-completion measured against the definition you agreed in Q19. Run the data export (Q29) before the PoC ends. Score everything, then decide.
The discipline costs four weeks. The Gartner cancellation statistic is the cost of skipping it.
How Sthambh Helps Enterprises Evaluate and Deploy Agentic AI
Sthambh works with CTOs and Heads of AI across Singapore, Hong Kong, the UK, and the US on exactly this decision. We run vendor evaluations as a structured engagement: adapting this checklist to your regulatory context, sitting on your side of the table in vendor sessions, designing the proof-of-concept eval sets, and modelling three-year TCO across the shortlist. Because we also build production agent systems on custom stacks, we can price the build-versus-buy alternative honestly rather than defaulting to whichever answer a platform vendor or an SI’s resale margin prefers.
Where clients proceed to deployment, the same team carries the work through pilot-to-production hardening: identity and permission architecture, audit logging that satisfies MAS, HKMA, FCA, or US sectoral examiners, and the evaluation harness that keeps the agent honest after go-live.
If you are shortlisting agentic AI vendors this quarter, book an agentic AI discovery call and we will walk your team through the scoring matrix against your actual shortlist.
FAQs
Q. How long should an agentic AI vendor evaluation take?
A. Four to six weeks for the structured process: one week of written screening, one week of live sessions, and two to four weeks of paid proof of concept on a real workflow. Compressing below four weeks usually means skipping the PoC, which is the only stage that tests production behaviour. Stretching beyond eight weeks typically signals an unclear use case rather than a thorough evaluation.
Q. What is agent washing and how do I detect it?
A. Agent washing is the rebranding of chatbots, RPA flows, or scripted assistants as “agentic AI” without genuine autonomy, tool use, or reasoning. Gartner estimates only around 130 of the thousands of vendors claiming agentic capability are genuine. The fastest detection method is Question 1 of this checklist: ask the vendor to run a multi-step task in your environment with the scripted demo disabled. Washed products break on the first input variant they were not configured for.
Q. Should we run the evaluation ourselves or use an external advisor?
A. Run it yourself if you have in-house agent engineering experience and a compliance team that has handled AI procurement before. Bring in an advisor if this is your first agentic purchase, if you operate in a regulated sector, or if the contract value exceeds the cost of getting the decision wrong. The most expensive option is delegating the evaluation to the vendor’s own pre-sales team, which is what an unstructured process does by default.
Q. How many vendors should be on the shortlist?
A. Issue the written checklist to six to eight vendors, run live sessions with three, and take no more than two into a paid proof of concept. Wider funnels at the PoC stage spread your evaluation engineering too thin to score any vendor properly, and the PoC is where the real signal lives.
Q. What is the single most predictive question on the checklist?
A. Question 14, decision replay. A platform that can reproduce a specific agent decision from months ago, with the exact model version and retrieved context, has necessarily solved versioning, audit logging, trace storage, and governance architecture. Vendors rarely build replay capability for marketing reasons; it only exists where engineering discipline is real.
Q. Do these questions apply if we are building agents in-house instead of buying?
A. About 25 of the 30 apply directly; you become the party who must answer them. The security, governance, reliability, and identity questions define your own architecture requirements, and the commercial questions become internal TCO modelling. Teams that cannot answer Q13 or Q15 for their own build are not ready to ship it, regardless of vendor involvement.
Q. How should the checklist change for regulated industries?
A. Raise the governance category weight from 20% to 30%, taking the increase from capability, and make Q16 and Q17 disqualifying rather than scored. A Singapore bank should also map vendor answers against MAS outsourcing and technology-risk expectations; a UK firm against FCA operational resilience and SS1/23; a US healthcare buyer against HIPAA business-associate terms. The questions stay the same; the pass threshold moves.
Nikhil Khandelwal
Co-founder & CTO, Sthambh
