03 · Evaluating AI Vendors¶
Choosing an AI vendor at team or organizational scale carries stakes far beyond the Module 1 (Level 2) individual comparison — contracts, data processing agreements, and long-term dependency all enter the picture. This module builds a vendor evaluation framework for that scale.
1. Evaluation dimensions beyond individual tool comparison¶
| Dimension | Question | Why it matters at team/org scale |
|---|---|---|
| Contractual data terms | Is there a signed data processing agreement (DPA), and what does it actually guarantee? | Verbal or marketing claims aren't enforceable; only the contract is |
| Security posture | Does the vendor have relevant certifications (e.g., SOC 2) and a documented incident history? | A breach at the vendor becomes your incident too |
| Financial stability | Is the vendor likely to exist and support this product in 2-3 years? | Losing a core vendor mid-dependency is costly to unwind |
| Support and SLA | What response time and support tier applies at your scale? | Individual-tier support is often inadequate for team-critical tools |
| Exit and portability | Can you export your data and workflows if you leave? | Determines real switching cost, not just sticker price |
| Roadmap alignment | Is the vendor's direction compatible with your future needs? | A vendor pivoting away from your use case is a hidden risk |
2. A structured vendor scorecard¶
| Criterion | Weight | Score (1-5) | Notes |
|---|---|---|---|
| Fit for core use cases | High | Based on a real pilot (Module 1), not vendor demos | |
| Data/security terms | High | Verified against the actual contract, not the marketing page | |
| Cost at projected scale | High | Modeled against realistic usage growth, not current usage | |
| Support quality | Medium | Test actual response times during evaluation, don't take claims at face value | |
| Exit/portability | Medium | Confirm export formats and any lock-in mechanisms | |
| Roadmap alignment | Low-Medium | Best-effort judgment; weight lower since it's inherently uncertain |
3. Red flags in vendor evaluation¶
| Red flag | Why it matters |
|---|---|
| Refuses to provide a written DPA or security documentation | A vendor unwilling to commit data terms in writing is a hard stop for anything beyond public data |
| Pricing that's opaque until you're deep into a sales process | Often signals costs will scale unpredictably |
| No clear data export path | Signals a lock-in strategy; treat as elevated switching cost |
| Marketing claims not matched by the actual contract language | The contract is what's enforceable; a mismatch is a warning sign, not a technicality |
| Pushing to skip a pilot and go straight to a long-term contract | Legitimate vendors accommodate a real evaluation period |
4. A due-diligence process¶
| Step | Action |
|---|---|
| 1. Shortlist | Narrow to 2-3 vendors using the Module 1 (Level 2) framework applied at team scale |
| 2. Request documentation | Security certifications, DPA template, SLA terms, in writing |
| 3. Run a paid or scoped pilot | Real usage under real contract terms, not just a sales demo |
| 4. Score with the section 2 scorecard | Weighted, written down, compared across vendors |
| 5. Legal/security review | Have the actual contract reviewed before signing, not just the summary |
| 6. Decision and documented rationale | Record why you chose this vendor — useful for future re-evaluation |
5. Common pitfalls¶
| Pitfall | Fix |
|---|---|
| Evaluating only capability, skipping contractual terms | Capability is necessary but not sufficient; a great tool with bad data terms is disqualifying for sensitive use |
| Letting the sales relationship, not the scorecard, drive the decision | Score first, then let relationship quality be a tiebreaker only |
| No renewal re-evaluation | Contract terms, pricing, and competitive landscape change; re-run the scorecard before renewal, not just at initial signing |
| Skipping legal review to move faster | The cost of an unreviewed bad clause vastly exceeds the time saved skipping review |
Worked example¶
A company evaluating an AI-powered customer support vendor shortlists two finalists after an initial capability comparison. One offers stronger out-of-the-box accuracy but stalls when asked for a signed DPA and SOC 2 report; the other is slightly less capable but provides full security documentation immediately and accommodates a 30-day paid pilot under contract. Using the scorecard, the security/contract-term weighting tips the decision to the second vendor, and the company documents the rationale so a future team revisiting the choice understands why raw capability wasn't the deciding factor.
How It Actually Works¶
A signed data processing agreement matters more than a vendor's public claims for a structural reason: what a model provider does with your data is governed entirely by contract and configuration, not by anything inherent in the underlying AI technology that would force one behavior or another. The same base model can be served under a consumer plan that logs requests and may use them for future training, or under an enterprise plan, at the same company, configured for zero data retention and a contractual no-training guarantee — the technology doesn't change between these; only the terms attached to a specific account do. This is why a vendor's marketing page ("we care about your privacy") carries no enforceable weight on its own — only the specific contract signed for your specific plan tier does, which is exactly what a DPA review is checking.
Dependency and lock-in risk, further down a typical vendor-evaluation framework, has a technical dimension worth naming explicitly: prompts, templates, and integrations built against one vendor's API are not portable to another vendor's API without rework, because each provider defines its own request format, its own function-calling schema, and often subtly different model behavior even for superficially similar prompts. Two "equally capable" models can require differently-worded prompts to get comparable output, because each was trained and tuned on different data with different conventions baked into its instruction-following behavior. This means a switching-cost estimate in a vendor evaluation isn't just about contract termination fees — it includes real re-engineering and re-tuning work, which is a fair reason concentration risk deserves its own line item independent of the contract's exit clause.
Exercise¶
Pick a real or plausible AI vendor your team might adopt. Build the section 2 scorecard for it and one competitor, actually attempting to verify (or note as unverified) each row — especially the contractual data terms. Identify any section 3 red flags you find.