Vendor Diligence Like a CTO

Half of vendor relationships fail in the first year, but not because of missing features. After evaluating 200+ vendors for portfolio companies, we've identified the five factors that predict success or expensive failure. Here's the diligence framework that spots red flags before you sign.

Common Failure Modes Nobody Talks About

Feature comparisons dominate vendor selection, yet features rarely cause failures. Our data shows the real killers:

Bus factor disasters (35% of failures): Their lead engineer leaves. Response times triple. Nobody knows your integration. Six months later, you're migrating off their platform.

SLA reality gaps (30% of failures): They promise 99.9% uptime. The fine print excludes "scheduled maintenance" every weekend. Your customers don't care about semantic differences when they can't log in.

Scale cliffs (25% of failures): Works great at 1,000 requests/day. Falls over at 10,000. Their solution? "Move to Enterprise tier" at 10x the cost. Your growth becomes their leverage.

Support theater (10% of failures): 24/7 support means an offshore team that can only reset passwords. Real issues require "escalation to engineering" with 5-day response times.

SLA Clauses That Matter (And Those That Don't)

Most SLAs are written by lawyers to minimize vendor liability, not ensure your success. Focus on these five clauses:

Uptime Definition

What to look for: Uptime measured from your perspective Red flag: "Uptime excludes scheduled maintenance" Better: "Uptime includes all user-impacting downtime"

Credit Structure

What to look for: Automatic credits, not "upon request" Red flag: "5% credit for <99% uptime" Better: "Graduated credits: 10% at 99%, 25% at 95%, 50% at 90%"

Performance Metrics

What to look for: P95 response time commitments Red flag: "Average response time <500ms" Better: "95th percentile response time <500ms"

Support Response

What to look for: Severity-based response times Red flag: "Best effort support" Better: "Sev 1: 1 hour, Sev 2: 4 hours, Sev 3: 1 business day"

Data Portability

What to look for: Full export in standard formats Red flag: "Data available upon written request" Better: "Self-service export of all data within 24 hours"

Bus Factor & Continuity Planning

The "bus factor" - how many people need to be hit by a bus before you're screwed - predicts vendor stability better than revenue metrics.

Bus Factor Scorecard

Rate each factor 0-2 points:

• Multiple people know your account: 0-2 points
• Documentation publicly available: 0-2 points
• Support team >5 people: 0-2 points
• Been in business >3 years: 0-2 points
• Engineering team >10 people: 0-2 points

Score interpretation:

• 8-10: Low bus factor risk
• 5-7: Moderate risk, negotiate protections
• 0-4: High risk, consider alternatives

Observability and Friday Receipts

The best vendors provide transparency without you asking. We call it the "Friday Receipt Test":

Can they send you weekly:

• Uptime percentage for your services
• P95 response times
• Error rates by endpoint
• Ticket response times
• Upcoming changes that might impact you

Vendors who can't provide this data aren't measuring it. Vendors who aren't measuring it can't improve it.

Pilot Rubric with Exit Criteria

Never go straight to annual contracts. Our pilot framework de-risks vendor relationships:

30-Day Pilot Structure

Week 1: Basic integration

• Can you connect successfully?
• Does authentication work?
• Is documentation accurate?

Week 2-3: Real usage

• Deploy actual use case
• Measure performance
• Test support responsiveness

Week 4: Scale test

• 10x your expected load
• Break things on purpose
• Measure recovery time

Exit Criteria (Define Before Starting)

Clear exit criteria prevent awkward "it's not working out" conversations:

1. Performance: P95 latency >2x promised
2. Reliability: >2 severity-1 incidents
3. Support: >24hr response on critical issues
4. Scale: Degradation at <5x current load
5. Cost: Actual usage >30% above quoted

Vendor Evaluation Framework

Total Score Interpretation:

• 80: Strong proceed signal

• 60-80: Proceed with specific protections
• <60: Find alternatives

Real Vendor Evaluation Example

Let's walk through a recent evaluation:

Vendor: Authentication-as-a-Service startup Initial appeal: 50% cheaper than Auth0 Our evaluation:

1. Bus Factor Test: Failed - 3 person team, lead engineer owns all knowledge
2. SLA Review: No performance SLAs, only uptime
3. Pilot Results: 3 outages in 30 days
4. Support Test: 48-hour response to critical issue
5. Scale Test: Performance degraded at 2x load

Decision: Passed despite price advantage Result: Competitor who chose them spent 4 months migrating after repeated outages

Negotiation Leverage Points

Once you've scored a vendor, use these leverage points:

If they score 60-70:

• Demand monthly contracts until proven
• Require performance bonds for SLA misses
• Negotiate 30-day termination rights
• Get executive escalation paths

If they score 70-80:

• Push for better credit terms
• Lock in pricing for 24 months
• Add data export guarantees
• Include migration assistance clause

If they score 80+:

• Focus on growth pricing protections
• Negotiate volume discounts early
• Consider longer-term commitments
• Build strategic partnership terms

Now Do This

Protect your next vendor decision with these immediate actions:

Ready to evaluate vendors systematically? Our Vendor Diligence Scorecard provides the complete framework with automatic scoring and risk flags. For running effective pilots, see our two-week pilot guide.

Building vs buying? Our delivery risk ledger helps you assess internal build risks against vendor risks.