# GPT Model Analysis for FarmMatch Property Evaluation

**Date**: October 13, 2025
**Research Updated**: Latest pricing from October 2025
**Purpose**: Identify optimal AI model for property analysis balancing cost vs quality

## Executive Summary

**RECOMMENDATION**: Switch to **Gemini 2.0 Flash** for **95% cost savings** OR **GPT-4o-mini** for **70% savings** with better quality.

### Quick Comparison
| Option | Cost per 89 props | vs Current | Quality | Complexity |
|--------|------------------|-----------|---------|------------|
| **Current (GPT-3.5-turbo)** | $0.1535 | baseline | ⭐⭐⭐ | ✓ Active |
| **Gemini 2.0 Flash** | **$0.0067** | **-95%** | ⭐⭐⭐⭐ | Medium (new API) |
| **GPT-4o-mini** | **$0.0482** | **-70%** | ⭐⭐⭐⭐ | Easy (drop-in) |
| **o3-mini** | $0.2848 | +85% | ⭐⭐⭐⭐⭐ | Easy (reasoning) |

**Best Value**: Gemini 2.0 Flash ($7 vs $154 annually at current volume)
**Easiest**: GPT-4o-mini (same OpenAI API, 70% cheaper)
**Best Quality**: o3-mini (if budget allows for reasoning tasks)

---

## Current Setup Analysis

### Current Configuration
```python
model="gpt-3.5-turbo-1106"
temperature=0.0
avg_tokens_input=2,960 per property
avg_tokens_output=164 per property
```

### Actual Performance Data (Last Run: 89 properties)
- **Total Input Tokens**: 263,371 (2,960 avg/property)
- **Total Output Tokens**: 14,558 (164 avg/property)
- **Total Cost**: $0.1535
- **Cost per Property**: $0.001725
- **Duration**: 360 seconds (4.05 sec/property)

---

## Model Comparison Matrix (October 2025)

### OpenAI Models
| Model | Input/M | Output/M | Per 89 Props | vs Current | Quality | Context | Speed |
|-------|---------|----------|--------------|-----------|---------|---------|-------|
| GPT-3.5-turbo (current) | $0.50 | $1.50 | $0.1535 | baseline | ⭐⭐⭐ | 16K | ⚡⚡⚡ |
| **GPT-4o-mini** | $0.15 | $0.60 | **$0.0482** | **-70%** | ⭐⭐⭐⭐ | 128K | ⚡⚡⚡ |
| GPT-4o-mini (Batch) | $0.075 | $0.30 | **$0.0241** | **-85%** | ⭐⭐⭐⭐ | 128K | ⚡ (24h) |
| **o3-mini** | $1.10 | $4.40 | $0.2848 | +85% | ⭐⭐⭐⭐⭐ | 128K | ⚡⚡ |
| o3-mini (cached) | $0.55 | $4.40 | $0.2087 | +36% | ⭐⭐⭐⭐⭐ | 128K | ⚡⚡ |
| o1-mini | $1.10 | $4.40 | $0.2848 | +85% | ⭐⭐⭐⭐⭐ | 128K | ⚡⚡ |
| GPT-4o | $2.50 | $10.00 | $0.8040 | +424% | ⭐⭐⭐⭐⭐ | 128K | ⚡⚡ |
| o3 | $10.00 | $40.00 | $3.4446 | +2,144% | ⭐⭐⭐⭐⭐⭐ | 200K | ⚡ |
| o1-preview | $15.00 | $60.00 | $4.8240 | +3,043% | ⭐⭐⭐⭐⭐⭐ | 128K | ⚡ |

### Anthropic Models
| Model | Input/M | Output/M | Per 89 Props | vs Current | Quality | Context | Speed |
|-------|---------|----------|--------------|-----------|---------|---------|-------|
| Claude 3.5 Sonnet | $3.00 | $15.00 | $1.0084 | +557% | ⭐⭐⭐⭐⭐ | 200K | ⚡⚡ |
| Claude 3.5 Sonnet (ext) | $6.00 | $22.50 | $1.9062 | +1,142% | ⭐⭐⭐⭐⭐ | 1M | ⚡⚡ |

### Google Models
| Model | Input/M | Output/M | Per 89 Props | vs Current | Quality | Context | Speed |
|-------|---------|----------|--------------|-----------|---------|---------|-------|
| **Gemini 2.0 Flash** | **$0.075** | **$0.30** | **$0.0067** | **-95%** | ⭐⭐⭐⭐ | 1M | ⚡⚡⚡ |
| Gemini 2.0 Flash (>128K) | $0.15 | $0.60 | $0.0482 | -70% | ⭐⭐⭐⭐ | 1M | ⚡⚡⚡ |

### Detailed Cost Breakdown (89 properties, avg 2,960 input + 164 output tokens)

| Model | Input Cost | Output Cost | **Total** | Savings | Annual (200/mo) |
|-------|-----------|-------------|-----------|---------|----------------|
| GPT-3.5-turbo | $0.1317 | $0.0218 | **$0.1535** | - | $368 |
| **Gemini 2.0 Flash** | $0.0020 | $0.0044 | **$0.0064** | **$0.1471** | **$15** ⭐ |
| **GPT-4o-mini** | $0.0395 | $0.0087 | **$0.0482** | **$0.1053** | **$116** |
| GPT-4o-mini (Batch) | $0.0197 | $0.0044 | **$0.0241** | $0.1294 | $58 |
| o3-mini | $0.2897 | $0.0641 | $0.3538 | -$0.2003 | $848 |
| o3-mini (cached) | $0.1449 | $0.0641 | $0.2090 | -$0.0555 | $501 |
| GPT-4o | $0.6584 | $0.1456 | $0.8040 | -$0.6505 | $1,929 |
| Claude 3.5 Sonnet | $0.7901 | $0.2184 | $1.0085 | -$0.8550 | $2,420 |

---

## Cost Projections for FarmMatch

### Scenario 1: Current Activity (200 properties/month)
| Model | Monthly Cost | Annual Cost | vs Current |
|-------|-------------|-------------|-----------|
| GPT-3.5-turbo | $0.35 | $4.15 | baseline |
| **GPT-4o-mini** | $0.11 | **$1.28** | **-$2.87/year** |
| GPT-4o-mini (Batch) | $0.05 | **$0.64** | **-$3.51/year** |

### Scenario 2: Scaled Growth (1,000 properties/month)
| Model | Monthly Cost | Annual Cost | vs Current |
|-------|-------------|-------------|-----------|
| GPT-3.5-turbo | $1.73 | $20.75 | baseline |
| **GPT-4o-mini** | $0.53 | **$6.39** | **-$14.36/year** |
| GPT-4o-mini (Batch) | $0.27 | **$3.19** | **-$17.56/year** |

---

## GPT-4o-mini: Why It's the Clear Winner

### Technical Advantages
1. **Multimodal Capabilities** - Can process images (useful for property photos)
2. **Larger Context Window** - 128K tokens (vs 16K for GPT-3.5)
3. **Better Reasoning** - Outperforms GPT-3.5 on MMLU, MGSM, HumanEval
4. **Newer Training Data** - Knowledge cutoff October 2023 (vs Sept 2021)
5. **Structured Outputs** - Better JSON parsing and schema adherence

### Benchmark Scores (Source: OpenAI)
| Benchmark | GPT-3.5-turbo | GPT-4o-mini | Improvement |
|-----------|--------------|-------------|-------------|
| MMLU | 70.0% | **82.0%** | +17% |
| MGSM (Math) | 52.4% | **87.0%** | +66% |
| HumanEval (Code) | 48.1% | **87.2%** | +81% |

### Practical Benefits for FarmMatch
- **Better Dutch Language Understanding** - Improved NLP for property descriptions
- **More Consistent Scoring** - Better instruction following = less hallucination
- **Handles Edge Cases** - Better at ambiguous property descriptions
- **Future-Proof** - GPT-3.5 will eventually be deprecated

---

## Batch API Deep Dive

### What is Batch API?
- Asynchronous processing with 24-hour turnaround
- **50% discount** on both input and output tokens
- Higher rate limits (separate pool)
- Ideal for non-urgent bulk processing

### When to Use Batch API?
✅ **Good for:**
- Weekly full updates (analyze all 200 properties)
- Re-analysis of existing properties
- Bulk historical data processing

❌ **Not good for:**
- Real-time analysis ("Analyze Only" button)
- Properties needing immediate scores
- Interactive workflows

### Batch API Implementation Complexity
- Requires uploading JSONL file
- Polling for job completion
- Downloading results
- **Estimated Dev Time**: 4-6 hours

### Cost-Benefit Analysis
- **Savings**: $0.024 per 89 properties vs real-time GPT-4o-mini
- **Break-even**: ~150 hours of dev time at $15/hour opportunity cost
- **Recommendation**: Implement if processing >10,000 properties/year

---

## Temperature Setting Analysis

### Current: temperature=0.0 (Deterministic)

**Pros:**
- Consistent scores for same property
- Reproducible results
- Good for testing

**Cons:**
- Can be overly rigid
- May miss nuanced interpretations
- Less creative in edge cases

### Recommendation: temperature=0.3

**Benefits:**
- Slight variation allows for nuanced scoring
- Still mostly deterministic (low randomness)
- Better handles ambiguous cases
- Property analysis benefits from slight flexibility

**Risk Mitigation:**
- Intelligent caching already handles duplicates
- Variation is minimal at 0.3 (vs 0.0)
- Can A/B test against temperature=0.0

---

## Implementation Recommendations

### Phase 1: Immediate Switch (Priority: HIGH)
**Action**: Switch from GPT-3.5-turbo to GPT-4o-mini
**Effort**: 15 minutes
**Savings**: 70% cost reduction immediately
**Risk**: Very low (drop-in replacement)

```python
# Change in analyze_from_urls_optimized.py line 217
model="gpt-4o-mini"  # was: "gpt-3.5-turbo-1106"
```

### Phase 2: Temperature Adjustment (Priority: MEDIUM)
**Action**: Increase temperature from 0.0 to 0.3
**Effort**: 5 minutes
**Benefit**: Better edge case handling
**Risk**: Low (can revert easily)

```python
temperature=0.3  # was: 0.0
```

### Phase 3: Batch API (Priority: LOW)
**Action**: Implement Batch API for bulk updates
**Effort**: 4-6 hours development
**Savings**: Additional 50% (15% of total cost)
**When**: Only if processing >10,000 properties/year

---

## Cost Estimation Function Update

Current pricing function uses outdated GPT-3.5-turbo rates. Update needed:

```python
def estimate_cost(input_tokens, output_tokens, model="gpt-4o-mini"):
    """Estimate cost for various GPT models"""
    pricing = {
        "gpt-3.5-turbo": {"input": 0.50, "output": 1.50},
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
        "gpt-4o-mini-batch": {"input": 0.075, "output": 0.30},
        "gpt-4o": {"input": 2.50, "output": 10.00},
        "gpt-4-turbo": {"input": 10.00, "output": 30.00},
    }

    rates = pricing.get(model, pricing["gpt-4o-mini"])
    input_cost = (input_tokens / 1_000_000) * rates["input"]
    output_cost = (output_tokens / 1_000_000) * rates["output"]
    return input_cost + output_cost
```

---

## Quality Assurance Plan

### Testing Strategy for GPT-4o-mini Migration

1. **Parallel Run** (Recommended)
   - Analyze 20 properties with both models
   - Compare scores side-by-side
   - Identify any major discrepancies
   - Cost: ~$0.02 extra

2. **A/B Comparison Metrics**
   - Score consistency (standard deviation)
   - Hallucination rate (check KPI validation overrides)
   - Reasoning quality (manual review of 10 samples)
   - Processing time

3. **Success Criteria**
   - No increase in hallucination rate
   - Similar or better score distribution
   - Reasoning quality ≥ current
   - Cost reduction ≥ 60%

---

## Alternative Models Considered

### Why NOT These Models?

**GPT-4o** ($0.80 per 89 properties)
- 15x more expensive than GPT-4o-mini
- Minimal quality improvement for this task
- Overkill for property scoring

**GPT-4 Turbo** ($3.44 per 89 properties)
- 64x more expensive than GPT-4o-mini
- Best-in-class quality but unjustified cost
- Better suited for complex reasoning tasks

**Claude 3.5 Sonnet** (Anthropic)
- Excellent quality but higher cost ($3.00/$15.00 per M tokens)
- Would require code changes (different API)
- Consider for future if OpenAI quality degrades

**Gemini 1.5 Pro** (Google)
- Competitive pricing ($1.25/$5.00 per M tokens)
- Free tier available (2M tokens/day)
- Worth exploring if >$50/month spend

---

## Final Recommendation

### Immediate Action Plan

1. **Switch to GPT-4o-mini** - Do this NOW
   - Update model name in code
   - Update cost estimation function
   - Run 20-property test batch
   - Deploy if tests pass

2. **Adjust Temperature to 0.3** - Do after GPT-4o-mini is stable
   - Test with 10 properties
   - Compare scores with temperature=0.0
   - Deploy if variation is acceptable

3. **Monitor for 1 week**
   - Track cost savings
   - Check hallucination rates
   - Review user feedback
   - Document any issues

4. **Consider Batch API** - Evaluate in 3 months
   - Only if processing >10k properties/year
   - Or if immediate response not needed
   - Requires 4-6 hours dev work

### Expected Outcomes
- **Cost Reduction**: 70% immediately
- **Quality Improvement**: 15-20% (based on benchmarks)
- **Processing Speed**: Similar or slightly faster
- **ROI Timeline**: Immediate (day 1)

---

## Monitoring & Optimization

### Key Metrics to Track
1. **Cost per Property** - Target: <$0.0006 (GPT-4o-mini)
2. **Hallucination Rate** - Target: <5% (KPI validation overrides)
3. **Processing Time** - Target: <5 sec/property
4. **Cache Hit Rate** - Target: >30% for re-analysis
5. **User Satisfaction** - Track property score accuracy

### Monthly Review Checklist
- [ ] Review cost trends
- [ ] Check OpenAI pricing updates
- [ ] Evaluate new model releases
- [ ] Test prompt optimizations
- [ ] Review hallucination cases
- [ ] Consider batch API if volume increased

---

## Appendix: Technical Details

### Current Token Usage Breakdown
- **Property Description**: ~1,500 tokens
- **Prompt Template**: ~1,200 tokens
- **Location Context**: ~200 tokens
- **System Instructions**: ~60 tokens
- **Total Input**: ~2,960 tokens/property

### Output Token Distribution
- **Criterion Scores**: ~80 tokens
- **Reasoning**: ~60 tokens
- **Risk Profile**: ~20 tokens
- **Total Output**: ~164 tokens/property

### Rate Limits (GPT-4o-mini)
- **RPM**: 500 requests/minute (vs 60 for GPT-3.5)
- **TPM**: 200,000 tokens/minute
- **RPD**: 10,000 requests/day

### Caching Strategy
- Enabled for both models
- Cache hit saves ~$0.0015 per property
- 30-40% hit rate expected for updates
- Reduces GPT-4o-mini cost to ~$0.0004/property

---

**Document Version**: 1.0
**Last Updated**: October 13, 2025
**Next Review**: November 2025