# Farm Match - Complete Workflow Guide

## 🆕 New Features Overview

### 1. **Advanced Criteria-Based Filtering**
Interactive map viewer with individual criterion filters

### 2. **Automated Favorites Sync**
Detect new and removed favorites automatically

### 3. **Smart Unfavoriting**
Remove properties that don't meet your minimum criteria thresholds

---

## 📋 Complete Workflow

### Initial Setup (One Time)

1. **Install Dependencies** (if not already done)
```bash
cd scraper
pip3 install playwright pandas requests beautifulsoup4 openai python-dotenv
playwright install chromium
```

2. **Set OpenAI API Key**
Create a `.env` file in the scraper directory:
```bash
echo "OPENAI_API_KEY=your_key_here" > .env
```

---

## 🔄 Regular Workflow

### Option A: Complete Automated Flow

```bash
cd scraper
python3 run_all.py
```

This will:
1. Check authentication
2. Scrape all favorites
3. Analyze with GPT
4. Remove bottom 50% by score

### Option B: Step-by-Step with New Features

#### Step 1: Sync Favorites & Extract Location Data
```bash
# Sync favorites
python3 sync_favorites.py

# Extract breadcrumbs for geocoding (NEW!)
python3 extract_breadcrumbs.py

# Geocode properties using breadcrumbs (NEW!)
python3 geocode_with_breadcrumbs.py
```

**What it does:**
- Fetches current favorites from Properstar
- Compares with previously tracked properties
- Shows new additions and removals
- Extracts location breadcrumbs from property pages
- Geocodes properties using smart fallback strategies
- Optionally analyzes new properties immediately

**Output:**
```
📊 SYNC REPORT
Current favorites: 190
Previously tracked: 186
🆕 New properties: 4
🗑️ Removed properties: 0
```

#### Step 2: Analyze New Properties (if any)

**Choose one or use multiple analysis methods:**

**A. GPT Analysis** (subjective, costs ~$0.02/property)
```bash
python3 analyze_from_urls.py
```
- Scrapes property pages
- GPT-3.5 evaluates 6 criteria subjectively
- Scores: Market Garden, Guest, Workshop, Rental, Location, Local Market

**B. Deterministic Analysis** (objective, FREE)
```bash
python3 deterministic_analyzer.py
```
- Extracts hard facts (land size, bedrooms, features)
- Rule-based scoring (no AI)
- Instant results

**C. Custom Criteria Analysis** (climate/location data, FREE)
```bash
python3 custom_criteria.py
```
- 8 criteria: Rainfall, Temperature, Climate Risk, Airport Access, Rural Character, Soil Quality, Water Availability, Airbnb Potential
- Uses free APIs (Open-Meteo, Nominatim)
- Extensible - add your own criteria easily

#### Step 3: Parse Individual Criteria & Merge Prices
```bash
python3 parse_criteria.py
```

**What it does:**
- Extracts individual criterion scores from GPT analysis
- Merges price data from extracted_property_urls.csv
- Adds columns for each criterion to CSV
- Creates `enriched_data.json` for web viewer with prices

**Output:**
```
📊 Criteria Coverage:
  market_garden: 174/186 properties (avg: 3.60)
  guest_accommodation: 174/186 properties (avg: 4.05)
  workshop: 174/186 properties (avg: 2.56)
  rental_units: 174/186 properties (avg: 3.48)
  location: 174/186 properties (avg: 3.09)
  local_market: 174/186 properties (avg: 3.71)
```

#### Step 4: Filter Properties Visually
```bash
./view_map.sh
```

Then open: http://localhost:8000/map_viewer_advanced.html

**Features:**
- 🌱 **Market Garden** slider (1-5)
- 🏡 **Guest Accommodation** slider (1-5)
- 🔧 **Workshop** slider (1-5)
- 🏘️ **Rental Units** slider (1-5)
- 📍 **Location** slider (1-5)
- 🛒 **Local Market** slider (1-5)
- ⚖️ **Overall Score** slider (-5 to +5)
- 💰 **Price Range** slider (€0 - €1.7M) ⭐ NEW
- 🎯 **Risk Profile** checkboxes (Low/Medium/High)
- 🔍 **Search** by title or location
- 📊 **Sort** by any criterion (including price)
- 🗺️ **Map View** / 📋 **List View** toggle

**Export Options:**
- 🗑️ **Export Filtered for Unfavorite** - Download list of properties that DON'T meet your criteria
- 📥 **Export Filtered List (CSV)** - Download properties that DO meet your criteria

#### Step 5: Smart Unfavoriting

##### Preview Mode (Dry Run)
```bash
python3 smart_unfavorite.py --dry-run
```

Shows what would be removed without making changes.

##### Basic Usage - Remove Low Scorers
```bash
python3 smart_unfavorite.py --overall-score 0.5
```

Removes all properties with overall score < 0.5

##### Advanced - Custom Criteria Thresholds
```bash
python3 smart_unfavorite.py \
  --overall-score 1.0 \
  --market-garden 4 \
  --guest 4 \
  --location 3 \
  --max-risk Gemiddeld
```

**Parameters:**
- `--overall-score`: Minimum weighted score (default: 0.0)
- `--market-garden`: Min market garden score 1-5 (default: 3)
- `--guest`: Min guest accommodation score 1-5 (default: 3)
- `--workshop`: Min workshop score 1-5 (default: 2)
- `--rental`: Min rental units score 1-5 (default: 3)
- `--location`: Min location score 1-5 (default: 3)
- `--market`: Min local market score 1-5 (default: 3)
- `--max-risk`: Max acceptable risk: Laag/Gemiddeld/Hoog (default: Gemiddeld)
- `--require-all`: ALL criteria must be met (default: at least ONE)
- `--dry-run`: Preview only, don't unfavorite

##### Example Scenarios

**Scenario 1: Remove Only Poor Performers**
```bash
python3 smart_unfavorite.py --overall-score 0 --dry-run
```

**Scenario 2: Keep Only Excellent Properties**
```bash
python3 smart_unfavorite.py \
  --overall-score 1.5 \
  --market-garden 4 \
  --guest 4 \
  --location 4 \
  --max-risk Laag \
  --dry-run
```

**Scenario 3: Focus on Market Garden**
```bash
python3 smart_unfavorite.py \
  --market-garden 4 \
  --location 3 \
  --overall-score 0.5 \
  --dry-run
```

**Scenario 4: Remove ALL That Don't Meet High Standards**
```bash
python3 smart_unfavorite.py \
  --require-all \
  --market-garden 3 \
  --guest 3 \
  --workshop 3 \
  --rental 3 \
  --location 3 \
  --market 3 \
  --max-risk Gemiddeld
```

---

## 🎯 Recommended Workflows

### Weekly Maintenance
```bash
# 1. Check for new favorites
python3 sync_favorites.py

# 2. If new properties, parse criteria
python3 parse_criteria.py

# 3. Review in web interface
./view_map.sh
# Open: http://localhost:8000/map_viewer_advanced.html

# 4. Remove low performers
python3 smart_unfavorite.py --overall-score 0 --dry-run
# If looks good, run without --dry-run
```

### Quarterly Deep Clean
```bash
# 1. Sync favorites
python3 sync_favorites.py

# 2. Parse criteria
python3 parse_criteria.py

# 3. Remove aggressive (keep only best)
python3 smart_unfavorite.py \
  --overall-score 1.0 \
  --market-garden 4 \
  --guest 3 \
  --location 3 \
  --max-risk Gemiddeld \
  --dry-run

# 4. Review and execute
```

### Finding Specific Property Types

**Best for Market Garden:**
```bash
# In web interface, set:
# - Market Garden: 4+
# - Location: 3+
# - Overall Score: 1.0+
```

**Best for Guest Accommodation:**
```bash
# In web interface, set:
# - Guest Accommodation: 4+
# - Location: 4+
# - Risk: Low or Medium only
```

**Best Overall Properties:**
```bash
# In web interface, set:
# - Overall Score: 1.5+
# - All criteria: 3+
# - Risk: Low only
```

---

## 📊 Understanding the Scoring System

### Individual Criteria Scores (1-5)
- **1** = Very unsuitable
- **2** = Unsuitable
- **3** = Average
- **4** = Suitable
- **5** = Excellently suitable

### Weighted Overall Score
Formula:
```
Score = Σ(criterion_score × weight) / Σ(weights) × risk_factor
```

**Weights:**
- Location: 3.0 (most important)
- Guest Accommodation: 2.5
- Market Garden: 2.0
- Workshop: 2.0
- Rental Units: 1.5
- Local Market: 1.5

**Risk Factors:**
- Low: 1.0 (no penalty)
- Medium: 0.9 (10% penalty)
- High: 0.7 (30% penalty)

**Score Interpretation:**
- `< 0`: Poor match
- `0 - 1.0`: Below average
- `1.0 - 1.5`: Good match
- `1.5 - 2.0`: Excellent match
- `> 2.0`: Outstanding match (rare!)

---

## 🔧 Troubleshooting

### "Module not found" errors
```bash
pip3 install pandas requests beautifulsoup4 openai python-dotenv playwright
```

### "auth.json not found"
The script will prompt you to log in manually. Once logged in, it saves the session.

### "No criteria columns found"
Run `parse_criteria.py` first to extract individual scores from GPT analysis.

### Map viewer shows no properties
Make sure `enriched_data.json` exists. Run `parse_criteria.py` to generate it.

### Smart unfavorite not working
- Check that you have valid `auth.json`
- Ensure you're running without `--dry-run`
- Type exactly `UNFAVORITE` when prompted

---

## 📁 File Reference

### Input Files
- `auth.json` - Properstar login session
- `.env` - OpenAI API key
- `prompt.txt` - GPT analysis prompt template

### Data Files
- `extracted_property_urls.csv` - Raw scraped favorites
- `analysis_output.csv` - Full analysis with criteria
- `enriched_data.json` - Processed data for web viewer
- `map_data.json` - Geographic data for map

### Script Files
- `sync_favorites.py` - Fetch and compare current favorites
- `favorites_scraper.py` - Scrape favorites from Properstar
- `analyze_from_urls.py` - GPT analysis of properties ($0.02/property)
- `deterministic_analyzer.py` - FREE rule-based analysis
- `custom_criteria.py` - FREE climate/location/market analysis (8 criteria) ⭐ NEW
- `parse_criteria.py` - Extract scores & merge price data
- `smart_unfavorite.py` - Criteria-based unfavoriting
- `auto_workflow.py` - Complete automation
- `remove_low_scores.py` - Legacy (use smart_unfavorite.py instead)
- `geocode_properties.py` - Basic geocoding (rarely needed)
- `improve_geocoding.py` - Enhanced geocoding (rarely needed)

### Web Interface
- `map_viewer.html` - Basic map viewer
- `map_viewer_advanced.html` - Advanced with criteria filters ⭐ NEW
- `view_map.sh` - Start local server

---

## 💡 Tips & Best Practices

1. **Always use --dry-run first** before unfavoriting
2. **Backup your CSV files** before major operations
3. **Set conservative thresholds** initially (don't remove too many)
4. **Review properties in web interface** before bulk unfavoriting
5. **Run sync_favorites.py weekly** to catch new listings
6. **Use OR logic** (default) for more lenient filtering
7. **Use AND logic** (--require-all) for very strict filtering
8. **Monitor your OpenAI API usage** - each property costs ~$0.001-0.003

---

## 🎓 Example Session

```bash
# Monday morning: Check for new properties
cd scraper
python3 sync_favorites.py
# Output: "🆕 New properties: 3"
# Choose 'y' to analyze immediately

# Parse the new data
python3 parse_criteria.py

# Review in browser
./view_map.sh
# Open http://localhost:8000/map_viewer_advanced.html
# Set filters to your preferences
# Export list of properties to unfavorite

# Preview removals
python3 smart_unfavorite.py --overall-score 0.5 --dry-run
# Output: "🗑️ Found 23 properties to unfavorite"

# Looks good? Execute it
python3 smart_unfavorite.py --overall-score 0.5
# Type: UNFAVORITE
# Output: "✅ Successfully unfavorited: 23"
```

---

**Happy property hunting! 🌿🏡**