# 🤖 Farm Match - Full Automation Guide

## ✨ One-Command Automation

Your Farm Match system can now run completely automated:

```bash
cd scraper
./auto_clean.sh
```

This single command will:
1. ✅ Fetch all current favorites from Properstar
2. ✅ Analyze new properties with GPT
3. ✅ Parse individual criteria scores
4. ✅ Unfavorite low-scoring properties automatically

---

## 🔧 Configuration

### Edit `config.json` to customize behavior:

```json
{
  "unfavorite_thresholds": {
    "overall_score": 0.0,           // Minimum weighted score
    "market_garden": 3,              // Min 1-5
    "guest_accommodation": 3,        // Min 1-5
    "workshop": 2,                   // Min 1-5
    "rental_units": 3,               // Min 1-5
    "location": 3,                   // Min 1-5
    "local_market": 3,               // Min 1-5
    "max_risk": "Gemiddeld",        // Laag/Gemiddeld/Hoog
    "require_all": false            // true = ALL criteria must meet threshold
  },
  "openai_model": "gpt-3.5-turbo-1106",
  "auto_unfavorite": true,          // Enable/disable unfavoriting
  "dry_run": false                  // true = preview only, don't unfavorite
}
```

---

## 🚀 Usage Modes

### Mode 1: Full Automation (Default)
Runs everything including actual unfavoriting:

```bash
cd scraper
./auto_clean.sh
```

**Requirements:**
- Will prompt you to type "YES" to confirm unfavoriting
- Then prompts to type "UNFAVORITE" when ready to execute

### Mode 2: Dry Run (Preview Only)
See what would be removed without making changes:

**Edit config.json:**
```json
{
  "dry_run": true
}
```

Then run:
```bash
./auto_clean.sh
```

### Mode 3: Analysis Only
Only sync and analyze, no unfavoriting:

**Edit config.json:**
```json
{
  "auto_unfavorite": false
}
```

### Mode 4: Manual with Python
```bash
/usr/bin/python3 auto_workflow.py
```

---

## 📅 Recommended Schedule

### Weekly Cleanup (Lenient)
```json
{
  "unfavorite_thresholds": {
    "overall_score": 0.0,
    "market_garden": 3,
    "guest_accommodation": 3,
    "workshop": 2,
    "rental_units": 3,
    "location": 3,
    "local_market": 3,
    "max_risk": "Gemiddeld",
    "require_all": false
  }
}
```

Run: `./auto_clean.sh` every Monday morning

### Monthly Deep Clean (Strict)
```json
{
  "unfavorite_thresholds": {
    "overall_score": 1.0,
    "market_garden": 4,
    "guest_accommodation": 4,
    "workshop": 3,
    "rental_units": 3,
    "location": 4,
    "local_market": 3,
    "max_risk": "Laag",
    "require_all": false
  }
}
```

Run: `./auto_clean.sh` first Monday of each month

---

## 🔄 Complete Workflow

```
START
  ↓
Sync Favorites from Properstar
  ├─ Detect new properties
  ├─ Detect removed properties
  └─ Update extracted_property_urls.csv
  ↓
Analyze with GPT-3.5
  ├─ Scrape new property pages
  ├─ Send to GPT for evaluation
  ├─ Score on 6 criteria (1-5)
  ├─ Assess risk profile
  └─ Update analysis_output.csv
  ↓
Parse Individual Criteria
  ├─ Extract scores from GPT analysis
  ├─ Add columns to CSV
  └─ Create enriched_data.json
  ↓
Check Thresholds (config.json)
  ├─ Filter properties below thresholds
  └─ Build unfavorite list
  ↓
Unfavorite Low Scorers
  ├─ Show what will be removed
  ├─ Ask for confirmation
  ├─ Open browser (Playwright)
  ├─ Visit each property
  ├─ Click unfavorite button
  └─ Report results
  ↓
END
```

---

## 🎯 Example Scenarios

### Scenario 1: Conservative Weekly Maintenance
Keep only properties scoring 3+ on most criteria:

```json
{
  "unfavorite_thresholds": {
    "overall_score": 0.0,
    "market_garden": 3,
    "guest_accommodation": 3,
    "workshop": 2,
    "rental_units": 3,
    "location": 3,
    "local_market": 3,
    "max_risk": "Gemiddeld",
    "require_all": false
  },
  "dry_run": false
}
```

### Scenario 2: Aggressive Filtering
Keep only excellent properties:

```json
{
  "unfavorite_thresholds": {
    "overall_score": 1.5,
    "market_garden": 4,
    "guest_accommodation": 4,
    "workshop": 3,
    "rental_units": 4,
    "location": 4,
    "local_market": 4,
    "max_risk": "Laag",
    "require_all": false
  },
  "dry_run": false
}
```

### Scenario 3: Focus on Market Garden
Prioritize agricultural potential:

```json
{
  "unfavorite_thresholds": {
    "overall_score": 0.5,
    "market_garden": 4,
    "guest_accommodation": 2,
    "workshop": 2,
    "rental_units": 2,
    "location": 3,
    "local_market": 3,
    "max_risk": "Gemiddeld",
    "require_all": false
  },
  "dry_run": false
}
```

### Scenario 4: Must Meet ALL Criteria
Very strict - every criterion must be good:

```json
{
  "unfavorite_thresholds": {
    "overall_score": 1.0,
    "market_garden": 3,
    "guest_accommodation": 3,
    "workshop": 3,
    "rental_units": 3,
    "location": 3,
    "local_market": 3,
    "max_risk": "Gemiddeld",
    "require_all": true
  },
  "dry_run": false
}
```

---

## ⚙️ Advanced Usage

### Test Configuration First
Always test with dry run before executing:

```bash
# 1. Edit config.json - set your thresholds
# 2. Set dry_run to true
# 3. Run automation
./auto_clean.sh

# 4. Review output
# 5. If happy, set dry_run to false and run again
```

### Run Individual Steps
If you want more control:

```bash
# Step 1: Sync only
/usr/bin/python3 sync_favorites.py

# Step 2: Analyze only
/usr/bin/python3 analyze_from_urls.py

# Step 3: Parse criteria only
/usr/bin/python3 parse_criteria.py

# Step 4: Unfavorite with custom params
/usr/bin/python3 smart_unfavorite.py \
  --overall-score 1.0 \
  --market-garden 4 \
  --dry-run
```

### Schedule with Cron (macOS/Linux)

Edit crontab:
```bash
crontab -e
```

Add weekly Monday 9am run:
```
0 9 * * 1 cd /Users/jonathan/SynologyDrive/Since\ Today/PROJECTEN/farmmatch/scraper && ./auto_clean.sh >> /tmp/farmmatch.log 2>&1
```

---

## 📊 Monitoring & Logs

### Check What Happened
The automation prints timestamped logs:

```
[2025-10-08 14:30:00] Starting automated property evaluation
[2025-10-08 14:30:15] ▶️ Syncing favorites from Properstar
[2025-10-08 14:30:45] ✅ Sync complete: 4 new, 0 removed
[2025-10-08 14:30:46] ▶️ GPT analysis
...
```

### Save Logs to File
```bash
./auto_clean.sh > farmmatch_$(date +%Y%m%d).log 2>&1
```

---

## 🛡️ Safety Features

1. **Confirmation Required**: Must type "YES" to proceed
2. **Second Confirmation**: Must type "UNFAVORITE" to execute
3. **Dry Run Mode**: Preview before executing
4. **Config File**: All settings in one place
5. **Preserve Data**: Never deletes CSV files, only updates

---

## 🔍 Troubleshooting

### "No auth.json found"
The first time you run, it will prompt you to log in to Properstar manually. This saves your session for future automated runs.

### "Module not found"
Install dependencies:
```bash
pip3 install playwright pandas requests beautifulsoup4 openai python-dotenv
playwright install chromium
```

### "OpenAI API error"
Check your `.env` file has valid API key:
```bash
echo "OPENAI_API_KEY=your_key_here" > .env
```

### No properties being unfavorited
Your thresholds might be too lenient. Check with dry run:
```bash
# Edit config.json - set dry_run: true
./auto_clean.sh
```

### Too many properties being unfavorited
Thresholds too strict. Lower the scores in config.json.

---

## 💰 Cost Estimate

**Per run (assuming 186 properties, ~10 new):**
- Scraping: Free
- GPT analysis of 10 new properties: ~$0.03
- Unfavoriting: Free

**Monthly (4 weekly runs):**
- ~$0.12 assuming 10 new properties per week

**Annual:**
- ~$1.50

Very affordable for automated property management! 💰

---

## 📈 Expected Results

After first automated run:
- ✅ All favorites synced
- ✅ ~10-50 properties unfavorited (depending on thresholds)
- ✅ Only high-quality properties remain
- ⏱️ Total time: 5-10 minutes

After weekly runs:
- ✅ New properties automatically evaluated
- ✅ Low scorers automatically removed
- ✅ Favorites list stays clean
- ⏱️ Total time: 2-5 minutes

---

## 🎓 Best Practices

1. **Start Conservative**: Use lenient thresholds initially
2. **Review First Run**: Check with dry_run before executing
3. **Weekly Schedule**: Keep favorites fresh
4. **Adjust Gradually**: Increase thresholds over time
5. **Backup CSV**: Save analysis_output.csv occasionally
6. **Monitor Results**: Review web interface after each run

---

## 📞 Quick Reference

### Start Automation
```bash
cd scraper
./auto_clean.sh
```

### Configure Thresholds
```bash
nano config.json
# or
open config.json
```

### Dry Run Test
```json
{ "dry_run": true }
```

### View Results
```bash
./view_map.sh
# Open: http://localhost:8000/map_viewer_advanced.html
```

---

**Your Properstar favorites are now self-cleaning! 🧹✨**