# 🚀 FarmMatch Full Update Guide

## One-Click Full Update Script

The `full_update.sh` script provides a **one-click solution** to completely update your FarmMatch property database from Properstar to the final map viewer.

---

## 🎯 What It Does

The script runs the complete 7-step pipeline:

1. **📥 Scrape Favorites** - Fetches all your favorited properties from Properstar
2. **🍞 Extract Breadcrumbs** - Extracts location data from property pages
3. **🗑️ Remove Dead Properties** - Removes properties with 404/410 errors
4. **📍 Extract GPS** - Extracts precise GPS coordinates from embedded maps
5. **🌍 Geocode Properties** - Converts addresses to GPS coordinates
6. **🎯 Analyze Criteria** - Evaluates each property against your criteria
7. **🗺️ Update Map Data** - Updates enriched_data.json for the map viewer

---

## 🔧 Usage

### Simple Usage:
```bash
cd "/Users/jonathan/SynologyDrive/Since Today/PROJECTEN/farmmatch/scraper"
./full_update.sh
```

### Or double-click:
Just double-click `full_update.sh` in Finder (if Terminal is configured to run .sh files)

---

## 📋 Prerequisites

Before running, make sure:

1. **HTTP Server Running** (for map viewer):
   ```bash
   python3 -m http.server 8000
   ```

2. **Criteria API Running** (for property analysis):
   ```bash
   python3 criteria_api.py
   ```
   - The script will auto-start this if not running

3. **Properstar Account** - You'll need to log in during Step 1 if session expired

---

## 🔐 Login Process

### First Run or Expired Session:

When you run the script, it will:
1. Check for saved authentication (`auth.json`)
2. If expired or missing, open a **Chromium browser window**
3. **You need to manually log in** to Properstar in that window
4. After login, the script continues automatically
5. Session is saved for future runs

### Subsequent Runs:

If `auth.json` exists and is valid, the script runs **completely automatically** without manual login.

---

## ⏱️ How Long Does It Take?

| Step | Typical Duration | Notes |
|------|------------------|-------|
| 1. Scrape Favorites | 30-60 seconds | + login time if needed |
| 2. Extract Breadcrumbs | 1-2 minutes | Depends on # of new properties |
| 3. Remove Dead Properties | 5-10 seconds | Fast check |
| 4. Extract GPS | 2-5 minutes | Playwright scraping |
| 5. Geocode Properties | 30-60 seconds | API rate limits apply |
| 6. Analyze Criteria | 10-20 seconds | Depends on complexity |
| 7. Update Map Data | 5-10 seconds | Fast file update |

**Total Time**: ~5-10 minutes for a typical update with 100-150 properties

---

## 📊 What You'll See

The script provides **detailed colored output**:

- 🔵 **Blue** - Step headers and info
- 🟢 **Green** - Success messages
- 🟡 **Yellow** - Warnings (non-critical)
- 🔴 **Red** - Errors (critical)
- 🟦 **Cyan** - Statistics and summaries

### Example Output:
```
============================================================================
🚀 FARMMATCH FULL UPDATE - Starting Complete Pipeline
============================================================================

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📥 STEP 1/7: Scraping Favorites from Properstar
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

✅ auth.json gevonden - gebruik opgeslagen sessie
🌐 Navigeer naar favorietenpagina...
📥 Found 115 favorites

✅ Step 1 Complete: Favorites scraped successfully
📊 Current property count: 115

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🍞 STEP 2/7: Extracting Breadcrumbs and Location Data
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
...
```

---

## 🎉 Final Summary

At the end, you'll see:
```
============================================================================
🎉 FULL UPDATE COMPLETE!
============================================================================

📊 Final Statistics:
   • Total properties: 115
   • With coordinates: 112
   • With price data: 89

✨ Your FarmMatch database is now fully updated!

Next steps:
   • Open map viewer: http://localhost:8000/map_viewer_advanced.html
   • Or run: open http://localhost:8000/map_viewer_advanced.html

Would you like to open the map viewer now? [y/N]:
```

---

## ❌ Error Handling

### If Step 1 (Favorites Scraping) Fails:
```bash
# Remove saved session and try again
rm auth.json
./full_update.sh
```

### If Criteria Analysis Fails:
```bash
# Make sure criteria API is running
python3 criteria_api.py

# Then re-run the script
./full_update.sh
```

### If Script Stops Mid-Way:
The script uses `set -e` which means it **stops on first error**. Check the error message and fix the issue, then re-run. The script is **idempotent** - safe to run multiple times.

---

## 🔄 When to Run This Script

Run the full update when:

1. **✅ You added new favorites** on Properstar (like property 105943623)
2. **✅ You want to check for removed properties** (404/410 cleanup)
3. **✅ You updated your criteria** and want to re-analyze all properties
4. **✅ It's been a while** since last update (weekly recommended)
5. **✅ You notice missing properties** in the map viewer

---

## 🔍 Finding Your New Property

After running the full update, property 105943623 will:
1. ✅ Be in `analysis_output.csv`
2. ✅ Have breadcrumbs and location data
3. ✅ Be geocoded to GPS coordinates
4. ✅ Be analyzed against your criteria
5. ✅ Appear in the map viewer
6. ✅ Be searchable via the new URL search field!

---

## 📁 Files Modified by This Script

The script updates these files:

### **Primary Data Files**
- **`analysis_output.csv`** - **Single Source of Truth** for all property data
  - Contains: URLs, titles, descriptions, locations, coordinates, KPIs, criteria scores
  - Used by: All downstream scripts (geocoding, analysis, map viewer)
  - Updated by: Steps 1-6 of the pipeline

- **`extracted_property_urls.csv`** - Breadcrumb and location data
  - Contains: URLs, breadcrumbs, location hierarchy, 404 status flags
  - Used by: Geocoding and location extraction
  - Updated by: Step 2 (breadcrumb extraction)

- **`enriched_data.json`** - Map viewer data source
  - Contains: All property data formatted for map viewer (coordinates, scores, criteria, KPIs)
  - Used by: map_viewer_advanced.html, criteria_manager.html
  - Updated by: Step 7 (parse_criteria.py)

### **Authentication & Configuration**
- **`auth.json`** - Saved Properstar session (auto-created on first login)
  - Contains: Encrypted session cookies for Properstar
  - Expires: After ~24 hours of inactivity
  - Delete this file if login fails

### **Tracking & Management**
- **`removed_properties.json`** - List of manually removed/unfavorited properties
  - Contains: URLs of properties you've explicitly removed via map viewer
  - Used by: Favorites scraper to filter out unwanted properties
  - Prevents: Re-scraping properties you've already rejected

### **Backups**
The script automatically creates timestamped backups before destructive operations:
- `analysis_output_backup_YYYYMMDD_HHMMSS.csv`
- `extracted_property_urls_backup_YYYYMMDD_HHMMSS.csv`
- `enriched_data_backup_YYYYMMDD_HHMMSS.json`

Backups are created when:
- Removing 404/410 properties (Step 3)
- Updating coordinates (Step 5)
- Re-analyzing criteria (Step 6)

---

## 💡 Pro Tips

1. **Run weekly** to keep data fresh and catch removed properties
2. **Keep HTTP server running** to instantly see updates in map viewer
3. **Check auth.json age** - if it's old, delete it for fresh login
4. **Use URL search** after update to verify new properties appear
5. **Check final statistics** to ensure geocoding is working well

---

## 🐛 Troubleshooting

### "Favorites scraping failed"
- Delete `auth.json` and log in again
- Check your Properstar account is active
- Verify internet connection

### "Properties not showing in map"
- Refresh browser (Cmd+R)
- Hard refresh (Cmd+Shift+R) to clear cache
- Check enriched_data.json was updated (check file timestamp)

### "Script hangs at Step 1"
- Look for Chromium browser window asking for login
- Click the browser window and log in to Properstar
- Script will continue automatically after login

---

## 📞 Need Help?

If the script fails or you encounter issues:
1. Read the error message carefully
2. Check the troubleshooting section above
3. Look at log files (criteria_api.log, gps_kpi_extraction.log)
4. Try running individual steps manually to isolate the issue

---

**Happy Property Hunting! 🌿🏡**