# 🗺️ Breadcrumb-Based Geocoding

**Status**: ✅ Scripts created, ready to use

---

## Problem Solved

### Before:
- Properties geocoded using ambiguous location names from favorites page
- "Cabo" → geocoded to **Brazil** instead of Spain 🇧🇷❌
- Missing context led to incorrect coordinates

### After:
- Extract breadcrumb hierarchy from property detail pages
- "Spain > Galicia > Lugo > Monforte de Lemos" → correct Spanish location 🇪🇸✅
- Full geographic context for accurate geocoding

---

## 📋 Scripts Created

### 1. Extract Breadcrumbs
**File**: [extract_breadcrumbs.py](../extract_breadcrumbs.py)

**What it does**:
- Visits each property detail page on Properstar
- Extracts breadcrumb navigation (e.g., "Spain > Galicia > Lugo > Monforte de Lemos")
- Saves breadcrumb data to `extracted_property_urls.csv`

**Usage**:
```bash
python3 extract_breadcrumbs.py
```

**Progress**: Saves every 10 properties, can resume if interrupted

**Example output**:
```
[1/186] https://www.properstar.nl/listing/96231490
  ✅ Spain > Galicia > Lugo > Monforte de Lemos

[2/186] https://www.properstar.nl/listing/86083770
  ✅ Spain > Valencian Community > Castellón > Useras
```

---

### 2. Geocode With Breadcrumbs
**File**: [geocode_with_breadcrumbs.py](../geocode_with_breadcrumbs.py)

**What it does**:
- Uses breadcrumb hierarchy for accurate geocoding
- Falls back to favorites location if no breadcrumb
- Prioritizes European locations (prevents Brazil/India false matches)
- Updates `analysis_output.csv` with lat/lon coordinates

**Usage**:
```bash
python3 geocode_with_breadcrumbs.py
```

**Geocoding priority**:
1. **Breadcrumb** (most accurate): "Monforte de Lemos, Lugo, Galicia, Spain"
2. **Favorites location** (fallback): "Cabo"

**Example output**:
```
[176/186] https://www.properstar.nl/listing/96231490
  📍 Breadcrumb: Spain > Galicia > Lugo > Monforte de Lemos
  ✅ Monforte de Lemos, Lugo, Galicia, España
  📍 42.5237, -7.5097
```

---

## 🚀 How to Use (Complete Workflow)

### Step 1: Extract Breadcrumbs
```bash
cd "/Users/jonathan/SynologyDrive/Since Today/PROJECTEN/farmmatch/scraper"
python3 extract_breadcrumbs.py
```

**Time**: ~15-20 minutes for 186 properties (1 second per property)

**Result**: `extracted_property_urls.csv` now has "Breadcrumb" column

---

### Step 2: Geocode With Breadcrumbs
```bash
python3 geocode_with_breadcrumbs.py
```

**Time**: ~5-10 minutes for properties without coordinates

**Result**: `analysis_output.csv` updated with accurate lat/lon

---

### Step 3: Regenerate enriched_data.json
```bash
python3 parse_criteria.py
```

**Result**: `enriched_data.json` now has correct coordinates

---

### Step 4: Refresh Browser
Open map and do **hard refresh**: `Cmd + Shift + R` (Mac) or `Ctrl + F5` (Windows)

**Result**: All properties show in correct locations on map! 🗺️✅

---

## 🔄 Integration with Auto Update

To integrate breadcrumb extraction into the automated pipeline, update [auto_scrape_favorites.py](../auto_scrape_favorites.py):

```python
# After Step 1 (favorites scraping)
run_command("python3 extract_breadcrumbs.py")

# Step 2: Availability check (as before)

# Step 3: Geocoding (use breadcrumb version)
run_command("python3 geocode_with_breadcrumbs.py")

# Step 4-5: Custom criteria, GPT analysis (as before)
```

---

## 📊 Expected Results

### Before Breadcrumbs:
```
Properties with coordinates: 3/186 (1.6%)
Incorrect locations: Many (Brazil, India, etc.)
```

### After Breadcrumbs:
```
Properties with coordinates: 180+/186 (96%+)
Accurate locations: Yes ✅
Geographic hierarchy: Complete (Country > Region > Province > City)
```

---

## 🎯 Benefits

1. **Accuracy**: Full geographic hierarchy prevents ambiguous location names
2. **European Focus**: Bounded geocoding prevents false matches in other continents
3. **Transparency**: Shows exact location hierarchy used for geocoding
4. **Resumable**: Saves progress, can stop and restart
5. **Future-Proof**: Properstar breadcrumbs are stable and reliable

---

## 🛠️ Technical Details

### Breadcrumb Extraction

**HTML Elements Checked**:
1. `.breadcrumb-container` (primary)
2. `nav[aria-label='breadcrumb']` (fallback)
3. `.breadcrumb` (fallback)

**Filtering**:
- Removes "Home", "Properstar", "Properties" from breadcrumb
- Keeps only meaningful location parts

### Geocoding Strategy

**Nominatim Settings**:
```python
viewbox=[(-10, 35), (40, 70)]  # Western Europe bounding box
bounded=True                     # Stay within Europe
timeout=10                       # 10 second timeout
```

**Result Validation**:
- Latitude between 35-70 (Europe)
- Longitude between -10 to 40 (Europe)

---

## 📝 Example: Cabo → Monforte de Lemos

### Problem Property
**URL**: https://www.properstar.nl/listing/96231490

**Before**:
- Location from favorites: "Cabo"
- Geocoded to: Cabo de Santo Agostinho, Brazil (-8.28, -35.03) ❌
- 10,000 km away from actual location!

**After**:
- Breadcrumb: "Spain > Galicia > Lugo > Monforte de Lemos"
- Geocoded to: Monforte de Lemos, España (42.52, -7.51) ✅
- Correct location in Galicia, Spain!

---

## ✅ Completion Checklist

- ✅ Created extract_breadcrumbs.py
- ✅ Created geocode_with_breadcrumbs.py
- ✅ Documented usage and benefits
- ⏳ Run extract_breadcrumbs.py (user action needed)
- ⏳ Run geocode_with_breadcrumbs.py (user action needed)
- ⏳ Regenerate enriched_data.json (user action needed)
- ⏳ Test on map (user action needed)

---

## 🚦 Next Steps

1. **Run extract_breadcrumbs.py** to get breadcrumb data for all 186 properties
2. **Run geocode_with_breadcrumbs.py** to geocode using breadcrumbs
3. **Run parse_criteria.py** to update enriched_data.json
4. **Refresh browser** to see properties in correct locations

After this, **Cabo will be in Spain, not Brazil!** 🎉