# Progress Bar Fix - Root Cause Found

## Problem

Progress bar stays at "Starting... (0%)" when running Full Update.

## Root Cause

The API server uses **Python 3.9** (which has Flask), but the scraper scripts need **Playwright** which isn't installed for system Python.

### Error Chain:
1. API calls `python3 auto_scrape_favorites.py now`
2. Script tries to import `playwright` → ModuleNotFoundError
3. Script crashes immediately
4. Progress file stays at 0%
5. UI shows stuck progress bar

## Solution Options

### Option 1: Install Playwright for System Python (Recommended)

```bash
# Install playwright for the Python being used by API
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python -m pip install playwright --break-system-packages

# Install browsers
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python -m playwright install chromium
```

### Option 2: Use Virtual Environment (Better Long-term)

```bash
cd /Users/jonathan/SynologyDrive/Since\ Today/PROJECTEN/farmmatch/scraper

# Create venv
python3 -m venv venv

# Activate
source venv/bin/activate

# Install all dependencies
pip install playwright pandas openai python-dotenv beautifulsoup4 flask flask-cors pydantic requests

# Install browsers
python -m playwright install chromium

# Update API to use venv Python
# In criteria_api.py, change subprocess calls to use venv/bin/python3
```

### Option 3: Quick Test with Direct Python

Find which Python has playwright:
```bash
# Check if any Python has playwright
python3 -c "import playwright; print('✅ Has playwright')" 2>/dev/null || echo "❌ No playwright"

# Try different Pythons
/usr/local/bin/python3 -c "import playwright; print('✅ Has playwright')" 2>/dev/null || echo "❌ No playwright"
```

## Immediate Workaround

While we fix the Python environment, you can run updates manually:

```bash
cd /Users/jonathan/SynologyDrive/Since\ Today/PROJECTEN/farmmatch/scraper

# Find Python with playwright (likely used before)
# Check recent successful runs - what Python was used?

# Run manually with that Python
# Example (adjust path as needed):
python3 favorites_scraper.py
```

## Verification

After installing playwright, test:

```bash
# Test import
python3 -c "import playwright; import asyncio; from favorites_scraper import scrape_favorites; print('✅ All imports work')"

# Test full pipeline
python3 auto_scrape_favorites.py now
# Should show progress updates, not crash immediately
```

## Why This Happened

The system has multiple Python installations:
- `/usr/bin/python3` - System Python 3.14
- `/Library/Developer/CommandLineTools/.../Python` - Xcode Python 3.9 (used by API)
- Possibly Homebrew Python

The API server was configured to use Python 3.9 (which has Flask), but scraper scripts need Playwright. These modules weren't installed for Python 3.9.

## Recommended Fix

**Install playwright for Python 3.9:**

```bash
# One command to fix everything:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python -m pip install --break-system-packages playwright pandas openai python-dotenv beautifulsoup4 requests pydantic

# Install browsers
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python -m playwright install chromium
```

Then restart API:
```bash
ps aux | grep criteria_api | grep -v grep | awk '{print $2}' | xargs kill
cd scraper
nohup /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python criteria_api.py > /tmp/criteria_api.log 2>&1 &
```

Test full update:
```bash
curl -X POST http://localhost:5002/api/scrape-favorites \
  -H "Content-Type: application/json" \
  -d '{"full_pipeline": true}'
```

Progress should now update properly!

## Files Modified

- ✅ `auto_scrape_favorites.py` - Made schedule import optional
- ✅ `criteria_api.py` - Fixed total_steps to 8
- ⚠️  Still needs: Playwright installation for Python 3.9

## Test After Fix

```bash
# Should work without errors:
python3 test_deployment.py

# Should show progress:
curl -X POST http://localhost:5002/api/scrape-favorites -H "Content-Type: application/json" -d '{"full_pipeline": true}'
# Get job_id from response, then:
curl http://localhost:5002/api/job-status/<job_id>
# Should show progress > 0
```