# FarmMatch Pipeline Logging Guide

## Overview

The FarmMatch system now has comprehensive logging for all pipeline operations:
- **Scraping** favorites from Properstar
- **Checking availability** of properties
- **Geocoding** properties
- **Analyzing** properties with custom criteria and GPT

## Log Files

All pipeline jobs create log files in `/tmp/`:

- **Job Logs**: `/tmp/farmmatch_job_<job_id>.log`
  - Contains stdout and stderr from the pipeline
  - Created when you click "Full Update" or run availability checks

- **Progress Files**: `/tmp/farmmatch_progress_<job_id>.json`
  - Contains job status and progress percentage
  - Updated as the pipeline progresses

## Viewing Logs

### Method 1: Using the Helper Script (Recommended)

```bash
# List all recent job logs
./view_job_log.sh

# View a specific job's log
./view_job_log.sh <job_id>

# Follow a job's log in real-time
./view_job_log.sh <job_id> follow
```

### Method 2: Direct File Access

```bash
# List all job logs
ls -lht /tmp/farmmatch_job_*.log

# View a specific log
cat /tmp/farmmatch_job_<job_id>.log

# Follow a log in real-time
tail -f /tmp/farmmatch_job_<job_id>.log
```

### Method 3: Through the UI

When you start a job from the Criteria Manager UI:

1. The API response includes a `log_file` path
2. You can copy this path and view it in terminal
3. The UI progress indicator shows current status

## What Gets Logged

### Full Update Pipeline (`/api/scrape-favorites`)

Logs include:
- **Step 1**: Scraping favorites from Properstar
  - Number of properties found
  - URLs extracted
  - Any authentication errors

- **Step 2**: Checking availability
  - Each property URL being checked
  - Active vs removed properties
  - HTTP status codes

- **Step 3**: Geocoding
  - Coordinates being fetched
  - API responses
  - Rate limiting info

- **Step 4**: Analysis
  - Custom criteria evaluation
  - GPT analysis results
  - Scores and reasoning

### Availability Check Only (`/api/check-availability`)

Logs include:
- Properties being checked
- Availability status (active/removed)
- Reasons for removal
- Updated data summary

## Finding Your Job ID

When you click "Full Update" or "Check Availability":

1. **In the UI Response**: Check browser console for API response
   ```javascript
   {
     "success": true,
     "job_id": "a1b2c3d4",  // This is your job ID
     "log_file": "/tmp/farmmatch_job_a1b2c3d4.log"
   }
   ```

2. **List Recent Jobs**:
   ```bash
   ./view_job_log.sh  # Shows all recent jobs with IDs
   ```

3. **Check Progress Files**:
   ```bash
   ls -lht /tmp/farmmatch_progress_*.json | head -5
   ```

## Troubleshooting with Logs

### Problem: Pipeline Stuck at 0% Progress

**Check the log**:
```bash
./view_job_log.sh <job_id>
```

**Common issues**:
- **No auth.json**: Error message about missing authentication
  - **Fix**: Run `./login.sh` or double-click "Login to Properstar.command"

- **No output at all**: Process crashed immediately
  - **Fix**: Check if required scripts exist, Python dependencies installed

### Problem: Scraping Fails

**Look for in log**:
- "401 Unauthorized" → Need to login again
- "timeout" → Network issues
- "No properties found" → Favorites list might be empty

### Problem: Analysis Not Running

**Check log for**:
- "No active properties" → All properties were removed
- OpenAI API errors → Check API key
- Import errors → Missing Python packages

## Log Retention

Logs are stored in `/tmp/` which may be cleared on system restart.

**To preserve important logs**:
```bash
# Copy log to permanent location
cp /tmp/farmmatch_job_<job_id>.log ~/farmmatch_logs/

# Or save with timestamp
cp /tmp/farmmatch_job_<job_id>.log ~/farmmatch_logs/job_$(date +%Y%m%d_%H%M%S).log
```

## Monitoring Active Jobs

### Check if a job is still running

```bash
# Via API
curl -s http://localhost:5001/api/job-status/<job_id> | python3 -m json.tool

# Via process list
ps aux | grep python3 | grep farmmatch
```

### Follow progress in real-time

```bash
# Terminal 1: Follow the log
./view_job_log.sh <job_id> follow

# Terminal 2: Watch progress file
watch -n 2 "cat /tmp/farmmatch_progress_<job_id>.json | python3 -m json.tool"
```

## Example: Debugging a Failed Update

1. **Start update** from UI → Get job ID (e.g., "a1b2c3d4")

2. **Follow the log**:
   ```bash
   ./view_job_log.sh a1b2c3d4 follow
   ```

3. **See error** (e.g., "FileNotFoundError: auth.json")

4. **Fix issue**:
   ```bash
   ./login.sh
   ```

5. **Retry** from UI

6. **Verify success** in log:
   ```bash
   ./view_job_log.sh <new_job_id>
   # Should see: "✅ Analysis complete!"
   ```

## Log Format

Logs contain colored output and emoji indicators:

- ✅ Success messages (green)
- ❌ Error messages (red)
- ⚠️  Warning messages (yellow)
- ℹ️  Info messages (cyan)
- 📊 Progress updates
- 🔍 Detailed operation logs

**Example log output**:
```
======================================================================
🎯 SCRAPING FAVORITES
======================================================================

🔍 Found 186 properties in favorites
📥 Extracting URLs...
✅ Saved to: extracted_property_urls.csv

======================================================================
🔍 CHECKING AVAILABILITY
======================================================================

[1/186] https://www.properstar.nl/listing/12345
  ✅ Status: 200 - Property is ACTIVE

[2/186] https://www.properstar.nl/listing/67890
  ❌ Status: 404 - Property REMOVED

...
```