# 🔧 FarmMatch Process Management Guide

## Problem: Background Processes and System Clutter

When running FarmMatch scripts, especially in development or debugging, background processes can accumulate and cause issues:
- **Duplicate processes** running simultaneously
- **Stuck processes** consuming resources
- **Conflicting operations** on the same files
- **Unclear system state** - which processes are actually running?

This guide explains how to avoid and fix these issues.

---

## ✅ Best Practices (Prevention)

### 1. **Always Use `full_update.sh`**

The full update script is designed to run safely:
```bash
./full_update.sh
```

**Why?**
- ✅ Checks for running processes before starting
- ✅ Runs steps sequentially (no overlaps)
- ✅ Handles errors gracefully
- ✅ Uses foreground execution (you see what's happening)

**Don't do this:**
```bash
# ❌ BAD - runs in background, hard to track
python3 favorites_scraper.py &
python3 extract_breadcrumbs.py &
python3 geocode_with_breadcrumbs.py &
```

---

### 2. **Check Process Status Before Running Scripts**

Before running any manual script, check if it's already running:

```bash
# Check for FarmMatch processes
ps aux | grep -E "(favorites_scraper|extract_breadcrumbs|extract_gps|geocode_|analyze_from_urls)" | grep python3
```

If you see output, a process is already running. Wait for it or kill it.

---

### 3. **Use the Cleanup Utility**

When in doubt, clean up all FarmMatch processes:

```bash
./cleanup_processes.sh
```

This safely kills:
- favorites_scraper.py
- extract_breadcrumbs.py
- fix_missing_breadcrumbs.py
- extract_gps_and_kpis.py
- geocode_with_breadcrumbs.py
- analyze_from_urls_optimized.py
- And more...

**When to use:**
- Before starting a full update
- When processes seem stuck
- After interrupting a script (Ctrl+C)
- When debugging

---

## 🚨 Common Issues and Fixes

### Issue 1: "Process Already Running"

**Symptom:**
```
Error: Another instance of favorites_scraper.py is running
```

**Fix:**
```bash
./cleanup_processes.sh
./full_update.sh
```

---

### Issue 2: Background Processes from Old Sessions

**Symptom:**
```
Background Bash 128bc8 (status: running) Has new output available
Background Bash 5e20f0 (status: running) Has new output available
```

**What happened:**
- You (or the system) started processes in background earlier
- They're still tracked but may have finished or gotten stuck

**Fix:**
```bash
# Kill actual Python processes
ps aux | grep python3 | grep farmmatch | awk '{print $2}' | xargs kill

# Or use the cleanup utility
./cleanup_processes.sh
```

---

### Issue 3: Script Stuck on "Waiting..."

**Symptom:**
```
🌐 Navigeer naar favorietenpagina...
[hangs here forever]
```

**What happened:**
- favorites_scraper.py opened a browser window waiting for login
- The browser window is hidden or you didn't see it

**Fix:**
1. Look for a Chromium/Playwright browser window
2. Log in to Properstar in that window
3. Script will continue automatically

**Or cancel:**
```bash
# Press Ctrl+C
# Then cleanup
./cleanup_processes.sh
```

---

### Issue 4: File Lock Errors

**Symptom:**
```
Error: Permission denied: 'analysis_output.csv'
Error: File is locked
```

**What happened:**
- Multiple processes trying to write to the same file
- Previous process crashed and left file lock

**Fix:**
```bash
# Kill all FarmMatch processes
./cleanup_processes.sh

# Check for file locks (macOS)
lsof | grep "analysis_output.csv"

# Force close if needed (replace PID)
kill -9 <PID>
```

---

## 🔍 Diagnostic Commands

### Check What's Running

```bash
# Show all FarmMatch Python processes
ps aux | grep python3 | grep -E "(favorites|extract|geocode|analyze)"

# Show with PIDs and command
ps aux | grep python3 | grep farmmatch | awk '{print $2, $11, $12, $13}'

# Count running processes
ps aux | grep python3 | grep farmmatch | grep -v grep | wc -l
```

### Check System Load

```bash
# CPU usage
top -l 1 | grep "CPU usage"

# Memory usage
top -l 1 | grep PhysMem

# Disk I/O
iostat -w 1 -c 5
```

### Check File Locks

```bash
# Files open by FarmMatch processes
lsof -c python3 | grep farmmatch

# Specific file
lsof | grep "analysis_output.csv"
```

---

## 🛠️ Manual Process Management

### Start a Process (Foreground - Recommended)

```bash
# You see all output, Ctrl+C stops it
python3 favorites_scraper.py
```

### Start a Process (Background - Use Carefully)

```bash
# Runs in background, harder to track
python3 extract_gps_and_kpis.py > gps.log 2>&1 &

# Get PID of last background process
echo $!

# Monitor log
tail -f gps.log
```

### Stop a Specific Process

```bash
# Graceful stop (SIGTERM)
kill <PID>

# Force stop if it doesn't respond (SIGKILL)
kill -9 <PID>

# Stop by name
pkill -f "extract_breadcrumbs.py"
```

### Stop All FarmMatch Processes

```bash
# Use the cleanup utility (recommended)
./cleanup_processes.sh

# Or manually
ps aux | grep python3 | grep farmmatch | awk '{print $2}' | xargs kill
```

---

## 📋 Process Lifecycle (Full Update)

Here's what happens when you run `./full_update.sh`:

```
1. Check for conflicts → Kill if user confirms
2. Run favorites_scraper.py (foreground) → Wait for completion
3. Run extract_breadcrumbs.py (foreground) → Wait for completion
4. Run 404 removal (foreground) → Wait for completion
5. Run extract_gps_and_kpis.py (foreground) → Wait for completion
6. Run geocode_with_breadcrumbs.py (foreground) → Wait for completion
7. Start criteria_api.py (background) → Keep running
8. Run analyze_from_urls_optimized.py (foreground) → Wait for completion
9. Run parse_criteria.py (foreground) → Wait for completion
10. Show summary and exit
```

**Key points:**
- Most steps run in **foreground** (sequential, no overlap)
- Only `criteria_api.py` runs in **background** (needed as API server)
- Each step waits for previous to complete
- No duplicate processes

---

## 🎯 Prevention Checklist

Before running any FarmMatch operation:

- [ ] Check if `full_update.sh` is already running
- [ ] Check if any Python FarmMatch scripts are running
- [ ] If in doubt, run `./cleanup_processes.sh`
- [ ] Close any lingering Playwright/Chromium browser windows
- [ ] Check available disk space (`df -h`)
- [ ] Check available memory (`top`)

---

## 🆘 Emergency Recovery

If everything is stuck/broken:

```bash
# 1. Kill ALL Python processes (nuclear option)
pkill python3

# 2. Kill ALL Chromium/Playwright browsers
pkill -f chromium
pkill -f playwright

# 3. Remove lock files (if they exist)
rm -f .*.lock
rm -f *.lock

# 4. Check for zombie processes
ps aux | grep "Z"

# 5. Restart from clean slate
./cleanup_processes.sh
./full_update.sh
```

---

## 📊 Monitoring Active Processes

### Real-time CPU and Memory

```bash
# Live view of all processes
top

# Live view filtered for Python
top | grep python3

# Update every 2 seconds
watch -n 2 'ps aux | grep python3 | grep farmmatch'
```

### Process Duration

```bash
# Show how long processes have been running
ps -eo pid,etime,comm | grep python3
```

### Log Monitoring

```bash
# Watch favorites scraper
tail -f favorites_scraper.log

# Watch GPS extraction
tail -f gps_kpi_extraction.log

# Watch criteria API
tail -f criteria_api.log

# Watch all logs simultaneously
tail -f *.log
```

---

## ⚡ Quick Reference

| Task | Command |
|------|---------|
| Clean all processes | `./cleanup_processes.sh` |
| Run full update | `./full_update.sh` |
| Check what's running | `ps aux \| grep python3 \| grep farmmatch` |
| Kill specific PID | `kill <PID>` |
| Force kill | `kill -9 <PID>` |
| Kill by name | `pkill -f "script_name.py"` |
| Monitor logs | `tail -f *.log` |
| Check file locks | `lsof \| grep "filename.csv"` |

---

## 🎓 Best Practices Summary

1. ✅ **Use `full_update.sh`** for all updates (not individual scripts)
2. ✅ **Run `cleanup_processes.sh`** before starting new operations
3. ✅ **Monitor logs** to see progress (`tail -f *.log`)
4. ✅ **Wait for completion** - don't start multiple updates simultaneously
5. ✅ **Use Ctrl+C carefully** - always run cleanup after interrupting
6. ❌ **Don't background processes manually** - use the full update script
7. ❌ **Don't run multiple updates** at the same time
8. ❌ **Don't ignore "process running" warnings**

---

## 🔗 Related Documentation

- [FULL_UPDATE_GUIDE.md](FULL_UPDATE_GUIDE.md) - Complete guide to full update script
- [QUICK_START.md](QUICK_START.md) - Quick reference for running updates
- [cleanup_processes.sh](cleanup_processes.sh) - Process cleanup utility

---

**Keep your FarmMatch system clean and running smoothly! 🌿**