# Troubleshooting Guide

This document provides solutions to common issues encountered when working with HarnessElo.

## Table of Contents

1. [HRNZ API Authentication Issues](#hrnz-api-authentication-issues)
2. [Mock Mode for Testing](#mock-mode-for-testing)
3. [Database Issues](#database-issues)
4. [Rating Computation Issues](#rating-computation-issues)

---

## HRNZ API Authentication Issues

### Problem: 403 Forbidden Error

**Symptom**: When attempting to ingest race data, you receive a `403 Forbidden` error:

```
Access forbidden (403) when accessing: racing/meetings

COMMON CAUSE: Your HRNZ account does not have API access permissions.
```

**Root Cause**: Your HRNZ account credentials are valid (basic authentication works, HR key is obtained successfully), but your account does not have permission to access the racing data API endpoints.

**Solution**:

1. **Contact HRNZ Support**:
   - Email: support@hrnz.co.nz
   - Subject: "Request for Racing Data API v1.1 Access"
   - Provide your account details and explain you need API access for harness racing data

2. **Verify Account Type**:
   - Some HRNZ account types may not have API access by default
   - You may need to upgrade your account or purchase an API subscription
   - Ask HRNZ what account level is required for API access

3. **Request Developer/Sandbox Account**:
   - If you're developing or testing, ask if HRNZ offers a developer/sandbox account
   - These accounts often have API access enabled for testing purposes

4. **Use Mock Mode (Temporary Workaround)**:
   - While waiting for API access, you can use mock mode to test the application
   - See [Mock Mode for Testing](#mock-mode-for-testing) below

### Problem: 401 Unauthorized Error

**Symptom**: Authentication fails with `401 Unauthorized`:

```
Authentication failed (401 Unauthorized)
```

**Root Cause**: Invalid credentials or account issues.

**Solutions**:

1. **Check Credentials**:
   ```bash
   # Verify your .env file has correct credentials
   cat .env | grep HRNZ_
   ```
   - Ensure `HRNZ_USERNAME` and `HRNZ_PASSWORD` are correct
   - Check for extra spaces or newlines in credentials

2. **Verify Account Status**:
   - Log into HRNZ website to confirm your account is active
   - Check if your password has expired
   - Verify account hasn't been suspended

3. **Clear HR Key Cache**:
   ```bash
   # Remove cached HR key and retry
   rm .hrkey_cache
   docker compose run --rm worker python -m apps.worker.cli ingest --date 2024-12-26
   ```

### Authentication Flow Explained

The HRNZ API uses a two-step authentication process:

1. **Step 1: Obtain HR Key**
   - Uses HTTP Basic Auth (username/password)
   - Endpoint: `GET /security/hrkey`
   - Returns an `hrKey` that is cached for 24 hours

2. **Step 2: Access Data Endpoints**
   - Uses both Basic Auth AND `X-HR-KEY` header
   - The HR key provides scope/permissions for accessing specific endpoints
   - Even with valid credentials, you need the right permissions

**Why Both?** The HR key provides fine-grained access control. Your account might have valid credentials but limited API scopes.

---

## Mock Mode for Testing

Mock mode allows you to test HarnessElo without access to the real HRNZ API. It generates realistic sample data for development and testing.

### Enabling Mock Mode

**Method 1: Environment Variable**

```bash
# Set in your .env file
HRNZ_MOCK_MODE=true
```

**Method 2: Command Line**

```bash
# Pass as environment variable to Docker
docker compose run --rm -e HRNZ_MOCK_MODE=true worker python -m apps.worker.cli ingest --date 2024-12-26
```

### What Mock Mode Provides

- **Realistic Data Structure**: Mock data matches the real HRNZ API format exactly
- **Meetings**: Generates 1-2 meetings per day for date range
- **Races**: 8 races per meeting
- **Starters**: 8-12 runners per race with realistic attributes
- **Deterministic**: Same dates produce same data (for reproducibility)

### Mock Data Characteristics

```
Meetings: 1-2 per day (Auckland, Christchurch)
Races per Meeting: 8
Runners per Race: 8-12
Race Distances: 2000m-3400m
Gaits: Alternating Trot/Pace
Barriers: 1-12
Handicaps: 0m, 10m, 20m, 30m...
```

### Example Mock Ingestion

```bash
# Ingest mock data for a date range
docker compose run --rm -e HRNZ_MOCK_MODE=true worker \
  python -m apps.worker.cli ingest --from 2024-01-01 --to 2024-01-31

# Output:
# ✓ Ingestion completed successfully
# Meetings: 31
# Races: 248
# Starters: 2,480
```

### When to Use Mock Mode

✅ **Use mock mode when:**
- You don't have HRNZ API access yet
- You're developing new features
- You're writing tests
- You want to test the system without API rate limits
- You're demonstrating the system

❌ **Don't use mock mode for:**
- Production deployments
- Actual race predictions
- Performance benchmarking (data is simpler than real data)
- Training ML models (data is synthetic)

### Switching Between Real and Mock Mode

Mock mode is controlled by a single environment variable, so you can easily switch:

```bash
# Development with mock data
HRNZ_MOCK_MODE=true docker compose up

# Production with real API
HRNZ_MOCK_MODE=false docker compose up
```

### Mock Mode Limitations

- **No Real Race Results**: Placings are deterministic (1-8 in order)
- **Simplified Data**: Missing some optional fields present in real data
- **No Historical Accuracy**: Not suitable for backtesting predictions
- **Limited Variety**: Same venues, simplified runner names

---

## Database Issues

### Problem: "relation does not exist"

**Symptom**: Error when ingesting data:

```
relation "meetings" does not exist
```

**Solution**: Run database migrations:

```bash
docker compose run --rm worker alembic upgrade head
```

### Problem: Connection Refused

**Symptom**: Cannot connect to database

**Solutions**:

1. **Check Database is Running**:
   ```bash
   docker compose ps
   # Should show db container as "Up"
   ```

2. **Restart Database**:
   ```bash
   docker compose restart db
   ```

3. **Check Database Logs**:
   ```bash
   docker compose logs db
   ```

### Problem: Migration Conflicts

**Symptom**: Alembic migration errors

**Solutions**:

1. **Check Current Revision**:
   ```bash
   docker compose run --rm worker alembic current
   ```

2. **View Migration History**:
   ```bash
   docker compose run --rm worker alembic history
   ```

3. **Reset and Re-apply** (⚠️ destroys data):
   ```bash
   # Drop all tables
   docker compose exec db psql -U harnesselo -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;"

   # Re-run migrations
   docker compose run --rm worker alembic upgrade head
   ```

---

## Rating Computation Issues

### Problem: No Ratings Generated

**Check Prerequisites**:

1. **Data Ingested**:
   ```bash
   docker compose exec db psql -U harnesselo -c "SELECT COUNT(*) FROM races;"
   # Should return > 0
   ```

2. **Run Recompute**:
   ```bash
   docker compose run --rm worker python -m apps.worker.cli recompute \
     --from 2024-01-01 --to 2024-01-31
   ```

3. **Check Ratings**:
   ```bash
   docker compose exec db psql -U harnesselo -c "SELECT COUNT(*) FROM rating_snapshots;"
   ```

### Problem: Ratings Non-Deterministic

**Symptom**: Re-running recompute produces different ratings

**Causes**:
- Random number generation in code (should not exist)
- Non-deterministic race ordering
- Floating point precision issues
- Async operations completing in different orders

**Debug**:

1. **Check Race Order**:
   ```sql
   SELECT id, race_datetime FROM races ORDER BY race_datetime LIMIT 10;
   ```

2. **Compare Ratings**:
   ```bash
   # First run
   docker compose run --rm worker python -m apps.worker.cli recompute --from 2024-01-01 --to 2024-01-31
   docker compose exec db psql -U harnesselo -c "SELECT entity_id, rating FROM rating_snapshots WHERE entity_type='HORSE' LIMIT 5;" > run1.txt

   # Clear and second run
   docker compose exec db psql -U harnesselo -c "DELETE FROM rating_snapshots;"
   docker compose run --rm worker python -m apps.worker.cli recompute --from 2024-01-01 --to 2024-01-31
   docker compose exec db psql -U harnesselo -c "SELECT entity_id, rating FROM rating_snapshots WHERE entity_type='HORSE' LIMIT 5;" > run2.txt

   # Compare
   diff run1.txt run2.txt
   ```

---

## Getting Help

If you encounter issues not covered in this guide:

1. **Check Logs**:
   ```bash
   docker compose logs worker
   docker compose logs api
   docker compose logs db
   ```

2. **Enable Debug Logging**:
   ```bash
   # Add to .env
   LOG_LEVEL=DEBUG
   ```

3. **Check GitHub Issues**: [https://github.com/your-repo/issues](https://github.com/your-repo/issues)

4. **Contact Support**: Create a GitHub issue with:
   - Error messages (full stack trace)
   - Steps to reproduce
   - Environment details (OS, Docker version)
   - Relevant logs

---

**Last Updated**: 2024-12-28