Imported from T-rav/hydraflow (.codex/skills/hf.test-audit/SKILL.md). Install upstream with npx skills add T-rav/hydraflow --skill hf.test-audit. Copyright stays with the author.

Test Audit Agent

Comprehensive test quality auditing agent that analyzes test files for adherence to established patterns, identifies anti-patterns, and suggests improvements based on the codebase's own testing standards.

Agent Purpose

Audit test files across all services (bot, tasks, control_plane, agent-service) to ensure:

1. Naming conventions - Files and tests follow established patterns
2. 3As structure - Arrange, Act, Assert clarity
3. Single responsibility - One logical assertion per test
4. Builders & Factories - Proper use and identification of missing opportunities
5. Anti-pattern detection - Multiple assertions, over-mocking, incomplete setups

Established Standards (From Codebase Analysis)

1. Factory Pattern

Location: bot/tests/utils/mocks/, bot/tests/utils/settings/, tasks/tests/factories.py

Pattern:

class LLMProviderMockFactory:
    """Factory for creating mock LLM provider objects"""

    @staticmethod
    def create_mock_provider(response: str = "Test response") -> MagicMock:
        """Create basic mock LLM provider"""
        provider = MagicMock()
        provider.get_response = AsyncMock(return_value=response)
        provider.model = "test-model"
        return provider

    @staticmethod
    def create_provider_with_error(error: Exception) -> MagicMock:
        """Create LLM provider that raises errors"""
        provider = LLMProviderMockFactory.create_mock_provider()
        provider.get_response = AsyncMock(side_effect=error)
        return provider

Standards:

• Static methods only
• Clear method names: create_<thing>, create_<thing>_with_<condition>
• Composition: specialized factories call basic factory
• Return typed objects (MagicMock, AsyncMock, or real objects)
• Docstrings for each method

2. Builder Pattern

Location: bot/tests/utils/builders/

Pattern:

class MessageBuilder:
    """Builder for creating message objects with fluent API"""

    def __init__(self):
        self._role = "user"
        self._content = "Test message"

    def as_user(self):
        """Set role as user"""
        self._role = "user"
        return self

    def as_assistant(self):
        """Set role as assistant"""
        self._role = "assistant"
        return self

    def with_content(self, content: str):
        """Set message content"""
        self._content = content
        return self

    def build(self) -> dict[str, Any]:
        """Build the message"""
        return {"role": self._role, "content": self._content}

Standards:

• Instance methods that return self for chaining
• as_<state>() methods for role/type
• with_<attribute>() methods for properties
• Final build() method returns the object
• Can include class methods for common scenarios: @classmethod def user_message(cls, content: str)

3. Test Naming Convention

Pattern: test_<method/feature>_<scenario>[_<expected_result>]

Examples from codebase:

• ✅ test_send_message_success - Method + Scenario
• ✅ test_send_message_with_blocks - Method + Condition
• ✅ test_conversation_id_with_thread - Feature + Condition
• ✅ test_invoke_agent_404_not_found - Method + HTTP status
• ❌ test_1 - Meaningless
• ❌ test_user_test - Redundant "test"

4. 3As Structure

Good Example from codebase:

@pytest.mark.asyncio
async def test_send_message_success(self):
    """Test successful message sending"""
    # Arrange
    channel = "C12345"
    text = "Hello, World!"
    thread_ts = "1234567890.123456"
    expected_response = {"ts": "1234567890.654321"}

    slack_service = SlackServiceFactory.create_service_with_successful_client(
        expected_response
    )

    # Act
    result = await slack_service.send_message(channel, text, thread_ts)

    # Assert
    assert result == expected_response
    slack_service.client.chat_postMessage.assert_called_once_with(
        channel=channel, text=text, thread_ts=thread_ts, blocks=None, mrkdwn=True
    )

Standards:

• Clear separation of setup, execution, verification
• Comments (# Arrange, # Act, # Assert) optional but helpful
• All arrange code before act
• All assertions after act
• No mixing of phases

5. Single Responsibility

Good:

def test_user_creation_sets_name():
    user = create_user({"name": "John"})
    assert user.name == "John"

def test_user_creation_sets_email():
    user = create_user({"email": "john@example.com"})
    assert user.email == "john@example.com"

Bad:

def test_user_creation():
    user = create_user({"name": "John", "email": "john@example.com"})
    assert user.name == "John"
    assert user.email == "john@example.com"
    assert user.is_active is True  # Testing 3 different things
    assert user.created_at is not None

6. Factory + Fixture Pattern

From conftest.py:

# Factory class (reusable logic)
class SettingsMockFactory:
    @staticmethod
    def create_basic_settings() -> MagicMock:
        settings = MagicMock()
        settings.slack.bot_token = "test-token"
        return settings

# Fixture (pytest integration)
@pytest.fixture
def mock_settings():
    """Mock application settings with test values."""
    return SettingsMockFactory.create_basic_settings()

Standards:

• Factories can be used independently or in fixtures
• Fixtures provide pytest integration
• Tests can call factory directly if they need customization
• Session-scoped fixtures for expensive setup

7. Test Coverage Analysis (NEW)

Critical - Untested Code Paths:

# ❌ CRITICAL - Critical function with no tests
# File: bot/services/payment_processor.py
def process_payment(user_id: int, amount: float, card_token: str) -> dict:
    """Process payment for user."""
    charge = stripe.Charge.create(
        amount=int(amount * 100),
        currency="usd",
        source=card_token
    )

    user = User.query.get(user_id)
    user.balance += amount
    user.save()

    send_receipt_email(user.email, charge.id)
    return {"status": "success", "charge_id": charge.id}

# No tests found for this function!
# Missing tests for:
# - Happy path (successful payment)
# - Error cases (invalid card, insufficient funds, stripe API error)
# - Edge cases (negative amount, non-existent user, amount = 0)
# - Side effects (user balance updated, email sent)

How to detect:

# Generate coverage report
pytest --cov=bot --cov-report=term-missing

# Find functions/files with <70% coverage
pytest --cov=bot --cov-report=html
# Open htmlcov/index.html and find files in red

# Analyze specific module
pytest --cov=bot.services.payment_processor --cov-report=term-missing

High - Missing Edge Case Tests:

# ✅ GOOD - Has happy path test
def test_divide_positive_numbers():
    assert divide(10, 2) == 5

# ❌ MISSING - No edge case tests
# Missing tests for:
# - divide(10, 0) → Should raise ValueError
# - divide(0, 10) → Should return 0
# - divide(-10, 2) → Should return -5
# - divide(10, -2) → Should return -5
# - divide(1, 3) → Should handle float precision

Medium - Untested Error Paths:

# Production code
def get_user(user_id: int) -> User:
    user = User.query.get(user_id)
    if not user:
        raise UserNotFoundError(f"User {user_id} not found")
    return user

# ✅ Test exists for happy path
def test_get_user_success():
    user = get_user(123)
    assert user.id == 123

# ❌ MISSING - No error path test
# Missing test:
def test_get_user_not_found():
    with pytest.raises(UserNotFoundError, match="User 999 not found"):
        get_user(999)

Detection Patterns:

# CRITICAL severity
- Functions in critical modules (payment, auth, security) with 0% coverage
- Error handling code (try/except blocks) never executed in tests
- Database operations (create, update, delete) with no tests

# HIGH severity
- Functions with <50% coverage
- Missing tests for all error paths
- No tests for input validation

# MEDIUM severity
- Functions with 50-70% coverage
- Missing tests for edge cases (null, empty, boundary values)
- Integration paths not covered

# Detection Commands:
pytest --cov=bot --cov-report=term-missing | grep "0%"
grep -r "def " --include="*.py" bot/services/ | wc -l  # Count functions
grep -r "def test_" --include="*.py" bot/tests/ | wc -l  # Count tests
# Compare ratio: should be at least 3 tests per function

8. Test Performance & Reliability (NEW)

Critical - Flaky Tests:

# ❌ CRITICAL - Time-dependent test (flaky)
def test_async_processing():
    start_processing(task_id=123)
    time.sleep(0.5)  # Flaky! Sometimes 0.5s isn't enough
    result = get_result(task_id=123)
    assert result.status == "complete"

# ✅ GOOD - Wait with timeout
import pytest

@pytest.mark.asyncio
async def test_async_processing():
    await start_processing(task_id=123)

    # Poll with timeout
    for _ in range(10):  # Max 5 seconds
        result = await get_result(task_id=123)
        if result.status == "complete":
            break
        await asyncio.sleep(0.5)
    else:
        pytest.fail("Processing did not complete in 5 seconds")

    assert result.status == "complete"

# ✅ BETTER - Use async/await properly
@pytest.mark.asyncio
async def test_async_processing():
    result = await start_processing(task_id=123)
    # Awaits completion, no sleep needed
    assert result.status == "complete"

# ❌ CRITICAL - Random behavior (non-deterministic)
def test_random_selection():
    items = [1, 2, 3, 4, 5]
    selected = random.choice(items)
    assert selected > 3  # Fails 40% of the time!

# ✅ GOOD - Seed random or mock it
def test_random_selection():
    items = [1, 2, 3, 4, 5]
    random.seed(42)  # Deterministic
    selected = random.choice(items)
    assert selected == 5  # Always same result

# ✅ BETTER - Mock random
from unittest.mock import patch

def test_random_selection():
    items = [1, 2, 3, 4, 5]
    with patch('random.choice', return_value=4):
        selected = random.choice(items)
        assert selected == 4

High - Slow Tests:

# ❌ HIGH - Test takes 5 seconds (too slow)
def test_user_creation():
    # Creates full user with profile, permissions, settings
    user = create_full_user_profile()  # 3 seconds
    time.sleep(2)  # Unnecessary wait
    assert user.created

# ✅ GOOD - Fast test with minimal setup
def test_user_creation():
    # Only create what's needed
    user = User(name="Test", email="test@example.com")
    user.save()  # < 0.1 seconds
    assert user.id is not None

# ❌ HIGH - Real API calls in tests
def test_fetch_weather():
    weather = weather_api.get_current("London")  # Real HTTP call!
    assert weather.temperature > 0

# ✅ GOOD - Mock external services
from unittest.mock import patch

def test_fetch_weather():
    with patch('weather_api.get_current') as mock_get:
        mock_get.return_value = {"temperature": 20}
        weather = weather_api.get_current("London")
        assert weather["temperature"] == 20

Medium - Test Isolation Issues:

# ❌ MEDIUM - Tests depend on execution order
def test_create_user():
    user = User.create(name="Alice")
    assert user.id == 1  # Assumes no other users exist!

def test_get_user():
    user = User.get(1)  # Depends on test_create_user running first!
    assert user.name == "Alice"

# ✅ GOOD - Each test independent
def test_create_user():
    user = User.create(name="Alice")
    assert user.id is not None  # Don't assume specific ID

def test_get_user():
    # Setup own data
    user = User.create(name="Bob")
    fetched = User.get(user.id)
    assert fetched.name == "Bob"

# ❌ MEDIUM - Shared state between tests
class TestUserService:
    cache = {}  # Shared across all tests!

    def test_cache_user(self):
        self.cache[1] = User(id=1, name="Alice")
        assert len(self.cache) == 1

    def test_get_from_cache(self):
        # Depends on test_cache_user!
        assert 1 in self.cache

# ✅ GOOD - Each test has own state
class TestUserService:
    def setup_method(self):
        self.cache = {}  # Fresh cache per test

    def test_cache_user(self):
        self.cache[1] = User(id=1, name="Alice")
        assert len(self.cache) == 1

    def test_get_from_cache(self):
        # Setup own data
        self.cache[1] = User(id=1, name="Alice")
        assert 1 in self.cache

Detection Patterns:

# CRITICAL severity
- time.sleep() in tests (flaky timing)
- random.choice/randint without seeding (non-deterministic)
- Tests that fail intermittently (flaky)
- Tests depending on external services (network calls)

# HIGH severity
- Tests taking > 2 seconds
- Real database operations without transactions
- Shared state between tests (class attributes)
- Tests that must run in specific order

# MEDIUM severity
- Tests taking 0.5-2 seconds
- Missing setup/teardown for cleanup
- Tests modifying global state

# Detection Commands:
pytest --durations=10  # Show 10 slowest tests
pytest -v | grep "FLAKY"  # If using pytest-flaky
pytest --lf  # Rerun only failed tests (detect flaky tests)
pytest -x  # Stop on first failure (detect order dependency)

9. Assertion Quality (NEW)

High - Weak Assertions:

# ❌ HIGH - Very weak assertion
def test_user_creation():
    user = create_user()
    assert user  # Only checks user is not None!

# ✅ GOOD - Specific assertions
def test_user_creation():
    user = create_user()
    assert user.id is not None
    assert user.name == "Test User"
    assert user.email == "test@example.com"
    assert user.is_active is True

# ❌ HIGH - Generic assertion
def test_api_response():
    response = api.get_user(123)
    assert response is not None  # Too generic!
    assert response.get("status") == "success"  # Could be wrong key

# ✅ GOOD - Explicit structure validation
def test_api_response():
    response = api.get_user(123)
    assert response["status"] == "success"  # Will fail if key missing
    assert response["user"]["id"] == 123
    assert response["user"]["name"] == "John Doe"
    assert "created_at" in response["user"]

# ❌ HIGH - Assert True (meaningless)
def test_validation():
    result = validate_email("test@example.com")
    assert True  # Always passes!

# ✅ GOOD - Assert actual value
def test_validation():
    result = validate_email("test@example.com")
    assert result is True  # Or: assert result

Medium - Missing Edge Case Assertions:

# ✅ Has basic test
def test_parse_date():
    date = parse_date("2023-12-25")
    assert date.year == 2023

# ❌ MISSING - No edge case tests
# Missing assertions for:
# - Empty string: parse_date("") → Should raise ValueError
# - Invalid format: parse_date("25-12-2023") → Should raise ValueError
# - Leap year: parse_date("2024-02-29") → Should parse correctly
# - Invalid date: parse_date("2023-02-30") → Should raise ValueError
# - Null input: parse_date(None) → Should raise TypeError

# ✅ COMPREHENSIVE - All edge cases covered
def test_parse_date_valid():
    date = parse_date("2023-12-25")
    assert date.year == 2023
    assert date.month == 12
    assert date.day == 25

def test_parse_date_empty():
    with pytest.raises(ValueError, match="Empty date string"):
        parse_date("")

def test_parse_date_invalid_format():
    with pytest.raises(ValueError, match="Invalid date format"):
        parse_date("25-12-2023")

def test_parse_date_leap_year():
    date = parse_date("2024-02-29")
    assert date.day == 29

def test_parse_date_invalid_date():
    with pytest.raises(ValueError, match="day is out of range"):
        parse_date("2023-02-30")

def test_parse_date_null():
    with pytest.raises(TypeError):
        parse_date(None)

Medium - Missing Negative Tests:

# ✅ Has positive test
def test_authenticate_success():
    token = authenticate("user", "correct_password")
    assert token is not None

# ❌ MISSING - No negative tests
# Missing tests for:
# - Wrong password
# - Non-existent user
# - Empty credentials
# - SQL injection attempts
# - Account locked
# - Expired password

# ✅ COMPREHENSIVE - Positive and negative cases
def test_authenticate_success():
    token = authenticate("user", "correct_password")
    assert token is not None
    assert len(token) == 32

def test_authenticate_wrong_password():
    with pytest.raises(AuthenticationError, match="Invalid credentials"):
        authenticate("user", "wrong_password")

def test_authenticate_nonexistent_user():
    with pytest.raises(AuthenticationError, match="User not found"):
        authenticate("nonexistent", "password")

def test_authenticate_empty_credentials():
    with pytest.raises(ValidationError, match="Username required"):
        authenticate("", "password")

def test_authenticate_sql_injection():
    with pytest.raises(AuthenticationError):
        authenticate("admin' OR '1'='1", "password")

Detection Patterns:

# HIGH severity
- assert user (checks only truthiness)
- assert response is not None (too generic)
- assert True / assert result (meaningless)
- No assertions in test (test does nothing)

# MEDIUM severity
- Single assertion for complex object
- No edge case tests (boundary values)
- No negative tests (error cases)
- Generic assertions (assert len(x) > 0 instead of specific)

# LOW severity
- Could add more specific assertions
- Could test more attributes

# Detection Commands:
grep -r "assert True" --include="*.py" tests/
grep -r "assert.*is not None" --include="*.py" tests/
grep -r "def test_" --include="*.py" tests/ -A10 | grep -v "assert"

10. Test Maintainability (NEW)

High - Parameterization Opportunities:

# ❌ HIGH - Repeated test structure (DRY violation)
def test_validate_email_valid_gmail():
    assert validate_email("user@gmail.com") is True

def test_validate_email_valid_yahoo():
    assert validate_email("user@yahoo.com") is True

def test_validate_email_valid_custom():
    assert validate_email("user@custom-domain.com") is True

def test_validate_email_invalid_no_at():
    assert validate_email("usergmail.com") is False

def test_validate_email_invalid_no_domain():
    assert validate_email("user@") is False

# ✅ GOOD - Parameterized test
import pytest

@pytest.mark.parametrize("email,expected", [
    ("user@gmail.com", True),
    ("user@yahoo.com", True),
    ("user@custom-domain.com", True),
    ("usergmail.com", False),  # No @
    ("user@", False),  # No domain
    ("@gmail.com", False),  # No user
    ("", False),  # Empty
    ("user @gmail.com", False),  # Space
])
def test_validate_email(email, expected):
    assert validate_email(email) == expected

# ❌ HIGH - Multiple similar tests
def test_calculate_discount_10_percent():
    assert calculate_discount(100, 0.1) == 90

def test_calculate_discount_25_percent():
    assert calculate_discount(100, 0.25) == 75

def test_calculate_discount_50_percent():
    assert calculate_discount(200, 0.5) == 100

# ✅ GOOD - Single parameterized test
@pytest.mark.parametrize("price,discount,expected", [
    (100, 0.1, 90),
    (100, 0.25, 75),
    (200, 0.5, 100),
    (50, 0.0, 50),  # No discount
    (100, 1.0, 0),  # 100% discount
])
def test_calculate_discount(price, discount, expected):
    assert calculate_discount(price, discount) == expected

Medium - Test Brittleness:

# ❌ MEDIUM - Brittle test (coupled to implementation)
def test_process_order():
    order = Order(items=[Item(id=1), Item(id=2)])

    # Brittle: Assumes specific internal method calls
    with patch.object(order, '_validate_items') as mock_validate:
        with patch.object(order, '_calculate_total') as mock_calc:
            with patch.object(order, '_apply_discount') as mock_discount:
                order.process()
                mock_validate.assert_called_once()
                mock_calc.assert_called_once()
                mock_discount.assert_called_once()

# ✅ GOOD - Test behavior, not implementation
def test_process_order():
    order = Order(items=[Item(price=100), Item(price=50)])
    result = order.process()

    # Test outcomes, not how it's done
    assert result.total == 150
    assert result.status == "processed"
    assert len(result.items) == 2

# ❌ MEDIUM - Hardcoded values (brittle)
def test_get_users():
    users = get_users()
    assert len(users) == 5  # Breaks if users change
    assert users[0].name == "Alice"  # Breaks if order changes

# ✅ GOOD - Test properties, not exact values
def test_get_users():
    users = get_users()
    assert len(users) > 0
    assert all(user.name for user in users)  # All have names
    assert all(user.email for user in users)  # All have emails

Medium - Test Data Management:

# ❌ MEDIUM - Hardcoded test data scattered across tests
def test_create_order():
    user = {
        "id": 123,
        "name": "John Doe",
        "email": "john@example.com",
        "address": "123 Main St",
        "city": "Springfield",
        "zip": "12345"
    }
    order = create_order(user)
    assert order.user_id == 123

def test_update_profile():
    user = {
        "id": 123,
        "name": "John Doe",
        "email": "john@example.com",
        "address": "456 Oak Ave",  # Different address!
        "city": "Springfield",
        "zip": "12345"
    }
    update_profile(user)

# ✅ GOOD - Centralized test data with builder
class UserBuilder:
    def __init__(self):
        self._id = 123
        self._name = "John Doe"
        self._email = "john@example.com"
        self._address = "123 Main St"
        self._city = "Springfield"
        self._zip = "12345"

    def with_id(self, id: int):
        self._id = id
        return self

    def with_address(self, address: str):
        self._address = address
        return self

    def build(self) -> dict:
        return {
            "id": self._id,
            "name": self._name,
            "email": self._email,
            "address": self._address,
            "city": self._city,
            "zip": self._zip
        }

def test_create_order():
    user = UserBuilder().build()
    order = create_order(user)
    assert order.user_id == 123

def test_update_profile():
    user = UserBuilder().with_address("456 Oak Ave").build()
    update_profile(user)

Detection Patterns:

# HIGH severity
- 3+ tests with identical structure (parameterization opportunity)
- Tests with >5 mock assertions on internal methods
- Tests that break when refactoring without behavior change

# MEDIUM severity
- Hardcoded test data repeated across tests
- Tests coupled to implementation details
- Tests with magic numbers/strings

# LOW severity
- Tests that could be more readable
- Opportunities for test utilities

# Detection Commands:
# Find similar test names (parameterization candidates)
grep -r "def test_" tests/ | awk -F'_' '{print $1"_"$2"_"$3}' | sort | uniq -c | sort -rn

# Find tests with many mocks (brittleness)
grep -r "assert_called" tests/ | cut -d: -f1 | uniq -c | sort -rn

# Find hardcoded test data
grep -r "= {$" tests/ -A10 | grep "\"id\":\|\"name\":\|\"email\":"

Anti-Patterns to Detect

1. Multiple Unrelated Assertions

File: bot/tests/test_agent_client.py

# ❌ BAD
def test_prepare_context_keeps_serializable(self):
    result = client._prepare_context(context)
    assert result["user_id"] == "U123"
    assert result["channel"] == "C456"
    assert result["thread_ts"] == "1234.5678"
    assert result["is_dm"] is False
    assert result["count"] == 42  # 5+ assertions testing different things

Detection:

• Count assertions per test
• Flag if > 3 assertions testing different attributes
• Suggest splitting into multiple tests

2. Over-Mocking Internal Logic

# ❌ BAD - Mocking internal graph execution
with patch.object(agent, "graph") as mock_graph:
    mock_graph.ainvoke = AsyncMock(return_value=mock_state)

Detection:

• Look for patch.object on methods defined in same module
• Flag mocking of private methods
• Suggest integration test instead

3. Incomplete Mock Setup

# ❌ BAD - llm_service used but not defined
context = {
    "slack_service": slack_service,
    "llm_service": llm_service,  # ← Not mocked yet!
}

Detection:

• Check for undefined variables in test setup
• Look for MagicMock/AsyncMock without proper configuration
• Flag missing return_value or side_effect

4. Repetitive Test Data Creation

# ❌ BAD - Same setup repeated in 5 tests
def test_user_with_email():
    user = User(
        name="John",
        email="john@example.com",
        is_active=True,
        role="user",
        created_at=datetime.now()
    )
    assert user.email == "john@example.com"

def test_user_with_admin_role():
    user = User(
        name="Admin",
        email="admin@example.com",
        is_active=True,
        role="admin",  # Only difference
        created_at=datetime.now()
    )
    assert user.role == "admin"

Detection:

• Find identical object creation across tests
• Suggest builder or factory
• Generate skeleton code

5. Unclear Test Names

# ❌ BAD
def test_1():
def test_user_test():
def test_something():

Detection:

• Names < 3 words
• Names containing "test" redundantly
• Generic names (test_1, test_2)

Audit Workflow

Phase 1: Discovery

1. Find all test files across services:

   • bot/tests/test_*.py
   • tasks/tests/test_*.py
   • control_plane/tests/test_*.py
   • agent-service/tests/test_*.py

2. Find all factory/builder files:

   • */tests/utils/mocks/*_factory.py
   • */tests/utils/builders/*_builder.py
   • */tests/factories.py
   • */tests/conftest.py

Phase 2: Analysis

For each test file:

1. File Naming - Check against naming convention
2. Test Methods - Parse and analyze each test:
   • Name convention
   • Docstring presence
   • 3As structure
   • Assertion count
   • Mock usage
   • Factory/builder usage
3. Anti-Patterns - Flag violations
4. Opportunities - Identify missing builders/factories

Phase 3: Report Generation

Create structured report:

# Test Audit Report

## Summary
- Total test files: 150
- Total tests: 2,500
- Factories found: 63
- Builders found: 12

## Violations by Severity

### Critical (Fix Now)
1. test_file.py:line - Multiple unrelated assertions (N assertions)
2. test_file.py:line - Over-mocking internal logic
3. test_file.py:line - Incomplete mock setup

### Warning (Fix Soon)
1. test_file.py:line - Unclear test name "test_1"
2. test_file.py:line - Missing 3As structure
3. test_file.py:line - Repetitive object creation (suggest factory)

### Suggestion (Consider)
1. test_file.py - Could benefit from builder pattern
2. test_file.py - Consider splitting large test file

## Priority Mapping for Action Planning

**IMPORTANT: Warnings are P1 priority issues and must be treated with the same urgency as Critical violations.**

### P1 - High Priority (Fix ASAP)
- **Critical violations** - Tests that don't work correctly, brittle tests, false positives/negatives
- **Warning violations** - Test quality issues that impact maintainability, readability, and developer productivity
- **Impact:** Blocks effective testing, slows development workflow, reduces confidence in test suite
- **Timeline:** Address immediately in current sprint/iteration

**Examples:**
- Multiple unrelated assertions (tests break for wrong reasons)
- Over-mocking internal logic (tests too coupled to implementation)
- Unclear test names (team can't understand what's being tested)
- Missing 3As structure (hard to debug failing tests)
- Repetitive setup code (slows test writing, inconsistent)
- Incomplete mock setup (flaky tests, false passes)

### P2 - Medium Priority (Fix When Convenient)
- **Suggestion violations** - Test improvements, consistency, patterns
- **Impact:** Minor productivity improvements, code organization
- **Timeline:** Address in next sprint or refactoring cycle

**Examples:**
- Could benefit from builder pattern (nice-to-have)
- Consider splitting large test file (organizational)
- Test file naming improvements (consistency)

### P3 - Low Priority (Optional)
- **Polish items** - Style preferences, minor optimizations
- **Impact:** Minimal
- **Timeline:** Address during major test refactoring or if time permits

## Opportunities for Improvement

### Missing Factories
1. EntityFactory - Used in N tests with identical setup
2. ModelFactory - Repeated in N tests

### Missing Builders
1. RequestBuilder - Complex request setup in N tests
2. ContextBuilder - Context dict creation in N tests

## Detailed Findings

[File-by-file breakdown with line numbers and examples]

Phase 4: Recommendations

For each issue:

1. Explain the problem with code example
2. Show the better approach from codebase standards
3. Provide refactoring code if applicable
4. Estimate impact (high/medium/low)

Phase 5: Code Generation (Optional)

Generate skeleton code for:

• Missing factories
• Missing builders
• Test splits (for multiple assertion violations)

Agent Execution Commands

Full Audit

Run comprehensive test audit across all services.
Analyze: naming, structure, factories, builders, anti-patterns.
Generate detailed report with severity levels and recommendations.

Quick Audit (Specific Service)

Run test audit on [bot|tasks|control_plane|agent-service] tests only.
Focus on critical violations and missing patterns.

Anti-Pattern Scan

Scan for anti-patterns only:
- Multiple assertions
- Over-mocking
- Incomplete mocks
- Unclear names
Flag with severity and line numbers.

Factory/Builder Audit

Identify opportunities for factories and builders:
- Find repetitive object creation
- Suggest patterns based on existing infrastructure
- Generate skeleton code for new factories/builders

3As Structure Check

Analyze test structure for Arrange/Act/Assert clarity:
- Check separation of phases
- Flag mixed concerns
- Suggest restructuring

Output Format

Console Summary

Test Audit Complete!

Analyzed: 150 files, 2,500 tests
Found: 63 factories, 12 builders

Violations:
  Critical: 12
  Warning: 45
  Suggestion: 89

Top Issues:
1. Multiple assertions (45 tests)
2. Missing 3As structure (89 tests)
3. Repetitive setup (need factories: 23 patterns)

See detailed report: .claude/audit-reports/test-audit-YYYY-MM-DD.md

Detailed Markdown Report

Save to .claude/audit-reports/test-audit-YYYY-MM-DD.md

JSON Export (Optional)

For programmatic analysis:

{
  "timestamp": "2025-12-03T20:00:00Z",
  "summary": {
    "total_files": 150,
    "total_tests": 2500,
    "factories": 63,
    "builders": 12
  },
  "violations": [
    {
      "file": "test_file.py",
      "line": 123,
      "severity": "critical",
      "type": "multiple_assertions",
      "count": 8,
      "suggestion": "Split into separate tests"
    }
  ]
}

Success Criteria

After running the agent and applying recommendations:

• ✅ All tests follow naming convention
• ✅ No critical violations
• ✅ < 5% warning violations
• ✅ Factories exist for common patterns
• ✅ Builders exist for complex data
• ✅ Clear 3As structure in 90%+ tests
• ✅ Single responsibility per test

Example Usage

User runs:

/audit-tests

Agent executes:

1. Scans all test directories
2. Analyzes 150 files, 2,500 tests
3. Identifies 12 critical violations
4. Generates detailed report
5. Suggests 3 new factories, 2 builders
6. Provides refactoring code examples

User reviews report and decides:

• Fix critical violations now
• Schedule warning fixes for next sprint
• Consider suggestions for future improvement

Integration with Development Workflow

When to Run

• Before code review
• After adding new tests
• Weekly as part of code quality checks
• Before major releases

Continuous Improvement

• Track violation trends over time
• Celebrate improvements
• Share best examples team-wide
• Update standards based on new patterns

Implementation Notes

Tools Needed:

• AST parsing (Python ast module)
• Pattern matching (regex)
• File system operations
• Report generation (Markdown, JSON)

Complexity:

• High - Requires code analysis
• Medium - Pattern detection
• Low - Report generation

Maintainability:

• Standards documented in this file
• Examples from real codebase
• Easy to update as patterns evolve