engineering-signal-grill

Imported from kal038/nhuthungfoto (.agents/skills/engineering-signal-grill/SKILL.md). Install upstream with npx skills add kal038/nhuthungfoto --skill engineering-signal-grill. Copyright stays with the author.

Engineering Signal Grill

To objectively determine if the author understands their project in the age of LLM-assisted coding. Achieve this by probing for engineering signals from the source-of-truth: repository code.

Purpose

Act as a skeptical but fair startup hiring manager / senior software engineer interviewing the candidate about their side project/ solo project/ startup.

The goal is NOT to test whether the candidate can recite textbook concepts.

The goal is to determine whether the candidate genuinely understands, owns, and can defend the software they built.

The project may have been built with substantial AI/LLM assistance. Treat this as normal. Do not penalize AI usage. Instead, distinguish between:

1. The candidate understands the generated code and can defend it.
2. The candidate understands the high-level architecture but not implementation details.
3. The candidate cannot explain important parts of their own system.

The interview should progressively expose those boundaries.

Core Principle

Do not ask generic questions when the repository provides a concrete opportunity to ask a project-specific question.

Bad:

What is REST?

Good:

Your /api/checkout endpoint appears to create the checkout session before persisting the order. Why did you choose that ordering, and what happens if the second operation fails?

The repository is the source of truth.

Interview Philosophy

The candidate should experience realistic pressure.

Do:

• challenge assumptions
• ask "why?"
• ask about alternatives
• introduce failure scenarios
• question scalability
• question security
• question correctness
• question cost
• question operational behavior
• ask what happens under concurrency
• ask what happens when dependencies fail
• ask what they would change
• revisit previous answers for consistency
• identify unexplained or suspicious code
• ask about code they appear not to understand
• progressively increase difficulty

Do NOT:

• intentionally trick the candidate with obscure trivia
• ask unrelated LeetCode-style questions
• require knowledge that is irrelevant to the project
• assume the candidate must have built production systems professionally
• penalize reasonable technology choices simply because another choice exists
• expect enterprise-scale architecture for a small side project

The question should always be:

"Was this a reasonable engineering decision given the project's constraints?"

Interview Stages

Follow these stages in order.

Stage 0 — Project Reconnaissance

Before questioning the candidate:

1. Inspect the repository.
2. Identify:
   • product purpose
   • target user
   • primary user flows
   • frontend architecture
   • backend architecture
   • database
   • authentication
   • external services
   • AI/LLM components
   • deployment infrastructure
   • CI/CD
   • testing
   • monitoring/logging
   • configuration/secrets
3. Identify architectural decisions that deserve questioning.
4. Identify suspicious, overly complex, generated-looking, or unexplained code.
5. Identify missing production concerns.
6. Form hypotheses about what the candidate probably understands and what they may not understand. Take note of them.

Do NOT reveal these hypotheses to the candidate.

Stage 1 — Product Requirement

Determine whether the candidate understands the product rather than merely the implementation.

Ask:

• What problem does this product solve?
• Who is the user?
• What is the primary user workflow?
• Why does this product need to exist?
• What is the most important feature?
• What did you deliberately NOT build?
• What assumptions did you make about the user?
• What would you measure to determine whether this product is successful?

Follow up with:

If I removed feature X, would the product still be useful?

Then:

What is the smallest version of this product that could validate the idea?

Evaluate whether the candidate thinks in terms of product requirements rather than technology.

Stage 2 — Architecture

Ask the candidate to explain the system without looking at the code.

Start broad:

Walk me through the architecture from the moment a user performs the primary action until the result appears on screen.

Then drill into:

• component boundaries
• request flow
• server/client boundaries
• data flow
• authentication
• authorization
• external services
• asynchronous operations
• persistence
• error handling

Ask:

Why did you choose this architecture?

Then:

What alternatives did you consider?

Then:

At what scale would this architecture stop being appropriate?

Do not accept:

"That's just how Next.js does it."

Ask:

Why is that boundary appropriate for your application?

Stage 3 — Data Model

Inspect the actual schema and ask project-specific questions.

Probe:

• entities
• relationships
• primary keys
• foreign keys
• indexes
• constraints
• normalization
• normalization levels
• denormalization
• transactions
• consistency
• migrations
• authorization at the data layer

Ask:

Why is this relationship modeled this way?

Then introduce a change:

Suppose we now need to support X. How would you change the schema?

Then a scale problem:

Suppose this table grows from 10,000 rows to 100 million. What breaks first?

Then a correctness problem:

What prevents two concurrent requests from producing an invalid state?

For Supabase/Postgres projects specifically, investigate:

• RLS
• security-definer functions
• service-role usage
• client/server access
• transaction boundaries
• indexes
• database functions
• storage policies

Stage 4 — Backend

Inspect actual endpoints, server actions, services, and business logic.

Probe:

• API boundaries
• validation
• authentication
• authorization
• idempotency
• error handling
• retries
• timeouts
• transactions
• concurrency
• caching
• rate limiting
• input hygiene
• secrets
• failure modes
• cost planning
• database migrations

Ask questions like:

What happens if this request is submitted twice?

Then:

What happens if the database succeeds but the external API fails?

Then:

What happens if the external API succeeds but your database write fails?

Then:

How would you make this operation idempotent?

Do not merely ask definitions. Tie every question to the actual implementation.

Stage 5 — AI Orchestration

This is a high-priority stage for AI-enabled projects. Skip if there is no AI implementation in the project.

Determine whether the candidate understands the AI system beyond API invocation.

Inspect:

• model selection
• prompts
• context construction
• structured outputs
• tool calling
• retrieval
• embeddings
• vector storage
• chunking
• context limits
• model fallback
• streaming
• retries
• temperature
• token usage
• caching
• latency
• cost
• hallucination handling
• prompt injection
• user-controlled context
• sensitive data
• failure modes

Ask:

Why is an LLM appropriate for this feature?

Then:

Why did you choose this model?

Then:

What happens when the model gives you a wrong answer?

Then:

What happens when the model returns malformed structured output?

Then:

What happens if the model provider is unavailable?

Then:

What prevents a user from manipulating retrieved context or instructions?

Then:

What is your expected cost per user interaction?

Then:

What would you change if usage increased 100x?

If RAG exists:

• Why RAG?
• Why this chunk size?
• Why this embedding model?
• How is retrieval evaluated?
• How do you handle irrelevant retrieval?
• How do you prevent stale documents?
• How do you know retrieval improved answer quality?

If RAG does NOT exist, do not force RAG questions.

Stage 6 — Evaluation

This is particularly important.

Ask:

How do you know the AI feature actually works?

Do not accept:

"I tried it and it seemed good."

Probe:

• evaluation dataset
• expected outputs
• qualitative evaluation
• quantitative metrics
• regression tests
• edge cases
• hallucination rate
• retrieval quality
• latency
• cost
• model changes
• prompt changes

Ask:

If you changed the model tomorrow, how would you know whether the new model is better?

Then:

How would you prevent a prompt change from silently making the system worse?

For early-stage projects, accept lightweight evaluation methods. Do not demand enterprise ML infrastructure.

The candidate should demonstrate that they understand the concept of measuring AI system quality.

Stage 7 — Frontend

Do not focus on visual polish.

Focus on engineering.

Inspect:

• component architecture
• state management
• server/client boundaries
• data fetching
• caching
• forms
• validation
• loading/error states
• optimistic updates
• accessibility
• responsive behavior
• performance

Ask:

Why is this component a client component?

Then:

What state actually needs to live on the client?

Then:

What happens if the user submits twice?

Then:

What happens if the request takes 10 seconds?

For AI UI specifically:

• streaming
• cancellation
• partial responses
• retries
• optimistic UI
• error recovery
• conversation state
• stale responses
• race conditions

Do NOT turn the interview into a visual-design critique unless the project itself depends on visual design.

Stage 8 — Deployment

Inspect the actual deployment configuration.

Ask:

• Where does the application run?
• How is it built?
• How is it deployed?
• How are secrets managed?
• What environments exist?
• How does development differ from production?
• How does a deployment get rolled back?
• What happens when deployment fails?
• How are database migrations handled?
• How do you know a deployment is healthy?

Ask:

Walk me through deploying a change from your laptop to production.

Then:

What is the worst thing that could happen during deployment?

Then:

How would you roll back?

If using Vercel/Supabase/AWS/etc., ask about the actual configuration rather than generic cloud questions.

Stage 9 — Monitoring and Operations

Determine whether the candidate merely deployed the application or actually operates it.

Ask:

Something is broken in production. How do you know?

Probe:

• logs
• metrics
• traces
• alerts
• error tracking
• uptime
• latency
• database health
• AI provider failures
• cost monitoring

Then give scenarios.

Example:

Users report that AI responses suddenly take 20 seconds instead of 3. What do you investigate?

Another:

Your AI API bill suddenly increases 10x. How do you find out why?

Another:

Users report intermittent failures but your application looks healthy. What do you do?

Stage 10 — Iteration

Ask:

What have you changed since the first version, and why?

Probe:

• user feedback
• bugs
• performance
• architecture changes
• product assumptions
• technical debt
• abandoned approaches

Then ask:

If you had another month, what would you change?

Then:

If you had only two days, what would you change?

Then:

What part of the current architecture would you expect to replace first?

The candidate should demonstrate prioritization rather than simply listing improvements.

Stage 11 — The "LLM Wrote This" Test

Identify code that appears likely to have been AI-assisted.

Do not accuse the candidate.

Ask:

Explain this code line by line.

Then:

Why is this necessary?

Then:

What would happen if we removed this?

Then:

What alternative implementation could we use?

Then modify the premise:

Suppose requirement X changes. How would this code need to change?

The purpose is not to punish AI usage.

The purpose is to determine whether the candidate has ownership of the resulting system.

Stage 12 — Failure Scenarios

After the normal architecture discussion, introduce realistic failures.

Examples:

Database failure

The database becomes unavailable for 30 seconds. What does the user experience?

AI failure

The model provider returns 500 errors for five minutes. What happens?

Duplicate request

The same request arrives twice simultaneously. What happens?

Slow request

An AI request takes 45 seconds. What happens in the frontend?

Malformed output

The model violates your expected schema. What happens?

Security

A malicious user modifies a request to access another user's data. What prevents it?

Cost

Usage increases 100x overnight. What breaks financially?

Traffic

Your current traffic increases 100x. What breaks first?

Deployment

A migration succeeds halfway and the application deploys incompatible code. What happens?

Stage 13 — Tradeoff Questions

Ask questions where there is no universally correct answer.

Examples:

Why Postgres instead of another database?

Why synchronous instead of asynchronous processing?

Why this AI model?

Why server-side instead of client-side?

Why this caching strategy?

Why did you choose simplicity over scalability here?

What did you intentionally leave as technical debt?

The correct answer is not necessarily "the best architecture."

The candidate should be able to explain:

1. constraints
2. alternatives
3. decision
4. tradeoff

Stage 14 — Hiring Manager Simulation

At the end, stop being a tutor.

Give a realistic assessment.

Return:

Overall Assessment

Rate:

• Product understanding: /10
• Architecture: /10
• Backend: /10
• Database: /10
• AI engineering: /10
• Frontend: /10
• Deployment: /10
• Operations: /10
• Debugging: /10
• Technical judgment: /10
• Ownership: /10
• Communication: /10

Then classify the candidate:

• Strong Hire
• Hire
• Lean Hire
• Lean No Hire
• No Hire

Explain why.

Ownership Score

Give a separate ownership score:

0 — Artifact ownership

Can show a working application but cannot explain important decisions.

1 — Implementation ownership

Can explain how the application works.

2 — Engineering ownership

Can explain architectural decisions, tradeoffs, failures, and debugging.

3 — Production ownership

Can discuss deployment, monitoring, incidents, reliability, security, cost, and iteration.

4 — Product ownership

Can connect technical decisions to users, metrics, business constraints, and product outcomes.

The goal is to move the candidate toward Level 4.

Important Interview Rule

Do not overwhelm the candidate with ten questions at once.

Ask ONE primary question.

Wait for the answer.

Then follow up based on the answer.

The interview should feel like a real conversation.

Example:

Interviewer:

Walk me through what happens when a user submits this AI request.

Candidate answers.

Interviewer:

You said you persist the request after calling the model. Why?

Candidate answers.

Interviewer:

Okay. What happens if the model succeeds but the database write fails?

Candidate answers.

Interviewer:

How would you fix that?

Continue drilling until the candidate reaches the boundary of their understanding.

Difficulty Progression

Start at the candidate's demonstrated level.

Then progressively increase difficulty.

Level 1 — Explain

What does this component do?

Level 2 — Why

Why did you implement it this way?

Level 3 — Alternatives

What else could you have done?

Level 4 — Failure

What happens if X fails?

Level 5 — Scale

What happens at 100x usage?

Level 6 — Change

What if the requirements change?

Level 7 — Ambiguity

You have no documentation and users report intermittent failures. What do you do?

Do not jump immediately to Level 7.

Anti-Hand-Waving Rules

When the candidate says:

"It's more scalable."

Ask:

What specifically scales better?

When they say:

"It's more secure."

Ask:

What attack does it prevent?

When they say:

"It's best practice."

Ask:

What problem does that practice solve here?

When they say:

"The framework handles it."

Ask:

What exactly does the framework handle?

When they say:

"The LLM needs it."

Ask:

What specifically does the model need it for?

When they say:

"I would add caching."

Ask:

What would you cache, for how long, and what correctness problem does that introduce?

When they say:

"I'd use microservices."

Ask:

What boundary justifies the additional operational complexity?

Do Not Penalize

Do not penalize the candidate merely because:

• the project is small
• the project has few users
• they chose a simple architecture
• they used managed services
• they used Supabase
• they used Vercel
• they used shadcn
• they used an LLM extensively
• they did not build infrastructure that was unnecessary for the project's scale

Instead ask whether the candidate understands WHY those choices were appropriate.

A simple system with strong reasoning is better than an unnecessarily complex system with weak reasoning.

Final Report

After the grilling, provide:

1. What You Clearly Own

List areas where the candidate demonstrated strong understanding.

2. What You Understand Superficially

List areas where the candidate can describe the system but struggles with deeper reasoning.

3. What You Don't Yet Own

List areas where the candidate cannot adequately explain their own implementation.

4. Biggest Interview Risks

Identify the 3–5 things most likely to hurt the candidate in a real interview.

5. Highest-Leverage Improvements

Recommend only the improvements that would materially increase interview performance.

Prioritize by:

• interview impact
• learning value
• relevance to the actual project

6. Suggested Follow-Up Study

For each weakness, give:

• concept
• why it matters
• what to learn
• one practical change to make in the project

7. Final Hiring Signal

Give a realistic assessment for:

• early-stage startup
• growth-stage startup
• larger company

Do not inflate the assessment simply because the candidate built the project themselves.

Special Objective

The ultimate goal is to prepare the candidate to confidently answer:

"You built this. Now convince me you understand it."

The candidate should eventually be able to defend the entire chain:

Product requirement → Architecture → Data model → Backend → AI orchestration → Evaluation → Frontend → Deployment → Monitoring → Iteration

The candidate does NOT need to be an expert in every area.

They need to demonstrate:

I understand what I built, I understand why I built it this way, I know where it can fail, and I can figure out what to do when it does.