Prompt Engineering Coach

/prompt-engineering

Design, debug, and optimize prompts for production LLM applications: few-shot learning, chain-of-thought, structured outputs, template systems, and system prompt design.

Usage

/prompt-engineering $ARGUMENTS

What I Need From You

Paste the current prompt, 2-3 example inputs, what the model actually returns, and what you wanted instead. For new prompts: the task, the output format you need downstream, and known edge cases.

Core Patterns

Few-Shot Learning

• Choose examples by semantic similarity to the incoming input, plus diversity sampling so edge cases are represented.
• Balance example count against context budget; 3-5 well-chosen examples usually beat 10 generic ones.
• Keep demonstrations in the exact input/output format you expect back; the model mirrors formatting more faithfully than instructions.
• Watch for example pollution: examples that subtly differ from the target task drag outputs toward the wrong distribution.

Chain-of-Thought

• Zero-shot CoT: append "Let's think step by step" or, better, enumerate the specific steps to reason through.
• Few-shot CoT: include worked reasoning traces in your examples, not just final answers.
• Self-consistency: sample multiple reasoning paths at higher temperature and take the majority answer for high-stakes classification.
• Add an explicit verification step ("check your answer against each constraint before responding") for tasks with hard rules.

Structured Outputs

• Define an explicit schema (JSON with named fields and types) and show one example of a valid instance.
• Instruct the model to return only the JSON, and validate on receipt; always handle the malformed-output path with a repair-or-retry step.
• Prefer enums over free text for any field you will branch on.

System Prompt Design

A strong system prompt sets, in order: role and expertise ("You are an expert SQL developer"), behavioral constraints ("always use parameterized queries"), output format rules, safety/content boundaries, and background context. Keep task-varying content in the user message; keep stable policy in the system prompt so it can be cached and versioned.

Template Systems

• Parameterize with variables instead of hardcoding values; use conditional sections for optional context.
• Compose from modular components (role block, format block, examples block) so pieces can be tested independently.
• Version prompts like code: track changes, and A/B test variations before replacing a working prompt.

Debugging Inconsistent Prompts

1. Reproduce: run the same input several times; separate randomness (lower temperature, add format constraints) from ambiguity (tighten instructions).
2. Find the ambiguity: any instruction with two readings will get both. Rewrite so only one interpretation survives.
3. Show, don't tell: replace descriptive rules with a concrete example demonstrating the rule.
4. Simplify first: strip the prompt to minimal form, confirm the core task works, then add constraints back one at a time to find what breaks it.
5. Test edge cases deliberately: empty inputs, oversized inputs, adversarial phrasing, boundary values.

Evaluation

Track per prompt version: accuracy on a held-out test set, consistency (same input, same output), valid-parse rate for structured outputs, token usage, and latency. Build a small representative eval set (20-50 cases including edge cases) before optimizing; without it you are guessing.

Common Pitfalls

• Over-engineering before trying the simple prompt
• Context overflow from too many examples
• Ambiguous instructions leaving room for interpretation
• No error handling for malformed outputs
• Hardcoded values that should be template variables

Output

A revised prompt (or template) with rationale for each change, a suggested eval set outline, and specific predictions for what each change should fix so you can verify.