AI Training & Evaluation
I design the prompts and golden-truth rubrics that reveal exactly where advanced AI models fail. Every evaluation turns a hidden weakness into precise data that makes the next model smarter.
Representative image only. The original interfaces, data and materials from this engagement are confidential and protected by non-disclosure agreements.
Project Snowman
Led a team of AI evaluators in developing complex prompts and golden-truth scoring rubrics to test advanced language models across reasoning, specialized knowledge, instruction-following, and writing style. My role combined hands-on model evaluation with team mentorship, quality assurance, and the development of project guidelines to strengthen evaluation standards across the team.
This project focused on identifying performance gaps in advanced LLMs through structured prompt engineering and model evaluation. We tested four core capabilities: reasoning, specialized knowledge, instruction-following, and writing style, with an emphasis on how models handled complex questions involving multiple requirements.
The objective was to design prompts that produced meaningful model failures, then build scoring frameworks that could identify and measure those weaknesses. Each evaluation contributed structured data intended to support further model training and improvement.
Prompt Engineering & Model Testing
Engineered layered prompts that tested LLMs across reasoning, specialized knowledge, instruction-following, and writing style. Each task combined realistic scenarios with carefully chosen constraints to expose model weaknesses without introducing ambiguity or artificial complexity.
Calibrated prompt difficulty to produce meaningful failures, distinguishing genuine gaps in model performance from minor errors or flaws in the test itself.
Golden-Truth Rubrics & Evaluation
Developed scoring rubrics grounded in golden-truth answers, breaking complex prompt requirements into distinct, objective criteria that could be graded without guesswork. Each rubric had to meet several standards: self-containment, so evaluators had the information needed to score each criterion; clarity, to eliminate ambiguous expectations; and atomicity, to separate requirements that measured different capabilities.
I also removed redundancies that could penalize the same failure twice, checked logical dependencies between criteria, and verified factual accuracy and prompt–rubric alignment. Classified criteria across knowledge, reasoning, writing style, and final-answer requirements, while distinguishing major from minor scoring weights. These standards helped ensure that each score reflected a specific model strength or weakness rather than an inconsistency in the rubric.
Edge Cases & Scoring Judgment
Navigated complex edge cases where the distinction between right and wrong was less clear, particularly in niche knowledge and reasoning tasks. To limit subjective scoring, we favoured questions with five or fewer defensible answers or answers supported by broad expert consensus.
Major versus minor criteria presented another challenge: deciding when an implied requirement was essential to answering the prompt or merely a useful addition. For example, a recipe might not ask for cooking instructions, but most experts would consider them essential to a complete answer. These distinctions shaped how we weighted criteria and judged model failures.
Reasoning required its own scoring boundaries. Rather than judging how strong or sophisticated an explanation sounded, we assessed whether the requested reasoning was present. A weak explanation could still pass, while reasoning built on false facts or flawed logic could fail. This helped keep evaluations grounded in clear, defensible standards rather than individual reviewer preferences.
Team Leadership & Quality Assurance
Reviewed evaluators’ prompts, rubrics, and scoring decisions to identify errors that could compromise evaluation accuracy. Common issues included ambiguous criteria, missing golden-truth details, redundant scoring, inaccurate dependencies, insufficient model failure rates, and prompts that were either too broad or overly constrained.
Provided detailed feedback that explained the source of each problem and how to correct it. I also mentored writers through revisions, tracked their progress, and helped them develop stronger judgment in prompt engineering, rubric construction, and model assessment.
Project Guidelines & Methodology Development
Spearheaded the development of our Prompt Playbook, a comprehensive guide covering prompt-engineering techniques, rubric-design standards, scoring principles, and examples of common evaluation errors. Drawing on recurring issues from quality reviews, I helped translate complex requirements and edge cases into practical guidance for the team.
The playbook served as a shared reference for improving prompt quality, building more precise rubrics, and maintaining consistent evaluation standards.
The project produced structured evaluation data identifying weaknesses in model reasoning, knowledge, instruction-following, and writing style. By linking each failure to a defined scoring criterion, we could isolate performance gaps and provide more precise feedback for subsequent model training.
My work as Team Lead also strengthened the evaluation process through quality reviews, mentorship, and the development of the Prompt Playbook. These contributions established clearer standards for prompt engineering, rubric design, and scoring consistency, supporting more reliable training data and a more effective team evaluation process.
Representative image only. The original interfaces, data and materials from this engagement are confidential and protected by non-disclosure agreements.
Project Echo
Tested and evaluated a speech-to-speech AI model through natural, multi-turn conversations involving personal advice, emotional support, everyday assistance, and role-play. My work combined conversational testing, audio and transcript analysis, and detailed rubric development, with particular emphasis on the subtle judgments required to distinguish genuine model failures from subjective differences in tone, delivery, and conversational style.
The project examined how well a voice-based AI could understand and respond to people in real conversations, where meaning extends far beyond the words being spoken. A model might understand a request perfectly yet miss the hesitation behind it, offer sound advice in a tone that feels completely out of place, or lose track of an earlier instruction as the conversation unfolds.
Our objective was to uncover these failures through realistic spoken interactions, then translate them into precise evaluation data covering audio understanding, audio generation, lexical content, and conversational performance.
Multi-Turn Voice Testing & Critical Failure Identification
Recorded natural, multi-turn conversations designed to expose clear, meaningful model failures without resorting to trick questions, false information, or artificial scenarios. I tested how the model handled emotional cues, changing tones, layered instructions, conversational memory, and role-play consistency, ending each interaction at the point where a critical failure emerged.
I also refined transcripts to match the recordings exactly, preserving filler words, hesitations, and background sounds that could influence how the model interpreted the exchange.
Multidimensional Rubric Design & Evaluation
Developed detailed, turn-by-turn rubrics using 18 evaluation tags across four dimensions: audio understanding, audio generation, lexical content, and conversation. Together, these measured everything from speech comprehension, emotional cues, and acoustic awareness to voice naturalness, factual accuracy, reasoning, instruction-following, and context retention.
Each criterion had to be self-contained, unambiguous, atomic, and independently gradable, with no redundant requirements or vague expectations. I assigned major or minor weights, distinguished objective from subjective criteria, and calibrated responses using No Issues, Minor Issues, and Major Issues ratings. Rubrics also had to meet strict coverage requirements across turns, ensuring that evaluations captured the full range of model capabilities rather than concentrating on a few obvious weaknesses.
Nuanced Judgment & Evaluation Calibration
Some of the most demanding work involved failures that were easy to sense but difficult to score. A model might give accurate advice in an emotionally mismatched voice, follow the substance of an instruction while missing its stylistic constraints, or maintain factual accuracy while slipping out of character. These cases required careful judgment about severity, subjectivity, and the relationship between spoken delivery and verbal content.
I often revisited individual scoring decisions, refined criteria to remove competing interpretations, and brought particularly subtle edge cases into team discussions to establish a defensible, consistent assessment. The challenge was to preserve the nuance of human communication while making each judgment clear enough for another evaluator to reproduce.
Produced detailed, turn-level evaluation data that captured weaknesses across the model’s ability to hear, interpret, speak, reason, and sustain natural conversation. By connecting critical failures to specific audio, language, and conversational criteria, the work provided more precise signals for subsequent voice-model training and refinement.
The project also demonstrated the value of rigorous human judgment in AI evaluation. Through careful rubric construction, repeated calibration, and collaborative discussion of difficult edge cases, nuanced conversational failures could be translated into clearer, more consistent assessments without reducing human communication to a simple checklist.
Representative image only. The original interfaces, data and materials from this engagement are confidential and protected by non-disclosure agreements.
Project AgentSet
Designed and validated multi-step AI agent workflows in a sandboxed email management environment, combining prompt engineering, hands-on task execution, JSON-based verification, and natural-language scoring rubrics. My work focused on testing whether AI assistants could perform actions with accuracy, maintain the correct system state, and use contextual reasoning to determine appropriate next steps without explicit instructions.
This project focused on training and evaluating an AI assistant designed to manage emails and carry out user requests within a Gmail-like interface. Beyond understanding instructions, the model needed to perform the correct actions, handle information across multiple steps, and produce results that could be verified against the application’s underlying data.
A key objective was to test more than mechanical task completion. We wanted to evaluate whether an assistant could interpret the context of an email, recognize what a user needed, and make sound decisions when the next action was implied rather than stated.
Scenario Design & Prompt Engineering
Developed realistic email management scenarios that required the AI assistant to carry out sequences of actions within a sandbox environment. Each prompt established a user request, relevant email context, and an expected outcome, creating a controlled test of the assistant’s ability to manage tasks across multiple steps.
Designed scenarios with varying levels of complexity, from direct instructions to requests that required interpreting message content, connecting information, and deciding how to proceed.
Sandbox Execution & JSON Verification
Performed the requested actions within the sandbox to establish the correct workflow and expected system state. Used JSON-based checks to verify that each action had produced the intended change, rather than relying on what appeared to have happened in the interface.
This included validating individual steps, their sequence, and the resulting data against the expected outcome. The process helped distinguish an assistant that claimed to complete a task from one that had executed it.
Critical Thinking & Contextual Inference
Extended scenarios beyond step-by-step instructions by creating opportunities for the assistant to draw reasonable conclusions from email content. For example, rather than telling the model every action to take, a prompt could require it to recognize that an email contained a pending request, identify the relevant information, and determine the next logical step.
The challenge was to test whether the assistant understood the task well enough to act on context, rather than follow a prescribed sequence. These scenarios introduced a reasoning component to workflow validation, testing the model’s ability to interpret intent, connect details, and distinguish appropriate actions from unsupported assumptions.
Natural-Language Rubrics & Evaluation
Translated each verified workflow into a structured scoring rubric, using clear, human-readable criteria to describe the actions required for successful completion. Each criterion corresponded to a verifiable step or system-state change, connecting the expected behaviour to its JSON-based evidence.
I used the validated workflow and golden reference state to define what success looked like, while separating individual requirements so that errors could be identified without obscuring which step had failed. The rubrics also accounted for tasks involving contextual inference, distinguishing actions supported by the user’s request and email content from actions that exceeded the available instructions.
Produced realistic AI agent test scenarios, verified reference workflows, and structured evaluation rubrics that connected natural-language instructions to observable system behaviour. By combining sandbox execution with JSON-based validation, the project supported more reliable assessment of tool use, action accuracy, workflow completion, and system-state consistency.
My emphasis on contextual inference also expanded the scope of evaluation beyond mechanical task execution, introducing scenarios that tested whether an AI assistant could use reasoning and judgment to determine the next appropriate action. This helped capture an important distinction in agent performance: following instructions versus understanding what needs to be done.
Representative image only. The original interfaces, data and materials from this engagement are confidential and protected by non-disclosure agreements.
Project Mist
Engineered complex research prompts designed to expose gaps in LLM information retrieval, evidence synthesis, and multi-step reasoning. My work involved extensive independent research, repeated model testing, source verification, and detailed reasoning reports to establish a single, defensible answer for each question.
This project tested whether advanced LLMs could answer seemingly simple questions that required extensive research to solve. Each question involved several hard-to-find facts scattered across different sources, forcing the model to locate the right information, connect the evidence, and follow a logical chain toward one correct answer.
Questions covered five categories: Entertainment, Health, Shopping, Information Seeking, and Law. Each had to demand at least one hour of human research and produce five failed model attempts, without becoming so broad that multiple answers could qualify.
Research & Prompt Engineering
Designed layered questions that combined obscure facts, indirect clues, and interconnected conditions. The challenge was to make each question difficult enough to expose gaps in the model’s research and reasoning, while keeping its requirements precise enough to support only one answer.
Each question also had to pass a search difficulty check: the answer could not appear on the first page of Google. This prevented questions from qualifying on model failure alone when the information was too easy to retrieve.
Model Testing & Failure Analysis
Tested each question across five model attempts, with five failures required for the prompt to qualify. Reviewed responses to identify where models overlooked evidence, relied on incorrect assumptions, or failed to connect information needed to reach the answer.
Source Verification & Reasoning Chains
Conducted independent research using reliable, authoritative sources to establish the correct answer. Built a step-by-step reasoning chain supported by direct source quotations, showing how each fact contributed to the final conclusion.
The evidence had to do more than support a plausible answer. It needed to establish that every condition was satisfied and that no competing answer could meet the same requirements.
Answer Uniqueness & Time-Invariant Validation
Verified that each question had one definitive answer by investigating alternative possibilities and checking them against every clue. Another requirement was time invariance: the answer had to remain correct regardless of when the question was asked.
This meant avoiding facts that could change, such as current rankings, product availability, or evolving legal conditions, and grounding questions in fixed, verifiable information.
Research Reports & Documentation
Produced detailed reports documenting the correct answer, authoritative sources, supporting quotations, and the complete reasoning chain. Each report had to demonstrate why the answer was unique, why it would remain unchanged over time, and how the evidence supported each step of the conclusion.
The reports also documented model failures and validated that the question met the project’s research difficulty and search requirements.
Developed validated research challenges that exposed gaps in LLM information retrieval, evidence synthesis, and multi-step reasoning. The combination of five failed model attempts, authoritative source verification, and a single time-invariant answer ensured that each task tested a meaningful limitation rather than an ambiguous or poorly constructed question.
My research reports provided traceable evidence for every conclusion, allowing reviewers to verify the answer, follow the reasoning, and identify where the model’s approach fell short. Together, these evaluations produced structured data for testing and improving how AI models research complex questions and connect information across sources.
Representative image only. The original interfaces, data and materials from this engagement are confidential and protected by non-disclosure agreements.
Project Beacon
Engineered complex research prompts to test how advanced LLMs interpret information that has changed over time, distinguishing current facts from outdated knowledge and applying historical information to the correct time period. My work combined temporal prompt engineering, authoritative source research, model failure analysis, and rubric-based evaluation to assess factual accuracy, reasoning, information retrieval, and response quality.
Project Beacon focused on a specific weakness in AI research: the ability to retrieve information without recognizing whether it still applies. Models were challenged to reconcile facts across different time periods, account for changes in laws, regulations, policies, and other time-sensitive information, and determine which version was relevant to the user’s question.
The objective was to expose temporal reasoning failures, particularly when models relied on current information to answer historical questions, applied outdated rules to newer situations, or overlooked changes that affected the correct answer.
Temporal Prompt Engineering & Scenario Design
Developed layered questions that required models to research how facts, rules, and conditions had changed over time. Scenarios drew on historical legislation, education policies, taxation, institutional regulations, and other subjects where the correct answer depended on the date and circumstances involved.
Rather than asking models to recall isolated historical facts, I designed questions that required them to connect multiple conditions, identify which rules applied during a specific period, and reconcile conflicting information from different sources.
Research & Historical Source Verification
Conducted independent research using authoritative sources, including historical records, legislation, government publications, and institutional documentation. Verified when policies or regulations took effect, whether exceptions applied, and how subsequent changes affected earlier rules.
This required distinguishing the date a source was published from the period its information described, particularly when newer sources summarized or replaced older requirements.
Model Testing & Failure Analysis
Tested prompts to identify failures in temporal knowledge, information retrieval, and contextual reasoning. Reviewed model responses to determine whether errors came from missing historical facts, relying on outdated information, overlooking policy changes, or applying the wrong rules to a given time period.
Documented how the model likely reached an incorrect conclusion, identifying the missing evidence or faulty connections that prevented it from arriving at the correct answer.
Research Reports & Evidence Documentation
Produced detailed evaluation reports establishing the correct answer through verified sources and a documented reasoning chain. Each report explained the model’s failure, identified the information it had missed or misapplied, and provided supporting evidence for the correct conclusion.
The research also had to account for changes across time, showing why particular facts applied to the scenario while others did not.
Rubric Design & Response Evaluation
Developed structured scoring rubrics that translated each research task into clear, measurable evaluation criteria. Rubrics assessed source quality, factual retrieval, temporal accuracy, reasoning, and final-answer correctness, along with the overall helpfulness of the response, including whether the model provided relevant context or further assistance.
This allowed evaluations to distinguish between different types of failure. A model might retrieve credible sources but overlook a critical historical change, identify the correct facts but apply them to the wrong period, or reach the right answer without providing the reasoning or support needed to make it useful.
Produced research-backed evaluation tasks that exposed gaps in LLM temporal knowledge, historical fact retrieval, and multi-step reasoning. By testing how models reconciled information across changing laws, policies, and conditions, the project identified failures that conventional fact-based questions could overlook.
My evaluation reports documented the correct answer, supporting evidence, likely causes of model failure, and missing information, while scoring rubrics provided a consistent framework for assessing research quality, temporal reasoning, accuracy, and helpfulness.