Experiments Overview
What We Tested
Same task, same IDE, same model — run twice. Once with the agent using only web search, once with Context Plugins providing live API source, models, and controllers.
Legacy · ASP.NET
nopCommerce
Mature open-source e-commerce platform. Plugin architecture, decade of production use, strict backward-compatibility expectations.
Modern · .NET 9
eShop
Reference architecture from the .NET team. Services-based design, .NET Aspire orchestration, clean separation of concerns.
Code Quality
What Improved Beyond the Numbers
Average code quality scores across all 4 experiments, rated 1–5.
Architecture
2.5→5.0
Clean layering, hexagonal / adapter patterns preserved
Modularity
2.5→5.0
Adapter pattern preserved; UI/business logic kept separate
Error Handling
3.3→4.5
SDK-aware exceptions, structured logs, validation at boundaries
Testability
3.3→4.5
Abstract sealed SDK clients behind interfaces for testable code
Design Patterns
3.0→5.0
Adapter + Builder patterns, typed results, consistent constructors
Readability
2.8→4.5
Focused functions, clear SDK types, major LOC reduction
Security
4.0→4.3
No URL injection, proper credential validation at startup
Configuration
4.0→4.3
OAuth via SDK, zero hardcoded secrets; detect token leaks in logs
EXP 01 nopCommerce · Migration
Migrating HTTP Calls to PayPal Server SDK
Replace hand-rolled HttpClient calls with the official SDK without breaking OAuth, order creation, capture, refund, or void flows.
Prompt Used
Plan and execute a migration to the PayPal Server SDK v2.0.0 via NuGet package in the PayPalCommerce plugin.
✕ Agent Only
- Hallucinated that
OAuthAuthorizationControllerdidn't exist in the SDK - 29+ compile errors from guessed model shapes, enums, namespaces
- Markup sanitization corrupted generics —
List<T>becameList - Selected deprecated SDK despite explicit v2.0.0 requirement
- Logging silently disappeared; entire branch had to be reset
✓ Agent + Context Plugins
- Immediately identified SDK-supported vs unsupported flows
- Proposed hybrid migration strategy from the first prompt
- Isolated SDK usage cleanly in HTTP client layer
- Preserved entire architecture — only 2 files changed
- Zero hallucinations, clean compilation from start
Results at a Glance — Agent Only vs Agent + Context Plugins
Prompt Iterations
Total prompts required for a production-ready integration — lower is better
Manual Fixes
Human interventions needed to correct agent output — lower is better
Errors Encountered
Total compile and runtime errors surfaced during the session — lower is better
Tokens Consumed
Total tokens used across the full session — lower is better
EXP 02 nopCommerce · New Integration
Building PayPal Checkout from Scratch
With the existing PayPal plugin removed, integrate PayPal as a payment method where the buyer approves first, and then the payment is captured on order confirmation.
Prompt Used
I need a PayPal payment option added to the checkout page as a plugin that can be enabled/disabled like the other plugins in the solution.
Architecture: Implement a PayPal payment plugin that matches the repository's existing plugin architecture and avoids breaking changes.
Flow
1- The shopper can choose PayPal as a payment method at checkout.
2- When selected, create an order via PayPal's Orders API and send the shopper to approve the order.
3- After buyer approval, capture the order to complete the payment.
1- The shopper can choose PayPal as a payment method at checkout.
2- When selected, create an order via PayPal's Orders API and send the shopper to approve the order.
3- After buyer approval, capture the order to complete the payment.
✕ Agent Only
- Critical: URL injection allowed order capture on ID mismatch
- Payments processed for explicitly cancelled orders
- Missing configuration key management and DI setup
- Order capture logic completely absent — no payments completed
- Multiple prompts required to patch security holes
✓ Agent + Context Plugins
- Verified its knowledge proactively with tool calls
- Zero security vulnerabilities — proper order ID validation
- Correct DI registration and configuration management
- All checkout workflows completed in a single pass
- Builds and compiles correctly from first attempt
Results at a Glance — Agent Only vs Agent + Context Plugins
Prompt Iterations
Total prompts required for a production-ready integration — lower is better
Manual Fixes
Human interventions needed to correct agent output — lower is better
Errors Encountered
Total compile and runtime errors surfaced during the session — lower is better
Tokens Consumed
Total tokens used across the full session — lower is better
EXP 03 eShop · Migration
Migrating HTTP Baseline to PayPal SDK
Migrate a direct HTTP implementation to the official PayPal Server SDK in a modern .NET 9 service-based application.
Prompt Used
I need you to migrate all the PayPal specific infrastructure code to the new PayPal SDK. Follow the AGENTS.md file for requirements. Ignore all the Optional requirements.
You should add natural breaks within the plan so that:
- I know whether or not to delegate the task to a new agent
- I know whether or not to parallelize a task to multiple agents
- I know where there is ambiguity or that a decision needs to be made by me
✕ Agent Only
- Assumed it had complete SDK knowledge — incorrectly
- GitHub access limitations → incomplete placeholder classes
- Confused between different PayPal SDK variants throughout
- 15 compile errors requiring manual namespace corrections
- 33% hallucination rate; official docs had to be manually provided
✓ Agent + Context Plugins
- Verified its knowledge proactively with tool calls
- Fetched accurate, up-to-date documentation
- Zero hallucinations, zero compile errors
- Applied hexagonal architecture with proper abstraction
- SDK isolated in dedicated adapter — business logic untouched
Results at a Glance — Agent Only vs Agent + Context Plugins
Prompt Iterations
Total prompts required for a production-ready integration — lower is better
Manual Fixes
Human interventions needed to correct agent output — lower is better
Errors Encountered
Total compile and runtime errors surfaced during the session — lower is better
Tokens Consumed
Total tokens used across the full session — lower is better
EXP 04 eShop · New Integration
Adding PayPal to a Modern Checkout
Add a two-part payment flow — buyer approves at checkout, payment captured after stock confirmation — with no prior PayPal integration.
Prompt Used
I want you to integrate PayPal in the checkout flow, where the order is created at checkout and approved by the user.
After approval, the user clicks on "Confirm Order", and then the order is captured after confirming stock availability.
✕ Agent Only
- Hallucinated API version → immediate compile error
- Authorization silently failed — misdiagnosed as config issue
- URL injection:
?paid=1query param triggers false approvals - Incorrect Razor component usage broke checkout button
- 12 compile errors; manual doc provision needed to unblock
✓ Agent + Context Plugins
- Verified its knowledge proactively with tool calls
- Correct PayPal Server API v2 usage throughout
- Zero hallucinations, zero compile errors
- PaymentProcessor as single integration point — clean encapsulation
- Eliminated 200+ lines of manual HTTP boilerplate
Results at a Glance — Agent Only vs Agent + Context Plugins
Prompt Iterations
Total prompts required for a production-ready integration — lower is better
Manual Fixes
Human interventions needed to correct agent output — lower is better
Errors Encountered
Total compile and runtime errors surfaced during the session — lower is better
Tokens Consumed
Total tokens used across the full session — lower is better
See Context Plugins in Action
Browse APIs with live Context Plugins and see what authoritative API context looks like for your stack.
Browse API Showcase×