Forem: Amanda

Choosing a model means measuring cost vs quality on your data

Amanda — Sun, 22 Mar 2026 16:01:27 +0000

I wanted to evaluate model-based extraction in a way that would tell me more than benchmarks alone. The scenario is building an AI recruiting agent to help match candidates to job postings. To do this, we need to ingest job postings from career pages, aggregators, social media posts, and other messy sources. Every posting needs to be parsed into structured JSON: title, company, salary range, requirements, benefits.

I set up a comparison with a small dataset of 25 job postings across three model tiers to answer a practical question: does the quality difference between a more expensive model and a budget model justify the cost over time?

All three models perform competitively on standard benchmarks, which is exactly why I couldn't rely on them to make this call.

Setup

For this exploration, I used Baseten's Model APIs. You can use whatever model provider you like.

I picked three models across the cost spectrum tiered by model positioning (priced March 2026):

Tier	Model	Active Params	~Input $/1M tokens
High	DeepSeek V3.1	671B / 37B active	$0.50
Mid-tier	Nvidia Nemotron 3 Super	120B / 12B active	$0.30
Budget	OpenAI GPT-OSS-120B	120B / 5.1B active	$0.10

I generated a dataset of 25 job postings with Claude, designed to reflect the kinds of messy variation you see in real job posting data: informal listings, non-English postings, missing or no fields, hourly rates vs. annual, multiple currencies. For production, this type of data would likely come from multiple sources and be larger.

The extraction prompt asks for valid JSON with ten fields: title, company, location, work model, salary min/max/currency, requirements, nice-to-haves, and benefits. Temperature is set to 0. For the purpose of this exploration, the same system prompt was used for the entire evaluation.

For scoring, scalar fields (title, company, location, etc) are compared after normalization with exact match for strings and a 5% tolerance band for numbers. Array fields (requirements, nice-to-haves, benefits) are scored using exact normalized item matches and F1 score. The overall accuracy per posting is a weighted average across all fields, with title and requirements weighted highest because those matter most for this recruiting agent use case.

This is a deliberately strict metric. For example, a model returning "senior engineer" instead of "senior software engineer" would not get full credit under this scorer even if a recruiter or downstream system might treat those as the same role family. This is a choice, however, and exact-match extraction accuracy is not the same thing as business usefulness.

I included one reasoning model, Nemotron, because when you send a prompt to a reasoning model it will "think" first and that output is wrapped in think tags. This is something to consider when building your parser. DeepSeek V3.1 is technically a hybrid model that supports both thinking and non-thinking modes. I didn't specify, so it ran in the default mode (non-thinking).

Example reasoning output might look like this:

<think>
The posting mentions "$150k - $180k" — I should normalize this to annual integers.
The location says "SF Bay Area" — should I interpret this as San Francisco?
The posting mentions "3 days in office" — this implies Hybrid, not On-site...
</think>
{"title": "Senior Engineer", "company": "Acme Corp", ...}

Reasoning models also affect cost because those thinking tokens count toward output. Across three runs, Nemotron averaged roughly 735 output tokens per call compared to 141 for DeepSeek and 481 for OpenAI, which is a big part of why it ended up as the most expensive option in this comparison.

The results

I ran the comparison three times on the same dataset to account for variation in model runs. One clear pattern across runs is that DeepSeek was always first while GPT-OSS-120B and Nemotron were close with no clear winner for second place.

Metric	DeepSeek	Nemotron	OpenAI
Mean Accuracy Across 3 Runs	74.0%	69.6%	70.4%
Accuracy Range Across Runs	74.0-74.0%	68.5-70.6%	70.1-70.6%
JSON Valid Rate	25/25 in every run	25/25 in every run	25/25 in every run
Avg Latency	0.70s	1.63s	2.13s
Avg Cost/Posting	$0.00042	$0.00068	$0.00029
Est. Cost/100K Posts	$42.00	$68.33	$28.67

All three models produce valid JSON 100% of the time.

Where the models actually differ by field

The aggregate scores tell a partial story. Here's the per-field breakdown averaged across all three runs:

Field	DeepSeek	Nemotron	OpenAI
title	88.0%	84.0%	81.3%
company	80.0%	76.0%	80.0%
location	44.0%	32.0%	38.7%
work_model	77.3%	74.7%	72.0%
salary_min	82.0%	80.7%	80.7%
salary_max	84.0%	80.7%	80.7%
salary_currency	84.0%	80.0%	84.0%
requirements	54.6%	53.0%	50.1%
nice_to_have	74.0%	67.3%	71.6%
benefits	80.3%	75.0%	75.1%

A few things stand out. Location was low for everyone, 32-44% across the board. These postings include things like "SF Bay Area," "remote (US only)," and locations in Portuguese, so that is not surprising. DeepSeek has a slight edge across many categories.

Nemotron's weakest spots were location and requirements. While I don't know the single cause for this, it's a useful reminder that extra reasoning tokens do not automatically translate into better structured extraction.

Requirements and location were difficult for all the models.

Human review

In general, automated scoring is not enough to confidently choose a model for your agent. How much you validate and against which fields will vary by use case. You may want to review all fields in a subset of data, or you may have one field that must be 100% correct and choose to audit that field across everything.

Human review might reveal that your automated scoring weights don't reflect what actually matters for your use case.

In my case, because this was a small exploratory dataset, I reviewed a subset of outputs outside the repo with extra attention on fields that scored lower, especially work_model and location. The repo is meant as a companion for readers to run themselves, not as a checked-in record of my manual review.

A few interesting findings:

When a posting did not name a real company in the main content, such as a recruiter email or something ambiguous like "stealth startup," all three models either left the company unresolved or returned placeholder-like values such as "Stealth Startup." That is probably the right behavior for a strict extraction pipeline, but it might not be the behavior we want.

In a posting with dual currency salary bands, each model handled it differently. One took the first band, one mixed values across both bands, and one returned nothing. This could potentially be handled with different field design as I was only looking for salary min and max with no flexibility for the dual currency scenario.

In listings with a specific city that did not state remote, in-office, or hybrid, all models tended to set work_model to null. This is another example where whether or not this is acceptable is a product choice a human needs to make.

Cost at scale

At 100K postings per month:

High(DeepSeek V3.1): ~$42/month
Mid-tier (Nemotron): ~$69/month
Budget (GPT-OSS-120B): ~$29/month

The budget model saves you $13/month over the higher cost model for roughly a 3.6-point accuracy drop. Nemotron costs more than both while scoring lower on average. The thinking tokens make it the worst value for this particular task.

If we scale this to 1M postings, the spread becomes roughly $420 vs $683 vs $287 per month, which makes the cost penalty for the reasoning model more visible.

What's next

Ultimately this dataset is too small to come to a clear conclusion, but does offer some interesting areas to explore further against real data.

For this use case of structured extraction from messy text at volume, the numbers make the OpenAI budget model worth exploring further. Across three runs it stayed fairly close to DeepSeek on aggregate score while remaining the cheapest option, which is still an attractive tradeoff.

Now, you may be thinking about the latency of the budget model as it was consistently slower. This would matter more if this were user-facing and synchronous. In this case, there is no reason the end user needs to trigger extraction and wait on it directly, so batching is a reasonable fit.

I would be more cautious about claiming OpenAI is definitively better than Nemotron on quality alone. Under this strict scorer, those two were close enough that second place flipped once across the three runs. Ultimately, I would still skip the reasoning model for this kind of extraction. Nemotron's thinking was sometimes useful extracting from ambiguous formatting, but for structured output the extra cost was not justified by the measured quality here.

Final thoughts

I was definitely surprised by these findings and expected a more definitive "winner".

Now, this exploration is 25 postings which is a small sample size. Given the strict scoring mehtod in a dataset this small, each mark swings the accuracy score more than in a larger dataset. Generated data also misses other things that you will find grabbing this same type of data from real world sources like unexpected artifacts. With more data, more runs, and a more rigorous "human in the loop" step, we would see something different.

I also used the same system prompt for all models and test runs. Prompt variations could impact results and are worth exploring.

What you evaluate will depend on your final product as well. Your structured extraction problem might have different failure modes and need different scoring weights than mine.

The main takeaway here is that benchmarks alone won't tell you which model handles your messy data the best. Build something against what your model really needs to perform well at and see what comes back.

If you would like to run this analysis yourself, the project is hosted on Github. If you have questions or want to chat, please get in touch with me on LinkedIn or X.

Why your solo agent workflow breaks down in a team build

Amanda — Sun, 15 Mar 2026 16:24:06 +0000

As I have been learning and building with AI, most of my discoveries and improvements have been around the concept of solo developer + agent.

Recently I had the opportunity to work with a small team on a new build and brought the approach that works for me to the team. While my flow worked, it blocked the team for too long and wasted precious time that my teammates could have spent building more meaningfully.

Here I want to explore the approach that works well for me solo and what I think changes when you apply that same approach to a team on a greenfield build.

I am not presenting this as the correct workflow. This is more an exploration of a failure mode I hit, the tradeoff I now see more clearly, and the experiment I would run differently next time.

Note: You will notice reading this that I do not share the specific build. This project was for a CodeTV episode that has not yet been released. I will update this once it is out.

How I build

When I start a new project with an agent, currently my focus is spending most of the time writing out a plan for the agent to follow up front. Something that gets me to a working application within the boundaries I set that I can then refine.

In essence I create a detailed plan across a few files for the agent to follow: a spec, a runbook, and a checklist, with an AGENTS.md that functions more as the high level order and file map. It covers scope, architecture, features, tech choices, and what is out of scope.

The goal is simple. Give the agent enough structure to move with confidence and produce a coherent first pass.

What happened when I applied this to a team

Naturally I brought this approach to my team with the theory that we could give the agent our requirements, have it spin us up a working starting point, and then divide and iterate from there.

We had a 6 hour build time and this felt like a good way to get the boring scaffolding out of the way so we could all apply our expertise and creativity where it mattered.

As a team we briefly reviewed the spec docs to make sure we agreed on the decisions and I gave it to the agent to build. In this case I used Codex 5.4.

Did it work? Sort of.

The result was a fully working baseline. It wasn't pretty but, it met the requirements and the logical flows needed very little refining. We only replaced one technical decision, not the agent's fault in this case, but one area where we needed to change direction. At this point my team focused on UI/UX and adding some creative flows.

The problem was that the first pass took over an hour and for a 6 hour build that meant the rest of the team was effectively blocked for a meaningful portion of the project timeline.

This is what failed. The agent produced good work, the spec was fine but the artifact created was too complete when we could have moved with a more minimal scaffold.

By the time we had the baseline we had lost too much of the build window for refinement, UI/UX creativity, and the kind of decisions we need humans to be able to make early.

Why this breaks in a team setting

As a solo developer, a highly complete first pass is often useful. You are optimizing for coherence and momentum.

On a team build especialy with a compressed timeline that same instinct can backfire.

What I didn't consider is that the first artifact does not just need to work or be super complete, it simply needed to give other people something stable to build against immediately.

A strong spec and a strong team start are not the same thing.

What I would try next time

If that first pass had taken 5 to 10 minutes, I would probably still think this was a great workflow.

But once the first pass starts taking a meaningful chunk of the total project time, the question changes:

What should an agent build first so the rest of the team can start contributing right away?

For a short build, I do not think the answer is "the whole MVP."

I think it is something closer to a shared scaffold that gives the team a coordination surface after the first agent pass.

This could include:

app shell
route structure
shared types
schema or model shapes
API contracts
page shells
placeholder states
mocked data

Jusr enough structure that people can work without inventing their own versions of the system. As we've all seen, an agent given the same instructions can produce different results.

Instead of this

Team writes spec

one agent builds the full working MVP
everyone else waits or does non-code tasks
team reviews and divides the remaining work

Perhaps it looks like this

Team writes spec

one agent builds the minimum scaffold
team splits work immediately
one stream pushes logic, one stream pushes UI/UX

In a little more detail, the division of work becomes:

one agent generates the shared scaffold
one human + agent builds the core interaction flow against mocked data
one human + agent works UI/UX and page feel
one human + agent focuses on integration edges like auth, persistence, and deployment

In our case the point of failure was that the first pass already included things like fully written API routes, test suites, authentication, notifications, and almost every requirement needed for the final product.

All examples of things that should have come later.

What I took away from it

I had spent so much time optimizing as a solo developer that I did not have the right mental model for a team build.

For a short build like ours, my thinking has definitely changed.

The first generated artifact should optimize for parallel human contribution, not maximum autonomous completeness.

That sounds obvious in hindsight, but it does not always feel obvious when you are staring at a capable model and a good spec and thinking "why not just have it build the whole thing now?"

Final thoughts

I do not think I have the perfect answer for what workflow is best in every scenario.

There will be projects where one developer plus agents gets to a strong MVP so quickly that bringing the rest of the team in later is the right move.

There will also be projects where everyone should start moving early, but against minimal shared artifacts like types and page shells instead of a full application.

What I am convinced of after this experience is that this should be a deliberate choice based on the constraints of the team, the project, and the shared goals you have.

Just because your agent can build the whole working application does not mean that is the best place to start.

A better question to ask might be...

What is the smallest useful thing we should generate first so that nobody spends the best part of the build waiting?

Catching agent repo drift before evals

Amanda — Thu, 05 Mar 2026 16:14:03 +0000

After covering basic linting checks in my previous post, there is another layer worth adding before the more costly behavioral evals.

You can catch repo drift and convention violations with deterministic checks before paying for slower behavioral eval runs.

Reference integrity check

One place to start is reference integrity, for example:

referenced code still exists
referenced code still contains real implementation

# reference integrity
if [ -n "$resolved_ref" ]; then
  case "$ref" in
    *.ts|*.js|*.tsx|*.jsx)
      if rg -q '^\s*(export\s+)?(async\s+)?(function|class|interface|type|const|let|var|enum)\s' "$resolved_ref"; then
        log_pass "$relctx -> $ref exists and has declarations"
      else
        log_error "$relctx -> $ref exists but has no clear declarations"
      fi
      ;;
  esac
else
  log_warn "$relctx references '$ref' which no longer exists"
fi

This is useful because files can still exist after a refactor, but stop being useful references for your agent.

Architecture drift checks

Agents tend to crawl directory structure to infer where new code should be placed. If the architecture rules drift, the agent may generate code in the wrong location.

# architecture rule in this repo: routes should live in src/routes
if [ -d "$REPO_ROOT/src/routes" ]; then
  log_pass "src/routes exists"
else
  log_fail "Missing src/routes"
fi

# architecture rule:  import from ./routes, not ./route
if rg -n "from ['\"]\\./routes/" "$REPO_ROOT/src/app.ts" >/dev/null 2>&1; then
  log_pass "app.ts imports routes from ./routes"
else
  log_fail "app.ts route imports do not match ./routes architecture"
fi

Instruction drift checks

This example check is looking for contradictory guidance which can confuse agents depending on prompt interpretation.

# ensure AGENTS.md route guidance matches repo structure
if rg -n "src/routes" "$REPO_ROOT/AGENTS.md" >/dev/null 2>&1 \
  && [ ! -d "$REPO_ROOT/src/routes" ]; then
  log_warn "AGENTS.md references src/routes but the directory no longer exists"
else
  log_pass "Route guidance matches repo structure"
fi

Deterministic Anti-Pattern Checks

I think we've all seen coding agents clearly favor certain frameworks based on training data. You can use these types of checks to enforce project conventions especially in cases of common violations.

Some examples of patterns could be:

no raw try/catch in service business logic
no NestJS decorators in an Express codebase
no chai in tests where Jest is the standard

This check will obviously vary greatly by project, but here are some example code snippets.

# no try/catch in services
if rg -n 'try\s*\{' "$REPO_ROOT/src/services/" >/dev/null 2>&1; then
  log_fail "Service files contain try/catch blocks (should use Result<T>)"
else
  log_pass "No try/catch in service files"
fi

# no chai in tests
if rg -n "from ['\"]chai['\"]" "$REPO_ROOT/src/test/" >/dev/null 2>&1; then
  log_fail "Test files import chai, should use Jest"
else
  log_pass "Test files don't use chai"
fi

# no NestJS decorators in routes/services
if rg -n '@(Controller|Get|Post|Injectable)' "$REPO_ROOT/src/routes/" "$REPO_ROOT/src/services/" >/dev/null 2>&1; then
  log_fail "Found NestJS decorators in Express layers"
else
  log_pass "No NestJS decorator drift"
fi

These checks enforce repo contracts before you run behavioral LLM-as-judge evals.

When to add an eval platform

This post is intentionally script-first and focused on things that can be checked deterministically in your environment.

Capability drift is different and needed to measure if the agent is getting better or worse over time. This gets into behavioral evals and deserves its own exploration.

Regardless, keep your more deterministic repo checks. They are a low lift and valuable!

I'd love to hear more about how you are approaching checks and evals in your projects. Leave me a comment below.

Practical linting for agent context files

Amanda — Mon, 02 Mar 2026 15:00:33 +0000

As more and more developers are adding agent context files, skills, AI PR review, and other AI indicators to their repos I noticed the same questions coming up:

How do I test these files?
How do I know my updates are meaningful?
How do I measure impact?

Your agent treats a context file like the truth so drift and inconsistency over time can introduce unexpected behavior. The non-deterministic nature of agents can feel intimidating, but before you even get to writing behavioral checks there are some simple automatic and deterministic things you can do that are already familiar.

How agent context linting is different

Traditional linting might check syntax and style answering the question "Is this valid JavaScript?" but linting for your context files is answering different questions.

Some examples could be:
Is this guidance specific enough for a model to follow?
Are the file references up to date?
Are my rules and terminology consistent throughout context files?

Basic Structural Checks

Quick checks for basic structure:

minimum or maximum line and word count (as a proxy for token consumption). This is important as agents need enough information to do the job while not bloating the context window.

# Example: fail if AGENTS.md is too long for your standards

max_lines=300
line_count=$(wc -l < AGENTS.md | tr -d ' ')

if [ "$line_count" -gt "$max_lines" ]; then
  echo "ERROR: AGENTS.md is $line_count lines (max $max_lines)."
  echo "Suggestion: move detailed procedures into focused docs and reference them from AGENTS.md."
  echo "Example split: docs/review-checklist.md, docs/testing-standards.md"
  exit 2
fi

check for required frontmatter in skills.md

# Example: required frontmatter in SKILL.md
#replace .goose with your agent

skill_file=".goose/skills/context-eval/SKILL.md"

if ! head -1 "$skill_file" | grep -q "^---"; then
  echo "ERROR: $skill_file missing YAML frontmatter"
  exit 2
fi

if ! grep -q "^name:" "$skill_file"; then
  echo "ERROR: $skill_file missing required 'name' field"
  exit 2
fi

if ! grep -q "^description:" "$skill_file"; then
  echo "ERROR: $skill_file missing required 'description' field"
  exit 2
fi

echo "PASS: frontmatter check"

Other checks could be no TODO markers that may confuse agents and basic checks that any file references still exist.

Instruction Quality Checks

Generic phrases like "follow best practices" are not useful to your agents so flagging weak wording patterns to ensure concrete instructions is another strategy.

This can be accomplished in your own scripting, but here you may want to consider other tooling meant for validating against prose like Vale

Vale is a prose linter similar to ESLint, but for natural language in your markdown and context files. You define patterns and configure Vale to scan and report severity as a suggestion, warning, or error.

For agent context, this is useful for checking non-deterministic language that agents don't work well with versus file structure.

Example:

Before: "Follow best practices for error handling."
After: "In src/services/, use Result<T> from src/common/result.ts and avoid raw try/catch for business logic."

# .vale.ini
StylesPath = styles
MinAlertLevel = warning

[*.md]
BasedOnStyles = ContextQuality
ContextQuality.WeakReviewVerbs = NO

# styles/ContextQuality/GenericPhrases.yml
extends: existence
message: "Replace generic phrase '%s' with project-specific guidance."
level: warning
ignorecase: true
tokens:
  - be helpful
  - follow best practices
  - be concise

Next steps

Linting can catch some obvious quality issues early and is quick to set up. This can run on every PR while your more costly behavioral evaluations can run nightly (or whatever cadence makes sense for your project).

Remember linting for agents has similar expectations to linting your code. You can expect fast feedback on edits, fewer instruction quality issues, and less drift. You should not rely on linting to predict agent behavior.

Vibe coding as a love language

Amanda — Sun, 18 Jan 2026 13:58:57 +0000

I never used to be a person who coded on the weekends.

Not because I didn't have ideas, but because it felt unfair to my partner to sit holed up at my desk hacking away all weekend instead of spending time with him.

Coding with AI tooling has enabled me to scratch the itch of creating silly things with no business value just for fun - and be able to share that experience because the barrier to add value is just natural language and now creating a program can be a fun game or puzzle we do together.

This weekend I wanted to recreate the viral speed reading video as an application that pulled in unique articles but also tests you on reading comprehension. I wanted to do this because after I took that test I felt like I was the fastest reader ever! But no clue if I knew what I just read.

Since I wanted to do this with my husband and he has zero technical background (or interest), I started with goose desktop. Together we crafted the initial prompt around the idea.

This was a very different way of working for me. No real plan, I didn't specify the tech. It was about having fun and making something together.

We started very simple...

...with this prompt.

I want to make a website based on the viral speed reading test. This site should pick a wikipedia article at random for the test and have an input at the end where the player has to put in what they understood about what they just read. The site should then be able to judge the reading comprehension. Help me design a plan to implement this. The first step will be understanding what the viral speed reading test is so you will need to research that

We used Sonnet 4.5 as there was no reason to call on anything more robust here. Not surprisingly, the model chose to create a NextJS application. React is king - no argument from me!

Once my husband saw it working, he had ideas! He is a comms professional with a background in journalism so immediately wanted the font to be more readable and fix that the red letter focal point so it wouldn't bounce around as much.

He was pointing at the screen and telling me what to tell goose to do. We iterated a bit together like this and ended up with a great starting point.

I wanted to add a way to pull a fresh article and a loading state in case the first article returned wasn't long enough. So I added that next.

Here's the test and restart flow.

While this was truly a for fun exercise, it was also interesting for me to see how someone very far removed from any tech understanding use a tool like this and how he problem solved. He was more likely to describe how he wanted the application to feel or focus on the design. Very different from the developers I focus on for work!

As a developer, it was definitely hard for me to walk away from the "finished" project. I could see the rough edges and started thinking about user experience issues - what happens if there's network issues, what happens if it takes 20 tries to find something long enough, how should it handle duplicate content, does it work on mobile....on and on...

At the end of the day though, this wasn't about making a polished product. This was about a new world of fun that you can share with anyone, even your most non-technical friend or partner. Perhaps the future of social interactions includes ephemeral games we spin up and play over an evening together.

There is a lot of hate in the world for AI and lots of valid points to explore there but in many ways this technology can bring us closer to people we love and find new ways to connect with them.

Dynamic MCP Server discovery with goose

Amanda — Tue, 13 Jan 2026 13:55:00 +0000

A common sentiment I hear about MCP servers is that you have to have them all enabled prior to your session to be able to use them and that's wasteful.

I realized in talking to some engineers I work with that some of the options to dynamically use MCP Serves might not be known to everyone.

I often use the goose agent internally and a feature that is very useful for anyone with multiple MCP Servers is the Extensions Manager.

This tool allows goose to search for and enable relevant MCP Servers depending on your needs. This works even for servers that require auth as long as you have configured them.

In this example I have the Github MCP Server configured with my required PAT token and disabled.

I then ask goose to complete a task that requires Github. The first step goose takes is to see if there is an MCP tool available to help. Due to Github being an authed server, I have to confirm a second time and then goose can run with my task.

While I used goose dektop for visuals here, you can use this in the terminal if that's where you like to work.

Check it out and let me know other ways you are using MCP

How I added experimental MCP Apps support to Apollo MCP Server with Goose

Amanda — Sat, 10 Jan 2026 21:27:22 +0000

The challenge: contribute to my companies codebase in a language I do not write using an agentic dev workflow.

As a developer advocate, I'm often creating demo and educational code versus contributing to our product codebases. Over the holiday break I wanted to challenge myself to fully lean into agentic coding by adding experimental support for MCP Apps draft spec to the Apollo MCP Server (my experimental repo here).

It was time to test everything I teach. in a real world scenario in a language I do not write...Rust.

The Apollo MCP Server is open source and is used as a way to expose GraphQL operations as MCP tools without writing any additional code. We recently added support for the OpenAI Apps SDK so evolving into MCP UI apps was a great next step to challenge myself on.

Constraints

The agent had to add experimental support for MCP Apps without breaking existing functionality or interfering with the OpenAI Apps implementation. To validate my changes, I also needed a separate repository with a prototype MCP UI App for my Luma Community Analytics tool.

The Tools

These are the tools and techniques I leveraged to make this a successful build.

goose CLI and Desktop
Markdown files to provide instructions and important information to the agent including MCP Spec details and general project goals and rules
Apollo MCP Server
MCP Jam for testing

The Build

I needed to work with the agent to get to a point where we had a flow that followed a research/plan → code in chunks → test → report deviations → repeat process. This was the core idea I had to make this a successful test.

My research started with a simple prompt:

"research all files in the apollo-mcp-server local repo to understand the OpenAI apps implementation and the details of the draft MCP UI Apps spec. Then create a plan to add expirimental support for MCP apps that preserves the open ai SDK version"

This wasn't a magical plan that was perfect the first time. Iteration on this plan was the most "human in the loop" work that I completed for this project. In my experience spending time on a planning phase is very important when working in an existing codebase especially in the case of adding a new/experimental feature that should not impact the rest of the project.

One of the features of my plan that increased the success of my agent was to have each chunk of the plan followed by full testing of the code. This way, errors were caught along the way and fixed rather than having a mess at the end. I also requested that the final report output have any details where the agent needed to deviate from the plan based on errors or other information.

Once it was complete, I built the rust binary and added it to an MCP Apps project for testing.

Did it work the first time? No.

However, I was able to quickly debug with goose to find the few loose ends that did not get caught such as a query parameter used in our OpenAI SDK version that was not actually MCP App compliant but didn't flag on a test.

With the app working, I was able to do a preliminary test in MCP Jam which is an alternative to the MCP Inspector. It's a great experience and the regular inspector does not currently support apps.

At this point... I had to wait. Remember MCP Apps is in draft and the agent ecosystem hadn't begun to support it yet. The Goose team at this point was almost there but it was the holiday break so I actually shut my laptop. (go touch grass?)

The final test!

Goose team released draft spec support in early January in v1.19.0! I immediately rushed to test in the Goose desktop agent but ran into a few issues.

Back to goose again to debug!

Goose helped me discover 2 small bugs in the early release of MCP Apps support in goose and I was able to report quickly to the team so they could quickly release a patch.

You read that right. I made goose debug goose!

Ultimately I ended up with the early prototype of my Luma events dashboard which will be a great tool for folks internally here at Apollo to understand community metrics for events I host.

Learnings

You don't have to be an expert in a language to contribute a meaningful feature to a codebase
Plan, Iterate, Plan
Agents working in small chunks and always testing means they can catch errors early, fix and move on while you grab a coffee
Is perfection the goal? In my experience, not yet but I was able to move so much faster as a developer. The server updates and new app were completed in half a day with me starting from zero knowledge of the draft spec or codebase.

Note: After I completed this I saw a post by Angie Jones for a Research -> Plan -> Implement flow with recipes that she released over the holiday break too. If I were starting this flow again today, this is where I would start. Check the docs here.

Can you do this? Yes! You can test out my Luma Analytics MCP App Protoype or build your own. I'd love to hear more about how you leverage agentic coding in the comments.

Meet the Builders: Highlights from the MCP Server Builder Meetup

Amanda — Tue, 08 Jul 2025 16:33:39 +0000

On June 18th, we hosted our very first MCP Server Builder Meetup in San Francisco, bringing together engineers, tinkerers, and early adopters to explore the future of building AI-native developer experiences with Model Context Protocol (MCP).

This was more than just a showcase of cool demos. It marked the launch of a new community of developers building for and with LLMs using MCP servers. The energy in the room made one thing clear: this community is ready to build what's next.

Keep reading for a recap of all the demos with links to the full sessions. Our next events in the series are on July 29th in NYC and July 31st in San Francisco. We hope to see you there. RSVP and subscribe in our Luma calendar.

Jordan Bergero (Block): Building Reliable MCP Servers with Goose & MCP Tool Layering in Square MCP Server

Jordan shared how his team turned an internal hack week project into a production-grade MCP server at Square, which now powers real developer workflows.

Key Highlights:

Introduced Goose: an open source, LLM-agnostic tool for running and testing MCP servers locally in a GUI or terminal experience.
Explained Square’s "layered approach" (discover → plan → execute) to reduce API confusion. This structured layering helps make results predictable and reproducible.
Goose read a raw .txt file of invoice notes, parsed out customer and payment data, called multiple Square API endpoints (customer create, order create, invoice publish), and emailed real invoice all from a single prompt.

Why it matters: An impressive real-world example of turning API chaos into structured tool behavior.

Check out Jordan’s full session on YouTube and connect with them on LinkedIn.

Melissa Herrera (Langflow): Langflow as a Visual MCP Client and Server

Melissa brought the energy and delivered a lightning-fast tour of Langflow, a visual IDE for building agent workflows with drag-and-drop ease.

Key Highlights:

MCP Client & Server: Langflow can act as both an MCP client (calling tools via agent workflows) and a server (exposing your flows as tools), letting you chain, compose, and republish.
Multi-agent architecture: She showcased a resume enhancer app that parsed a resume, queried live job market data via Tavily, and returned improvements all orchestrated across multiple agents.
Tool Reuse: Demonstrated turning any Langflow component into a reusable tool and exporting it as part of a server bundle.

Why it matters: For devs or visual thinkers, Langflow is a developer-friendly launchpad for building composable, LLM-native tools.

Check out Melissa's full session on YouTube and connect with them on LinkedIn.

Lizzie Siegle (Cloudflare): Podcast Generator with Workers + MCP

Lizzie brought some joy (and a surprise) with her talk on building and deploying fun, voice-powered MCP apps on Cloudflare Workers.

Key Highlights:

Serverless podcast generator: Combined Claude, Workers AI, and a Cloudflare D1 SQL database to build a podcast generator that outputs both audio and script content.
End-to-end deployment: Demonstrated how to go from zero to a deployed MCP server using Cloudflare’s click-to-deploy button. Most code was auto-generated, including durable object support.
Tool listing & persistence: Saved generated podcast metadata and audio URLs to SQL, and exposed a tool to query prior results.

Why it matters: Cloudflare’s infra stack is well-suited for lightweight AI applications. If you want a fast way to host, persist, and serve LLM workflows globally, this is a great blueprint.

Check out Lizzie's full session on YouTube and connect with them on LinkedIn.

Tobin South (WorkOS): Securing MCP Servers

Tobin delivered a high-impact talk focused on the security foundations of MCP infrastructure.

Key Highlights:

Highlighted common authentication pitfalls in community-deployed MCP servers and the growing need for OAuth and SSO.
Gave a crash course in OAuth and SSO for MCP, including how to support dynamic client registration properly.
Demoed a secure, OAuth-powered MCP server for ordering custom swag.

Why it matters: If you’re shipping MCP servers to users or enterprises, this talk is your must-watch.

Check out Tobins full session on YouTube and connect with them on LinkedIn.

Michael Watson (Apollo): Token-Efficient MCP Servers with GraphQL

Watson closed the night with a deep dive into how token efficiency and schema-first tooling can make LLM interactions smarter and cheaper.

Key Highlights:

Token bloat audit: Analyzed GitHub's MCP server responses which produced 8.5k+ tokens per tool call caused by duplicate and irrelevant fields.
Selective GraphQL queries: Used GraphQL selection sets to omit unneeded data, reducing token usage by 75% to about 2,000 tokens.
Hot-reloadable tools: Showed how the Apollo MCP server can load tools directly from .graphql files with no build step needed.
Live demo: Used Goose to introspect GitHub’s GraphQL schema and auto-generate a working tool in minutes, then hot-loaded it into the server.

Why it matters: GraphQL-first design gives you clean, typed, and cost-efficient tooling which is a superpower for scaling MCP use.

Check out Watson's full session on YouTube and connect with them on LinkedIn.

What’s Next: Join the MCP Server Builder Community

This was just the beginning. Our mission is to support and grow a community of developers building smarter MCP tools.

Check out our MCP Server

Apollo's own MCP server is open source and we have a full tutorial on Odyssey that will get you up and running building your own tools.

Join future events

We host meetups regularly. Check out our Luma event page to RSVP for the next one in SF, NYC, or virtually.

Get involved

Got something to show? Built a server? Join our talks or share in the Apollo Community.

From REST to GraphQL in minutes with prebuilt Connectors

Amanda — Fri, 13 Jun 2025 15:25:32 +0000

We’ve all been there: vague API docs, an outdated OpenAPI spec, and a half-buried list of endpoints that leave you guessing. With prebuilt REST Connectors, you skip the guesswork. Just download the schema files, run your graph locally, and start querying live data with GraphQL.

In this guide, we’ll use the Space Devs Launch Library 2 in the Connectors Community repo as an example to show how easy it is to integrate a REST API into your graph in minutes and how you can tailor it to meet your needs.

Prerequisites

-Create an Apollo Studio account. This allows you to create and manage your graph, providing the necessary credentials, APOLLO_KEY and APOLLO_GRAPH_REF (more on those later…) which the Apollo Router uses to fetch the supergraph schema and run locally.
-Install and authenticate Rover CLI which we will use for configuring our graph and running Apollo Router locally.

Setup

In your working directory, initialize your schema with:

rover init

Select the option to create a new graph.

Then “Start a graph with one or more REST APIs”.

Next you will name your graph and Rover will generate you a new ID for your graph and APOLLO_KEY. Once your APOLLO_KEY and GRAPH_REF are generated,you can start the router with the command provided, but I would recommend putting these values in an VSCode > settings.json. The configuration below will ensure these environment variables are loaded into any new VSCode terminal windows you open:

{

 "terminal.integrated.profiles.osx": {

   "graphos": {

     "path": "zsh",

     "args": ["-l"],

     "env": {

       "APOLLO_KEY": "<YOUR_KEY>",

       "APOLLO_GRAPH_REF": "<GRAPH_REF>"

     }

   }

 },

 "terminal.integrated.defaultProfile.osx": "graphos"

}

Rover generated a supergraph.yaml file for you. You can delete this.

Using thespacedevs prebuilt connector

Next, navigate to thespacedevs folder in the Connectors Community repo. This Connector is an implementation of the Space Devs Launch Library 2. Download all of the graphql files here and the supergraph.yaml. You can add these to the root of your project folder. Rover generates test schema files and a supergraph. Anything that is duplicated or you do not need can be removed.

Next, open your terminal and run the following.

rover dev --supergraph-config supergraph.yaml

That’s it! Your graph is live at http://localhost:4000. Pop it open in your browser to explore the queries in Apollo Sandbox.

Modifying a prebuilt connector

You can use a prebuilt Connector as is or modify it for your application. For example, I’d like to add more information about what a valid search string looks like for upcoming launches.

In launches.graphql find the upcoming launches query and place this comment above it.

"""
Fetches a list of upcoming launches by agency name. 
Try Starlink, Nasa, or Exa for example.
"""
upcomingLaunches(search: String, limit: Int = 5, offset: Int = 0): LaunchConnection

   @connect(

     source: "llv2"

     http: {

       GET: "/launches/upcoming/?search={$args.search}&limit={$args.limit}&offset={$args.offset}&format=json"

     }
...rest of code

Save your file and you will see your new comment as context in the query builder.

What’s Next?

Once you have the prebuilt Connector ready for your use case now you are ready to integrate it to a client. To continue exploring where to use this Connector, take a look at this tutorial to learn how to replace traditional REST API calls in a React and Next.js application. Or write your very first MCP tool using the Apollo MCP server. There’s even a ready to go example using The Space Devs Api for MCP you can use.

There are many prebuilt Connectors ready to use or extend including Stripe, OpenAI, AWS Lambda, and more. Motivated to build your own Connectors to share with the community? We welcome contributions! Learn how to submit your Connector in the Connectors Community repo.

Simplify Your REST API Logic in React with Connectors for REST APIs and GraphQL

Amanda — Thu, 29 May 2025 20:10:27 +0000

If you’ve built a React or Next.js app that talks to multiple REST APIs, you’ve probably got a file like actions.ts set up as a central spot for fetch calls to public services like the USGS Earthquake API or Nominatim’s reverse geocoder. It works, but the code often ends up repetitive, brittle, and difficult to maintain or scale when different endpoints need to talk to each other.

In this post, we’re going to take that same setup and clean it up with a GraphQL layer powered by Apollo Connectors. Instead of orchestrating data in actions.ts, we’ll define a GraphQL schema that does the work for us. Using a declarative configuration, we’ll unify earthquake and location data in a single query with no need for custom resolvers or custom backend logic. The goal: to make your frontend simpler and your data fetching smarter, without giving up the REST services and patterns you already know.

Prerequisites

Create an Apollo Studio account. This allows you to create and manage your graph, providing the necessary credentials APOLLO_KEY and APOLLO_GRAPH_REF (more on those later…), which the Apollo Router uses to fetch the supergraph schema and run locally.
Install and authenticate Rover CLI, which we will use for configuring our graph and running Apollo Router locally.

Setup

In the directory you want to host your schema in run:

rover init

Follow the prompts in the CLI to create a new project.

Rover will generate a command for you that contains your APOLLO_KEY and GRAPH_REF. You can start the Router with the command provided, but I would recommend putting these values in an VSCode > settings.json.

{
 "terminal.integrated.profiles.osx": {
   "graphos": {
     "path": "zsh",
     "args": ["-l"],
     "env": {
       "APOLLO_KEY": "<YOUR KEY>",
       "APOLLO_GRAPH_REF": "<YOUR GRAPH REF>"
     }
   }
 },
 "terminal.integrated.defaultProfile.osx": "graphos"
}

Next create a file at the root called router.yaml and place this code in it. This is to override CORS errors while working in dev. This is not for use in production.

cors:
 allow_any_origin: true

If you haven’t already open your project in VSCode or your IDE of choice and from the terminal run:

rover dev --router-config router.yaml --supergraph-config supergraph.yaml

Building the Schema

Create a new file called earthquake.graphql and paste the following code.

Note: If you prefer, the completed schema can be found here.

@link(url: "https://specs.apollo.dev/federation/v2.10", import: ["@key"]) # Enable this schema to use Apollo Federation features
@link( # Enable this schema to use Apollo Connectors
   url: "https://specs.apollo.dev/connect/v0.1"
   import: ["@connect", "@source"]
 )
@source(
   name: "usgs"
   http: { baseURL: "https://earthquake.usgs.gov/fdsnws/event/1/" }
 )


type EarthquakeProperties {
 mag: Float
 place: String
 time: Float
 updated: Float
 url: String
 title: String
}


type EarthquakeGeometry {
 #this returns 3 values - lon,lat,depth in km
 coordinates: [Float]
}


type Earthquake {
 id: ID!
 properties: EarthquakeProperties
 geometry: EarthquakeGeometry
 }


type Query {
 recentEarthquakes(limit: Int! lat: String! lon: String! maxRadius: Int!): [Earthquake]
   @connect(
     source: "usgs"
     http: {
       GET: "query?format=geojson&latitude={$args.lat}&longitude={$args.lon}&maxradiuskm={$args.maxRadius}&limit={$args.limit}"
     }
     selection: """
       $.features
      {
       properties{
         mag
         place
         time
         updated
         url
         title
       }
       geometry{
        coordinates
       }
       id
     }

     """
   )
}

The first two directives following @ link are required at the top of any file to enable Connectors and federation. The @ source directive is used to point to our base URL for USGS earthquakes.

Underneath this there are three types: Earthquake, EarthquakeProperties, and EarthquakeGeometry. Here we are defining what we want to make available in our schema. You can build this however you want and include as much of the REST API as is valuable for your application. Here you will also define all the types and what values are nullable.

Finally you will see the query type. In this schema we only have one query, but you can build out as many as you need. For example, while this calls recent earthquakes and takes in four parameters, you may also want to include a query for retrieving one earthquake by Id. What you design in this schema is dependent upon the needs of your application and frontend team. Inside this query, you will see the @connect directive which is where you declare what populates this query and how it will return in the selection set below.

Adding a second REST API

Next, we want to add extended location details for each earthquake using Nominatim. To do this, add in another source directive below the USGS one.

@source(
   name: "location"
   http: {
     baseURL: "https://nominatim.openstreetmap.org/"
     headers: [{ name: "User-Agent", value: "testing" }]
   }
 )

Nominatim requires user-agent headers but the value passed can be anything you want.

Next, in your Earthquake type add a new field “display_name” and the following code:

type Earthquake {
 id: ID!
 properties: EarthquakeProperties
 geometry: EarthquakeGeometry
 display_name: String
 @connect(
   source: "location"
   http: {
     GET: "reverse?lat={$this.geometry.coordinates->slice(1,2)->first}&lon={$this.geometry.coordinates->first}&format=json"
   }
   selection: """
   $.display_name
   """
 )
}

Here we use a Connector for REST to declare that display_name comes from Nominatum. Later in the React app, we’ll cover how you can alias this to what you are using on your frontend. You can also alias here in the schema if you choose. The $this key references the parent Earthquake object.

Your schema is ready, the last thing we need to do before testing it is to update your supergraph.yaml to point to your file. It should look like this:

subgraphs:
 earthquake:
   routing_url: http://localhost:4000
   schema:
     file: earthquake.graphql
federation_version: =2.10.0

To learn about configuring your Router, you can head over to the documentation to see more options especially relevant once you are ready to go to production.

Testing the Query

Head to http://localhost:4000 to create and test the query for your app.

Once you have confirmed that your graph is working as expected, it’s time to move to the React app to see what we need to modify. Leave your local Router running as you will need it to be able to test your application.

Modifying the React App

Clone the repo and install the dependencies, run the project.

Open actions.ts so we can investigate the API calls.

There are two functions here, one to call USGS and a second to then call Nominatim on each returned value. There is also some code here to create the necessary object for our frontend.

export async function searchEarthquakes(
 latitude: number,
 longitude: number,
 maxRadius: number,
 limit = 20
): Promise<Earthquake[]> {
 try {
   // Build USGS Earthquake API URL
   const url = `https://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson&latitude=${latitude}&longitude=${longitude}&maxradius=${maxRadius}&limit=${limit}`;


   // Fetch earthquake data
   const response = await fetch(url);


// rest of code omitted here. . . . .


   return earthquakesWithLocation;
 } catch (error) {
   console.error("Error fetching earthquake data:", error);
   return [];
 }
}

Integrating your GraphQL query doesn’t have to be complicated. If you’re comfortable writing a query and calling fetch, you already know 90% of what you need. Head back to the GraphQL sandbox and click the 3 dots to the right of your query. Then select copy to cURL.

Note: If you would rather follow along with the final version, take a look at this branch.

In actions.ts inside the try/catch block at the top, paste your curl. This is a little difficult to read so if you are in VSCode, this is a good place to use copilot to make this more readable. The end result should look like this.

const query = `
    query RecentEarthquakes($limit: Int!, $lat: String!, $lon: String!, $maxRadius: Int!) {
      recentEarthquakes(limit: $limit, lat: $lat, lon: $lon, maxRadius: $maxRadius) {
        id
        locationDetails: display_name
        properties {
          mag
          place
          time
          updated
          url
          title
        }
        geometry {
          coordinates
        }
      }
    }
  `;


  const variables = {
    limit,
    maxRadius,
    lat: latitude.toString(),
    lon: longitude.toString(),
  };


  const response = await fetch("http://localhost:4000/", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      query,
      variables,
    }),
  });
if (!response.ok) {
    throw new Error(`GraphQL API error: ${response.statusText}`);
  }


  const {data} = await response.json();


  const earthquakes = data.recentEarthquakes ?? [];
   console.log(earthquakes)
  return earthquakes;

Here you have your query, the parameters needed from the frontend, and a fetch call to your graph. Notice we are providing an alias to display_name to locationDetails to match our frontends shape.

Let’s clean up the old code.

In the try/catch block you can remove the rest of the code.

// Build USGS Earthquake API URL
   // const url = `https://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson&latitude=${latitude}&longitude=${longitude}&maxradius=${maxRadius}&limit=${limit}`;


   // // Fetch earthquake data
   // const response = await fetch(url);


   // if (!response.ok) {
   //   throw new Error(`USGS API error: ${response.statusText}`);
   // }


   // const data = await response.json();
   // const earthquakes = data.features as Earthquake[];


   // Get location details for each earthquake
   // const earthquakesWithLocation = await Promise.all(
   //   earthquakes.map(async (quake) => {
   //     try {
   //       // Get location details from Nominatim
   //       const locationDetails = await getLocationDetails(
   //         quake.geometry.coordinates[1], // latitude
   //         quake.geometry.coordinates[0] // longitude
   //       );


   //       // Add location details to earthquake properties
   //       return {
   //         ...quake,
   //         locationDetails: locationDetails ?? undefined,
   //       };
   //     } catch (error) {
   //       console.error(
   //         `Error getting location details for earthquake ${quake.id}:`,
   //         error
   //       );
   //       return quake;
   //     }
   //   })
   // );


   // Log the first earthquake with location details for debugging
 //   if (earthquakesWithLocation.length > 0) {
 //     console.log(
 //       "First earthquake with location details:",
 //       JSON.stringify(
 //         {
 //           id: earthquakesWithLocation[0].id,
 //           hasLocationDetails:
 //             !!earthquakesWithLocation[0].locationDetails,
 //           locationDetails:
 //             earthquakesWithLocation[0].locationDetails,
 //         },
 //         null,
 //         2
 //       )
 //     );
 //   }


 //   return earthquakesWithLocation;

You can also remove the entire function call for getLocationDetails. We are now calling exactly what we need in one GraphQL query so we do not need this second function or API call here.

async function getLocationDetails(
//   latitude: number,
//   longitude: number
// ): Promise<string | null> {
//   try {
//     // Add a small delay to avoid rate limiting
//     await new Promise((resolve) => setTimeout(resolve, 100));


//     const url = `https://nominatim.openstreetmap.org/reverse?format=json&lat=${latitude}&lon=${longitude}`;


//     const response = await fetch(url, {
//       headers: {
//         "User-Agent": "EarthquakeSearchApp/1.0",
//       },
//       // Ensure we don't cache the response
//       cache: "no-store",
//     });


//     if (!response.ok) {
//       console.error(
//         `Nominatim API error: ${response.status} ${response.statusText}`
//       );
//       return null;
//     }


//     const data = await response.json();


//     // Log the response for debugging
//     console.log(
//       `Location details for ${latitude},${longitude}:`,
//       JSON.stringify({ display_name: data.display_name }, null, 2)
//     );


//     return data.display_name ?? null;
//   } catch (error) {
//     console.error("Error getting location details:", error);
//     return null;
//   }
//}

By modifying this file to use our new GraphQL query, we eliminated over 50 lines of redundant code without changing any UI logic. It’s a reminder that simplicity scales and a practical example of how to keep your GraphQL integration dead simple.

Another interesting thing to notice here is that searchByPlace is still intact and using REST. Using Graphql is not all or nothing, you can adopt it in your applications when it makes sense and use it alongside your other REST API calls. This allows you to adopt Graphql slowly without breaking other parts of your application which is especially important if you work with multiple teams.

Save your files and navigate to http://localhost:3000 to see your updates in action.

Wrapping up

While this example uses Apollo Router and Connectors for REST APIs to keep things simple and local, in a production app you’d typically pair this setup with Apollo Client on the frontend. Apollo Client handles caching, state management, and reactive updates, all the things you’d expect in a mature GraphQL application.

But the key point here is GraphQL doesn’t have to be all-or-nothing or hard to adopt. By adding a lightweight GraphQL layer over your existing REST services using Connectors, you can reduce boilerplate, simplify frontend data fetching, and set yourself up for more scalable, maintainable code. And when you’re ready, tools like Apollo Client make it easy to fully integrate this into your application architecture.

To get started building your first Apollo Connector for REST APIs today, check out the documentation or get started with pre-built connectors.

Authenticated tests with Playwright, Prisma, Postgres, and NextAuth

Amanda — Tue, 06 Aug 2024 13:47:51 +0000

Testing authenticated pages is hard. Lets demystify it!

In EddieHub community projects it's common to place authenticated pages behind a GitHub SSO login as it's the perfect fit for an Open Source project but when it comes time to test these pages it proved challenging to create a test that could pass the login flow as close to the user experience as possible.

This flow was based on inspiration from this post which has a similar approach but with a mock server which was not needed for our test.

This implementation was finished up on a Livestream. If you wanna check it out, you can see the replay on YouTube.

And finally before we dive in - this work and blog is a community effort with huge thanks to Eddie Jaoude and Dan B for getting this code finalized and merged. If you are interested in geeking out in the Eddiehub community with us, you can check out the Eddiehub Github Org.

The Project

This code is implemented in a project called The Open Source HealthCheck. This project helps repo owners check quickly if their projects are following best practices and quickly identifies places for improvement.

The following tutorial will show you how mock GitHub OAuth login is implemented in this project.

The Stack

NextJS / NextAuth
Postgres
Prisma
Playwright

Note that this tutorial assumes you have this tech stack set up and playwright configured with your database to run in github actions.

Interested in a version of this using MongoDb and Mongoose? Check out the code here in BioDrop. The code is almost the same except for the creation of the database user and some schema modifications.

How to implement : Setup auth

This code will mock the initial login request to GitHub by NextAuth. To read more about the NextAuth GitHub provider, check out the docs.

Prefer to see the full implementation instead of a tutorial, head to the repo

Let's get started...

Create a file called auth.js in tests/setup/auth.js - this file makes sure that all the needed values for the auth process are in place for the test login/logout

Setup a test user in your db. This user will not be stored in your production db but only used in the test environment. What is required to create a user will depend on your schema.

const login = async (
  browser,
  user = {
    name: "Authenticated User",
    email: "authenticated-user@test.com",
  },
) => {
const date = new Date();
  let testUser;

  const userData = {
    email: user.email,
    name: user.name,
    image: "https://github.com/mona.png",
    emailVerified: null,
  };

  try {
    testUser = await prisma.user.upsert({
      where: { email: user.email },
      update: userData,
      create: userData,
    });

    if (!testUser) {
      throw new Error("Failed to create or retrieve test authenticated user");
    }
  } catch (e) {
    const error = "Test authenticated user creation failed";
    console.error(error, e);
    throw new Error(error);
  }

2. Next, we need to creat a valid JWT for GitHub OAuth. The values used here are expected by GitHub including the profile image, access token (this is fake but in a similar structure), user email and name (in the spread ...user) and the sub which is the mock GitHub id. For more information on GitHub OAuth, check out the docs.

  const sessionToken = await encode({
    token: {
      image: "https://github.com/mona.png",
      accessToken: "ggg_zZl1pWIvKkf3UDynZ09zLvuyZsm1yC0YoRPt",
      ...user,
      sub: testUser.id,
    },
    secret: process.env.NEXTAUTH_SECRET,
  });

3. Now that you have the valid JWT, you can create an authenticated user session with Prisma. Of note here is to make sure that your expiry date is in the future. By setting this date to a future date, GitHub won't be called to obtain new tokens while your tests are running.

const session = {
    sessionToken,
    userId: testUser.id,
    expires: new Date(date.getFullYear(), date.getMonth() + 1, 0),
  };

  try {
    await prisma.session.upsert({
      where: {
        sessionToken: sessionToken,
      },
      update: session,
      create: session,
    });
  } catch (e) {
    const error = "Test authenticated session creation failed";
    console.error(error, e);
    throw new Error(error);
  }

4. Next you will create the mock account and insert it into the database. Note here that the access token you use should match the one you used in JWT creation. These values will depend on your database schema.

 const account = {
    type: "oauth",
    provider: "github",
    providerAccountId: testUser.id,
    userId: testUser.id,
    access_token: "ggg_zZl1pWIvKkf3UDynZ09zLvuyZsm1yC0YoRPt",
    token_type: "bearer",
    scope: "read:org,read:user,repo,user:email,test:all",
  };

  try {
    await prisma.account.upsert({
      where: {
        provider_providerAccountId: {
          provider: "github",
          providerAccountId: testUser.id,
        },
      },
      update: account,
      create: account,
    });
  } catch (e) {
    const error = "Test account creation failed";
    console.error(error, e);
    throw new Error(error);
  }

5. Now that everything is setup you have what you need to create a new browser context, add your session token, and browse to a page as an authenticated user.

const context = await browser.newContext();
  await context.addCookies([
    {
      name: "next-auth.session-token",
      value: sessionToken,
      domain: "127.0.0.1",
      path: "/",
      httpOnly: true,
      sameSite: "Lax",
      secure: true,
      expires: -1,
    },
  ]);

  const page = await context.newPage();

  return page;
};

6. Finally, in a separate function create a logout flow that clears out the cookie.

const logout = async (browser) => {
  const context = await browser.newContext();
  await context.clearCookies();

  const page = await context.newPage();

  return page;
};

How to implement : Write the tests

1. In a separate file add.spec.ts in tests/account/repo you will write your tests. Where you do this depends on your approach. In the HealthCheck implementation, all tests from login through authenticated page tests are located in the same file. At the top of the file import your login/logout from the setup

Write a test for "guest cannot login" to ensure an unauthenticated user cannot reach pages they should not. Here you will call logout first to ensure there is no saved authenticated session.

test("Guest user cannot access add repo", async ({ browser }) => {
  const page = await logout(browser);
  await page.goto("/account/repo/add");
  await expect(page).toHaveURL(/\//);
});

3/. Next write tests for your logged in users. In the HealthCheck repo, we wanted to ensure that they could see the authenticated nav and get to an authenticated page. Your tests will depend upon the needs of your app.

test("Logged in user can access add repo", async ({ browser }) => {
  const page = await login(browser);
  await page.goto("/account/repo/add");
  await expect(page).toHaveURL(/account\/repo\/add/);
});

test("Logged in user can see add user nav button", async ({ browser }) => {
  const page = await login(browser);
  await page.goto("/");
  await page.getByRole("link", { name: "Add" }).click();
  await expect(page).toHaveURL(/account\/repo\/add/);
});

Thanks for following along. You can find the full solution here.

If this interests you, stay tuned for part 2 of this blog where we setup a mock server to make GitHub api calls after login.

The Value of Community Conferences

Amanda — Wed, 17 Jul 2024 01:12:02 +0000

Ever since returning from the last JSConf Budapest, a question has been circling in my mind: how can we effectively sell the idea of community conferences to companies? It's a topic that seems to resonate with many in the tech field, from developers to developer advocates.

Why Community Conferences Matter

Those of us in tech who have been to many events big and small know there's something unmatched about a well curated community conference. They're not just events; they're breeding grounds for innovation, learning, and networking. You leave with deep knowledge, friendship, and opportuities you really can't find at larger events.

Anyone you ask in tech will agree that community led events matter and need support but the reality is they are struggling. Beloved conferences like JSConf Budapest have seen their last event already while others are struggling to keep above water.

So if everyone agrees that these types of events are important and necessary for developers, then what gives?

Sponsors aren't sponsoring. Without larger corporate sponsors it's impossible for smaller events to keep costs reasonable.

Why your company should sponsor community conferences

1. Cost: The cost of the sponsorship itself will be much less than the one boasting 5 - 10,000 attendees and a yacht party. In addition to the base cost, there will be less human cost, asset costs, and swag costs as you bend your budget to figure out how to have a cooler booth than the company next to you.

2. Flexibiltiy: You will be working directly with the community to plan your presence. While many community conferences aren't keen on vendor talks taking the main stage that doens't mean you can't show off your product in other ways. If you don't know how you can fit into the event, it's easy to get in touch with the organizers and a good developer relations team will be able to guide you on the best way to engage meaningfully in this space.

3. Stand out from the crowd: Big conferences mean big burnout for attendees and most of your interactions will be the 10 seconds folks give you to make your pitch while they grab your swag and move on. And while many bigger conferences have space for vendor talks, this is generally becuase they are multi-track and you will be competing with big names in the tech speaker space.

A personal example. My first big conference as a speaker - I was given a difficult slot in a difficult location and had 4 people in the audience. This conference had 1000's of attendees.

4. More impactful conversations: Quantity != Quality. While the overall number of attendees may be lower. At a community conference you have the opportunity to really get to know attendees personally and professionally. Instead of pitching your product with the usual script or demo, you are able to have deep technical conversations with devs about what they are thinking about, building, and getting excited about. People are there to learn and be inspired and the conversations during community event downtime are always top tier.

5. Brand bonus points: Your product isnt that special (oh, you have AI now too?). I mean, maybe it is - but tech is flooded with cool tools you are absolutely in competition with. Stand out by supporting communities developers love in a meaningful way. Devs talk to each other and while we want tools that solve our problems, we also love to support companies doing good in the community. If you can be both and awesome tool and a great community member, your brand wins.

Don't get me wrong

A note about all this - still do the big conferences but be more selective about this budget. I would argue by diversifying your approach to include MORE community driven events like smaller single track conferences, meetups, and whatever cool events devs are dreaming up that you will see a better ROI than blowing all your budget on the flashiest event you can find.

A Thank You to the Community

As I reflect on the experiences and insights gained from JSConf Budapest, I can't help but express my gratitude. Thank you to the organizers, the speakers, and every attendee who contributes to making these conferences a rich source of knowledge and community spirit.

In conclusion, selling the concept of community conferences to companies should be a natural step. After all, the advantages are clear. It's about recognizing the importance of these events and the long-term value they bring to individuals and organizations alike. Thank you, once again, to everyone who plays a part in these incredible tech gatherings.

Drop your favorite community led event in the comments and reach out to me to continue the conversation on Twitter or LinkedIn