Forem: zvone187

How to secure AI-coded (vibe coded) applications

zvone187 — Thu, 25 Sep 2025 16:26:36 +0000

Overview

For the past two years, I’ve been deep in the world of AI, building Pythagora and trying to push it toward building production-ready applications. Recently, things have taken a big turn. With the release of Claude Code with Sonnet 4, we’ve hit a turning point. You can now spin up really complex applications, complete with database operations, third-party integrations, and basically anything you can dream up, just by chatting. It feels like live coding, but with AI at your side.

Initially, my focus while working on Pythagora was on enabling people to build apps by chatting, and on what the UX should look like. Now, my attention has shifted to something more important: how do we actually secure web apps once they’re built? For this, I believe we need a mindset shift. Security isn’t something we can just offload to AI or abstract away—it has to stay firmly in human hands.

At Pythagora last year, we were the target of multiple hacker attacks, both on our system and on the systems of vendors we used. When one of our vendors got hacked, the hackers got our OpenAI API and stole over $30k worth of tokens.

After seeing how vibe coded apps are getting exploited like in this case where a Lovable built app exposed all its users' data to everyone, I wanted to see how we can ensure that apps built with Pythagora are secure.

So, I embarked on a journey to map out all potential vulnerabilities and see how we can enable less technical builders to build completely secure applications.

In this post, I'll share what I've learned about security in no-code apps, but also what we actually did as a team. We’ll look at some real examples of vulnerabilities that pop up, and then I’ll walk you through my solution for making vibe-coded applications truly secure. Secure enough that you could confidently build business-critical systems with nothing but vibe coding.

Web app vulnerabilities

We can segment vulnerabilities into the following categories:

1. Insecure HTTPS connection - today, this is a table-stakes security measure, and most websites have it by default. If you open a website that starts with http:// instead of https://, aside from your browser clearly warning you that this website is insecure, all data you enter on this website may be intercepted by a hacker, allowing them to read everything you entered. In contrast, all HTTPS websites encrypt the data so that, if a hacker intercepts it, they will see just gibberish.

2. Authentication - this is the first crucial part of making a web app secure. Seemingly, all AI coding tools have this covered, but if AI is building something for you that you don't check every single line of code, the question is how to make the authentication 100% secure. Take a look at the screenshot below, in which, if AI hallucinates just one line of code and removes the requireAdmin keyword, you will expose all your customers to any crawler that's scanning the internet.

Imagine you create a page that is accessible by anyone (eg. Home page) on which you add names of your teammates. When you decide to deploy this app, you want to remove this page so no one can see employees in your company. When you remove that page, the endpoint is left open, so anyone can get the employee data.

3. Authorization (access control) - once you are sure that your app cannot be accessed by anyone outside of the company, you must ensure that people within the company don't get access to data that they shouldn't have access to.

What if one user gets access to the data that they shouldn't see - for example, in an HR management app, you MUST NOT allow everyone to see HR's paycheck slips for an employee.

4. Codebase vulnerabilities - there is a whole list of potential codebase vulnerabilities - here are a few of the most important ones:

Compromised libraries - the entire web stack is built on top of libraries (code plugins that other people built), most of which are open sourced. Using open-source libraries is nice, but it also provides hackers with full insight into how the library works. So, a library developer might introduce a vulnerability (e.g. a back door to the system that enables hackers to access any app that uses that library). You might not have anything to do with this library, and you may not even know you're using it (it could be a library that a library you've installed is using), but you still get affected.
Sensitive data exposure - if you're not careful how your backend is structured, it might expose some of your API keys to your users (doesn't even need to be a hacker) so if your user finds your API key, they can accure a massive bill on the service you're using (eg. if you're using OpenAI API key for LLM requests).
Injections - if the codebase is not structured correctly, it can enable input data to be run on the server. The most famous is SQL injection, where, if your code inserts a user's input field into an SQL query, if you put DROP DATABASE as your name and on the server, you run INSERT INTO users (email) VALUES (DROP DATABASE) - this query will completely delete your entire database. A similar case happened in an app coded with Replit.

AI writes A TON of code

Everyone knows that AI writes way too much code - much more than it's supposed to...if it were a human. In theory, yes, AI writes more code than what was asked of it, but that code isn't necessarily bad. It's much better than many developers' code that I've reviewed. It adds a lot of logs, error handling, and extra functions.

Pythagora can easily write 10k lines of code (LOC) in a day of work. This is essentially impossible for humans to review. Especially when LLMs get faster, we're likely looking at producing 50k LOC in a day. So, how do we manage all the new code that's being produced?

It's clear that AI is able to write working code so we need to find a way to live in this new world where it is actually possible to produce 50k working LOC.

We shouldn't look for ways to review all those lines of code or reduce the number of lines, but rather, we should find a way to secure AI coded apps without having to review every single line of code regardless of how much code AI writes.

Hallucinations

In reality, the problem is not the amount of code but rather hallucinations. Imagine that AI misses one crucial line of code. In the case below, simply removing "requireAdmin", AI exposed all customers of a company to the public.

So, the question is...

How to use vibe coded apps for business

For me to trust an AI coded app, the first requirement is that I need to be 100% sure that my data is safe. That is 100% - not 99%.

When we first started to build internal apps at Pythagora, I realized the anxiety. I was super insecure about whether there were any exposed endpoints. I didn't just want to see the app working; I wanted to be 100% sure that our users' data is secure.

The anxiety just grew when someone else, especially less technical people in the team, would build a tool that used users' data so I started to think about what would it take for me to trust an app that Pythagora wrote.

You want to offload coding to AI, but keep security in human hands

If I could be 100% sure that the data cannot be leaked or mismanaged, I would not need anyone from my team to review the code that AI built and I would approve any tool release that makes sense.

In the sentence above, the key part is "100%" which AI cannot provide, so humans have to provide a guarantee for some part - security.

Over the past few months, I structured a set of security layers that, if provided, would give me the ease to approve a release of a tool within my team.

These layers should ensure that, regardless of what AI writes, our data remains secure without requiring a review of every single line of code.

✅ 1. Isolate the codebase from non-authenticated users

Vulnerable use cases:
- API endpoint ends up being exposed to the world
- AI hallucinates and removes access control from an API endpoint
- A human creates a publicly available page with sensitive data, then removes the page, but doesn't remove the API endpoint

The first step is to isolate our AI coded apps from anyone outside of the team. That means that API requests that are coming into the codebase already need to be authenticated. A non-authenticade request MUST NOT trigger even a single line of code.

This can be done by enforcing authentication on the infrastructure layer.

Essentially, all vibe-coded tools embed authentication into the AI-generated code, which means that AI can modify the authentication and potentially mismanage it. If it hallucinates in the wrong place, it can expose our entire system to hackers.

In contrast, I wanted to be sure that authentication works regardless of what AI generates. This is where the idea for Pythagora's isolated security system came from. With it, the authentication isn't implemented into the AI generated codebase but in the reverse proxy (specifically NGINX) that sits on the server in front of the applications and authenticates the user before allowing the API request to reach the app. This way, it doesn't matter what happens in the codebase; a user who is not authenticated cannot even reach the app that someone from your team vibe-coded.

In other words, Pythagora's security system completely isolates all your AI-coded apps from anyone who is not authenticated.

Here is a video in which I go deeper into the details about how it works.

🟡 2. Transparent data access

Vulnerable use cases:
- Data is accessible to users within the company who shouldn't have access to that data
- AI coded app creates a destructive database query

For me to allow anyone less technical to deploy an app inside Pythagora, I need to be 100% confident about what happens on the backend of this app.

Today, all vibe coded backends are a black box where, if you don't do a full code review of the entire app, you don't know what happens there.

Code review is unrealistic for AI coded apps since you knock out tens of thousands of LOC within a day. The developer reviewer would need another full day to review it. That's why we need to think differently.

We can still give humans the ability to review what the app is doing without going through each and every LOC. For that, we need to see what we want to review in a code review:

API endpoints - which user has access to which endpoints
Database queries - what DB queries are made within which API endpoint
3rd party requests - which API endpoints trigger which 3rd party requests, and what data does it send

Here is a backend visualization that enables you to easily view all backend endpoints and understand the functionality of each one.

🟡 3. Codebase vulnerabilities

Vulnerable use cases:
- The library that AI installed gets compromised
- AI creates code that's vulnerable to SQL injection
- AI creates code that leaks an API key

These vulnerabilities are mostly done by static code analysis. Some systems automatically scan for these vulnerabilities like Github - if you push code to Github and have an API key exposed or if you're using a compromised library, Github will raise an alert and not let you make your code public.

This is the part that will come last on the roadmap and will perform the following checks before an app is deployed:

Static code analysis.
Automated pen test scan.
AI codebase scan by Pythagora's security agent, powered by Claude Code. It is looking for the following vulnerabilities:
- Injections (SQL, OS, LDAP) - check for places that enable a user to inject, for example, an SQL query that will automatically be run
- Secret keys exposure - check if the secret keys are propagated outside places where they need to be utilized and if any secret key is being propagated to the frontend.
- Cross-site scripting (XSS) and insecure decentralization - check for any places where a string (eg. user name) can be automatically run.

Approval process - code review is outdated in the world of AI

The final part of ensuring an application is fully secured involves not only the security measures but also enabling technical leaders to be confident in the software being deployed within their organization. The final step is to set up the approval process, allowing leaders to review the changes made, such as API endpoints and database queries, and sign off on the software release. We still have some time before reaching this point, but it is coming - Pull Requests in GitHub will soon become obsolete in the world of AI.

Conclusion

In this blog post, we explored typical vulnerabilities that web apps suffer from and how we can defend ourselves from these vulnerabilities in times when AI writes so much code that it's impossible for humans to verify every single line of code.

My goal is to start a discussion about what is needed in order to secure AI coded applications and how we can bring vibe coding into business critical use cases.

If you have any thoughts on this, I would love to hear from you either in the comments below or by reaching me at zvonimir@pythagora.ai.

If you still haven't, check out our product, Pythagora. It gives you the power to build astonishing tools, not just demos, within a few hours of chatting with AI. From today, you can deploy your apps securely and be assured that your data is safe.

How We Made Sure Big Companies Can’t Steal Our Code

zvone187 — Thu, 30 Jan 2025 20:00:31 +0000

As a developer, I love open source. I think it’s the backbone of innovation, collaboration, and community-driven progress. But if you’re a small team or a startup, sharing your work as open source comes with risks. Big risks.

When we started building Pythagora in 2023., it was one of the first agentic systems where AI agents work together to create entire codebases - so, we wanted to share it with the world by showing and inspiring others to build complex systems. However, we knew we needed to protect our innovation from being exploited by larger companies.

In this post, I’ll share the challenges we faced while choosing a license, why traditional open-source licenses didn’t work for us, and how we found a solution in the Functional Source License (FSL). If you’ve ever struggled to balance openness with sustainability, this is for you.

The Challenge of Open-Source Licensing

When we started in 2023, we were excited to share our work. Open source had always been part of our DNA, but we had to pause and ask ourselves tough questions:

How do we protect our code from being rebranded and commercialized by big companies?
Which license encourages collaboration without compromising sustainability?
Can we avoid complex, overly restrictive hybrid licenses? We’ve all seen stories like Elasticsearch vs. Amazon. It’s a cautionary tale for smaller teams and startups using permissive licenses. Initially, we tried GPL, but it felt too restrictive. Licenses like MIT and Apache were too permissive. What we needed was something fair, balanced, and protective.

Why We Chose Sentry & Functional Source License (FSL)

After exploring different licensing options, we came across the Functional Source License (FSL) from Sentry. It’s part of the growing Fair Source movement, which empowers companies to share their core technology responsibly without losing control of their roadmap or innovation.

Here’s why FSL was the perfect fit for us:

1. Two-Year Grace Period

The FSL gives us a two-year buffer where our code is free to explore, use, and modify—but competitors can’t commercialize it. By the time the grace period ends, we’ll have evolved far beyond the initial release, ensuring we stay ahead.

2. Transparency Without Exploitation

With FSL, anyone can:

Read our code.
Use, modify, and share it under simple conditions.
But here’s the key condition: you can’t directly compete with the project.

3. Simplicity

Unlike some licenses that require legal expertise to interpret, FSL is simple and accessible. It strikes the perfect balance for startups like us: encouraging collaboration while protecting our work.

How FSL Works for Startups

For a startup like Pythagora, the Functional Source License is a game-changer. It allows us to:

Share Our Innovation: Developers can freely explore and build on our work.
Prevent Exploitation: We remain protected from big players who might otherwise rebrand and sell our code.
Foster Collaboration: The simplicity of FSL fosters trust and community engagement.

By adopting FSL, we’re aligning with the values of open source while addressing the realities of being a small team in a competitive landscape.

What This Means for Developers

If you’re a developer exploring GPT Pilot, here’s what FSL means for you:

Transparency: You can freely read our code and understand how it works.
Flexibility: You can use, modify, and share GPT Pilot with minimal restrictions. & Fairness: The two-year grace period ensures we can continue to innovate while giving back to the community.

Licensing is one of the hardest decisions for open-source startups, but the Functional Source License (FSL) has allowed us to share our work responsibly while protecting it from exploitation.

We hope our experience inspires other startups to explore Fair Source licensing as a way to balance openness, sustainability, and collaboration.

If you’re curious about GPT Pilot, check out our blog post or explore the codebase.

Pythagora_v1 release reflection – what we learned, current state, next steps

zvone187 — Tue, 01 Oct 2024 16:34:45 +0000

Today, we’re launching Pythagora v1, so we wanted to reflect on our progress in the past year since we released the first version of GPT Pilot (the open source brain of Pythagora) and in the past six months since we released Pythagora beta.

In this post, you’ll read about how we got carried away with hype, thinking AI will be able to build apps much easier than it really can, how flashy demos deceived everyone as well as us, how we switched our focus from the technology to the DX (Developer experience), what can you build with Pythagora, why are we keeping the core technology open source, and what are our next steps.

History of Pythagora

When we started building Pythagora, our North Star was clear: enable developers to create production-ready applications using natural language. I’ve been a developer for the past 15 years, and my dream is to be able to build applications without having to deal with debugging. So, when I realized how good LLMs can code (you can read about my first experience with coding with ChatGPT in this blog post), I immediately started looking into how we can automate this process with my cofounder, Leon.

After a couple of months of experimentation, we created GPT Pilot. This was in September 2023. The community reacted well, so we continued working on it and soon started fundraising. At this time, we needed to create a roadmap for what we plan to build next year. We ended up building most of the roadmap, but the capabilities we set GPT Pilot to have were missed by a huge margin.

We predicted that, in 6 months (around May 2024), GPT Pilot will be able to build a 30k lines of code app in 2 days!

Learnings from Beta to v1

Not only that it wasn’t able to build a 30k LOC app in any amount of time, but it was so hard to use that we needed to do a reality check.

We realized that a bunch of other projects started to come up (most notably Devin) that promised the world while all they did was create a nice, flashy demo.

In the early days, we packed Pythagora with features like automated tests and autonomous code reviews, aiming for maximum autonomy. But reality hit hard. The technology just wasn’t there yet. So, we decided to take a step back.

We removed many components of Pythagora (like automated tests and self code review), refocused our efforts on user experience, and started investigating how people can use Pythagora to build any app they will end up actually using. So, we began introducing more and more human-in-the-loop interactions.

LLMs are still a technology in its infancy, and we must treat it as such. Many people think we can just say two sentences, like “Build me a clone of Reddit”, to AI and let it build the entire app with the infrastructure. Many decisions need to be made about how the app should work. This is exactly what we’re focused on. We want Pythagora to manage the codebase and ask humans whenever they need to make a decision.

Current state of Pythagora

With today’s release, we believe we’re breaking the new boundaries for us and the rest of the world. With Pythagora, people can build apps with up to 5000 lines of code ONLY by writing in natural language. You can see apps built with Pythagora here. All code in all these apps was written exclusively by Pythagora, and humans wrote no lines of code.

In reality, you can build even bigger apps with even 10k LOC. However, above the 5k LOC, the experience gets trickier, but we’re continuously working to push that boundary. Nevertheless, 5k LOC is quite a solid app size, which makes it truly useful. This is our north star – we want Pythagora to build apps that actually end up in production and not in some random folder on a computer.

That is why we added autonomous deployment so you can deploy the created app in one click online and share it with others.

We believe Pythagora is currently one of the most capable tools for building applications through plain English instructions, minimizing the need to write code manually. But hey, if you know of a tool that is able to build apps with this complexity without requiring humans to write a single line of code, we’d genuinely love to hear about it. Drop a comment or ping me directly at zvonimir@pythagora.ai

That said, we truly believe in that common phrase that AI will not replace humans, but humans using AI will. Pythagora is not a fully autonomous AI developer that wanders off into the wilderness of software development and returns with a fully built app – it is a tool that helps people build apps stellar fast without writing and managing the codebase. You just explain what you want to build and talk to Pythagora, as it constantly asks you questions to understand if it’s going in the right direction.

Managing expectations

First off, let’s temper expectations around zero-shot app creation. While it’s tempting to imagine an AI that builds an entire app from a few sentences, it’s not just about the technological hurdles. Even if the tech gets there—and it likely will in the next few years—is that what people really want?

Imagine working with a freelancer developer today to whom you can give a project and wait for them to come back with working software. Now, imagine telling that human developer your app idea in three sentences, and they disappear for a month, returning with a fully built app based on their interpretations. Chances are, it won’t align with your vision. Development is an iterative process filled with decisions that need to be reviewed, discussed, and tweaked.

We believe that humans will always be the decision-makers and creators. Our mission with Pythagora is to offload the grunt work to AI while keeping you in the driver’s seat. Think of humans as a control center operator—a concept echoed by Milo Medin from Benchmark Capital on a 20VC podcast—where AI handles the heavy lifting, but the human guides the direction.

In practical terms, this means creating a seamless user experience where AI does the coding autonomously but reaches out whenever a decision needs to be made. We want to avoid constant context-switching, where you must check in with the AI every few minutes or wait hours for it to return with updates. Rather, we want to enable a constant experience like with any other digital tool where you work with Pythagora for hours or days but get weeks or months of work done.

Looking Forward: The Future of Pythagora

We are very much focused on improving the agentic system, open sourced GPT Pilot, so it can solve more problems and build larger apps faster. However, we are even more focused on how to engage the human, the developer behind the keyboard whose goal is to make their ideas a reality. We are rigorously focused on improving the Developer Experience and making Pythagora a new extension of human creativity.

With that, we will be more focused on building the knowledge base and learning materials that will help everyone become a better builder of software applications. You can watch a 15 minute crash course on Pythagora here:

We are very passionate about building useful software tools to make our and other people’s lives easier, and we want others to be able to do that as well, without having to spend 15 years learning the ins and outs of how to write code.

I almost never had to spend effort on memory allocation while it was a requirement for every developer just a decade before I became an engineer. I hope that happens to writing and managing the countless lines of code in evergrowing codebases.

Conclusion

Launching Pythagora v1 is both a celebration and a stepping stone. We’re proud of how far we’ve come, but we’re even more excited about where we’re headed. The landscape of AI in code generation is evolving rapidly, and while there are challenges, these challenges make our team get up in the morning and march forward to solve them.

We’re committed to bridging the gap between AI capabilities and developer needs, focusing on creating tools that enhance your workflow rather than dictate it. As we continue to develop Pythagora, your feedback is invaluable. Let’s shape the future of coding together. You can sign up for early access to Pythagora v1 here.

Thanks for taking the time to read this, and I wish you happy coding…I mean, building 🙂

PS If you know of any other tools pushing the boundaries in AI code generation, share them! The more we collaborate, the better tools we’ll all have.

What I learned in 6 months of working on a CodeGen dev tool GPT Pilot

zvone187 — Thu, 29 Feb 2024 16:39:45 +0000

For the past 6 months, I’ve been working on GPT Pilot (https://github.com/Pythagora-io/gpt-pilot) to understand how much we can really automate coding with AI, so I wanted to share our learnings so far and how far it’s able to go.

When I started building GPT Pilot, I wrote this blog post on how it is envisioned. Now, I want to share my revised thinking and go into what worked and what didn’t work.

Finally, you’ll see examples of apps that were created with GPT Pilot and how you can create an app with a real AI pair programmer.

What is the idea behind GPT Pilot?

GPT Pilot is envisioned as a real AI developer – not an autocomplete or a chat bot. Rather, it is a developer who creates a plan for how your app or feature should be built and starts coding. It wants to do most of the coding by itself, but when it gets stuck, it needs clarification about the given requirements, or requires a code review, it asks you for help.

Is AI like a junior developer? Or…

I often see CodeGen GPT-4-based tools that say they are building an AI junior developer. Somehow, I’ve always had a problem with that because when I use ChatGPT for coding, it gives me answers and ideas that only a super-senior person could give – something that absolutely no junior dev would even be able to grasp. Still, no LLM can build an app nearly as well as a senior developer can, but the knowledge GPT-4 has about coding is way beyond any junior developer. I would say that GPT-4 has so much knowledge about every part of software development like it’s the most senior developer in the world but with the memory of a goldfish. I picture it as a superhuman robot that just stands in the middle of a room and can only do a single small action at a time, but it cannot combine many actions and work repetitively. You must tell it exactly what it should do next. This is what we’re after with GPT Pilot – we want to create a framework of thinking for the LLM that gets that superhuman robot to continuously work by revising its previous actions, have a feedback loop, and determine what should it do next in order to finish the end goal, which is to build a production-ready application.

In the blog post I mentioned above, I outlined the main pillars on which GPT Pilot was built. But these have changed a bit based on our team’s learnings, so here are the revised pillars:

A human is needed to supervise the AI not only because AI is not good enough but also because you might want to change how something works or looks after it’s implemented. It’s common for a developer or product manager, once they see what an implementation looks like, to decide to change it. Or, you realize there are more edge cases than you initially anticipated and think it’s easier to refactor your current implementation than to fix every issue. The problem is when you finish the entire app and then try to refactor – this is when it becomes much harder because every change will impact all the other features. On the other hand, if you do the refactor before you commit your changes, you’ll be able to proceed with the next features on top of well-written code. This is why it’s crucial for an AI developer to have a human in the loop whenever a task is implemented. This way, the human can review the implementation of each task (just like a code review before merging a PR) before GPT Pilot continues onto the next task. If a human tells GPT Pilot what is wrong, it will be much easier to fix the issues within the task itself. At the same time, the LLM has the context of what needs to be done in the task and what has been done so far.
AI can iterate over its own mistakes. I have a feeling that many people judge ChatGPT’s ability to write code by how well it delivers the first time you ask it to code something. If it doesn’t produce working code, many will think it’s not impressive. In reality, humans almost never write working code on the first try. Instead, you write code, run it, see the errors, and iterate. This is exactly what GPT Pilot enables GPT-4 to do – after it writes code, GPT Pilot can run the code, take the output, and ask the LLM if the output is correct, if something should be fixed, and if so, how.
Software development can be orchestrated. There are many repetitive routines that all developers go through when building an app. One of the routines can be – write code, run it, read the errors, change code, rerun it, etc. Another higher-level one can be – take a task, implement it, test the implementation (repeat until all tests pass), send it for review, fix the issues (repeat until the reviewer approves), and deploy. Many of these routines can be orchestrated if we have an intelligent decision-maker in the loop (like an LLM).
The coding process is not a straight line. When we created the first version of GPT Pilot, we thought it would need to iterate over tasks, implement code, fix it, and move on. In reality, you don’t continuously progress when coding an app – you rewrite your code all the time. Sometimes, you refactor the codebase because, after the initial implementation, you realize there is a better way to implement something. Other times you do it because of a change in requirements. Like I mentioned in #1, after you see that a solution isn’t working, you sometimes need to roll back a bunch of changes, think about an alternative solution to the problem, and try solving it that way. To make GPT Pilot, or any other AI developer, work at scale, it needs to have a mechanism that will enable it to go back, choose an alternative path, and reimplement a task.

What did we learn?

LLMs, in general, are a new technology that everyone is trying to understand – how it works, what can be done better, how to do proper prompt engineering, etc. Our approach is to focus on building the application layer instead of working on getting LLMs to output better results. The reasoning is that LLMs will get better, and if we spend weeks optimizing a prompt, it might be completely solved with the new version of GPT. Instead, we’re focusing on what the user experience should look like and which mechanisms are needed to control the LLM to enable it to continuously work, getting closer and closer to the final solution. So, here are our learnings so far:

The initial description of the app is much more important than we thought. Our original thinking was that, with the human’s input, GPT Pilot would be able to navigate in the right direction and get closer and closer to the working solution, even if the initial description was vague. However, GPT Pilot’s thinking branches out throughout prompts, beginning with the initial description. And with that, if something is misleading in the initial prompt, all the other info that GPT Pilot has will lead in the wrong direction. So, when you correct it down the line, it will be so deep into this incorrect way that it will be almost impossible to get it onto the right path. Now, as I’m writing this, it seems so obvious, but that is something we needed to learn – to focus much more on the initial description. So, we built a new agent called “Spec Writer,” which works with you to break down the project requirements before it starts coding.
Coding is not a straight line. As I mentioned above in the pillars section, refactoring happens all the time, and GPT Pilot must do so as well. We haven’t implemented a solution for this yet, but it will likely work by adding the ability for GPT Pilot to create markers around its decision tree so that whenever something isn’t working, it can review markers and think about where it could have made a wrong turn.
Agents can review themselves. My thinking was that if an agent reviews what the other agent did, it would be redundant because it’s the same LLM reprocessing the same information. But it turns out that when an agent reviews the work of another agent, it works amazingly well. We have 2 different “Reviewer” agents that review how the code was implemented. One does it on a high level, such as how the entire task was implemented, and another one reviews each change before they are made to a file (like doing a git add -p).
LLMs work best when they can focus on one problem compared to multiple problems in a single prompt. For example, if you tell GPT Pilot to make 2 different changes in a single description, it will have difficulty focusing on both. So, we split each human input into multiple pieces in case the input contains several different requests.
Verbose logs help. This is very obvious now, but initially, we didn’t tell GPT-4 to add any logs around the code. Now, it creates code with verbose logging, so that when you run the app and encounter an error, GPT-4 will have a much easier time debugging when it sees which logs have been written and where those logs are in the code.
Splitting the codebase into smaller files helps a lot. This is also an obvious conclusion, but we had to learn it. It’s much easier for GPT-4 to implement features and fix bugs if the code is split into many files instead of a few large ones. Just like we, humans, think – it’s much easier to process smaller chunks of code rather than big ones. We do that by creating levels of abstraction – functions, classes, etc. One of the easiest ways to get the LLM to create a more manageable abstraction is just to tell it to create more modular code and split it into more files. It works amazingly well, and the end result is also better for us, developers.
For a human to be able to fix the code, they need to be clearly shown what was written in the task and the idea behind it. GPT Pilot’s goal is to do 90% of all coding tasks and leave the other 10% to a human. This 10% usually comprises fixes or small changes that the LLM struggles to notice, but for the human, it might be a simple change. However, the problem is that it’s not easy to tell the human what is not working and what code they should look at. If GPT Pilot writes 3,000 lines of code, the human developer, if they want to help GPT Pilot, needs to understand the entire codebase before diving into the actual code. In future versions of GPT Pilot, the human developer will have a detailed explanation of what code has been added to the current task and the reasoning behind it. This way, you will be able to help GPT Pilot.
Humans are lazy. LLMs are better off asking humans questions rather than letting humans think about all the possible errors. Again, it’s very obvious looking back, but one of the things we noticed was that people were much more willing to answer questions that GPT Pilot asked them instead of having an open-ended input field where they could write unconstrained feedback.
It’s hard to get an LLM to think outside the box. This was one of the biggest learnings for me. I thought you could prompt GPT-4 by giving it a couple of solutions it had already used to fix an issue and tell it to think of another solution. However, this is not as remotely easy as it sounds. What we ended up doing was asking the LLM to list all the possible solutions it could think of and save them into memory. When we needed to try something else, we pulled the alternative solutions and told it to try a different, but specific, solution.

Apps created with GPT Pilot

Currently, you can create simple but non-trivial apps with GPT Pilot. We’ve also seen people create apps with GPT Pilot that are very impressive, such as an app that can fine tune a ResNet model to count palm trees and then, when you upload an image, count the trees in it. Here are a couple of apps we created, along with the code, stats, and demos:

Prompt Lab (DEMO)

Think of this as OpenAI Playground on steroids – it enables you to load a conversation from a JSON array or enter it manually, run the conversation with the LLM X number of times, save the conversation to the database, and load it back in. We actually use this app when we engineer a prompt and want to see how it behaves. The Playground is not good enough because it’s time consuming to repetitively rerun a single prompt. With Prompt Lab, you can choose how many times to run it and let the app run the prompt repeatedly. Once it’s finished, you can analyze the results. This might be useful for people who are building an AI app and need to optimize their prompts.

⏳ Time spent: ~2 days
💾 Github repo

SQLite database analyzer tool (DEMO)

This is also an app we use internally to analyze a local SQLite database. It pulls the data from the database in a format that’s very specific to the GPT Pilot database structure, but it can easily be modified to fit other structures. It reads and uploads your SQLite database, splits the rows by a specific field, unpacks the rows into values, loads the LLM conversation data into a form, and enables you to simply change one of the messages and submit the LLM request to GPT-4 to see how the results will look. This way, you can analyze the conversations GPT Pilot’s agents have with the LLM and easily explore what would happen if the prompts were different.

⏳ Time spent: ~2 days
💾 Github repo

Code Whisperer (DEMO)

Code Whisper is a fun project we created as an example to showcase. The idea is that you can use it to ask the LLM questions about your codebase. You paste in the link to a public Github repo. Then, it clones the repository, sends the relevant files to the LLM for analysis, which creates a description for each file about what the code does, and saves those descriptions into the database. After that, you can ask the app questions about the codebase, and the codebase shows you the response. In this demo, we use GPT-3.5.

⏳ Time spent: 7 hours
💾 Github repo

Star History (DEMO)

I’ve been releasing open-source projects for years now, and I’ve always wanted to see how fast my Github repo is growing compared to other successful repositories on https://star-history.com/. The problem is that on Star History, I’m unable to zoom into the graph, so a new repo that has 1,000 stars cannot be compared with a big repo that has 50,000 because you can’t see how the bigger repo does in its beginning. So, I asked GPT Pilot to build this functionality. It scrapes Github repos for stargazers, saves them into the database, plots them on a graph, and enables the graph to be zoomed in and out.

⏳ Time spent: 6 hours
💾 Github repo

Conclusion

I hope you gained some insight into the current state, problems, and findings that we deal with at GPT Pilot.

So, to recap:

We think that a real AI developer tool should be based on the following pillars. Human is needed to supervise the AI, we should enable the AI to iterate over its own mistakes, software development can be orchestrated, and we should aim to implement the orchestration layer on top of LLMs, and the AI developer should be able to refactor the codebase because, in reality, the coding process is not a straight line.

We think that a real AI developer tool should be based on the following pillars:

A human is needed to supervise the AI
We should enable the AI to iterate over its own mistakes
Software development can be orchestrated, which should be implemented on a layer on top of LLMs
**The AI developer should be able to refactor the codebase **because, in reality, the coding process is not a straight line

So far, we’ve learned that:

The initial app description is much more important than we thought
Coding is not a straight line, agents can review themselves
LLMs work best when they focus on one problem compared to multiple problems in a single prompt
Verbose logs do miracles
Splitting the codebase into smaller files helps a lot
For a human to be able to fix the code
They must be clearly shown what has been written and the idea behind it
Humans are lazy
It’s hard to get the LLM to think outside the box

What do you think? Have you noticed any of these behaviors in your interactions with LLMs, or do you have a different opinion on any of these?

I would be super happy to hear from you, so leave a comment below or shoot me an email at zvonimir@gpt-pilot.ai.

You can try creating an app with AI with GPT Pilot here:

If you liked this post, it would mean a lot if you star the GPT Pilot Github repo, and if you try it out, let us know what you think.

🤜💥🤛 GPT-4 vs Claude-2 context recall analysis

zvone187 — Tue, 05 Dec 2023 15:10:48 +0000

In today’s rapidly advancing AI world, one of the limiting factors of modern Large Language Models (LLMs) is the context size. But it would also be interesting to know how well the LLMs can use the context they have – their context recall, or the reliability with which the LLM can access information in its context.

To set the stage, the context is the data fed to the LLM for it to produce output, basically representing the LLM’s “working memory.” While there are techniques to work around the current size limitation – most notably Retrieval-Augumented Generation (RAG) – ultimately, all the relevant information about the task-at-hand must fit into the context.
Context sizes are improving, with the recent update of the GPT-4 model (gpt-4-1106-preview) bumping the context size to 128 thousand tokens and Claude 2 upgrading its context to 200 thousand tokens.

I’m working on an AI dev tool GPT Pilot that uses LLMs a lot. So, I was interested in context recall - however, it becomes more apparent at larger context sizes. In other words, how well can the LLM find the information it needs that is in the context? Less than ideal, as it turns out.

The context contest

I was interested in exactly how well this context recall works for different LLMs, specifically for GPT-3.5, GPT-4 and Claude. I constructed a context of the desired size with a piece of data buried inside it, asked the LLM to find it, and measured how often it succeeds.

This research follows the “haystack test” Greg Kamradt published when the update GPT-4 came out (twitter, code). That test provided useful insight into (the lack of) context recall performance. But it was performed on a very small sample test (limiting its statistical significance) and was initially limited to GPT-4 (he has since published an updated version that also uses Claude 2.1). Moreover, the test data consists of essays that were likely already used pretraining LLMs, and the results were evaluated by GPT-4, potentially introducing confounding variables into the mix.

To dive deeper, I wanted to measure pure context recall on random data never before seen by an LLM and measure it directly (as the probability of success). I also wanted to run the test in more iterations to achieve more statistically significant results. The results were surprising!

Methodology

In the test, I constructed an artificial data set – a randomly generated CSV file with two columns, “key” and ”value,” and as many rows that would fit into the context (minus some padding for prompt, query, and response so that total number of tokens is under the limit).

This was constructed for 8, 16, 32, 64, 96, 128, and 192 thousand tokens. The set was split into 5 equal parts (quintiles) of 20% size of the total CSV length:

Quintile 0: Near the start of the context
Quintile 1: In the first half of the context
Quintile 2: Around the middle of the context
Quintile 3: In the second half of the context
Quintile 4: Near the end of the context

I randomly chose a key from the target quintile and asked the LLM to find the corresponding value (from the entire set). This was repeated 30 times and then calculated the resulting score for that context size and quintile, as a percentage of correct responses (ie the correct value was found).

Results

As of writing this article, GPT (and especially GPT-4) is the undisputed champion of LLMs in terms of reasoning power. Let’s see how well it performs in terms of recall.

(Note for readers in a hurry: I put a handy chart comparing all the results at the bottom of this article).

GPT-3.5

GPT-3.5 performed poorly on the tests. While I didn't test on the 4k that it was originally built with, results for 8k context size didn’t perform very well, and using all 16k produced outright atrocious results.

GPT-4

GPT-4 was flawless on 8k context and preformed really well with 16k context. It was somewhat worse with 32k and 64k (roughly on par with gpt-3.5 on 8k), and rather poorly on 96k and 128k contexts.

Claude

Results for Claude were surprising. While it's understandable that Claude has at least somewhat different approach to solving the context problem, the graphs do tell rather different story than those of GPT series.

Claude 2 performed flawlessly on 8k, really well on 16k, 32k, 64k, and 96k contexts (on par with GPT-4 16k), and not too shabby on 192k! It was much slower than GPTs on large contexts, though. Unfortunately, I didn’t focus time the requests, but on large contexts, Claude seems several times slower than GPT-4 – like it was doing RAG or something else behind the scenes.

Claude Instant 1.2

As expected, Claude Instant did somewhat worse than both Claude 2 and GPT-4, but it was markedly better than GPT-3.5.

Synthetic data vs the real world

How should we interpret these results? For example, what exactly does 73% recall performance mean for us when using these models in the real world?

It’s important to remember these tests measure the absolute ability of LLM to (ideally) perfectly remember every little detail from a big data set. While it’s useful for us to be able to evaluate performance, in many uses, it’s not as big of an issue for a few reasons:

Real-world data is usually duplicated in one way or another (in other words, compressible), meaning it’s probably easier for an LLM to remember real-world data than purely random strings with very high entropy.
In the real world, if we want to look up the data as-is, we’d use a database, not a LLM. The context is a guide to the LLM on what to do and how to do it, not a trivia quiz.

In other words, these results show the hard performance limits on the context recall and are a useful guide when thinking about context sizes we want to employ in our use cases. But the real-world situation is both messier and more forgiving.

Anecdotally, in some of the use cases I looked at, the models gave okay results at sizes where they scored 75% or more on this test.

I also didn’t shoot for hard, statistically sound measurements (3-sigma confidence) because that would be measuring at higher precision (and at a much higher cost) than what’s really useful.

Future work

As noted, I intentionally used synthetic random data to measure the recall. Using real-word data would probably give a somewhat different results and would be an interesting followup study.

I also kept the conversation chain short: all the data was in one (system) message, and the user query was in the second message. It would be interesting to see if having the context split across multiple smaller messages impacts the performance in any way.

Finally, I also haven’t tested any open source LLMs. Most are limited to 4k context, so it wouldn’t be a fair comparison. However, it would be interesting to see a comparison of the leading open source LLMs regarding context recall performance.

Conclusions

With the above caveats out of the way, who won the context contest?

Based on the context limit alone, Claude 2 is the winner, followed by GPT-4. When using small context sizes (relative to what the models suggest), both models perform really well.

Unless you’re dealing with small data that can comfortably fit inside a 4k context size, my recommendation is that you avoid GPT-3.5 and Claude Instant 1.2.

That’s it for this post - I hope you find this insightful. If you have a different experience with any of these LLMs, let me know what you found out.

🌟🌟🌟

Also, it would mean A LOT if you check out GPT Pilot. We’re working on a dev tool that tries to offload 90+% of coding tasks from the developer to the LLM. It’s completely open source so if you star the Github repo, it would mean a lot to us. Thank you 🙏

🌟🌟🌟

GPT Pilot - a dev tool that writes 95% of coding tasks [Part 2/3 - Coding Workflow]

zvone187 — Tue, 03 Oct 2023 15:19:22 +0000

This is the second blog post in a 3-part series where I explain we created GPT Pilot – the AI coding agent that’s designed to work at scale and build production-ready apps with a developer’s help. In part #1 of this series, I discussed the high-level overview of GPT Pilot. The idea is that AI can now do 95% of all coding that we, developers, are doing. See how I used ChatGPT to code out an entire Redis proxy in 2 hours, which would usually take 20-30 developer hours. However, an app is of no use if it doesn’t fully work or solve the user’s problem. So, until real AGI arrives, you need a developer.

So, this is how GPT Pilot came to life. It is designed to do 95% of the required coding and asks developers for reviews, such as when it becomes stuck and cannot move forward or needs something outside the app like an API key.

In this post, I walk you through the entire process GPT Pilot goes through when coding an app. I share diagrams to provide a visual representation of everything that’s going on behind the scenes in GPT Pilot. I’m a visual person, so I always create diagrams. To understand how GPT Pilot’s coding works, there are 3 concepts – context rewinding, recursive conversations, and TDD. See my introduction where I described in them in part #1 of this series.

The GPT Pilot coding workflow contains 8 steps:

Take the next development task in line
Break down the task into development steps
Take the next development step
Fetching of currently implemented code
Write code for the current step
Run the code or a command
Test the new code changes
Debug the development step or go to the next step

Coding workflow is my favorite part of GPT Pilot so let's dive in. Here is a diagram of how it looks like visually:

#1 Task breakdown

Two important concepts will be mentioned throughout this blog post – development tasks and development steps.

GPT Pilot works in a way that, after breaking down the specifications for developing an app, it creates development tasks that will lead to a fully working app. Development tasks are basically high-level descriptions of what needs to be done that a developer will take and start implementing. Think of them as tasks in Jira (btw, I hate Jira…not sure if anyone relates, but I just wanted to let it out of my system). Here is an example of a development task:

In the diagram above, you see 3 task properties:

description: what needs to be implemented to fulfill this task
user_review_goal: how can the lead developer determine if the current task is finished – a crucial pillar of GPT Pilot is a developer must be involved throughout the coding process so that you can ensure the development process is going as planned and understand the codebase along the way
programmatic_goal: the kind of automated test GPT Pilot should write to test if this entire development task works as expected. After a development step, GPT Pilot writes unit tests, and after a development task, it writes integration or E2E tests.

Now, when you start developing a task from Jira (development task), you will split it into smaller chunks (we call them development steps) that are actionable items you would set out to implement into the codebase. Each development step can be one of the following:

Command run – a command that needs to be run on the machine, such as a command to install a dependency, start the app to check if the previously implemented steps work, or create a folder.
Code change – the most important development step that explains what exactly needs to be implemented in the actual code to fulfill the current step. It can contain new code that needs to be written or code that needs to be changed. The way it works is the code change is a detailed, human-readable description of what needs to be implemented. It contains both the code that needs to be implemented and the description of what it is being used for. This is very similar to when you ask ChatGPT to code something. It will give you the code as well as the explanations of why it wrote that code.
- The reason for this is that code implementation is not so simple. Sometimes, we need to add a snippet into existing code or change the existing implementation. That is why we separated the outline of the coding change (this development task) and the actual implementation of this change that the CodeMonkey agent is dedicated to. I will go deeper into that in the #3 Coding section. Here is an example of a code change:
Human intervention – a development step that AI cannot do by itself and needs human help in fulfilling the step. Then, GPT Pilot asks the developer to do something, and when he/she is done, they write “continue,” and GPT Pilot will continue with the implementation. Here are some reasons why human intervention might be needed:

There is a needed API key (e.g., Twitter API key to fetch data from Twitter)
GPT Pilot became stuck in a debugging process, and it either filled the entire context length or the recursion conversation was too deep and became unproductive to continue down the recursion depth.
GPT Pilot needs a verification if something works as expected – e.g., GPT Pilot is not sure if Mongo is installed properly on the machine and might ask the developer to run some sudo commands and see if it works as expected.

#2 Fetching of currently implemented code

It’s easy for AI to write a new file that contains code, but in reality, that is rarely the case. For the most part, we write into the existing files and either change the existing code or add new code. Now, AI can do this easily if you give it all of the existing code and instructions for what needs to be implemented. The problem arises when an app scales and the codebase becomes so large that it cannot fit into the LLM context. And this is actually a very common case – at least until we have LLMs with 1M tokens, which doesn’t seem to be coming soon.

When you work on a task in a big codebase, you usually look at a smaller part of the codebase (maybe 1,000 lines) and work only with that subset of code to implement the task.

So, to address this issue and make GPT Pilot truly scalable so that it can create and upgrade large production-ready codebases, we must create a way for the AI to select the smaller part of the codebase (e.g., those 1,000 lines) on which it will implement the current task. Once it’s finished, we can simply add the finished lines back into the original codebase. Let me start explaining this by telling you what happens when GPT Pilot writes code and creates new files and folders. For each file and folder it must create, it needs to write a description of what the idea is behind the file or folder it wants to create. For example, it might want to create a folder utils for which it will write:

Contains utility modules that provide generic, reusable solutions to common problems encountered throughout the application.
These utilities are not specific to the app's core domain but offer auxiliary functionality to support and streamline the primary codebase.
They encapsulate best practices, reduce code repetition, and make the overall code cleaner and easier to maintain. Examples include functions for data formatting, error handling, debugging tools, string manipulation, data validation, and other shared operations that don't fit within specific modules or components of the app.

Now, for each function GPT Pilot creates, it writes a description of what the function is supposed to do – that is a pseudocode for the entire codebase.

Now that you know what happens when GPT Pilot writes code, you can understand how it fetches the relevant code for each development step.

Before GPT Pilot codes each step, it first fetches the relevant part of the codebase in a completely separate LLM conversation. That conversation is set up in 3 steps.

AI is given the development step description along with the entire project file/folder structure and descriptions for each file and folder. From this, LLM tells us which files are relevant for the mentioned step.
After narrowing down the necessary files, we give the LLM pseudocode for each file it listed and ask it to tell us which functions are relevant for the current development step.
Once we know the pseudocode it selected, we can fetch the actual code and put it into the original conversation, where the LLM will write the description of what needs to be implemented.

If the app becomes extremely huge, we can improve this by first giving the LLM the folders, from which it will select folders, and then we give it relevant files. Before each of these steps, we can also rewind the conversation to the beginning to leave more room in the context.

Here is a diagram of what this looks like:

#3 Coding

Now that we can create an LLM message that contains all code necessary for someone to implement a specific task, we can start with the actual coding process. This happens in a 2-part process:

First, the LLM writes the description of what needs to be implemented along with the code. If the entire file needs to be coded, LLM’s response will contain all the code, but if only a part of the code inside a file needs to be changed, LLM will tell us things like After the Mongo setup, add the following lines of code... As you can imagine, by this being stochastic rather than deterministic, we need to ensure that the written code is inserted into appropriate places or changed correctly.

Here is where the CodeMonkey agent steps in. It is called code monkey because it doesn’t make any decisions but rather simply implements the code that the Developer agent writes. It is given the code relevant for the current task (that is previously selected by LLM in the code-fetching phase) and the description that the Developer agent created in development step #1. Then, the only thing it needs to return are the completely coded sections/files that we can just insert/replace in the codebase.

#4 Testing

There are 2 places where testing is done – (1) after each development task when GPT Pilot creates integration tests that test if the high-level features work as intended and (2) after each development step when it creates smaller unit tests that ensure all functions work as expected.

GPT Pilot has 3 different types of test it can do:

Automated tests are the preferred way of testing a step or task because they will be used in a regression test suite so that GPT Pilot can be sure that new code changes don’t break old features. However, automated tests are not always the most optimal way to test new code.
Command run is a test where we run a specific command and give the output to the LLM, which then tells us if the implementation was successful. For example, we don’t need to create an automated test that will check if we can run an app with npm run start – for that, a simple command run is enough to check if we successfully set up our environment.
Human intervention is the final way to test the app, and it is needed whenever AI cannot test the implementation itself. This is needed, for example, when there are some visual aspects (e.g., CSS animations) that must be checked to see if they work correctly.

After running each test, if successful, GPT Pilot takes on the next task or step and continues with coding, but when the test fails, GPT Pilot needs to debug the error.

#5 Debugging

The debugging process needs to be so robust so that it can be started on any bug that arises, regardless of the error. It also needs to be able to debug any issue that happens during the debugging process. This is where recursive conversations come in, which are conversations with the LLM that are set up in a way that they can be used “recursively.”

Let’s look at the example in the image below. It represents a flow that GPT Pilot goes through when working on a development task that has 5 development steps. In this example, during the development of step #3, an error occurs – let’s say it implements a specific code change but after running a test, it fails. Then, it goes into the recursion level #1 to debug this issue. It breaks down what needs to be done to fix this issue into 2 steps, but during the implementation of the first step, another error happens. For example, a needed dependency for fixing the error #1 doesn’t exist. GPT Pilot then goes into the recursion level #2, which it breaks down into 3 steps. In the third step, another error occurs. Then, it goes to the third recursion level, which has only 1 step. Once that step is successfully executed, GPT Pilot goes back to the recursion level #2 and finishes debugging error #2. After that, it goes back to debugging error #1, and finally, after error #1 is fixed, it goes back to the development step #3 after which it continues the app implementation.

When the recursions go 5 levels deep, GPT Pilot will stop the debugging process and ask the developer to fix the initial issue it started with. Once the developer resolves this issue, they write the results to GPT Pilot. Then, it can continue the development process as if it debugged the issue itself.

Conclusion

In the first post of this series, I discussed the high-level overview of how GPT Pilot works. In this post, I described the GPT Pilot Coding Workflow, including:

How Developer and CodeMonkey agents work together to implement code (write new files or update existing ones),
How recursive conversations and context rewinding work in practice, and
Rewinding the app development process and restoring it from any development step.

In the final post, I will dive deep into how all the agents are structured. We built the agents modularly because we know they will evolve over time. Please head over to GitHub, clone the GPT Pilot repository, experiment with it, and send me your feedback. I want GPT Pilot to be as helpful to developers as possible, so let me know what you think, how it can be improved, or what works well. Add comments at the bottom this post or email me at zvonimir@gpt-pilot.ai.

🌟🌟🌟
Finally, we're trying to raise funds to continue developing GPT Pilot, so it would mean A LOT if you could star GPT Pilot Github repository and/or share it with your friends. Thank you 🙏
🌟🌟🌟

GPT Pilot - a dev tool that writes 95% of coding tasks

zvone187 — Wed, 06 Sep 2023 14:54:26 +0000

An MVP for a scalable dev tool that writes production-ready apps from scratch as the developer oversees the implementation

In this blog post, I will explain the tech behind GPT Pilot - a dev tool that uses GPT-4 to code an entire, production-ready app.

The main premise is that AI can now write most of the code for an app, even 95%.

That sounds great, right?

Well, an app won’t work unless all the code works completely. So, how do you make that happen? Well, this post is part of my research project to see if AI can really do 95% of developers' coding tasks. I decided to use GPT-4 to make a tool that writes scalable apps with the developer's oversight.

I will show you the main idea behind GPT Pilot, the crucial concepts it's built upon, and its workflow up until the coding part.

Currently, it’s in an early stage and can create only simple web apps. Still, I will cover the entire concept of how it can work at scale and demonstrate how much of the coding tasks AI can do while the developer acts as a tech lead overseeing the entire development process.

Here are some example apps that I created with it created:

Ok, let's dive in.

How does GPT Pilot work?

First, you enter a description of an app you want to build. Then, GPT Pilot works with an LLM (currently GPT-4) to clarify the app requirements, and finally, it writes the code. It uses many AI Agents that mimic the workflow of a development agency.

After you describe the app, the Product Owner Agent breaks down the business specifications and asks you questions to clear up any unclear areas.
Then, the Software Architect Agent breaks down the technical requirements and lists the technologies that will be used to build the app.
Then, the DevOps Agent sets up the environment on the machine based on the architecture.
Then, the Tech Team Lead Agent breaks down the app development process into development tasks where each task needs to have:
- Description of the task (this is the main description upon which the Developer agent will later create code)
- Description of automated tests that will need to be written so that GPT Pilot can follow TDD principles
- Description for human verification, which is basically how you, the human developer, can check if the task was successfully implemented
Finally, the Developer and the Code Monkey Agents take tasks one by one and start coding the app. The Developer breaks down each task into smaller steps, which are lower-level technical requirements that might not need to be reviewed by a human or tested with an automated test (eg. install some package).

In the next blog post, I will write in more detail about how Developer and Code Monkey work (here's a sneak peek diagram that shows the coding workflow), but now, let's see the main pillars upon which the GPT Pilot is built.

3 Main Pillars of GPT Pilot

I call these the pillars because, since this is a research project, I wanted to be reminded of them as I work on GPT Pilot. I want to explore what's the most that AI can do to boost developers' productivity, so all improvements I make need to lead to that goal and not create something that writes simple, fully working apps but doesn't work at scale.

Pillar #1. Developer needs to be involved in the process of app creation

As I mentioned above, I think that we are still far away from an LLM that can just be hooked up to a CLI and work by itself to create any app by itself. Nevertheless, GPT-4 works amazingly well when writing code. I use ChatGPT all the time to speed up my development process - especially when I need to work on some new technology or an API or if I need to create a standalone script. The first time I realized how powerful it can be was a couple of months ago when it took me 2 hours with ChatGPT to create a Redis proxy that would usually take 20 hours to develop from scratch. I wrote a whole post about that here.

So, to enable AI to generate a fully working app, we need to allow it to work closely with the developer who oversees the development process and acts as a tech team lead while AI writes most of the code. So, the developer needs to be able to change the code at any moment, and GPT Pilot needs to continue working with those changes (eg. add an API key or fix an issue if an AI gets stuck).

Here are the areas in which the developer can intervene in the development process:

After each development task is finished, the developer should review it and make sure it works as expected (this is the point where you would usually commit the latest changes)
After each failed test or command run - it might be easier for the developer to debug something (eg. if a port on your machine is reserved but the generated app is trying to use it - then you need to hardcode some other port)
If the AI doesn't have access to an external service - eg. in case you need to get and add an API key to the environment

Pillar #2. The app needs to be coded step by step

Let's say you want to create a simple app, and you know everything you need to code and have the entire architecture in your head. Even then, you won't code it out entirely, then run it for the first time and debug all the issues at once. Instead, you will split the app development into smaller tasks, implement one (like add routes), run it, debug, and then move on to the next task. This way, you can fix issues as they arise.

The same should be in the case when AI codes.

Like a human, it will make mistakes for sure, so for it to have an easier time debugging and for the developer to understand what is happening in the generated code, the AI shouldn't just spit out the entire codebase at once. Instead, the app should be generated and debugged step by step just like a developer would do - eg. setup routes, add database connection, etc.

Other code generators like Smol Developer and GPT Engineer work in a way that you write a prompt about the app you want to build, they try coding out the entire app and give you the entire codebase at once. While AI is great, it's still far away from coding a fully working app from the first try so these tools give you the codebase that is really hard to get into and, more importantly, it's infinitely harder to debug.

I think that if GPT Pilot creates the app step by step, both AI and the developer overseeing it will be able to fix issues more easily, and the entire development process will flow much more smoothly.

Pillar #3. GPT Pilot needs to be scalable

GPT Pilot has to be able to create large production-ready apps and not only on small apps where the entire codebase can fit into the LLM context. The problem is that all learning that an LLM has is done in-context. Maybe one day, the LLM could be fine-tuned for each specific project, but right now, it seems like that would be a very slow and redundant process.

The way that GPT Pilot addresses this issue is with context rewinding, recursive conversations, and TDD.

Context rewinding

The idea behind context rewinding is relatively simple - for solving each development task, the context size of the first message to the LLM has to be relatively the same. For example, the context size of the first LLM message while implementing development task #5 has to be more or less the same as the first message while developing task #50. Because of this, the conversation needs to be rewound to the first message upon each task.

For GPT Pilot to solve tasks #5 and #50 in the same way, it has to understand what has been coded so far along with the business context behind all code that's currently written so that it can create the new code only for the task that it's currently solving and not rewrite the entire app.

I'll go deeper into this concept in the next blog post, but essentially, when GPT Pilot creates code, it makes the pseudocode for each code block that it writes, as well as descriptions for each file and folder it needs to create. So, when we need to implement each task, in a separate conversation, we show the LLM the current folder/file structure; it selects only the code that is relevant to the current task, and then, we add only that code to the original conversation that will write the actual implementation of the task.

Recursive conversations

Recursive conversations are conversations with the LLM that are set up in a way that they can be used "recursively". For example, if GPT Pilot detects an error, it needs to debug it, but let's say that another error happens during the debugging process. Then, GPT Pilot needs to stop debugging the first issue, fix the second one, and then get back to fixing the first issue. This is a crucial concept that, I believe, needs to work to make AI build large and scalable apps. It works by rewinding the context and explaining each error in the recursion separately. Once the deepest level error is fixed, we move up in the recursion and continue fixing errors until the entire recursion is completed.

TDD (Test Driven Development)

For GPT Pilot to scale the codebase, improve it, change requirements, and add new features, it needs to be able to create new code without breaking previously written code. There is no better way to do this than working with TDD methodology. For all code that GPT Pilot writes, it needs to write tests that check if the code works as intended so that whenever new changes are made, all regression tests can be run to check if anything breaks.

I'll go deeper into these three concepts in the next blog post, in which I'll break down the entire development process of GPT Pilot.

Next up

In this first blog post, I discussed the high-level overview of how GPT Pilot works. In posts 2 and 3, I show you:

How do Developer and Code Monkey agents work together to implement code (write new files or update existing ones).
How recursive conversations and context rewinding work in practice.
How can we rewind the app development process and restore it from any development step.
How are all agents structured - As you might've noticed, there are GPT Pilot agents, and some might be redundant right now, but I think these agents will evolve over time, so I wanted to build agents in a modular way, just like the regular code.

But while you are waiting, head over to GitHub, clone GPT Pilot repository, and experiment with it. Let me know if you are successful, and while you’re there, don’t forget to star the repo - it would mean a lot to me.

Thank you for reading 🙏, and I’ll see you in the next post!

If you have any feedback or ideas, please let me know in the comments or email me at zvonimir@pythagora.ai, and if you want to get notified when the next blog post is out, you can add your email here.

How to kickstart automated test suite when there are 0 tests written and the codebase is already huge

zvone187 — Thu, 29 Jun 2023 14:07:09 +0000

You know, there’s a certain anxiety that creeps in when you take a look at a codebase you’re just starting to work on, only to realize it’s a vast uncharted wilderness. Not a single test has been written to guard against the unexpected. It’s like walking on a tightrope over a chasm, knowing a single misstep could send your entire project plummeting into chaos.

If you worked on a codebase with 0 tests, you know that it can be a daunting task to think about covering the entire codebase with tests from scratch where none currently exist. The process demands an almost Herculean effort: you’d have to pour over every function, method, and component, brainstorm all potential edge cases, structure the test suite code, and get it all running smoothly. And that’s not even touching on the time it takes to reach meaningful coverage. We’re talking weeks, perhaps months, before you can sit back and say, “Yes, we’ve hit 80% or 90% coverage.”

This is why I’m excited to share what I’ve been working on for the past couple of months. This journey takes us to a place where the realm of automated testing meets the magical world of AI. Meet Pythagora, an open-source dev tool that’s about to become your new best friend.

Throughout this blog post, I’m going to show you how to kickstart automated testing with Pythagora which harnesses the power of AI to generate tests for your entire codebase, all with a single CLI command, and hopefully get your codebase to 80% – 90% code coverage in a single day.

Creating a test suite from scratch

We all know the saying, “Rome wasn’t built in a day”. The same could be said for a comprehensive, effective test suite. It’s a meticulous, demanding process, but once you’ve traversed this rocky road, the sense of accomplishment is profound. Let’s journey together through the necessary steps involved in creating a test suite from scratch and reaching that coveted 80% – 90% code coverage.

Laying the groundwork

In the first stage, you’re like a painter in front of a blank canvas. The world is full of possibilities, and you’re free to create a masterpiece. Your masterpiece, in this case, involves choosing the types of tests you want to write, finding the right testing framework to use, and adopting the best practices suited for your specific environment. Are you considering unit tests, integration tests, E2E tests, or a blend of all three?

While this initial setup is often viewed as the “easy” part, it is by no means a walk in the park. Time, research, and perhaps a few cups of coffee are required to make informed decisions.

Diving into the details

Once you’ve got your basic structure in place, it’s time to roll up your sleeves and delve deep into the nitty-gritty. Now, you’ll need to go through your entire codebase, one function at a time, and write tests for each. Your task here is to ensure that your tests touch all lines of code within each function, method, or component. This task is akin to exploring an intricate labyrinth. You need to traverse every path, turn every corner, and ensure no stone is left unturned.

Writing these tests is a detailed, time-intensive step. It’s not just about writing a few lines of code; it’s about understanding the function’s purpose, its expected output, and how it interacts within your application.

Exploring the edge cases

After the initial round of testing, you might breathe a sigh of relief. Hold on, though; there’s still an important piece of the puzzle left. It’s time to dive into the wild, unpredictable world of edge cases. This part might not increase your code coverage percentage, but it’s crucial in testing the robustness and resilience of your code.

These so-called negative tests help evaluate how your code reacts to various inputs, particularly those on the fringes of expected behavior. From empty inputs to values that push the limits of your data types, these tests are designed to mimic user behavior in the real world, where users often have a knack for pushing your code in directions you never thought possible.

Creating a test suite from scratch is a Herculean task. But rest assured, every effort you put in is a step towards creating a more robust, reliable, and resilient application. And remember, you’re not alone. We’ve all been there, and with a tool like Pythagora, the journey is not as daunting as it may seem.

Generating Tests with one CLI command

On the other hand, with Pythagora, what you can do is enter:

npx pythagora --unit-tests --path ./path/to/repo

Pythagora will navigate through all files in all folders, conjuring up unit tests for each function it encounters. Now, you can sit back and relax or go grab lunch and let it run for a while until it finishes writing tests.

Ok, but wait, what the hell is Pythagora??

What is Pythagora

I’ve always dreamed of a world where automated tests could be created for me. But the reality isn’t that simple. No one knows your code quite like you do, making it challenging for another to draft effective automated tests for it. The results often fall short of what you’d achieve yourself.

However, everything changed when ChatGPT entered the scene. As I tinkered with this technology, I found myself wondering, “Could we harness the power of ChatGPT for writing automated tests?” Curiosity piqued, I delved deeper, experimenting with its capabilities, and what I discovered blew me away. ChatGPT demonstrated an incredible ability to comprehend code, offering a glimpse of a promising new avenue in automated testing.

And thus, an idea for Pythagora was born.

Pythagora is an open-source dev tool, crafted with one mission in mind: making automated testing autonomous. I envision a world where developers, such as you and me, can focus on creating features without getting bogged down in the mire of test writing and maintenance. To achieve this vision, it’s using GPT-4.

Currently, Pythagora has the prowess to write both unit and integration tests. However, for the purposes of this blog post, we’ll concentrate on its ability to generate unit tests.

Installation

To install Pythagora, you just need to do npm i pythagora. That’s it! Pythagora is now at your service.

Configuration

Once Pythagora is installed, you’ll need to configure it with an API key. This can either be an OpenAI API key or a Pythagora API key.

To use an OpenAI API key, you should run the following command:

npx pythagora --config --openai-api-key <OPENAI_API_KEY>

It’s important to note that, if you choose to use your own OpenAI API key, you must have access to GPT-4.

Alternatively, you can obtain a Pythagora API key from this link. Once you have it, set it up with the following command:

npx pythagora --config --pythagora-api-key <PYTHAGORA_API_KEY>

Commands

If you prefer to generate tests for a specific file, use:

npx pythagora --unit-tests --path ./path/to/file.js

And if you have a particular function in mind, use:

npx pythagora --unit-tests --func <FUNCTION_NAME>

How does Pythagora work

Let’s peel back the curtain and take a peek into the engine room. What makes Pythagora tick?

At its core, Pythagora functions as an intrepid explorer, delving into the intricate labyrinth of your codebase. First, it maps all functions that are exported from your files so that it can call them from within the tests. Obviously, if a function is not exported, it cannot be called from the outside of its file. Btw, after generating tests a couple of times, it will make you think about your codebase and how can you structure it better so that more tests can be generated.

Once it identifies the exported functions, Pythagora takes another step into the rabbit hole: it investigates each function in turn, hunting down any additional functions called within. Picture it as the archaeologist of your codebase, gently brushing away layers of dust to expose the hidden connections and dependencies. In other words, it looks for all functions that are called from within the function being tested so that GPT can get a better understanding of what does a function, for which the tests are being written for, do.

Armed with this information, Pythagora prepares to utilize the power of AI. It packages the collected code and dispatches it to the Pythagora API. Here, the actual magic happens: a prompt is meticulously crafted and handed over to the GPT model. This interaction between the code, the API, and the AI model results in generating a comprehensive set of unit tests, ready to be deployed and put to work.

Both the API server and the prompts used are open-source. They’re available for you to delve into, scrutinize, and even contribute to if you so desire. You can find the Pythagora API server here while the prompts, key ingredients in the creation of unit tests, are housed in this folder.

Reviewing Tests

Once Pythagora writes all requested tests, it’s time for you to jump in and start reviewing them. This is a vital step in the process; it’s important to know what has been created and ensure everything aligns with your expectations.

Remember, Pythagora creates Jest-based tests. So, to run all the generated tests, you can just run:

npx jest ./pythagora_tests/

Now, a word of caution: Pythagora is still in its early stages. As with all young projects, it’s bound to have some hiccups along the way. So, you might encounter failing tests in your initial runs. Don’t be disheartened; consider this a part of the journey. With your review and the continuous improvements to Pythagora, these failed tests will soon be a thing of the past.

And let’s not forget the bright side. Even with these early-stage teething problems, Pythagora can get you to a place where your codebase has a substantial, potentially up to 90%, test coverage.

Committing Tests

The review process, especially for larger codebases, may take a few hours. Remember, you’re not only looking at the tests that passed but also at those that failed. It’s crucial to understand every test you’re committing to your repository. Knowledge is power, after all.

After a thorough review and potential tweaks, you’re ready to make your final move: committing the generated tests to your repository. With this last step, you would have successfully integrated a robust unit test suite into your project. And all of this is achieved with the power of Pythagora and a few lines of command in your terminal.

Example Tests on Lodash Repo

Alright, now that I’ve got your interest piqued, let’s delve into the real stuff – tangible examples of Pythagora in action. For the purpose of our demonstration, we selected a well-known open source project, Lodash.

Running just one Pythagora command was enough to generate a whopping 1604 tests, achieving an impressive 91% code coverage of the entire Lodash repository. But it’s not just the quantity of tests that’s impressive. Out of these, 13 tests unearthed actual bugs within the Lodash master branch.

If you’re curious to check these out yourself, we’ve forked the Lodash repository and added the tests generated by Pythagora. Feel free to explore them here.

Now let’s take a closer look at one of the tests that caught a sneaky bug:

test(`size({ 'a': 1, 'b': 2, 'length': 9 })`, () => {
  expect(size({ 'a': 1, 'b': 2, 'length': 9 })).toBe(3); // test returns 9
});

In this test, the size function of Lodash is supposed to return the size of a JSON object. But, GPT added a key named length, a little trick to see if Lodash might return the value of that key instead of the true size of the object. It appears that Lodash fell for this ruse, as the test failed by returning ‘9’ instead of the expected ‘3’.

This is a fantastic example of how Pythagora, powered by GPT, excels at uncovering tricky edge cases that could easily slip under the radar. By generating a large number of such intricate test cases automatically, Pythagora can be your trusty sidekick, helping you discover and fix bugs you might never have anticipated.

Conclusion

Well, there we have it, fellow developers. We’ve embarked on quite a journey today, traversing through the uncharted territories of a substantial codebase devoid of tests, and returning with an automated suite of tests crafted by our trusty AI-powered tool, Pythagora.

You’ve learned that even in the face of a daunting, test-less codebase, there’s no need for despair. The task of creating a substantial suite of tests need not be an uphill slog anymore. We’ve witnessed the magic of Pythagora as it examined a well-known open-source library, Lodash, and generated 1604 tests that covered a jaw-dropping 91% of the codebase.

We saw how Pythagora isn’t just about quantity, but also the quality of tests. It isn’t just creating tests for the sake of it, but intelligently finding edge cases and bugs that may have otherwise slipped through unnoticed. Pythagora unmasked 13 real bugs in the Lodash master branch – a testament to the power of AI in software testing.

Now, you should have a clearer understanding of why AI-powered testing tools like Pythagora are not just a luxury, but a necessity in today’s fast-paced development landscape.

So whether you’re dealing with an existing project with zero tests or starting a new one and looking to establish a solid testing framework from the outset, remember that you’re not alone. Pythagora is here to take the reins, helping you generate meaningful tests with ease, and saving you valuable time that can be better spent on developing great features.

Thank you for joining me on this journey, and I can’t wait to see how you utilize Pythagora in your projects. Happy coding!

P.S. If you found this post helpful, it would mean a lot to me if you starred the Pythagora Github repo and if you try Pythagora out, please let us know how it went on hi@pythagora.ai.

How to create an automated test suite - get from 0 to 90% code coverage with a single command and GPT-4

zvone187 — Tue, 13 Jun 2023 14:33:22 +0000

Creating a test suite from scratch can certainly seem like a daunting task, especially if you have an already-developed codebase with no tests. However, it is crucial to implement automated tests if you are looking to scale up your software project.

In this blog post, you'll see how to use the power of GPT-4 to create your test suite from scratch in just a couple of minutes with Pythagora. You can achieve code coverage of 90%; basically, with a single command.

If you are familiar with unit tests and other types of automated tests, feel free to skip to How to kickstart a test suite to see how to generate the entire test suite for your repo.

What type of tests to start with?

There are many types of tests while the main types are: unit tests, integration tests, and end-to-end (E2E) tests.

Unit tests: simplest type of tests. They usually test just a single function. However, they are the fastest type of tests and they can show exactly where the bug is in contrast to E2E tests that in most cases don’t show the location of the bug. Since they are the fastest type of tests, you can run them with a watcher (upon code changes) so you get feedback super fast.
Integration tests: these are the tests that test a bigger part of the codebase, usually multiple functions that are combined in some way. What they provide is a real world scenario how the codebase behaves in contrast to unit tests that test just functions in isolation from the rest of the system. They are slower than unit tests but are still fast so that a developer can run them before a commit or upon a bigger change in the codebase.
E2E (end-to-end): they mimic the user behaviour and test the entire system from the user action all the way until the reaction to the user action in the UI. That means that they test the frontend, the backend, the database and everything in between. Their main downside is that they are very slow to run since they need the entire UI spun up (eg. headless browser or a mobile app emulator) so they usually run within the CICD pipeline before deploying the code to production and not on the local machine of a developer

What tests should you write and in what percentage comparing to other types is still heavily debated topic. Some say that there should be unit tests the most, then integration tests and the fewest E2E tests. This is called Testing Pyramid - you can read more about it here. In that light, we’ll focus on creating unit tests since the are the simplest to make but still provide many benefits.

What to consider when writing automated tests?

If you have never written tests, you may believe that automated tests are only used to verify if a part of your code works as expected. While this is indeed one of the primary goals of automated tests, it is definitely not the only one.

Documentation: Automated tests serve as hands-on documentation for your codebase. When a new developer starts working on a part of the codebase that they've never seen before, if there are automated tests for that part of the codebase, they can run the tests and understand how the code works.
Debugging: Tests, especially unit tests, are important in identifying where a bug is in the code. This is one of the reasons why having only end-to-end or integration tests is not enough. Usually, different tests are all run together, so end-to-end and integration tests test how the system works in the real world, whereas unit tests detect where the bug might be occurring and test different edge cases.

How to kickstart a test suite

Getting started is always the hardest part, especially when you have an entire codebase but no tests. If this is the case, you will likely have to write hundreds or thousands of tests. This is where Pythagora comes in. With it, you can run a single command, take a break for lunch, and return to a fully written test suite. However, this is not the end of the process, as you still need to spend a few hours reviewing the tests. Nonetheless, with Pythagora, you can have an entire test suite ready in just one day. Let's see how it works.

1. First, install Pythagora with:

npm i pythagora

2. You will need to either have OpenAI API key (with GPT-4 access) or Pythagora API key that you can get here. When you have one of these, add it to config with:

npx pythagora --config --pythagora-api-key <PYTHAGORA_API_KEY>

npx pythagora --config --openai-api-key <OPENAI_API_KEY>

3. Finally, run the following command from the root of your repo:

npx pythagora --unit-tests --path ./

That's all for now. You can take a break and return once the test generation is complete. As a rough estimate, generating 100 tests will take approximately 30 minutes, so depending on the size of your project, it may take a bit longer.

Once you have the tests, it's important to review them. I recommend starting with the failed tests. You can run all of the generated tests with the following command:

npx jest ./pythagora_tests

This command will run all tests saved in the pythagora_tests folder, where Pythagora stores them. Check any failed tests to determine whether the failure is due to a syntax mistake (which GPT sometimes makes) or a bug in the code.

After reviewing any failed tests, evaluate all tests to determine whether they are meaningful enough to keep. It's not ideal to have too many tests so feel free to delete any that you think are not necessary.

Example repo

I gotta say, I was pretty surprised at how well GPT performed. I ran tests on a bunch of different repositories, and it was able to find some pretty tricky edge cases that would've been tough to find otherwise. The tests it generated were on point, and it found bugs right off the bat.

Here's a fork of the lodash repo where I ran the command above to generate tests with Pythagora. It took a minute (well, actually, like four hours), but the results were pretty impressive.

It generated 1604 tests that get code coverage to 90% and among those, GPT was able to catch 3 edge case bugs that might've flown under the radar otherwise. Now, those bugs might not seem like a big deal, but it's pretty impressive that GPT was able to catch them at all. Plus, it found 10 more regular bugs (thankfully, these are not in the live lodash version but are in the master branch).

Feel free to clone the demo repo and check out the tests yourself. If you want to see the tests in action, just do npm i and then run them as above with npx jest ./pythagora_tests.

Conclusion

In conclusion, in this blog post you saw how to kickstart your test suite from scratch, starting with understanding the importance of different types of tests to finally leveraging the power of Pythagora.

Pythagora, powered by GPT-4, helps automate the creation of your unit tests, and substantially improve your code coverage in a fraction of the time it would take manually. It's an exciting tool that has already proven its worth by catching edge case bugs in public repos like Lodash. If you're in the web dev field, I would encourage you to give Pythagora a try. With it, you can create an entire test suite with a single command.

If you found this post valuable, it would mean the world to me if you could support us by starring Pythagora Github repo.

And, if you try it out, please let me know what do you think. How do generated tests look like? Will you be committing the tests to your repo?

Generating negative tests with GPT-4

zvone187 — Thu, 20 Apr 2023 14:28:46 +0000

As developers, we strive to write efficient and error-free code, but let's face it - mistakes happen. In order to catch those pesky bugs before they wreak havoc on our applications, we rely on testing. While positive tests ensure our code works as intended, negative tests play a crucial role in validating that our applications are robust enough to handle unexpected input and edge cases. In this blog post, we'll explore the importance of negative tests and how we can leverage the power of GPT-4 to generate them.

I'll use Pythagora to generate positive test data which we will feed into GPT to create negative tests.

Negative tests

Negative tests, while sometimes overlooked, are an essential part of ensuring that our applications are robust and reliable.

So, what exactly are negative tests? These tests are designed to evaluate how well an application can handle unexpected inputs and conditions. They aim to make sure that the application won’t crash, produce incorrect results, or return errors when faced with inputs that deviate from what is expected. For example, if you make an API request to add -5 t-shirts to a cart. This looks quite irrelevant but exactly these kinds of requests will be the ones that crash your entire server.

The importance of negative tests cannot be overstated. They help us identify potential issues in our code, which might not be apparent during positive testing. By uncovering these issues, we can improve the performance, security, and overall quality of our applications, leading to increased customer satisfaction. In addition, negative tests can help us pinpoint areas of our code that may need improvement or refactoring.

Creating an extensive suite of negative tests can be quite time-consuming and requires a deep understanding of the application and its potential weak points. This is why the task often falls on the shoulders of the QA team. As expert bug hunters, they dedicate their time and attention to considering all the ways they can break the server and identify vulnerabilities.

Types of negative tests

Our primary focus here is on generating negative integration tests for APIs. To do this, we will manipulate various API request parameters, such as the request body and the URL path. By altering these values, we can create a range of negative test scenarios that can help us uncover potential issues in our applications.

When creating negative tests, it’s essential to consider the different values that could potentially cause problems. For instance, we might try using:

Empty or missing required fields
Invalid data types for fields (eg. number instead of a string and vice versa)
Invalid field values (e.g., exceeding character limits, malformed data)
Duplicated data in arrays
Extra or irrelevant keys in the payload
Potential security vulnerabilities (e.g., XSS, SQL injection)
Invalid or missing content-type headers
Missing or invalid authentication headers (e.g., Bearer token)
Incorrect data structure (e.g., array instead of an object, or vice versa)
JSON formatting issues (e.g., unmatched brackets, invalid Unicode characters)
I actually worked with GPT to list out a much bigger list of types of negative tests which I fully listed in this post.

By systematically exploring these variations, we can identify potential weak points in our application and ensure that it can handle unexpected input and conditions gracefully.

Now, imagine having 50 endpoints and going through all of these trying to create hundreds and hundreds of negative tests for all of them.

Thankfully, GPT-4 can do quite an amazing job with this!

How can GPT help create negative tests

By using GPT-4, we can create a range of negative test scenarios for a given API request, saving time and effort typically spent on brainstorming and manually designing these tests.

For this experiment we’ll try creating negative tests that completely break the server. So, the idea is to create an API request that will make the server return the status code higher than 500.

If you think about it, we’ll need to send a whole bunch of API requests that might do this so we’re going for the volume here.

Since GPT API is not cheap or fast, we want GPT to send as optimized data as it can. Each Pythagora test has ~3000 tokens so we’d be paying 9 cents per test so to get 1000 tests generated, we’d need to pay $90. So, my idea is to get GPT to send us only a list of parameters we want to change and we’ll create function that will augment the existing test data with the negative value GPT responded.

Creating integration tests with Pythagora

Pythagora is super easy to use so, we’ll just take a minute to create the data for a couple of integration tests. You can simply install it with npm i pythagora and then just run the command which will capture the API request data.

npx pythagora --init-script "npm run start"

Now, I’ll just make a couple of API requests. I used the browser and clicked around so that my frontend code makes API requests. It literally took me 5 seconds to capture 7 integration tests.

Ok, now that I have those, I’ll use the Pythagora test data to send it to GPT so it can create negative tests.

Tuning GPT to generate negative test data

First, we’ll tune the GPT-4 model by providing it with a prompt and examples of an API request along with a corresponding negative test, similar to those mentioned earlier.

First, the system message:

You are a QA engineer and your main goal is to find ways to break the application you’re testing.

Then, the actual prompt:

I will give you an API request data and I want you to give me the negative test data. Negative test data is a JSON object with an array of possible values for parameters in the API request data that might break the server – they will be used to create negative tests.
For example, if an API request data looks like this:
{
    "endpoint": "/api/category/add",
    "httpMethod": "POST",
    "body": {
      "name": "sdcvdvc",
      "description": "ccbvc",
      "products": [
        "63ed3108b81ccc28a4d9eaf2"
      ],
      "headers": {
            "authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjY0MjU2ZDljNDA4NzgzMzE0NGQ3MWMxNiIsImlhdCI6MTY4MDE3NDU0MiwiZXhwIjoxNjgwNzc5MzQyfQ.hy-D-AdsWSuUg3f1CH03FBHQFMFtwmklRwDCckvbzHk",
        "accept": "application/json, text/plain, */*"
    }
You would return something like this:
{
  "body.name": [...],
  "body.description": [...],
  "body.products": [...],
  "headers.authorization": [...],
  "headers.accept": [...],
  "endpoint": [...],
  "httpMethod": [...]
}
Make sure that each array consists only of values that could break the server and, when you answer, that you don’t say anything else except the JSON object.

Also, here are types of negative tests to help you think:
— here I added the list from the blog post I showed above which is too big to paste here —

First, respond only with “Got it” to process this message.

GPT responded with “Got it“. So, In the next message, I sent it the test data:

Here is API request data:
{
  "endpoint": "/api/address/add",
  "method": "POST",
  "body": {
    "isDefault": true,
    "address": "asasd",
    "city": "asdasd",
    "state": "asdasd",
    "country": "asdasd",
    "zipCode": "12312"
  },
  "query": {},
  "params": {},
  "headers": {
    "content-length": "106",
    "authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjY0MjU2ZDljNDA4NzgzMzE0NGQ3MWMxNiIsImlhdCI6MTY4MDE3NDU0MiwiZXhwIjoxNjgwNzc5MzQyfQ.hy-D-AdsWSuUg3f1CH03FBHQFMFtwmklRwDCckvbzHk",
    "content-type": "application/json",
    "accept": "application/json, text/plain, */*",
    "accept-encoding": "gzip, deflate, br",
    "cookie": "Webstorm-a7738556=37894940-6565-40a7-9ec5-b3a59a3f74e6; connect.sid=s%3AYt5uQ-ZhdJwLJja_sd814I2iAKkGpwzM.B%2F%2Btxr1eUYCZhBOzqQOsBsbTR8j%2FZn%2BbD1VFiJBRzAc"
  }
}
Give the values for negative tests but make sure that you don’t give values that might NOT break the server but only something that might trigger an error. Don’t start the response with “json" and don't end it with "” but return ONLY the valid JSON. Focus on adding proper brackets and quotes so that the response is a valid JSON. Make sure that you respond with only a valid, executable JSON object and nothing else (eg. you can’t write [...] because that cannot be executed). Also, try to give, at least, 10 values for each parameter.

Finally, after this message, GPT responds with the actual result that we’re looking for:

{
  "endpoint": [
    "/api/address/invalid_id",
    "/api/naddress/64256deb4087833144d71c25",
    "/api/address/",
    "/api/address/64256deb4087833144d71%25c25",
    "/api/address?_id=64256deb4087833144d71c25",
    "/api/address/123",
    "/api/address/64256deb4087833144d;&71c25",
    "/api/address/null",
    "/apiaddress/64256deb4087833144d71c25",
    "/api/address/\"64256deb4087833144d71c25\""
  ],
  "method": [
    "POST",
    "DELETE",
    "OPTIONS",
    "GET",
    "CONNECT",
    "invalid_method",
    "HEAD",
    "TRACE",
    "12345",
    "null"
  ],
  "body.isDefault": [
    "true",
    "",
    "null",
    "&^%$#",
    "{\"prop\": \"value\"}",
    "[true, true]",
    "1",
    false,
    30,
    "undefined"
  ],
  "body._id": [
    "",
    null,
    "a74$3#5r",
    "123",
    "64256deb4087833144d*&71c25",
    "{\"id\": \"64256deb4087833144d71c25\"}",
    "[64256deb4087833144d71c25]",
    1234567890,
    "eyJpZCI6IjY0MjU2ZDljNDA4NzgzMzE0NGQ3MWMxNiIsImlhdCI6MTY4MDE3NDU0MiwiZXhwIjoxNjgwNzc5MzQyfQ.hy-D-AdsWSuUg3f1CH03FBHQFMFtwmklRwDCckvbzHk",
    "64256deb 4087833144d71c25"
  ],
  "body.address": [
    "",
    null,
    "a*&^%$#",
    "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliqua.",
    "{\"address\": \"123 Street\"}",
    "[asasdasd]",
    1234567890,
    "하늘을 날아올라 나무에 눈을 맞추다",
    "漢字仮名交じり文",
    "SELECT * FROM users;"
  ],
  "headers.content-length": [
    "",
    null,
    "&^%$#",
    -1,
    50000,
    "\"\"",
    "\"123\"",
    "[221]",
    "{\"content-length\": 221}",
    "abcdefghijklmnopqrstuvwxyz"
  ],
  "headers.authorization": [
    "",
    null,
    "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==",
    "Bearer",
    "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IiIsImlhdCI6MTY4MDE3NDU0MiwiZXhwIjoxNjgwNzc5MzQyfQ.hy-D-AdsWSuUg3f1CH03FBHQFMFtwmklRwDCckvbzHk",
    "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.invalid_base64.encode",
    "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjY0MjU2ZDljNDA4NzgzMzE0NGQ3MWMxNiIsImlhdCI6MTY4MDE3NDU0MiwiZXhwIjoxNjgwNzc5MzQyfQ==.hy-D-AdsWSuUg3f1CH03FBHQFMFtwmklRwDCckvbzHk",
    "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjY0MjU2ZDljNDA4NzgzMzE0NGQ3MWMxNiIsImlhdCI6MTY4MDE3NDU0MiwiZXhwIjoxNjgwNzc5MzQyfQ.",
    "apikey 1234567890",
    "Digest username=\"Mufasa\", realm=\"testrealm@host.com\", nonce=\"dcd98b7102dd2f0e8b11d0f600bfb0c093\", uri=\"/dir/index.html\", qop=auth, nc=00000001, cnonce=\"0a4f113b\", response=\"6629fae49393a05397450978507c4ef1\", opaque=\"5ccc069c403ebaf9f0171e9517f40e41\""
  ],
  "headers.content-type": [
    "",
    "application/jpg",
    "text/html",
    "multipart/form-data",
    "text/xml",
    "image/png",
    "video/mp4",
    "text/css",
    "application/octet-stream",
    "audio/mpeg"
  ],
  "headers.accept": [
    "",
    "invalid_accept",
    "image/jpeg",
    "text/html",
    "multipart/form-data",
    "text/xml",
    "image/png",
    "video/mp4",
    "text/css",
    "application/octet-stream"
  ],
  "headers.accept-encoding": [
    "",
    "UTF-8",
    "invalid_encoding",
    "br",
    "deflate",
    "compress",
    "gzip;q=0",
    "identity",
    "null",
    "0"
  ]
}

This gives me the data which I can then process and create one test from each of the items in these arrays. I created a short function that takes in the original test and then iterates through the data GPT responded with. From each of the values in all these arrays, the function creates an entire test so that we try every single value GPT responded with.

Results

After setting everything up and running our experiment, the results were quite impressive. With this approach, we were able to generate 2001 tests without any manual effort. By using the power of GPT-4 with Pythagora, we were able to quickly create a comprehensive suite of negative tests for our API server.

Upon analyzing the generated tests, I discovered that there were three endpoints in the MERN e-commerce Github repo that could potentially break it. The reasons for the errors were:

Sending an array instead of an object for an added address
Sending emojis as a password field (this was pretty interesting one)
Sending an invalid Mongo object id in the URL

It’s worth noting that the process of generating these tests took around 30 minutes to generate all 2001 tests. But, you know – if you start the process before taking a break or moving on to another task, and when you return, you’ll have a vast array of tests ready to help you uncover potential issues in your system. So, I was pretty satisfied.

As for the cost, generating these tests using GPT-4 amounted to $1.5 in OpenAI tokens. Considering the amount of time and effort saved by automating the creation of these negative tests, this is quite reasonable IMO.

Try it yourself

Finally, if you want to try it out yourself, you can open negative_tests branch on the Pythagora Github repo. We didn’t want to put it to main until we test it out thoroughly.

Everything here is completely open sourced so you will have to use your own OpenAI API key. Make sure that you have OPENAI_API_KEY variable in your config. It will be used from the process environment variables like this:

const configuration = await new Configuration({
    apiKey: process.env.OPENAI_API_KEY,
});

Conclusion

This was a pretty cool thing to explore for me personally. You now understand how GPT-4 can be used to create negative integration tests for your API server by systematically altering various parameters and generating a wide range of test scenarios.

I introduced Pythagora, an open-source tool that captures server activity and generates automated integration tests, which we then sent to GPT-4 to create negative tests.

I’m working on Pythagora and I’m hoping to help developers spend less time on writing tests and more on creating the core codebase. It would mean the world to me if you could show support by starring the Pythagora Github repo.

If you try Pythagora, I’d be happy to hear your thoughts on it and, of course, help you out if you get stuck somewhere. Just let me know at zvonimir@pythagora.ai.

45 ways to break an API server (negative tests with examples)

zvone187 — Wed, 19 Apr 2023 13:12:16 +0000

As developers, we strive to write error-free code, but no one actually does so because...well, bugs. In order to catch those pesky bugs before they wreak havoc on our applications, we rely on automated testing. While positive tests ensure our code works as intended, negative tests play a crucial role in validating that our applications are robust enough to handle unexpected input and edge cases.

I'm working on Pythagora, an open source tool that writes automated integration tests by itself (well, with a bit of help from GPT-4) without you, the dev, having to write a single line of code. Basically, you can get from 0 to 80% code code coverage within 30 minutes (video).

We just created a feature that automatically generates negative tests from the entire test suite with a single command. While building that feature, I researched what are different ways one can break an API server to test if it handles errors gracefully so here it is - a comprehensive list of ways with which you can break your server, if it doesn't handle errors properly.

1. Empty or missing required fields

{
    "endpoint": "/api/users",
    "body": {
        "username": "",
        "email": ""
    },
    "method": "POST"
}

2. Invalid field values - exceeding character limits

{
    "endpoint": "/api/users",
    "body": {
        "username": "ThisIsAnIncrediblyLongUsernameThatExceedsTheCharacterLimit"
    },
    "method": "POST"
}

3. Invalid field values - malformed data

{
    "endpoint": "/api/users",
    "body": {
        "email": "invalid-email@"
    },
    "method": "POST"
}

4. Extra or irrelevant keys in the payload

{
    "endpoint": "/api/users",
    "body": {
        "username": "validuser",
        "extra_key": "irrelevant_value"
    },
    "method": "POST"
}

5. Incorrect or invalid HTTP methods

{
    "endpoint": "/api/users/123",
    "body": {},
    "method": "POST"
}

6. Invalid endpoint paths

{
    "endpoint": "/api/nonexistent_endpoint",
    "body": {},
    "method": "GET"
}

7. Query parameters instead of using the request body in POST requests

{
    "endpoint": "/api/users?username=testuser",
    "body": {},
    "method": "POST"
}

8. Missing or invalid authentication headers (e.g., API keys)

{
    "endpoint": "/api/users",
    "body": {},
    "method": "GET",
    "headers": {
        "Authorization": "Invalid API_KEY"
    }
}

9. Incorrect data structure - array instead of an object

{
    "endpoint": "/api/users",
    "body": [
        "username": "testuser",
        "email": "test@example.com"
    ],
    "method": "POST"
}

10. Incorrect data structure - object instead of an array

{
    "endpoint": "/api/users",
    "body": {
        "users": {
            "username": "testuser",
            "email": "test@example.com"
        }
    },
    "method": "POST"
}

11. JSON formatting issues - invalid Unicode characters

{
    "endpoint": "/api/users",
    "body": {
        "username": "test\uFFFFuser"
    },
    "method": "POST"
}

12. Duplicate keys in the payload

{
    "endpoint": "/api/users",
    "body": {
        "username": "testuser",
        "username": "duplicate"
    },
    "method": "POST"
}

13. Invalid or unsupported content types (e.g., sending XML instead of JSON)

{
    "endpoint": "/api/users",
    "body": "<user><username>testuser</username><email>test@example.com</email></user>",
    "method": "POST",
    "headers": {
        "Content-Type": "application/xml"
    }
}

14. Exceeding payload size limits

{
    "endpoint": "/api/users",
    "body": {
        "large_data": "A very large data string that exceeds the server's payload size limit..."
    },
    "method": "POST"
}

15. Invalid or expired authentication tokens

{
    "endpoint": "/api/users",
    "body": {},
    "method": "GET",
    "headers": {
        "Authorization": "Bearer expired_token"
    }
}

16. Using special characters in field values

{
    "endpoint": "/api/users",
    "body": {
        "username": "test!@#$%^&*()-user"
    },
    "method": "POST"
}

17. Sending nested objects instead of simple key-value pairs

{
    "endpoint": "/api/users",
    "body": {
        "user": {
            "username": "testuser",
            "email": "test@example.com"
        }
    },
    "method": "POST"
}

18. Sending data in the wrong data type (e.g., string instead of integer)

{
    "endpoint": "/api/users",
    "body": {
        "age": "25"
    },
    "method": "POST"
}

19. Sending null values for required fields

{
    "endpoint": "/api/users",
    "body": {
        "username": null
    },
    "method": "POST"
}

20. Using reserved keywords in field names

{
    "endpoint": "/api/users",
    "body": {
        "class": "user"
    },
    "method": "POST"
}

21. Sending incomplete or malformed multipart file uploads

{
    "endpoint": "/api/upload",
    "body": {
        "file": "incomplete_file_data"
    },
    "method": "POST",
    "headers": {
        "Content-Type": "multipart/form-data"
    }
}

22. Incorrect or missing URL encoding for special characters

{
    "endpoint": "/api/users?username=test user",
    "body": {},
    "method": "GET"
}

23. Sending the request body in GET requests

{
    "endpoint": "/api/users",
    "body": {
        "username": "testuser"
    },
    "method": "GET"
}

24. Invalid date or time formats

{
    "endpoint": "/api/users",
    "body": {
        "birthdate": "01-25-1990"
    },
    "method": "POST"
}

25. Using non-ASCII characters in field names

{
    "endpoint": "/api/users",
    "body": {
        "üsername": "testuser"
    },
    "method": "POST"
}

26. Sending deeply nested objects

{
    "endpoint": "/api/users",
    "body": {
        "user": {
            "profile": {
                "details": {
                    "nested": "too_deep"
                }
            }
        }
    },
    "method": "POST"
}

27. Using non-printable or control characters in field values

{
    "endpoint": "/api/users",
    "body": {
        "username": "test\u0008user"
    },
    "method": "POST"
}

28. Sending the same field multiple times with different values

{
    "endpoint": "/api/users",
    "body": {
        "username": "testuser",
        "username": "different"
    },
    "method": "POST"
}

29. Missing or invalid content-length headers for request bodies

{
    "endpoint": "/api/users",
    "body": {
        "username": "testuser"
    },
    "method": "POST",
    "headers": {
        "Content-Length": "invalid"
    }
}

30. Using spaces or special characters in field names

{
    "endpoint": "/api/users",
    "body": {
        "user name": "testuser"
    },
    "method": "POST"
}

31. Sending invalid or malformed JSONP callbacks

{
    "endpoint": "/api/users?callback=invalid(callback)",
    "body": {},
    "method": "GET"
}

32. Sending the payload as a single string instead of key-value pairs

{
    "endpoint": "/api/users",
    "body": "username=testuser&email=test@example.com",
    "method": "POST"
}

33. Sending boolean values as strings (e.g., "true" instead of true)

{
    "endpoint": "/api/users",
    "body": {
        "active": "true"
    },
    "method": "POST"
}

34. Using non-standard HTTP methods (e.g., PATCH, CONNECT)

{
    "endpoint": "/api/users/123",
    "body": {
        "username": "updateduser"
    },
    "method": "PATCH"
}

35. Sending unsupported HTTP version numbers

{
    "endpoint": "/api/users",
    "body": {},
    "method": "GET",
    "httpVersion": "HTTP/3.0"
}

36. Sending multiple authentication headers (e.g., both API key and token)

{
    "endpoint": "/api/users",
    "body": {},
    "method": "GET",
    "headers": {
        "Authorization": "Bearer token_value",
        "API-Key": "api_key_value"
    }
}

37. Sending unnecessary or invalid CORS headers

{
    "endpoint": "/api/users",
    "body": {},
    "method": "GET",
    "headers": {
        "Access-Control-Allow-Origin": "*"
    }
}

38. Sending conflicting query parameters and request body data

{
    "endpoint": "/api/users?username=testuser",
    "body": {
        "username": "different_user"
    },
    "method": "POST"
}

39. Using non-standard characters in authentication header values

{
    "endpoint": "/api/users",
    "body": {},
    "method": "GET",
    "headers": {
        "Authorization": "Bearer t@ken_value"
    }
}

40. Sending negative numbers for fields that should only accept positive values

{
    "endpoint": "/api/users",
    "body": {
        "age": -25
    },
    "method": "POST"
}

41. Sending timestamps in the future or past beyond expected range

{
    "endpoint": "/api/users",
    "body": {
        "birthdate": "01-25-1800"
    },
    "method": "POST"
}

42. Using HTML, JavaScript, or SQL code in field values to attempt code injection

{
    "endpoint": "/api/users",
    "body": {
        "username": "<script>alert('test')</script>"
    },
    "method": "POST"
}

43. Using different character encodings in the payload (e.g., UTF-8, UTF-16)

{
    "endpoint": "/api/users",
    "body": {
        "username": "téstuser"
    },
    "method": "POST",
    "headers": {
        "Content-Type": "application/json; charset=UTF-16"
    }
}

44. Sending arrays with mixed data types

{
    "endpoint": "/api/users",
    "body": {
        "values": [1, "string", true]
    },
    "method": "POST"
}

45. Sending field values as arrays or objects instead of simple data types (e.g., string, number)

{
    "endpoint": "/api/users",
    "body": {
        "username": ["testuser"]
    },
    "method": "POST"
}

That's it. I hope this list gave you new ideas to test and protect your server.

If you found this post valuable, it would mean the world to me if you could support us by starring Pythagora Github repo.

And, if you try it out, please let us know your feedback, we're happy to hear it.

How Pythagora Reduces Debugging Time and Supercharges Your Development Workflow

zvone187 — Tue, 18 Apr 2023 14:19:12 +0000

As developers, we know that testing is a crucial part of any development process. However, writing tests can be time-consuming, accounting for 20-30% of a developer's time. Pythagora, an open-source NPM package, changes the game by automating tests through server activity analysis. In this blog post, we'll dive into how Pythagora can help you reduce debugging time, make your development process more efficient, and create more reliable applications.

What is Pythagora?

Pythagora is an NPM package designed to create automated tests for your Node.js applications by analyzing server activity. It records all requests to your app's endpoints, along with responses and server actions, such as Mongo and Redis queries. Pythagora during testing simulates the server conditions from the time when the request was captured, allowing for consistent and accurate testing across different environments.

Setting Up Pythagora

Integrating Pythagora into your Node.js app is as simple as installing the package:

npm install pythagora

Once installed, you’re ready to start recording your integration tests with no additional setup or configuration.

To capture test data, run Pythagora in capture mode:

npx pythagora --init-command "my start command" --mode capture

After capturing your desired requests, simply stop your server and tests should be ready to go. Change the mode parameter to –mode test to run the tests:

npx pythagora --init-command "my start command" --mode test

When running tests, Pythagora creates an ephemeral database to restore data before each test execution. This ensures that tests can be run on any machine or environment without affecting the original database.

Reducing Debugging Time

With Pythagora, you no longer need to spend countless hours writing tests, as the package automatically captures and creates tests for you. This not only saves time but also ensures that your tests are consistent.

Pythagora intelligently monitors your application and generates test cases based on its observations. This eliminates the need for manual test creation and ensures that your tests are comprehensive and up-to-date with your codebase. By automating this process, developers can focus on writing high-quality code rather than worrying about creating and maintaining tests.

Example of data that Pythagora captures for each API request
Reviewing Failed Tests with the --review Flag

The --review flag in Pythagora is a convenient feature designed to make reviewing and maintaining tests more efficient and straightforward. By running the

npx pythagora --review

command, developers can easily review failed tests from the previous run, one at a time. This feature provides an interactive way to analyze test failures and make informed decisions about how to handle the discrepancies.

Interactive Test Review

When the --review command is executed, Pythagora presents each failed test individually, highlighting the differences between the test result and the captured test data. This clear visualization of discrepancies enables developers to easily identify the cause of the failure and decide on the most appropriate course of action.

The --review feature offers developers several options for handling test discrepancies. They can:

Accept new changes as the “new valid test”: If the changes are determined to be correct and the test should be updated accordingly, developers can accept the new changes as the new baseline for the test.
Delete the test due to changed functionality: If the application’s functionality has changed and the test is no longer relevant, developers can choose to delete the test altogether.
Skip checking the test and move to the next one: If developers want to postpone addressing the failed test and continue reviewing other tests, they can choose to skip the current test and move on to the next one.
Get that test run command: If developer wants to rerun only that test for debugging purpose this option will print out Pythagora command to run that specific test.
Quit: If developer wants to stop review process he can simply press “q”.

Efficient Test Maintenance

The --review flag in Pythagora significantly improves the efficiency of test maintenance compared to other tools. By providing a clear and interactive way to review test failures and offering flexible options for handling discrepancies, developers can quickly and easily maintain and update their tests, ensuring their application remains robust and reliable.

Debugging specific test

When a test fails, you can rerun the specific test by adding the --test-id <TEST_ID> parameter to the command. This allows you to easily debug the test with all the data it uses, making it simple to identify and fix any issues.

npx pythagora --init-command "node backend/server.js" --mode test --test-id 1fa3171b-42d4-485e-8403-4620c0d2145c

Pythagora also generates detailed code coverage reports, giving you valuable insights into which portions of your code are tested and which are not. These reports help you identify areas where your test coverage may be lacking, allowing you to concentrate your efforts on improving those sections. By being aware of any gaps in test coverage, you can ensure a more robust and reliable application.

Conclusion

Pythagora is a very promising NPM package that automates testing and reduces debugging time for developers. By recording server activity and simulating server conditions, Pythagora enables accurate and consistent testing across different environments. With its easy installation and setup, as well as its time-saving capabilities, Pythagora might soon be a must-have tool for any Node.js developer looking to supercharge their development workflow, so keep an eye on it!

Give Pythagora a try and see how it can revolutionize your testing process. Don’t forget to star the Pythagora repo to show your support and appreciation!