Forem: Doctolib

Cracking the code: How Copilot supercharged my last CTF and where it fell short

Thomas Betous — Mon, 30 Jun 2025 12:02:18 +0000

Over the years, I’ve always been drawn to riddles and brainteasers. It’s no surprise, then, that as a software engineer, I’ve always been interested in Capture The Flag (CTF) cybersecurity challenges. In these challenges, you need to find a solution (hack) to retrieve a secret string hidden somewhere. This could be in a website, social media, assembly code, images, or any medium that can conceal information. These challenges require a broad range of knowledge, particularly in computer science and software engineering, but also creativity and inventiveness. However, I never dared to try because, in my mind, CTFs were reserved for the elite: the seasoned hackers with skills far beyond my own.

Now, at 35, life is busier than ever. I recently became a father, and finding time for new pursuits feels almost impossible. But the itch to finally try a CTF remained. My main obstacles? A lack of time and the belief that I didn’t have enough knowledge.

That’s when generative AI, like ChatGPT, Gemini, Github Copilot sparked an idea: what if I tried hacking with AI as my sidekick? To my surprise, not only were CTF challenges more accessible than I’d imagined, but AI assistants also helped me save precious time by accelerating my learning and problem-solving. Thanks to this partnership, I completed the 404 CTF — one of France’s most famous competitions — and finished in a respectable 165th place out of more than 2,800 participants.

In this article, I’ll share how I used AI to supercharge my hacking journey, the tips I picked up along the way, and where the limits of AI became clear. As a disclaimer before diving in, while I used GitHub Copilot for this article, other technologies like Cursor, Claude, and similar tools are equally valid choices. You can even use several of them simultaneously, the end goal remains the same.

💬 Prompts I used the most and how I used them

Explain me the purpose of XXX?

GitHub Copilot is an excellent assistant that integrates seamlessly with VS Code. Microsoft’s plugin provides a powerful interface where you can query specific lines of code or use a dedicated console to focus Copilot’s attention on particular files or directories.

Since most CTF challenges begin with analyzing files, Copilot proved to be a perfect investigation tool. My typical workflow starts by asking Copilot to analyze the project, explain its purpose, and map out the function of each file. Then, I examine each file in detail, and whenever I encounter something unfamiliar, I ask Copilot to explain specific sections.

"Copilot, could you explain the purpose of this project? Could you describe the function of each file and directory?" — prompt example

For instance, one web security challenge included an nginx configuration file. While I don't regularly work with nginx configurations, by asking Copilot to explain specific instructions, I could quickly understand their purpose without leaving VS Code. This allowed me to learn just enough to efficiently identify potential attack vectors.

An important consideration is that files aren't always immediately readable by Copilot (or any AI assistant). Sometimes you need to be creative with file formatting. For example, when challenges provide binary files, Copilot can't interpret them directly. The solution is to export the binary's hexadecimal or assembly code to a text file. This conversion makes the content accessible to Copilot for analysis. While this approach is valuable for initial investigation, sometime specialized tools are necessary for deeper analysis. In my last example, after the initial examination with Copilot, I had to switch to BinaryNinja for a more thorough investigation of the binary file.

Another interesting example involves analyzing a pcapng file containing network captures. The challenge was to discover how a malicious user extracted a password from numerous network packets. While this file wasn't readable in VS Code and required Wireshark for viewing, it contained an overwhelming amount of data, including many irrelevant packets. To effectively analyze such data with an AI assistant, you first need to filter out the relevant packets (in my case, HTTP packets) through Wireshark and then export them to a more comprehensible format like JSON. It's essential to perform this initial filtering step, as otherwise, the AI assistant won't be able to analyze the data effectively due to the sheer volume of information.

Do you see any security issues or unusual elements?

After analyzing the structure of a challenge, there are times when I don't immediately spot relevant issues. In such scenarios, I ask Copilot to identify potential security vulnerabilities. Often, Copilot suggests multiple issues that provide a good foundation for starting to hack. This is where your skills as a hacker/developer become crucial, you need to evaluate and filter these suggestions, identifying which are viable and which aren't. You also need to find synergies and discover exploits that Copilot didn't explicitly mention. Frequently, challenges require exploiting multiple security vulnerabilities in combination rather than just a single issue.

"Copilot, do you see any security issue or unusual elements in route.py? Do you see any deprecated dependencies?" — prompt example

For example, in one web security challenge from the 404CTF, solving it required combining multiple vulnerabilities that Copilot helped identify such as issues in both the backend and proxy cache server. Out of the various potential problems Copilot pointed out, I had to analyze, select, and connect the relevant ones to reach the solution and extract the flag. This experience shows how AI can accelerate the investigative process, but still requires human intuition to piece everything together.

Finally, when Copilot identifies vulnerabilities, the method of exploitation isn't always obvious. You can continue the conversation with your AI assistant by asking how to exploit these vulnerabilities and requesting more detailed information. Don't hesitate to ask about specific tools or request step-by-step instructions for executing your hack. Those questions are always instructive and provides valuable learning opportunities.

"Copilot, what's a smuggle request and how can I do it ? Can you guide me step-by-step?" — prompt example

Write a Python script to accomplish XXX?

The analysis process generates numerous ideas. Generally, these ideas require some coding to create a proof of concept for a vulnerability. When you're unsure where to start and time is limited, asking an AI to write a small code snippet can be valuable. Copilot can adapt to your repository structure and generate files in the appropriate locations with the requested code. While not essential, this approach saves time.

One crucial point to notice is that you need to be precise with your requirements; otherwise, the generated code may not align with your needs. Don't hesitate to specify the programming language, preferred libraries, and your desired outcome. The goal isn't to obtain perfect code, but rather to get something you can easily customize to suit your specific needs.

"Copilot, can you write a Python script with pwntools that connects to [IP] and [PORT], receives two numbers, adds them, and sends back the result." — prompt example

🔄 It's an iterative process

This process is often iterative. Basically, after the global analysis, I enter a loop where I filter the data I have, analyze it, ask questions, inquire about security issues, and reflect on the results. Then, I reiterate again and again until I start to assemble pieces of the puzzle to solve the challenge.

💔 What's the drawback of using AI for a CTF

Using AI is exhausting

Using AI allows me to learn quickly, but it's exhausting. While you can access specific information and documentation faster than ever, it involves processing a lot of knowledge in a short time. You need to read extensively, maintain focus, and review every suggestion and generated code snippet. It's a different style of work. Counter-intuitively, working with an AI assistant actually consumes more energy than working alone. While one might think that having a computer do the thinking would require less effort, the reality is quite different.

Working without an AI assistant is a steady hike; working with one is a sprint - you move faster, but it takes a lot more energy.

AI == cheat?

A legitimate question might be: is using AI cheating in a CTF? For the easiest challenges, like the Intro or Easy levels in 404CTF, AI can sometimes hand you the solution on a silver platter. But as soon as you move up to Medium or harder, the problems get too complex for Copilot or any AI to just solve for you.

That's why I don't see AI as a cheat code. In reality, it's just one tool among many, like the ones real hackers use to break systems in the wild. Sometimes it helps, sometimes it doesn't, and it rarely cracks the tough challenges on its own. In cybersecurity, there are no shortcuts, just different methods and tools to reach your goal. And honestly, no hacker would refuse a useful tool just because it feels "unfair."

AI mostly raises the floor, not the ceiling. It helps beginners level up faster, but expert won't suddenly become stronger thanks to AI alone.

AI can be censored for certain types of questions

Sometimes I encountered limitations with Copilot, likely due to ethical considerations. When attempting to create attacks, Copilot sometimes refuses to provide assistance. As a workaround, I needed to specify in my initial prompt that I was working in a CTF context. However, even with this clarification, Copilot might still censor certain responses.

It's amusing to realize that you sometimes have to find creative ways to "hack" Copilot itself. It reminds me of Asimov's novels, where clever loopholes are found in the famous three laws of robotics. In a similar way, you need to think outside the box to get Copilot to do what you want.

AI can push you to do some unsecure actions

While asking Copilot how to exploit vulnerabilities, I sometimes received recommendations for tools whose safety I wasn't entirely confident about. When investigating these tools on GitHub, regardless of their quality, they often had few stars, indicating limited usage, and were frequently unmaintained with commits dating back years. This didn't inspire much confidence.

For example, when I was looking for a password generator based on social knowledge, Copilot recommended tools like pypasswords. While I won't judge this tool's quality, it may be good but pypasswords' last commit was 5 years ago and the project only had 1 star. Without looking in more details, it can be dangerous to install such a library.

That's why, as personal advice, I recommend not blindly trusting what your AI suggests when it comes to running commands or installing libraries. Always verify what you're about to execute.

AI can lead you down the wrong path

I think it's part of the experience, but I've lost count of the times Copilot has hallucinated issues where there were none, or assured me everything was fine when there was actually a subtle problem. I once spent hours trying to craft a smuggling request that turned out to be impossible - simply because Copilot suggested it might work. But as I mentioned earlier, this is a valuable lesson: don't blindly trust everything your AI suggests.

That said, I'd put this into perspective based on your goals with CTFs. If your aim is to learn new skills, sometimes following the wrong leads can be incredibly educational.

🚀 Beyond the CTF

Using AI to solve CTFs is fun, and I strongly encourage you to try it. You'll sharpen your AI skills for analysis and programming while learning extensively about specific domains. Personally, I focused mainly on web security since I specialize in web development. The knowledge I gained is valuable and helps me spot issues I might have missed before. It's all about honing your skills.

At Doctolib, we actively encourage software engineers to embrace AI, not to work less, but to free up time for higher-value tasks that truly matter. This philosophy mirrors exactly what I experienced during my CTF journey.

Just as I used AI to accelerate my learning and problem-solving in cybersecurity challenges, at Doctolib, we leverage tools like GitHub Copilot, Cursor, and Claude Code to streamline routine coding tasks, accelerate debugging, and enhance code exploration. The goal isn't to replace our expertise, but to augment it allowing us to focus on architectural decisions, complex problem-solving, and innovative solutions that improve healthcare for millions of patients.

Whether it's analyzing nginx configurations in a CTF challenge or refactoring critical healthcare infrastructure at Doctolib, AI serves as an amplifier of human intelligence. The key lesson from both contexts remains the same: AI is most powerful when it enhances human judgment, not when it replaces it.

Thank you for reading. If you have any questions or insights to share, feel free to reach out or leave a comment.

Special thanks to Justin Rabiller for reviewing this article and providing valuable insights.

Beyond Barnum: crafting meaningful requirements in System Design

Thomas de Saint Exupery — Wed, 29 Jan 2025 11:10:26 +0000

Make it scalable, but not too complex. And of course, everything needs to be secured and compliant.

— Every System Design Ever

If you've ever participated in a system design interview – whether as an interviewer or candidate – these words probably sound familiar. They're the software engineering equivalent of a horoscope: vague enough to apply to any system, yet specific enough to feel meaningful. Just as horoscopes tell you that you're "sometimes outgoing but also enjoy quiet time alone" (who doesn't?), these requirements seem to provide guidance while actually saying very little.

Welcome to the Barnum effect in system design.

As a staff engineer at Doctolib who has been conducting system design interviews for over a year, I've seen this pattern repeat itself: candidates nod sagely at these generic requirements, only to struggle when it comes to making concrete design decisions.

But here's the thing: real systems aren't built on truisms. You can't tell your infrastructure team to make it "not too complex" or ask your security team to just make “everything secure." These Barnum statements are the comfort food of system design – they feel good but provide little nutritional value for your actual engineering decisions.

The anatomy of void requirements

Make it scalable

What's our growth projection? Are we expanding to new countries? Each territory brings its own complexity: different currencies, tax systems, time zones, etc. There are many other dimensions of scalability often overlooked: data volume growth over time, number of supported third-party integrations, variety of client applications or even organizational scalability as teams grow and specialize. These dimensions radically change our technical choices.

but not too complex,

What does "complex" mean in your context? Sometimes, a "complex" solution might be exactly what's needed to solve a complex problem. Also "Not too" is a classic example of a truism - a statement that's so obviously true it adds no value to the discussion.

And of course, everything needs to be secured,

What threats are we protecting against, and what's the actual impact of a breach? Overemphasizing security can actively harm user experience or system maintainability. Like "not too complex", this is another truism - no one would argue for an unsecured system, making the requirement meaningless without specifics.

and compliant.

Compliant with what regulations, in which jurisdictions, and for what type of data? The choice between AWS, Azure, or on-premise hosting could hinge entirely on specific compliance requirements. This requirement often appears as a checkbox item, ignoring that compliance is a complex spectrum of legal, technical, and operational considerations.

From void to valuable: four principles for better requirements

After reading these Barnum-like requirements, you might be wondering "How do I write better ones?" Here's a confession: reviewing my own system designs from the past two years, I discovered I had fallen into the same trap. I sometimes used these vague requirements to subtly steer architecture decisions in the direction I preferred. To protect myself from this bias and improve the quality of my designs, I strictly follow these principles:

🎯 Focus on outcomes, not solutions
- Define what success looks like rather than prescribing how to achieve it.
🧪 Requirements must be testable
- If you can't test it, you can't verify it, and therefore it's not a requirement.
↔️ Use ranges and degrees instead of absolutes
- Express constraints through both ranges and precise language (must/should/may) to communicate flexibility and priorities.
⚖️ Define necessary and sufficient conditions
- Understand both the minimum needed to succeed and when additional effort yields no value.

To illustrate them, I'll share conversations between two engineers - one proposing requirements, let's call them Pat, the other - Sam - helping to refine them. You might recognize yourself in either role: I know I've played both parts and I’ll continue. That’s why I need to be challenged by others to cover my blind spots.

Principle 1: Focus on outcomes, not solutions 🎯

Crafting requirements

Software engineering literature often distinguishes between functional and non-functional requirements. But the real insight isn't in sorting requirements into these categories - it's understanding their origins and purposes.

Functional requirements emerge from business outcomes: they describe what problems we need to solve. They come from product vision, user problems, and business goals. "Doctors can view their schedule offline" is a clear outcome we can build for.

Non-functional requirements are different - they're rarely explicitly requested but always implicitly expected. As technical experts, it's our responsibility to identify, propose, and quantify them. When a product owner says "doctors should view their schedule," they might not specify "with less than 200ms latency" - but we know that slow response times would make the feature useless. These requirements help us choose between technically viable solutions and find the one that best serves our users.

These non-functional requirements serve as a mood board for technical quality: security, maintainability, observability, and many more. They represent our professional standards rather than specific features.

Requirements translate desired outcomes into actionable information that helps us evaluate and choose between possible solutions. Each solution comes with its own trade-offs, but we can only assess these trade-offs meaningfully when we clearly understand what we're trying to achieve.

For example, "support expansion to 3 new countries" might lead us to consider:

Multi-region deployment
Data residency solutions
Internationalization approaches.

The hidden solution trap

⚠️ Sometimes we already have a solution in mind and work backwards to write requirements that will inevitably lead to that solution.

❌ Once I wrote: "The system must maintain strict consistency across all operations", thinking: "Let’s use PostgreSQL".
✅ Real requirement: "Users must never see outdated prices during checkout"

❌ another time, I wrote: "The system must support independent deployment of components", thinking: "I want microservices".
✅ Real requirement: "Teams must be able to release features independently"

Requirements should create space for architectural discussions, not circumvent them. When we find ourselves writing requirements that can only be satisfied by one specific solution, it's often a sign we're working backwards from a preferred technology rather than forward from actual problems.

Principle 2: Requirements must be testable 🧪

This is a fundamental truth in software engineering: if you can't test it, you can't verify it. And if you can't verify it, it's not really a requirement - it's a wish.

From untestable to testable:

❌ Don't : The code should be maintainable
✅ Do : A new developer should make their first production commit within their first week
❌ Don't : The application should handle high load ->
✅ Do : The system maintains response times under 200ms for the 99th percentile while processing 100 concurrent requests

Notice how testable requirements naturally become more specific and actionable. They create clear acceptance criteria and enable objective discussions about whether they've been met.

By the way, even architectural decisions can be tested - as Neal Ford describes in Software Architecture: The Hard Parts, fitness functions help us verify our architectural characteristics through automated tests.

Principle 3: Use ranges and degrees instead of absolutes ↔️

In software engineering, precision in language helps communicate constraints and priorities. That's why we use specific terms:

🔒 MUST: Absolute requirement, non-negotiable
📌 SHOULD: Highly desirable but not mandatory
✨ MAY: Optional, nice to have

Just like mechanical engineering uses tolerances, software requirements work better with realistic ranges.
They not only acknowledge that perfect precision is neither possible nor necessary but also help us express our level of confidence - especially crucial when dealing with external dependencies.

For example, when specifying API response times:

In-memory search must return within 5ms ± 1ms (precise, highly controlled)
Full-text search should take 0.5-2s (moderate variability, Historical data available)
System should serve 10k-100k users (Low Confidence for new Market Entry Based on comparable markets)

Principle 4: Define necessary and sufficient conditions ⚖️

This dialogue illustrates a key principle: requirements have both a lower and upper bound. While necessary conditions are rarely missed (failure makes them obvious), sufficient conditions are often overlooked - leading to wasted optimization efforts.

✔️ Necessary conditions define the minimum viable threshold - below this, the solution fails its purpose. This creates a clear "MUST" requirement that helps eliminate unviable solutions early. For example, if document upload takes more than 5 seconds, users won't use it, making any solution that can't meet this threshold unviable.

✅ Sufficient conditions identify the point beyond which additional improvement has no impact on users or business outcomes. It's not about trade-offs - if users can't perceive a difference, any additional optimization is objectively worthless, regardless of how easy or hard it might be to achieve. I want to insist on this. The sufficient condition isn't a compromise - it's the point where further improvements become meaningless.

When displaying product inventory in e-commerce
- ✅ Sufficient conditions : 5-second consistency delay
- Users take longer than this to add items to cart, faster synchronization creates no benefit
When displaying search results
- ✅ Sufficient conditions : 100 results per page
- Analytics show users never scroll past result 80
When processing expense claims
- ✅ 48-hour validation delay
- Payroll cycles operate monthly, faster processing doesn't accelerate reimbursement

Between these boundaries lies our practical engineering target. Understanding this helps prevent the common pitfall of over-engineering in pursuit of unnecessary perfection.

Conclusion

Requirements aren't just a box to check in your system design process - they're the foundation that enables and justifies rational technical decisions. They serve a dual purpose: first guiding our design choices, then later justifying these decisions to others.
We didn't choose Redis because it was trendy, but because we needed sub-millisecond access to session data. We didn't adopt Kubernetes because everyone else did, but because our deployment requirements demanded independent scaling of components. These requirements become the rational foundation of our architecture - not "because the tech lead said so" but because specific, measurable criteria led us there.

🎁 My Go-to Checklist

As a gift for reading to the end, here's what I keep on my mental sticky note:

When you hear "we need X" → ask "what problem are we solving?"
When requirements feel vague → "how would we test this?"
When unsure about numbers → start with ranges
When optimizing → check if you're past "good enough"
When justifying decisions → point to requirements, not opinions

Keep these in your back pocket - they've saved me from many circular architecture discussions!