Forem: Escape - API discovery and security

Workshop - How to Write Custom Security Tests

Antoine Carossio — Fri, 15 Mar 2024 11:25:46 +0000

Custom rules help you test the custom business logic of your applications and gain full control of your API security posture.

Don't know why you need custom security tests and where to start? We'll be organizing a workshop on this topic

The goal is to guide you through advanced techniques and offer practical insights to improve your security protocols.

You'll learn:

The importance of custom security tests and how to write them
Setting up rules for various API vulnerabilities
Creating rules based on bug bounty or pentesting reports
Fine-tuning Escape rules to catch issues specific to your APIs
Ensuring your rules adapt to each API update and newly discovered APIs
By the end of this workshop, you'll have a clear path to establish a robust API security posture for your organization's APIs.

📅 When? 11:00 a.m. EST on Thursday, March 21st.

Can't make it? You can still sign up, and you'll receive a session recording once it ends.

Methodology: How we discovered over 18,000 API secret tokens

Antoine Carossio — Wed, 24 Jan 2024 16:40:36 +0000

Hey there!

It's just the beginning of the year, but our security research team has been working hard to identify current API security challenges.

So for the Escape team, January went under the tagline "API secret sprawl" (we thought it was more fun than"Dry January", and we hope you agree with us 😉).

If you're in the security field, you might mention that the beginning of the year usually marks the release of all "State of.. " reports, taking a look at the past year's trends. Instead of collecting opinions or our tool data, we decided to look at the real world. That's how the whole project started, and we were shocked at what it led to.

Our security research team scanned 189.5M URLs and found more than 18,000 exposed API secrets. 41% of exposed secrets were highly critical, i.e. could lead to significant financial risks for the organizations, as exposed financial tokens and API keys included $20 million in vulnerable Stripe tokens.

Unlike other reports, Escape’s web crawler analyzed applications in their actual usage scenarios, examining everything from APIs to frontends, including elements that run in the background, like JavaScript. This approach shows how and where API secret keys and tokens are exposed in real-world settings, not only in code repositories.

You can review the complete results in our comprehensive report. Meanwhile, in this article, we'll show you the methodology that guided us to these impressive findings.

Methodology

Development of the web spider

To tackle the complex task of scanning 1 million domains, we developed a specialized web spider. This tool was built using Golang, chosen for its excellent input/output (I/O) throughput, good productivity, and strong support for concurrency. These features of Golang were crucial for efficiently processing large volumes of web data.

For networking, we relied on a library named fasthttp (fasthttp GitHub), known for its high performance. fasthttp was instrumental in enabling our spider to handle numerous network requests swiftly and effectively.

To interpret and analyze JavaScript found on web pages, we used tree-sitter (Tree-sitter), a parser generator tool and an incremental parsing library.

It helped us build a robust mechanism to understand and process JavaScript code, which is a critical component in modern web applications.

The Golang-based spider was containerized and designed to listen on a Kafka (Redpanda) stream. This setup allowed for scalable and efficient handling of data streams.

In terms of secret analysis, we incorporated an existing Python-based service that we regularly use at Escape. This tool, which employs natural language processing, was accessed via gRPC.

The whole solution was deployed on a Kubernetes cluster, leveraging the orchestration.

Data gathering strategy

For our comprehensive analysis, we chose to examine the 1 million most popular domains.

The list of these domains was sourced from the Majestic Million dataset, which ranks websites based on the number of referring subnets. This ranking offered us a diverse set of domains to study, spanning various sectors and sizes.

We acknowledge a potential bias in our approach. Typically, larger domains with more resources might have better security measures, possibly leading to fewer instances of secret sprawl. In contrast, smaller websites or those without dedicated security teams might be more prone to such issues. However, our study focused on the most popular domains without specifically addressing this bias.

Alternatively, with over 365 million domain names reported across the internet, our sample size becomes relatively small, potentially leading to greater volatility in the number of findings.

The data collection was a one-time process, based on the latest available list from the Majestic dataset. During the collection, we encountered several limitations. To respect legal and ethical boundaries, we deliberately excluded certain types of domains. This included governmental, educational, and health-related domains, as regular users are not typically authorized to explore these. This decision ensured that our study adhered to the ethical norms of web crawling and data collection.

By focusing on domains that are accessible to the general public, our study provides insights into the state of secret sprawl in the broader, more publicly engaged segments of the internet. This focus enables a comprehensive evaluation of security practices and the challenges encountered in a diverse array of online platforms and environments. Through this lens, we gain a deeper understanding of how secret sprawl impacts various sectors and what this means for the broader digital security posture.

Data collection process

Our data collection process was a significant undertaking, both in terms of scale and technical complexity. To manage this, we deployed our containerized web spider on a Kubernetes cluster. The cluster was capable of scaling up to 150 concurrent worker instances. This level of scalability was crucial for effectively managing the immense task of scanning 1 million domains, allowing us to distribute the workload efficiently and process a vast amount of data.

The collection spanned over 69 hours, with our system analyzing an average of 4 domains per second. This pace resulted in a total sum duration of approximately 30,686,535 seconds for the entire operation. On average, each domain, including its subdomains, was analyzed in about 32 seconds. This comprehensive approach ensured that we not only looked at the primary domain but also dived into the numerous subdomains associated with each, providing a more complete picture of the web landscape.

In total, our process led us to visit 189,466,870 URLs. This extensive coverage was key to ensuring that our analysis was as thorough and inclusive as possible. By examining such a large number of URLs, we were able to gain deep insights into the current state of secret sprawl across a wide spectrum of the internet.

Also, we started this project by making a new tool as a test. It was impressive how quickly this tool was made – just three days by one engineer. Combining this quick tool development with the project's computing cost of only about $100 shows how, in today's world, we can get big results, build solutions without spending a lot of money or time.

Data cleanup and verification

One of the most challenging aspects of our study was, for sure, the data cleanup and verification process. While we could not verify the tokens ourselves, we made sure each one was classified accurately. A common pattern we noticed is that many tokens have specific prefixes. For instance, Stripe tokens have various prefixes, but we focused particularly on live secret keys, identified by the prefix 'sk_live_'.

To improve the accuracy of our findings, we refined our heuristics to filter out only classified information. This meant paying special attention to high-entropy keys, which often represent proprietary, undocumented tokens or false positives, and filtering them even more from our primary dataset. This approach helped us focus on the most relevant and potentially impactful tokens.

Verification of the tokens was a crucial step, and it was carried out by the token owners themselves. We alerted the owners only when our system was highly confident about the findings. This was a delicate balance to maintain – ensuring the accuracy of our alerts without the ability to test the keys ourselves. We had to be very sure before alerting affected parties to avoid any false alarms.

This cleanup and verification process was an intricate part of our study, requiring a nuanced understanding of token patterns and careful judgment to minimize false positives. Our method aimed to provide reliable and actionable insights to those whose security might have been compromised.

Conclusion

Securing all your APIs is hard. It’s even harder when your keys and tokens get exposed involuntarily in real-world settings - from APIs to frontends. Your organization is now not only prone to data breach risks but also to severe financial implications.

Our study reveals that API secret sprawl extends across a diverse array of websites, industries, and domain types. Even modern tech industries are not exempt. Organizations must respond fast, adopting best practices to secure themselves against potential threats.

Not sure where to start? Centralizing token management, enforcing rotation policies, segmenting access, intensifying security training, and leveraging automated testing tools are essential steps to mitigate these risks. We've compiled the comprehensive list in our report.

Need guidance tailored to your specific needs? Feel free to reach out!

Building your Product Security Roadmap

Antoine Carossio — Fri, 10 Nov 2023 12:26:14 +0000

Whether you're in the first few weeks of your ProdSec journey or building your Product security program for 2024, you might feel lost trying to wrap your head around all the attack surface your org faces and identifying some of the biggest security gaps. We're organizing our first webinar at Escape to help you learn how to deal with these challenges.

You'll learn:

The Essentials: Understand the foundational elements of product security and why they matter.
Risk Assessment: Identify potential vulnerabilities and threats specific to your organization.
Best Practices: Discover tried and tested methods for securing your applications effectively.
Compliance and Governance: Learn how to align your security roadmap with industry standards and regulations.
Incident Response: Prepare for the unexpected with a well-defined plan for handling security incidents.

By the end of this webinar, you'll have a clear path forward to establish a strong security posture for your organization's products!

📆 When? 9:30a.m. PST on Tuesday, the 28th of November.

Can't make it? No worries, sign up, and you'll receive a session recording once it ends.

Leveraging Temporal for resilient remote procedure calls (RPC)

Gautier — Wed, 18 Oct 2023 12:51:23 +0000

Our stack at Escape is written in multiple languages because each team has specific needs. We use TypeScript for its vibrant ecosystem, Python for cybersecurity research and Go for performance-sensitive tasks. To orchestrate cross-language task orchestration, we first developed a simple request-response protocol over HTTP, but it wasn't sustainable as the Escape codebase grew rapidly. We evaluated several technologies to replace our homegrown protocol, and two emerged as the most promising options: Connect and Temporal. The title gives it away, but the reason is far from obvious

Temporal vs Connect (gRPC)

On the one hand, Connect is a gRPC-compatible implementation that allows direct communication between two processes. On the other hand, Temporal is a centralized task orchestration system. Temporal does not even advertise itself as cross-language capable! You'd have to dig into their GitHub repos to know that it's possible.

Since we already use Protobuf internally, any gRPC implementation would have been the expected next step, but our requirements took us in the opposite direction.

Cross-language type safety: This requirement should favor gRPC over all technologies since it is the direct implementation of this requirement. However, we have already satisfied this requirement for any technology by using Protobuf to securely serialize and deserialize messages.
Observability: This is where Temporal's centralized model starts to shine in this comparison. Having an orchestrator allows logs and results to be in the same place. In addition, Temporal ships with a simple and clean web UI that allows a developer to see what has been exchanged between all the processes involved.

Temporal's lean user interface (UI)

Reliability and scalability: Temporal puts all pending tasks into queues, meaning that a process restart will not crash the remote procedure call, just delay it a bit. The queuing mechanism also allows multiple workers to handle the task out-of-the-box, whereas the same feature in gRPC would have required the deployment out a load balancer.

It turned out that we needed more than just a request-response protocol. We also had use for Temporal features outside of RPC, and since Escape is a small company, we preferred to introduce one complex tool rather than two.

What is Temporal?

Temporal is a distributed, scalable, and reliable platform for building durable workflow applications. It provides a programming model that makes it easy to write code that can handle complex and long-running workflows, even in the face of failures.

Temporal is used by companies like Netflix, Uber, and Snapchat to build critical applications such as order processing, payment processing, and customer onboarding.

A Temporal application is built around two concepts: workflows and activities.

👉 We have illustrated core Temporal concepts in TypeScript, but the same applies to all languages that Temporal supports: Go, Java, PHP, Python, and JavaScript, albeit with different syntax. Our example is a user registration confirmation email.

Workflows are deterministic sequences of instructions that can be resumed and monitored with minimal effort. They execute side effects through activities, which can be non-deterministic.

import { proxyActivities, sleep } from "@temporalio/workflow";
const { sendConfirmationEmail, isEmailConfirmed } = proxyActivities({
  startToCloseTimeout: "1 minute",
});

// A `registerUser` workflow, that sends up to 3 confirmation emails
export const registerUser = async (email: string) =&gt; {
  await sendConfirmationEmail(email);

  await sleep("1 day");

  if (await isEmailConfirmed(email)) return;
  await sendConfirmationEmail(email);

  await sleep("2 days");

  if (await isEmailConfirmed(email)) return;
  await sendConfirmationEmail(email);
};

A registration workflow that sends up to 3 confirmation emails

Activities are units of work performed asynchronously by a Temporal worker. Activities are typically used to perform complex tasks, such as calling an external service, transcoding a media file, or sending an email message. They can be stateful, whereas workflows are completely stateless.

// Sends a confirmation email to a given email
export const sendConfirmationEmail = async (email: string) =&gt; {
  const url = `http://localhost/users/confirm?email=${email}`;
  return mailer.sendMail({
    from: "support@example.com",
    to: email,
    subject: "Welcome!",
    html: `<h1>Welcome!</h1>
      <p><a href="${url}">Confirm your email</a></p>`,
  });
};

// Returns whether the user has confirmed their email by clicking the link
export const isEmailConfirmed = async (email: string) =&gt; {
  const { confirmed } = await database.user.findUnique({
    where: { email },
  });
  return confirmed;
};

The two activities used by the registration workflow: sending a mail and querying the database

A workflow execution can last for days, even months, thanks to the reentrant property of workflows: when its worker stops, the workflow resumes exactly where it was before, replaying all the instructions in order on a new worker. You can sleep for days without worrying that your process will be restarted!

There is much more to learn about what Temporal is and what it can do, but this short section should have given you an idea of the power of the tool. See their documentation, including schemas, for more information about what Temporal is.

Temporal for cross-language RPC

What may not be obvious from the examples above is that workflows and activities can be written in different languages, as can the code that starts a workflow and the workflow itself.

Thanks to this ability, we built RPC over Temporal:

const name = "TypeScript";
// Launch a Python workflow from TypeScript code:
const result = await temporal.workflow.execute("SayHello", {
  args: [name],
  taskQueue: "hello-task-queue",
  workflowId: `SayHello-${name}`,
});
console.log(result); // Prints "Hello TypeScript, Python here!"

Initiating a remote procedure call over Temporal in TypeScript

from temporalio import workflow

@workflow.defn
class SayHello:
    @workflow.run
    async def run(self, name: str) -&gt; str:
        # Respond to all requests with a friendly Hello:
        return f"Hello {name}, Python here!"

The simplest "Hello World" workflow written in Python

🔄 Our example is the execution of Python code from a TypeScript caller, but the other way around is perfectly feasible.

As long as you properly define workflow names and queues with the same names, Temporal will do all the heavy lifting of sending reliable messages, even across different programming languages. Temporal comes with sensible defaults, but all parameters, such as retries and timeouts, can be customized per user's needs.

Temporal can also be used to migrate between languages, allowing for both legacy and modern workers to run on the same queue, regardless of their implementation language. Although we have not yet used Temporal for this purpose, this potential use case also weighs in favor of Temporal.

Pros and cons

It may seem like we have nothing but compliments for Temporal, but all software has its drawbacks, Temporal included. Let's start by adding a few more qualities to the list compiled in this article:

Temporal is incremental: It's not an all-or-nothing technology; we've only started using it in a few places where we need it most.
It's easy for basic stuff: We only need a subset of Temporal's features, and we were able to learn them very quickly. Creating basic workflows like the ones above requires only limited knowledge of the Temporal API.

However, the following negative aspects can be noted:

The documentation is vast and incomplete: There are several official tutorial websites, and some pages have a large "WORK IN PROGRESS" banner at the top. It's hard to know what to learn and how to learn it.
There seem to be a lot of gotchas: There are many ways to do the same thing, and some may appear more idiomatic than others to experienced developers. The lack of clear guidelines makes it hard to know best practices, and we learn them the hard way, through trial and error.
Temporal Cloud is really expensive: It starts at $225/month, which means small businesses and hobbyists will have to host their instance themselves. Although well documented, setting up and keeping Temporal up to date is a technical and time-consuming task.

We still believe that choosing Temporal was the right choice for our technical stack and needs, but you may want to reconsider adopting Temporal in light of these drawbacks.

Closing words

We hope you enjoyed this article and found it enlightening. We're always excited to hear back and always open to feedback - feel free to send comments wherever you came across this article! If you want hands-on experience, you can find all the code of this article in a working environment in a GitHub repository.

Want to learn more about new technologies being implemented at Escape? We wrote many times about the introduction of new technologies in the Escape stack:

Using Protobuf with TypeScript, a way to ensure safe serialization and deserialization of network data.
Set up a design system using Storybook and Figma, how our frontend developers introduced tools to improve the brand consistency across Escape.
Code quality controls in our Node.js monorepo, how we created a consistent coding-style across a large codebase.

Introducing Goctopus: open-source, state-of-the-art GraphQL endpoint discovery & fingerprinting tool.

nohehf — Thu, 10 Aug 2023 16:18:17 +0000

Read this article more comfortably on Escape Blog_

In the fast-evolving domain of APIs, GraphQL has emerged as a powerful, data-oriented language. As its adoption soars, so does the need for robust tools to discover and fingerprint these APIs. Enter Goctopus, a Golang-based solution we developed at Escape to provide comprehensive, fast, and interoperable endpoint discovery and fingerprinting for GraphQL APIs. Engineered to overcome the limitations of existing tools, Goctopus extends its arms into realms of subdomain enumeration, route brute-force, authentication detection, schema introspection, and more, all while maintaining optimum speed and resource management. In this article, we dive into why we created Goctopus, what it does, and how it's pushing the boundaries of API security.

TL;DR: We built an open-source GraphQL endpoint discovery & fingerprinting tool: check out Goctopus on Github.

Unmasking APIs: The Importance of API Discovery & Fingerprinting

In the expansive realm of APIs, two practices stand out in their ability to shed light on the intricacies of your online presence: API Discovery and Fingerprinting.

API Discovery is the process of finding all the APIs an organization has in its ecosystem. The aim here is to unveil the totality of the digital surface that can be interacted with. In the era of microservices and distributed systems, discovering all your API endpoints is crucial for both maintaining service interoperability and ensuring security.

API Fingerprinting, on the other hand, takes this a step further. Once the endpoints are discovered, they're scanned to find specific information about them: technologies used, configuration, authentication details...

Endpoint discovery & fingerprinting are crucial tools in offensive and defensive security to:

Detecting Attack Surface: By understanding what endpoints exist and how they operate, you get an accurate view of your potential attack surface. This gives you the upper hand in securing your system, as you can preemptively locate and address vulnerabilities.
Internet-wide Research: If you're a security researcher or an organization keen on understanding the landscape of what's exposed on the internet, fingerprinting can give you a macro view. It helps to gather data about technologies, configurations, and security practices being used across various domains.
Classifying endpoints: Being able to determine which technologies and configurations are used on APIs is usually the first step in any security process (offensive or defensive).

For REST APIs there are a myriad of tools to perform those tasks. However, for GraphQL this is not the same story, as you will see in the next paragraph.

Identifying the Void: Why We Needed Goctopus

Regarding GraphQL APIs, we built Grafinder a while ago, and it has been serving us well for a few years. However, when we decided to implement our API Catalog, we needed a more robust, faster, all-in-one, and more interoperable solution.

Grafinder had some shortcomings that were hard to ignore. It was slow, hard to use programmatically, often missed inputs, lacked options, and its Python basis wasn't our preferred choice for this specific task. Moreover, we found ourselves needing a tool that could offer more comprehensive and efficient discovery and fingerprinting capabilities:

Broad and efficient discovery capabilities: (Subdomain enumeration, brute-force, script analysis).
Precise, fast and complete fingerprinting of GraphQL APIs: (Technology, authentication, schema, field suggestion attack).
Easy to use and enjoyable CLI.
Complete Package for programmatic usage.
Service interoperability: communicate with external services via webhooks.
Portability: run easily anywhere at any scale.
Versatile inputs: take a variety of inputs, domains, URLs, and IP addresses.
Speed: scan thousands of domains in minutes.
Scale: run on larger machines or scale horizontally.
Network resources: send a minimum amount of necessary requests and maximize throughput.

So, with those constraints in mind, we realized we needed a solution that could go beyond what was already there. A more holistic tool. A faster tool. A...well, better tool. We needed Goctopus.

Unleashing the Beast: Meet Goctopus

‌
Born out of the needs we've just discussed, Goctopus is our answer to the comprehensive, fast, and interoperable tool we need. Created with Golang for its superior speed and efficiency in networking, this octopus has its arms reaching everywhere. Let's delve into its various facets:

Discovery:

Subdomain Enumeration: Goctopus uses DNS records APIs via subfinder to enumerate subdomains.
Bruteforce: We built a custom list of popular GraphQL routes so that Goctopus has the best odds of finding endpoints on every domain it tests.
Script Analysis (Work-In-Progress): With this in the pipeline, Goctopus will soon be able to sniff out hidden GraphQL endpoints from Javascript & HTML files across the web.

Fingerprinting:

Authentication: It quickly identifies which endpoints require authentication and which are publicly exposed.
GraphQL Schema/Introspection: Goctopus checks if the schema is available or not.
Field Suggestions for Bruteforcing: If the schema is unavailable, this clever cephalopod can even figure out if the schema can be brute-forced using field suggestions attacks.
Engine Fingerprinting (Work-In-Progress): Once this feature is added, Goctopus will be capable of understanding the tech stack behind an endpoint.

Dx & usage:

CLI: Goctopus is easily usable as a CLI for manual operations.
Go package: It can be used as a Golang package programmatically to integrate into other tools.
Webhook: With this feature, Goctopus can forward its results via a webhook so that other services can retrieve them.
Variety of inputs: The tool accepts a wide variety of inputs: domains, URLs, and IP addresses.

Performance:

Speed: Designed for speed, Goctopus can scan thousands of domains in minutes.
Scale: This tool has no fear of the deep; it's built to scale on larger machines or spread horizontally across a cluster.
Network resources: Goctopus is mindful of its requests, only sending what's necessary.
Rate limiting: It also ensures not sending too many requests to a single endpoint concurrently to avoid rate limiting.

In a nutshell, Goctopus is what we envisioned when we realized the limitations of existing tools. It's comprehensive, fast, flexible, and extensible.

Quick Hands-On: Meet Goctopus

Interested in trying out Goctopus? Here's how to get started.

Installation: You have two options here. If Go is installed on your machine, just run go install -v github.com/Escape-Technologies/goctopus/cmd/goctopus@latest in your terminal. Prefer Docker? No problem, use docker run --rm -it escapetech/goctopus:latest <options> to run the latest version of Goctopus in a docker container.

Usage Example: Once installed, take Goctopus for a spin with this simple command goctopus -a example.com. This will start the discovery process on the specified domain with all features enabled (-a\ flag stands for "all").

For instance, running Goctopus on rickandmortyapi.com will result in the following:

‌

goctopus -a rickandmortyapi.com
                  _
  __ _  ___   ___| |_ ___  _ __  _   _ ___
 / _` |/ _ \ / __| __/ _ \| '_ \| | | / __|
| (_| | (_) | (__| || (_) | |_) | |_| \__ \
 \__, |\___/ \___|\__\___/| .__/ \__,_|___/ v0.0.14
 |___/                    |_|
[INF] Enumerating subdomains for 'rickandmortyapi.com'
[INF] Found 5 subdomains for 'rickandmortyapi.com' in 15 seconds 276 milliseconds
INFO[0016] Done fingerprinting rickandmortyapi.com
INFO[0016] Found: {"authenticated":false,"domain":"rickandmortyapi.com","schema_status":"OPEN","source":"rickandmortyapi.com","url":"https://rickandmortyapi.com/graphql"}
INFO[0016] Done. Found 1 graphql endpoints

‌

Visit our GitHub repo for detailed documentation and the latest release notes.

The Future of Goctopus: Contributing and Roadmap

We're actively looking for ways to enhance Goctopus, and we welcome contributions. Feel free to raise issues or PRs on our GitHub repo. We're excited to see where the community can take this project.

Looking ahead, we've got big plans. Improved subdomain enumeration, more intelligent endpoint discovery, engine fingerprinting - the roadmap for Goctopus is packed with exciting updates.

Goctopus at Escape: Providing Top-Notch Security for Our Users

Here at Escape, we're using Goctopus to provide enhanced security for our users.

API Catalog: We're utilizing Goctopus to provide users with detailed information about their attack surface. By continuously scanning for new APIs and updating our records, we can ensure an up-to-date vision and alert our users whenever they expose an unprotected API.

Suggestions: Goctopus is also integrated into our application creation process to automatically suggest our user's endpoints to secure to make our platform as smooth to use as possible.

‌

Kraken: We built an internal tool around Goctopus called Kraken to scale Goctopus to another level. It consists of an API, a database, and dozens of Goctopus instances managed by a Kubernetes cluster that allowed us to scan Millions of domains to gather data on +150k GraphQL endpoints. Stay tuned for an upcoming article on the topic!

‌

We invite you to join us in exploring what Goctopus can do. Try it out, contribute, and together, let's push the boundaries of API security! And don't forget to star the Goctopus repo on GitHub.

And if you want maximum security for your GraphQL APIs try out Escape security platform for free!

GraphQL errors: the Good, the Bad and the Ugly

Gautier — Thu, 05 Jan 2023 15:18:39 +0000

We, at Escape, have been using GraphQL for our apps for a long time, before many quality tutorials were available. Because we lacked experience, we made design mistakes on many aspects of our GraphQL API. This article reviews the evolution of how we return errors from our API, for consumption by the frontend and other internal services, emphasizing what could be improved on each step.

To illustrate this article, we will take the example of an account creation mutation:

type Mutation {
  register(email: String!, password: String!): User!
}

type User {
  id: ID!
  email: String!
}

The register mutation takes an email and a password and returns the newly created user.

Because the user creation might fail, we need to tell our API consumer (for instance the frontend) that something went wrong and what exactly went wrong. There are many ways to do it, and some work better than others.

The Ugly: the `errors` field

A GraphQL response is a JSON object with up to two keys: data and errors. We implemented error responses leveraging the latter. For instance, considering our register mutation from before, several errors could be raised:

The email already exists
The email uses a banned email provider
The password is too short
Unexpected runtime errors (e.g. a database connection failed)

The specification indicates that errors is an array of objects with a message field but allows additional details in an extensions field. We used extensions.code to convey a machine-interpretable response:

{
  "errors": [
    {
      // User-friendly error message
      "message": "Please provide a professional email address.",
      "extensions": {
        // Machine-friendly error code
        "code": "PROFESSIONAL_EMAIL_REQUIRED"
      }
    }
  ]
}

Several problems emerge from this approach. The most annoying one is that errors is an array: the consumer has to iterate over it to collect the errors. What to do if the array contains several errors but none that can be handled by our frontend? This led to hard to maintain switch inside for loops, with fallback cases afterwards.

On the plus side, all of our errors are returned using the same mechanism, leading to a simpler implementation on the backend, but we have a lot of room for improvement.

The Bad: an `Error` type

GraphQL has a neat type system with a feature named Union types. It allows returning several object types from the same resolver. Let's refactor our resolver a bit to include an error type:

type Mutation {
  register(email: String!, password: String!): RegisterResult!
}

union RegisterResult = User | Error

type User {
  id: ID!
  email: String!
}

type Error {
  message: String!
}

register may now return a user or an error.

Our type definition is getting significantly more complicated, but that's for good! The registration query would now look like this:

mutation {
  register(email: "gautier@example.com", password: "p4ssw0rd") {
    __typename
    # Query different fields depending on the response type
    ...on User {
      id
    }
    ...on Error {
      message
    }
  }
}

In case the user provides a password that is too short, the response payload would look like this:

{
  "data": {
    "register": {
      // This allows the consumer to know which fields are available
      "__typename": "Error",
      "message": "Password too short."
    }
  }
}

This approach requires us to classify errors in two categories: the ones we want in the response errors field, and the ones we return as an Error type. There are two concepts to account for when making this distinction:

Some errors can be returned for both queries and mutations, others are exclusive to this specific mutation.
Some errors are actionable, some are not.

We consider errors that can be returned for queries too because we want to keep queries short:

 This is short and neat
query {
  # User "1" might not exist but let's ignore it for now
  user(id: 1) {
    posts {
      title # ✨
    }
  }
}

# This is bloated and cumbersome
query {
  user(id: 1) {
    __typename
    ...on User {
      posts {
        # Consider possible errors on all fields, even the nested ones
        __typename
        ...on Post {
          title # 😫
        }
        ...on Error {
          message
        }
      }
    }
    ...on Error {
      message
    }
  }
}

Therefore, errors such as User not found, Unauthorized or runtime errors should still be returned in the errors response field, to have query and mutation responses handled the same way.

Furthermore, actionable errors are part of the normal execution flow. It makes sense for a registration to fail and having an error response type associated to it.

Let's take our list of errors from above and categorize them:

The email already exists: specific to this resolver and actionable → Error type
The email uses a banned email provider: same → Error type
The password is too short: same → Error type
Unexpected runtime errors: can happen for both queries and - mutations and hardly actionable for frontend users → errors field

These categories are quite simple to use in the frontend: if __typename is Error, display the error above the form, otherwise, show a generic “Something went wrong, please try again” error.

This error should be next to the password field, but the frontend has no way to know where to place it.

This solution is however less flexible than the previous one: we can only send one actionable error at a time, even when multiple errors could be returned at the same time. We might also want to be able to place the error message right next to its related form input.

The Good: structured errors

The API might return different kinds of errors, with specific data attached. Let's create structured errors for our actionable errors from before. The goal is to replace the generic Error type with more specific domain errors.

type Mutation {
  register(email: String!, password: String!): RegisterResult!
}

# Use different types for all possible actionable errors
union RegisterResult = User | ValidationError | ProfessionalEmailRequired

type User {
  id: ID!
  email: String!
}

# Malformed inputs
type ValidationError {
  # Allow several errors at the same time!
  fieldErrors: [FieldError!]!
}

type FieldError {
  path: String!
  message: String!
}

# Business specific errors (e.g. banned email providers)
type ProfessionalEmailRequired {
  provider: String!
}

When creating our error types, we took into consideration that some errors might be multiple: for instance, the validation step might throw several errors at the same time (email already used, password too short etc.). That is why the error contains an array of fieldErrors.

By requesting all the possible errors, it is now possible for the frontend to display contextualized errors (i.e. errors next to their input):

mutation {
  register(email: "gautier@example.com", password: "p4ssw0rd") {
    __typename
    # Query different fields depending on the response type
    ...on User { id }
    ...on ValidationError {
      fieldErrors { path message }
    }
    ...on ProfessionalEmailRequired { provider }
  }
}

{
  "data": {
    "register": {
      "__typename": "ValidationError",
      // Error messages specific to each input:
      "fieldErrors": [
        {"path": "email", "message": "This account already exists."},
        {"path": "password", "message": "Password too short."},
      ]
    }
  }
}

The frontend is now able to show several contextualized errors at the same time.

This architecture will also make some future considerations easier to implement, especially internationalization (i18n).

We are currently refactoring our API to use structured errors. It represents a substantial amount of work, but we are now able to display more precise error messages, especially for complex inputs and flows. Structured errors help us improve the user experience of our products.

HTTP errors

We spent a while talking about error types, but let's get back to the errors field to conclude. Sending errors in here allows keeping queries short and clear, and we still use it for Not found and Unauthorized errors. With a twist!

The GraphQL over HTTP specification states the following:

The server SHOULD use the 200 status code, independent of any GraphQL request error or GraphQL field error raised.

We decided to ignore this recommendation and attach semantic HTTP error codes to queries with errors. Yoga's error masking makes it really simple to transform JS error objects into GraphQL errors with the right HTTP code attached:

const yoga = createYoga({
  schema,
  maskedErrors: {
    maskError(error, message) {
      const cause = (error as GraphQLError).originalError;

      // Transform JS error objects into GraphQL errors
      if (cause instanceof UnauthorizedError)
        return new GraphQLError(cause.message, { extensions: { http: { status: 401 } } });

      if (cause instanceof NotFoundError)
        return new GraphQLError(cause.message, { extensions: { http: { status: 404 } } });

      // Default to 500 with a generic message
      return new GraphQLError(message, { extensions: { http: { status: 500 } } });
    },
  },
});

This enables the frontend to show the correct HTTP error page in case of a GraphQL error without even parsing the response.

We are not the first ones to write about returning GraphQL errors, you might be interested in these articles/documentations too:

Closing words

This is the end of this we-do-it-that-way article, we hope you enjoyed this format that allows to peep inside our development practices at Escape. We are continuously discovering new ways to design GraphQL APIs and we will keep writing about the different steps we took until reaching the state of the art. Please share your thoughts where you found this article, we have a lot to learn from your experiences!

Rendering emails with Svelte

Gautier — Thu, 15 Dec 2022 15:27:51 +0000

Emails are the cornerstone of automated internet communication. Create an account on a website? Email. Receive an invoice? Email. Sign up for an event? Email. As a developer, you will need to send emails at some point. And you will end up working with some of the most legacy web technologies.

We, at Escape, recently rebuilt our whole email stack from scratch to improve the developer experience: we used to send emails to preview them, whereas now, we have an instant feedback loop, leveraging a SvelteKit-powered dev server.

Emails are written with 2003 HTML

If you started web development before 2000, chances are you worked on websites designed with <table>-based layouts. It was the way to design complex two-dimensional layouts.

Well, unfortunately for us, email clients are still stuck in the dark ages. We, therefore, have four possibilities to write emails:

Write them by hand and learn the quirks of the old <table>-based layout system with loads of <!--[if mso]> and inline styles. This is way too slow for a company, but there is a lot to learn on the way. I might have picked this approach for a personal project.
Send plain text emails, like good ol' text messages. Not possible for Escape, but you might consider it.

Source: Email on Acid

Use a WYSIWYG email editor. There are a lot out there! Turns out emails are hard to design. It would work well for static emails with a few string interpolations, but not for dynamic emails with a lot of logic, which we need. Depending on what you want to achieve, this might be the best option.

Source: Unlayer.com

Use an XML-based email templating language. There are plenty too! This is the path we took because it is the most flexible and powerful option.

Source: MJML.io

Because we had some experience with MJML we decided to stick with it. It is battle-tested, has a lot of features, and is easy to learn.

We now need a way to make these emails dynamic, with logic and string interpolation. The title probably ruined the surprise, but hey, we chose Svelte.

MJML, Svelte, Vite, Square pegs and Round holes

Our challenge now is not only to write MJML with Svelte but also to have a simple way to preview and test emails. All the technologies we mentioned were never meant to work together, but there seemed to be a way.

Here is our plan:

1. We would write Svelte code with MJML elements:

<script lang="ts">
  export let name = "World";
</script>

<mj-section>
  <mj-column>
    <mj-text font-size="32px" color="#F45E43" font-family="helvetica">
      Hello {name}!
    </mj-text>
    <mj-divider border-color="#F45E43" />
  </mj-column>
</mj-section>

2. We would compile the Svelte code to HTML using the compiler's SSR capabilities:

const mail = (props) => `<mj-section>
  <mj-column>
    <mj-text font-size="32px" color="#F45E43" font-family="helvetica">
      Hello ${props.name}!
    </mj-text>
    <mj-divider border-color="#F45E43" />
  </mj-column>
</mj-section>`;

3. We would feed this to the MJML compiler:

const document = (body) => `<mjml>
  <mj-head><!-- Other head properties --></mj-head>
  <mj-body>${body}</mj-body>
</mjml>`;
const html = mjml2html(document(mail({ name: "World" })));

4. We would send the resulting HTML:

let transporter = nodemailer.createTransport();

await transporter.sendMail({
  from: "support@example.com",
  to: "client@example.com",
  subject: "Hello!",
  html,
});

Apart from that, we also want:

A way to preview the emails, and the easiest way to get live-reload in a Svelte project is to use SvelteKit.
Compile-time type checking for props, which is made possible by svelte2tsx.

Our setup will be in two parts:

Setting up a development server to create and preview emails.
Setting up a build pipeline to compile emails to HTML strings.

The dev server

SvelteKit offers a fantastic developer experience to work with Svelte and was just released as stable, so it is a no-brainer to use it.

You can clone the whole experiment from GitHub. We will go through the most interesting parts in the rest of the article.

You will find a complete SvelteKit project in packages/svelte-emails:

index.ts: This is our library entry point.

  // Export the renderer
  export { render } from "./lib/index.js";

  // Also export compiled Svelte components
  export * from "./mails/index.js";

lib/

Header.svelte: This is our common email header. MJML offers a lot of components out of the box.

<mj-section>
  <mj-column>
    <mj-text align="center" font-size="20px" font-family="Helvetica">
      <!-- Svelte slot here! -->
      <slot />
    </mj-text>
    <mj-divider border-color="#ff3e00" />
  </mj-column>
</mj-section>

index.ts: It contains the MJML rendering logic.

/** Renders a Svelte component as email-ready HTML. */
export const render = <Props>(
  component: new (...args) => SvelteComponentTyped<Props>,
  props: Props
) => {
  // Render the component to MJML
  const { html: body, css, head } = component.render(props);

  const mjml = `<mjml>
    <mj-head>
      ${head}
      <mj-style>${css.code}</mj-style>
    </mj-head>
    <mj-body>${body}</mj-body>
  </mjml>`;

  // Render MJML to HTML
  const { html } = mjml2html(mjml);

  return html;
};

mails/: This is the root HTTP directory, and it will also contain our emails.

index.ts: This file reexports all the emails.

export { default as HelloWorld } from "./hello-world/Mail.svelte";

hello-world/

Mail.svelte: Make a guess!

<script lang="ts">
  import Header from "$lib/Header.svelte";
  export let name: string;
</script>

<Header>Hello {name}!</Header>

<mj-section>
  <mj-column>
    <mj-button
      color="#fff"
      background-color="#ff3e00"
      font-family="Helvetica"
      href="https://svelte.dev"
    >
      Learn Svelte
    </mj-button>
  </mj-column>
</mj-section>

+page.server.ts and +page.svelte: These are our development email previews, powered by Vite.

export const load = async () => ({
  email: render(Mail, {
    // This is type-checked!
    name: "World",
  }),
});

That is quite a lot of code! Let's try it out:

# Start the dev server
yarn dev

Go to localhost:5173/hello-world to see the email preview, and edit anything to see it update in real-time.

The build pipeline

We now have a working development environment, but we need to build our emails for production. We will use Rollup to bundle our emails, and svelte2tsx to emit type declarations.

The rollup.config.js file defines our build pipeline:

export default {
  input: "src/mails/index.ts",
  plugins: [
    {
      /** Export component's types at the end of the build. */
      name: "rollup-plugin-svelte2dts",
      async buildEnd() {
        const require = createRequire(import.meta.url);

        // All the heavy lifting is done by svelte2tsx
        await emitDts({
          svelteShimsPath: require.resolve("svelte2tsx/svelte-shims.d.ts"),
          declarationDir: "build",
        });

        // We need to replace `.svelte` with `.svelte.js` for types to be resolved
        const index = "build/mails/index.d.ts";
        const code = await readFile(index, "utf-8");
        await writeFile(index, code.replaceAll(".svelte", ".svelte.js"));
      },
    },
    svelte({
      ...svelteConfig,
      compilerOptions: { generate: "ssr" },
      emitCss: false,
    }),
  ],
};

Run yarn build to transform the Svelte emails components into raw JavaScript.

Wrapping up

Using our built emails in a NodeJS app is as simple as:

import { render, HelloWorld } from "svelte-emails";

const html = render(HelloWorld, {
  // This is type-checked!
  name: "World",
});
console.log(html);

There is a demo package in the repo for you to try out.

This concludes this experiment. We are still tinkering with a few things to make it easier to use, but we hope you enjoyed this article. Feel free to ask questions or give feedback on how you are currently developing emails, we are eager to hear from you!

Achieving end-to-end type safety in a modern JS GraphQL stack

Gautier — Thu, 24 Nov 2022 15:18:47 +0000

In this article, we will create a simple GraphQL application, a message board, by combining many recent open-source technologies. This article aims to be a showcase of technologies that work well together rather than a complete tutorial on project setup. It is however a long read, so I recommend settling in with a cup of coffee, a comfortable chair, and a terminal.

This article was originally published as a two part series on escape.tech/blog.

What is end-to-end type safety?

Type safety is the property of a program that guarantees that all value types are known at build time. It prevents a lot of bugs from happening before running the program. The most common way to achieve type safety in a JavaScript project is to use TypeScript:

// Declare an object shape:
interface User {
  name: string;
  email: string;
}

const sendMail = (user: User) => {};

// This works:
sendMail({ name: "John", email: "john@example.com" });

// This doesn't:
sendMail("john@example.com");
// (x) Argument of type 'string' is not assignable to parameter of type 'User'.

When using TypeScript in a project, you get type safety in this very project. End-to-end type safety, on the contrary, is achieved when several projects interact together (e.g., with an API) in a type-safe way.

We will build a message board relying only on type-safe technologies: TypeScript for the API and the application, GraphQL as a way to interact between them, and a SQLite database.

Data and type flows

The data flow of an application is the way data travels and is transformed throughout the application. It is usually represented by a directed graph like this one:

Since data has a type, there also exists a type flow, which is the path of types through said application.

Here, our data flows from left to right, with the database as the data source. Our types follow the same path, with the database schema as the source of types. This is why I prefer code-first over schema-first GraphQL APIs: the data and type flows overlap.

I annotated the diagram with the technologies we will use in this article, and we will set up these technologies from left to right too.

Project setup

If you want to have everything working by the end of this article, you can follow the steps below. Otherwise, you can skip to the next section.

If you don't have Node or Yarn on your machine, you can install them easily with Volta:

# Install the latest versions of Node and Yarn
volta install node@latest
volta install yarn@latest

# Create a new project
mkdir typed-board && cd $_

# Setup a monorepo with Yarn 4
yarn init --private --workspace -2
yarn set version canary

# Enable the good ol' node_modules
echo 'nodeLinker: node-modules' >> .yarnrc.yml
echo 'node_modules/\nbuild/' >> .gitignore

We use Yarn 4 because it ships with a few tools to manage monorepos that we will use later.

Prisma

Prisma is an ORM for Node.js and TypeScript, focused on developer experience. Among all of its features, Prisma offers a top-notch type-safe database client.

# Create a new package for the GraphQL API
mkdir -p packages/api && cd $_

# Initialize a new project
yarn init --private

# Install a few dependencies to get going
yarn add --dev typescript @types/node tsx @tsconfig/node18-strictest-esm prisma
yarn add @prisma/client

The last command installs the following tools:

TypeScript, to check that our code is type-safe, and type definitions for Node.js;
TSX, to run TypeScript without compiling the code;
A tsconfig.json preset that ensures type safety;
Prisma, to interact with the database.

# Bootstrap a Prisma project
yarn prisma init --datasource-provider sqlite

This command creates a few files, but the most interesting one is prisma/schema.prisma. Prisma offers to describe a database through a schema file: we will use this file to have Prisma create the tables for us.

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "sqlite"
  url      = env("DATABASE_URL")
}

// Let's declare a Post table
model Post {
  id    Int    @id @default(autoincrement())
  title String
  body  String
}

This model declares that we want a Post table with three columns. Our database doesn't exist yet, so let's create it:

# Make prisma create a database conforming to the schema
yarn prisma db push

# Ignore the SQLite database
echo 'dev.db' >> .gitignore

Everything is up and running! Prisma created a SQLite database for us in packages/api/prisma/dev.db.

Let's try to interact with it; create a file named packages/api/src/index.ts with the following code:

import { PrismaClient } from "@prisma/client";

const prisma = new PrismaClient();

// Insert a new post
await prisma.post.create({
  data: {
    title: "Hello World",
    body: "This is the first post",
  },
});

// Print all posts
console.log(await prisma.post.findMany());

To run this code, we will need to complete the project setup:

1. Create a `tsconfig.json` file in `packages/api`

Let's make use of the preset we installed earlier:

{
  "extends": "@tsconfig/node18-strictest-esm/tsconfig.json",
  "compilerOptions": {
    "exactOptionalPropertyTypes": false,
    "outDir": "./build"
  }
}

2. Update the `package.json`

The following lines tell Node.js that we write ECMAScript modules rather than CommonJS modules. In other words, we will use import rather than require(). We also define two package scripts to make our lives easier.

{
  "name": "api",
  "dependencies": {
    "@prisma/client": "^4.6.1"
  },
  "devDependencies": {
    "@tsconfig/node18-strictest-esm": "^1.0.1",
    "prisma": "^4.6.1",
    "tsx": "^3.12.1",
    "typescript": "^4.9.3"
  },
  "private": true,
  // Add the following lines: (without this comment)
  "scripts": {
    "build": "tsc",
    "dev": "tsx watch --clear-screen=false src/index.ts"
  },
  "type": "module"
}

3. Fasten your seatbelt – we're ready for takeoff

# Type-check and build the package
yarn build

# Run the code in watch mode (every time you save a file, it will be re-run)
yarn dev

You should see your first post printed in the console. Hello World!

Prisma types all the arguments and return values, allowing TypeScript to catch typos and provide relevant autocompletion. You can ensure that TypeScript does its job by removing the title or body of the post.create call and then run yarn build in the directory; you should see something like this:

src/index.ts:7:3 - error TS2322:
  ...
    Property 'title' is missing in type '{ body: string; }' but required in type 'PostCreateInput'.

We're done with the database part; let's move on to the backend.

Pothos

Pothos is a breeze of fresh air when it comes to building GraphQL APIs. It is a library that lets you write code-first GraphQL APIs with an emphasis on pluggability and type safety. And it has an awesome Prisma integration! (I am genuinely excited about this one, it makes my life so much easier.)

We will add a GraphQL API on top of our database, with a query to get articles and a mutation to create a new one.

Let's install Pothos and Yoga in the packages/api directory:

# Install Pothos and Yoga
yarn add @pothos/core @pothos/plugin-prisma graphql graphql-yoga

# Setup the Prisma-Pothos integration
echo 'generator pothos {\nprovider = "prisma-pothos-types"\n}' >> prisma/schema.prisma
yarn prisma generate

And let's also create a few files to define a simple GraphQL API:

`src/schema.ts`

This file will contain our queries and mutations. It's a good practice to split the schema file into several files to allow it to scale, the Pothos documentation has a dedicated section about it, but we will keep it simple for now.

import SchemaBuilder from "@pothos/core";
import PrismaPlugin from "@pothos/plugin-prisma";
import type PrismaTypes from "@pothos/plugin-prisma/generated";
import { PrismaClient } from "@prisma/client";
import { printSchema } from "graphql";
import { writeFile } from "node:fs/promises";

// Instantiate the Prisma client
const prisma = new PrismaClient();

// Instantiate the schema builder with the Prisma plugin
const builder = new SchemaBuilder<{ PrismaTypes: PrismaTypes }>({
  plugins: [PrismaPlugin],
  prisma: { client: prisma },
});

// Declare a `Post` GraphQL type, based on the table of the same name
const PostType = builder.prismaObject("Post", {
  fields: (t) => ({
    // Expose only the underlying data we want to expose
    id: t.exposeID("id"),
    title: t.exposeString("title"),
    body: t.exposeString("body"),
  }),
});

builder.queryType({
  fields: (t) => ({
    // Declare a new query field, `posts`, which returns the latest posts
    posts: t.prismaField({
      // Pothos makes sure that `type` and `resolve` are of the same type
      type: [PostType],
      resolve: async (query) =>
        // Return the 10 latest posts
        prisma.post.findMany({ ...query, orderBy: { id: "desc" }, take: 10 }),
    }),
  }),
});

builder.mutationType({
  fields: (t) => ({
    // Declare a new mutation field, `createPost`, which creates a new post
    createPost: t.prismaField({
      type: PostType,
      // The mutation takes a `title` and `body` arguments
      // They are correctly typed as string in `resolve`
      args: {
        title: t.arg.string({ required: true }),
        body: t.arg.string({ required: true }),
      },
      resolve: async (query, _, { title, body }) =>
        // Create a post and return it
        prisma.post.create({ ...query, data: { title, body } }),
    }),
  }),
});

export const schema = builder.toSchema();

/** Saves the schema to `build/schema.graphql`. */
export const writeSchema = async () =>
  writeFile(
    new URL("build/schema.graphql", `file://${process.cwd()}/`),
    printSchema(schema)
  );

This is enough to declare a type, a query, and a mutation.

You will soon be able to read the resulting schema in build/schema.graphql; it will look like this:

# No need to copy this, it will be automatically generated!
type Post {
  id: ID!
  body: String!
  title: String!
}

type Query {
  posts: [Post!]!
}

type Mutation {
  createPost(body: String!, title: String!): Post!
}

`src/index.ts`

This file will be the entry point of our application. It creates the GraphQL server and starts it.

import { createYoga } from "graphql-yoga";
import { createServer } from "node:http";
import { schema, writeSchema } from "./schema.js";

// Create a Yoga instance with the schema
const yoga = createYoga({ schema });

// Start an HTTP server on port 4000
createServer(yoga).listen(4000, () => {
  console.log("Server is running on http://localhost:4000/graphql");
});

// Save the schema to `build/schema.graphql`
await writeSchema();

`src/post-build.ts`

This file is not necessary to run the application, but it will come in handy to have a simple way to generate the schema file.

import { writeSchema } from "./schema.js";

await writeSchema();
console.log("✨ Schema exported");

Update the build script in `package.json`

{
  "scripts": {
    // Update the build script with what follows:
    "build": "prisma generate && tsc && yarn node ./build/post-build.js",
    "dev": "tsx watch --clear-screen=false src/index.ts"
  }
}

And we're all settled! You can run yarn dev if it's not already running and go to localhost:4000/graphql to play with the GraphQL API. Behold the magnificent GraphiQL interface! It looks really nice compared to its previous version, doesn't it?

You can try fetching and inserting data with the following queries:

query {
  posts {
    id
    title
    body
  }
}

mutation {
  createPost(title: "Is this thing on?", body: "Sure it is!") {
    id
  }
}

Things are working well... Let's make them look good!

Svelte

I won't go into the details of why Svelte, but I like Svelte a lot. It feels great writing Svelte code. And did I mention that it also offers type safety?

The easiest way to set up a new Svelte website is with SvelteKit; let's go back to the packages directory and create a new SvelteKit project:

# Create a Svelte app in the `packages/app` directory
# You will have a few choices prompted:
#  - Template: Skeleton project
#  - Type checking: TypeScript
#  - Prettier, ESLint, etc.: Not needed, do as you wish
yarn create svelte@latest app

# Install the dependencies
cd $_ && yarn install

This creates a bunch of new files in the packages/app directory. Let's take a look at the most important ones.

src
- routes
- +page.svelte: this is the index page of the website and also a Svelte component
package.json: the package manifest, with the dependencies and scripts
svelte.config.js: this is the Svelte configuration file

The package.json comes with a few scripts out of the box:

dev: starts the development server
build: builds the application for production
check: checks the code for type errors (yay!)

You can run svelte dev to see the hello world, but we need a missing piece before we can do anything useful: the GraphQL client.

GraphQL Zeus

You can think of GraphQL Zeus as Prisma for the frontend: it writes GraphQL queries out of JavaScript objects and produces the proper return types.

In the packages/app directory, add the following dependencies:

# Install GraphQL Zeus
yarn add --dev graphql-zeus

# Mark the API as a dev dependency
yarn add --dev api@workspace

# Gitignore the generated files
echo 'src/zeus/' >> .gitignore

Let's update packages/app/package.json:

{
  "script": {
    // Build the GraphQL client right before the rest of application
    "build": "zeus ../api/build/schema.graphql ./src --es && yarn check --threshold warning && vite build"
    // There's also `yarn check` in there to catch type errors
  }
}

Run yarn build, and you should see a new src/zeus/ directory with a bunch of files. Let's put all the pieces together!

Typed Board

How do we create a message board out of all of this? Keep reading – we're almost there!

We have a few things to add to packages/app for everything to work:

`src/lib/zeus.ts`

Zeus is a powerful tool, but it was not made to be used for server-side rendering. We will solve this by creating a small wrapper around the generated client:

import type { LoadEvent } from "@sveltejs/kit";
import { Thunder, type ValueTypes } from "../zeus/index";

/** A function that allows using Zeus with a custom `fetch` function. */
const thunder = (fetch: LoadEvent["fetch"]) =>
  Thunder((query, variables) =>
    fetch("http://localhost:4000/graphql", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ query, variables }),
    })
      // Errors are not properly handled by this code, but you get the idea
      .then((response) => response.json())
      .then(({ data }) => data)
  );

/** A nice wrapper around the unfriendly `thunder` above. */
export const query = async <Query extends ValueTypes["Query"]>(
  query: Query,
  { fetch }: { fetch: LoadEvent["fetch"] }
) => thunder(fetch)("query")(query); // That's a lot of parentheses

/** Same, but for mutations. */
export const mutate = async <Mutation extends ValueTypes["Mutation"]>(
  mutation: Mutation
  // No need for a custom fetch function here, since mutations are
  // never sent during server-side rendering
) => thunder(fetch)("mutation")(mutation);

It will make our GraphQL queries much nicer to write.

`src/routes/+page.ts`

This file provides the data to the +page.svelte component, both on the client (on browser navigation) and on the server (when using server-side rendering).

import { query } from "$lib/zeus";
import type { PageLoad } from "./$types";

/** This gets called on both the server and the client to provide page data. */
export const load: PageLoad = async ({ fetch }) =>
  // Perform a GraphQL query
  query(
    {
      // Query the `posts` field with all three columns
      posts: {
        // It looks a bit like Prisma, doesn't it?
        id: true,
        title: true,
        body: true,
      },
    },
    // Giving fetch allows server-side rendering
    { fetch }
  );

`src/routes/+page.svelte`

<script lang="ts">
  import { invalidateAll } from "$app/navigation";
  import { mutate } from "$lib/zeus";

  // SvelteKit magic: forward `load` function return type 🪄
  import type { PageData } from "./$types";

  // `export` means that `data` is a prop
  export let data: PageData;

  // Variables bound to the form inputs
  let title = "";
  let body = "";

  /** Sends the `createPost` mutation and refreshes the page. */
  const createPost = async () => {
    await mutate({ createPost: [{ body, title }, { id: true }] });
    await invalidateAll();
    title = body = "";
  };
</script>

<main>
  <h1>Typed Board</h1>
  <form on:submit|preventDefault={createPost}>
    <h2>New post</h2>
    <p>
      <label>Title: <input type="text" bind:value={title} required /></label>
    </p>
    <p>
      <label>Body: <textarea bind:value={body} rows="5" required /></label>
    </p>
    <p style:text-align="center">
      <button type="submit">Post!</button>
    </p>
  </form>
  <!-- `data.posts` is fully typed! -->
  {#each data.posts as post}
    <article>
      <!--
        Our tools tell us that `post` is of type
        `{
          id: string;
          title: string;
          body: string;
        }`
        If you remove `title: true` from `+page.ts`, you will see an error below
      -->
      <h2>{post.title}</h2>
      <pre>{post.body}</pre>
    </article>
  {/each}
</main>

<style>
  /* Let's add a bit of CSS magic... */
  :global(body) {
    font-family: system-ui, sans-serif;
    background-color: #fff6ec;
  }

  main {
    max-width: 60rem;
    margin: 2rem auto;
  }

  article,
  form {
    background-color: #fff;
    overflow: hidden;
    margin-block: 1rem;
    padding-inline: 1rem;
    box-shadow: 0 0 0.5rem #3001;
    border-radius: 0.25rem;
  }

  label {
    display: flex;
    gap: 0.25rem 0.5rem;
    flex-wrap: wrap;
  }

  input {
    flex: 1;
  }

  textarea {
    width: 100%;
    resize: vertical;
  }

  button {
    padding: 0.25em 0.5em;
  }
</style>

And that's it! We now have a working message board with end-to-end type safety. If you make a type error in this project, the compiler will catch it at build time. Huh, I am missing the most critical part...

Wrapping up

It's time to set up the whole check my types build scripts. Thanks to Yarn 4, that's a matter of only one command. Add the following scripts in the root package.json:

{
  "name": "typed-board",
  "private": true,
  "workspaces": ["packages/*"],
  // Add these scripts to build the two packages in the right order:
  "scripts": {
    "build": "yarn workspaces foreach --topological-dev -pv run build",
    "dev": "yarn workspaces foreach -piv run dev"
  }
}

This build command triggers all the packages' build commands in the correct (topological) order, and they are all set up to catch type errors. I also added a dev command that starts all the dev servers in parallel, for convenience.

# Launch the whole build pipeline and check the code 👀
yarn build

# Launch all the dev servers at once
yarn dev

And this concludes this unusually long article. I hope you enjoyed it and that you will find it helpful. If you have any questions, feel free to ask them in the comments below.

Find your way in GraphQL applications - Paths explained

c3b5aw — Thu, 01 Sep 2022 14:36:22 +0000

👋 Hey GraphQL folks!

We just published a new article talking about GraphQL and it's path system.

We also introduce a new open-source python module: graphenum!

You can find the full code used in the tutorial here.

Make sure to take a look !

Forem: Escape - API discovery and security

Workshop - How to Write Custom Security Tests

Methodology: How we discovered over 18,000 API secret tokens

Methodology

Development of the web spider

Data gathering strategy

Data collection process

Data cleanup and verification

Conclusion

Building your Product Security Roadmap

Leveraging Temporal for resilient remote procedure calls (RPC)

Temporal vs Connect (gRPC)

What is Temporal?

Temporal for cross-language RPC

Pros and cons

Closing words

Introducing Goctopus: open-source, state-of-the-art GraphQL endpoint discovery & fingerprinting tool.

Unmasking APIs: The Importance of API Discovery & Fingerprinting

Identifying the Void: Why We Needed Goctopus

Unleashing the Beast: Meet Goctopus

Quick Hands-On: Meet Goctopus

The Future of Goctopus: Contributing and Roadmap

Goctopus at Escape: Providing Top-Notch Security for Our Users

GraphQL errors: the Good, the Bad and the Ugly

The Ugly: the errors field

The Bad: an Error type

The Good: structured errors

HTTP errors

Read more

Closing words

Rendering emails with Svelte

Emails are written with 2003 HTML

MJML, Svelte, Vite, Square pegs and Round holes

The dev server

The build pipeline

Wrapping up

Achieving end-to-end type safety in a modern JS GraphQL stack

What is end-to-end type safety?

Data and type flows

Project setup

Prisma

1. Create a tsconfig.json file in packages/api

2. Update the package.json

3. Fasten your seatbelt – we're ready for takeoff

Pothos

src/schema.ts

src/index.ts

src/post-build.ts

Update the build script in package.json

Svelte

GraphQL Zeus

Typed Board

src/lib/zeus.ts

src/routes/+page.ts

src/routes/+page.svelte

Wrapping up

Find your way in GraphQL applications - Paths explained

The Ugly: the `errors` field

The Bad: an `Error` type

1. Create a `tsconfig.json` file in `packages/api`

2. Update the `package.json`

`src/schema.ts`

`src/index.ts`

`src/post-build.ts`

Update the build script in `package.json`

`src/lib/zeus.ts`

`src/routes/+page.ts`

`src/routes/+page.svelte`