Forem: James Mikrut

SQL vs. NoSQL - cutting through the Tech Twitter noise

James Mikrut — Thu, 11 May 2023 14:42:51 +0000

We've done a lot of thinking around database support lately, and even hosted a community discussion on Discord. Right now we're MongoDB-only, but our most upvoted roadmap item is adding support for additional databases. And transitioning from NoSQL thinking to an SQL mindset has really spotlighted the concrete differences.

If you've ever wondered exactly what NoSQL is good for, and alternatively, why SQL might be a better fit for certain applications, and you're tired of Tech Twitter telling you what to think, then read on for a real-world comparison.

In this post, I'm going to cover the big challenges that we'll face when adapting our features and functionalities over from NoSQL to SQL, and along the way, I'll highlight the strengths and weaknesses of each paradigm with some objective ramifications.

Data Definition Language

Since we're currently solely on MongoDB, we don't need to do things manually like adding a new table column when you add a new field to your Payload config. But with SQL, we need to think about that (and make it painless).

DDL, or Data Definition Language, is how SQL databases define and maintain their structure. Even when you use something like Prisma, which takes care of a lot of the headache for you, you need to become intimately familiar with how Prisma migrations work. There, Prisma translates its own schema syntax into SQL migrations for you.

Don't know what any of this means? Go ahead and try to simply follow the "Create a column in a table" example. If you know SQL and if this is not new to you, then you're probably okay with this. But there's work involved here.

When you decide you want to store a new field in a NoSQL database like MongoDB, you literally have to do nothing besides just start storing the data. No migrations, no generated DDL, no schema / database "sync".

That's obviously a big win for NoSQL, but that also means that the NoSQL structure is significantly more loosey-goosey and that can lead to errors. Lots of devs (not to mention project requirements) straight up require the structural constraints of SQL, and allowing any type of data to be stored in a NoSQL document can present issues in terms of data integrity. So there are definitely pros and cons to both approaches.

Migrations

Fascinatingly, another very hotly requested Payload feature is a first-party way of handling migrations. And that got us thinking. When you look at the Prisma migration resources above, clearly, they are using the word "migration" to describe the "migration" of your database shape from A to B. This is the expected and most widely used meaning of "migration".

But we're on MongoDB, and that doesn't apply. But developers are still asking for migration support. There is a specific distinction here that I'd like to make. Migrations can mean many things in many different contexts.

Changing database shape

Most often, they are referred to as simply modifying your database structure (DDL). This is a requirement for SQL in general. For example, adding a column to a table is a manual ALTER that needs to be run at some point.

Transforming existing data

But there's a whole different side to the story. What about if you need to change or transform your existing data? What if you had a full_name field, and you needed to split it up into two separate fields, first_name and last_name? Stuff like this is super common, and requires manual, programmatic management. Whenever anyone has asked Payload for migration support, this is what they meant.

Due to this second aspect of "migrations", that means that the concept is relevant for both NoSQL and SQL databases alike. Application frameworks like Laravel handle this very well, with a full, true "migrations" workflow that allows you to handle both DDL and data-based transformations within the same paradigm, all based on database transactions. I personally think this is by far the best way to solve this and this is an often-overlooked aspect of ORMs in general. Some ORMs come with great programmatic migration support, while others solely handle the DDL side (the pain in the ass side) in raw, generated SQL. Excuse my French, but I've been spoiled here by MongoDB, and I spoiled myself intentionally.

If you end up listening to our community planning call, you'll see that we cover our goals and future vision in-depth on that call. We want to model ourselves after Laravel's consistent (solve for both) approach by not outputting raw SQL, but rather, outputting TS files that give you granular, functional control.

Relational data and performance

One of the biggest wins for SQL in the minds of engineers is its heavily structured relational architecture. Without a doubt, relationships are a first-class citizen in SQL. Much of the querying language itself is dedicated specifically to combining and relating data from separate tables.
But relationships can certainly be done in NoSQL, so what's the deal? What are the real-world differences?

First up, there is an entire array of functionality within SQL that actually enforces relational data integrity. Within MongoDB, those types of constraints would all need to be written manually, and you'll need to ultimately "own" the integrity of your data. In NoSQL, you might have a relationship to another document. You could go delete that document, but unless you've handled this yourself, the original relationship would remain, yet be completely invalid. That's not ideal, and that places more work on you.

Let's take Payload, for example. Surprise, surprise. We have a relationship field, and it can store IDs to other related documents which are seamlessly merged in when you retrieve documents from the DB. We leverage the dataloader pattern to batch together all "populations" required for a given query, returning them all super fast and with as few separate queries to the DB as possible.
We actually even outperform SQL-based frameworks quite a bit. In a purely relational test, we were 3x faster than Directus and 7x faster than Strapi while both were running Postgres, and we were on MongoDB.

Without knowing what's under the hood of either of those platforms, my takeaway here is that relationships are possible in both NoSQL and SQL. If you were to write out queries deliberately for very specific joins between tables, and have the ability to define queries so that a single query is made vs. dynamic populations, then SQL will likely outshine MongoDB in terms of relational performance. But - your application logic will determine which is faster, and it's easy to get it wrong with either side.

One thing to note here is that if you are using a GraphQL API, or any type of dynamic / expressive querying language, it's impossible to make a single query that returns all populated "relations". This is because it's up to the shape of the query to determine what to "populate". This is known as the N+1 problem. You first need to load the main document, and only then can you determine what to load after. If this applies to you, then most relational performance gains from SQL are non-applicable.

Handling complex data structures

One absolutely massive win that NoSQL has over SQL is that you can store complex data directly in a single document. It can be stored in the same shape as you want it to be in when you read it back from the database. Some SQL databases like Postgres have pretty great JSON support via JSON columns, but if you over-use JSON columns in a SQL database, you're not taking advantage of the strengths of SQL in the first place and might as well go with NoSQL.

So what's it look like to store a complex data structure in SQL, without losing the benefits of SQL? For a simple example, take Payload's Group field:

{
  "id": "some-id-here",
  "myGroup": {
    "mySubField": "hello",
    "anotherSubField": "goodbye"
  }
}

The above field can be stored directly in NoSQL as-is, which is fantastic. But what about SQL-based databases? Do we transform the nested architecture to a flattened column structure, using a double underscore to separate field "paths"?

id	myGroup__mySubField	myGroup__anotherSubField
`some-id-here`	`hello`	`goodbye`

This could functionally perform the same, but we'd need to "transform" the data on its way out from the database somehow back to its intended shape. Doable, but not nearly as elegant.

Alternatively, we could "join in" another table that includes columns that represent properties within the group. That'll become a bit less performant, because we're now doing a join that we didn't need to do in NoSQL land.

Finally, do we just say "screw it" and embed the group and all of its properties, from that point on, into a single JSON column?

Field-based localization

One specific feature that we support, which is simple in NoSQL, but more difficult in SQL is field-based localization. You might have 30 locales, and any given text field needs to be translated into each of the 30 locales.

{
  "id": "some-id-here",
  "myLocalizedField": {
    "en": "hello",
    "es": "hola",
    "de": "hallo"
  }
}

Again, in MongoDB, this is a cake-walk. Storing the locales all on a parent document is great, because they are fundamentally a part of that parent document.

But in SQL we've gotta get crafty again. You might have 30 locales or more. Does that mean 30 columns for every single text field, all in one database?

id	myLocalizedField__en	myLocalizedField__es	myLocalizedField__de
`some-id-here`	`hello`	`hola`	`hallo`

If you have too many columns in SQL, you'll start to see performance implications. But this needs to be solved one way or another. Again, here, we could jump to a JSON column but then we're losing a lot.

Array-based structures

In NoSQL, storing arrays is dead simple. Just save it on the doc and move on.

{
  "id": "some-id-here",
  "myArray": [
    {
      "subField": "hello"
    },
    {
      "subField": "goodbye"
    }
  ]
}

The well-accepted and correct way to handle array-based data in SQL is to simply create a new table for each array-based field (like Payload's Arrays and Blocks fields). Then you'd just join in each row of array data when you retrieve the record back. It's more complicated, sure, and we're losing some performance here because we have to make joins. But it's logical and at least the answer is clearer than the last few examples.

Takeaways

Ultimately I think having a healthy dose of skepticism when reading anything that Tech Twitter or Reddit tells you will do you a great deal of good as an engineer. The truth is that there are certain cases where SQL will perform better for you, and NoSQL will be a better fit elsewhere. But it's up to your own unique requirements.

Overall, I think it's safe to say that SQL will win for your project if you:

Know the shape of your data well in advance
Are confident that your schema won't change often or are comfortable dealing with lots of DDL migrations
Have a relatively "flat" and not complex schema
Require heavily relational structures

On the flip-side, NoSQL will win out if you:

Frequently manage complex, nested data structures like Groups, Arrays, Blocks, localized field data, and similar
Use relationships solely for simple structures like Categories, Tags, Authors, etc.
Want to avoid dealing with DDL stuff entirely and make your life easier
Are okay with manually enforcing data integrity and handling edge cases yourself, or are using a framework that handles that stuff for you on top of NoSQL

Ah, life. Everything's a compromise. As much as I'd like to go hard and declare a victor, I totally understand why Payload SQL support has been asked for, although I personally think MongoDB is still a better fit for most of our users. In the end, you'll be able to pick whatever's right for you.

Planning session recording

We recorded the whole Discord planning session that took place last week. To hear more about how we're approaching future SQL support in Payload, including ORM evaluation and more, give it a listen.

Where can you help?

First up, we want to hear from you! If you haven't yet, make your voice and vote known on our Roadmap GitHub Discussion.

From there, once we have our future database adapter pattern established, anyone can build an adapter. We'd love to see some community-generated database and ORM support and that will only speed up / pressure test what we're building.

We're really appreciative of the engagement we saw on that call and are super pumped about doing more of those in the future. The next one will likely be regarding replacing Webpack, but we'll see. Follow us on Twitter to keep in the loop or join our Discord to keep an eye on the conversation there.

On GitHub Issues and Engineering Efficiency

James Mikrut — Thu, 09 Feb 2023 19:20:39 +0000

Earlier this week, Payload hit zero open issues which is pretty wild to me. When is the last time you saw a repo with 9K stars and zero open issues?

Obviously it's unrealistic to keep our issues at zero, but I attribute our low issue count to a lot of different tactics that we employ at Payload. First and foremost, I want to take a second to think about what the word "issue" means. Without looking it up, to me, it means "something wrong." Obviously it's got lots of meanings and can imply lots of things but go ask your grandma what it means to her. She'll probably say something like I said.

Because of this, I've always had a problem with using issues for anything more than exactly that—bugs, things wrong with the software, etc. And I've been pretty adamant, for better or for worse, that Payload uses GitHub Issues solely for those types of things. I guess we engineers are suckers for semantics. Me included.

Payload's love for semantics means our issues are reserved for bugs. We're not the first to do this, but it's pretty common to go to a big open-source repo and see 1 fafillion open issues. Typically, issue labels are used to differentiate between enhancements, bugs, chores, "good first issues", etc. So if you are coming into a repo and you want to evaluate what you can expect from the product's stability / etc, you'd need to filter by label bug or similar. If we keep issues clean and true to what the word "issue" truly represents, they can paint a very accurate, crowdsourced, at-a-glance picture of what you can expect to run into with Payload.

But the noise that comes from mixing bugs with features, etc. creates cognitive dissonance. It makes it difficult to see what is truly considered as a bug (officially classified by the repo maintainers) and what are issues that are maybe real, but not yet officially classified as "bug". It also muddies up the waters for the team that needs to manage those issues, because they have to filter through a sea of noise to prioritize what they should be working on. To get an accurate picture you need to figure out what labels the repo uses to classify "confirmed bugs", what labels might be used for "unconfirmed bugs", and then what issues might simply just not be labeled yet, although potentially represent real issues that you could run into.

For new features, Q&A, help, and everything else, we point our community to our GitHub discussions or Discord #community-help.

GitHub apparently doesn't like this

Unfortunately, Payload's use of Discussions for features / roadmap items does indeed come with some drawbacks. If you're like me, you've noticed that if you Google for help with any given open-source repo, you'll end up in GitHub issues quite frequently. But when's the last time you Googled for help and ended up in GitHub Discussions? My theory is that something is amiss regarding how Google indexes GitHub content. While Issues get great SEO, Discussions barely surface themselves. I actually did a very rudimentary test of this in SF while at a GitHub event with a few GH and Payload employees and we confirmed my SEO suspicions.

This was a few months ago, and we did indeed report our findings up the chain at GH, so maybe it's improved. But I would love GitHub to rectify this issue.

Another thing that's sucky about using Discussions instead of Issues is that you can't use Discussions within GitHub Projects. I would love to use GitHub Projects for our roadmap. I'm a sucker for a Gantt chart. But we can't surface Discussions in Roadmaps at all. Big Bummer.

So we've made do with our Roadmap existing entirely in Discussions, holding out hope for GitHub to improve this at some point in the future. At that point, I'll drink a Bud Heavy nodding to GitHub for a job well done, and for sticking to my semantic guns.

Issues for Payload are top priority

If you've been around for a while, you'll notice that we're fastest at responding to issues. We do our best to respond to Discord threads, GitHub discussions, and issues all the same - but issues get our attention immediately. When an issue opens, the whole team has a tendency to just drop what we're doing and take a look at what's happening.

It's really nice for our engineers if our issues are as moderated as possible. By establishing the pattern that Payload issues are reserved to "bugs", this greatly reduces the amount of times that our engineers get distracted by help discussions and feature requests. Of course we still do our best to reply to all community help, especially including discussions, but generally discussions are totally OK with a response that comes within a day or so vs. an issue being addressed more quickly.

Even still, keeping issues as our top priority does indeed cause distractions. And that creates friction, breaking our engineering trains-of-thought quite frequently.

You could be knee deep in a new feature at Payload but then see our #dev-feed Slack channel light up with a new issue report. It's tempting to just shut down work on your new feature and go check out what's going on. I'm 100% guilty of this myself and sometimes feel like I'm on a hamster wheel, just running around day after day trying to close issues as diligently as I can.

But the problem goes deeper. It exacerbates itself if you consider that one engineer might need support from other engineers closer to the feature in order to properly solve it. Might need a rubber duck. Might just want to pair with someone to work through the best solution. Then all of a sudden you'll have two or more engineers distracted—their trains of thought interrupted by the issue butterfly.

Our attempt to keep focus - Bugfix Mondays

To alleviate these ever-so-alluring distractions, we have established a sort of "triage" approach to new issue reports. If it's significant, it gets addressed as soon as possible. But for all other issues, we'll respond and acknowledge—but then we will try and save the fix until Mondays.

On Mondays, every engineer at Payload spends the whole day working through issues together. In the morning, we all review each open issue together, assigning issues to those that are best suited to solve the issue. And then we go our separate ways - setting up reproduction environments, and picking through the Payload logic to determine what's necessary to fix the issue.

So far, this is has been hugely successful for a variety of reasons. The first is that we get to avoid distractions, which allows our engineering team to focus for the other 4 days of the work week.

But second, and possibly much more important, is that if everyone is in "fix it" mode at the same time, we can bounce back and forth to help each other, brainstorm solutions, and review fixes.

So if engineer A was assigned a bug, but engineer B built the original feature, engineer A can ping B to say "hey, I'd like your eyes on this when you have a sec" and then once B is done, they can jump over to help A.

Knowledge is shared, effectiveness is ensured, and distractions are basically expected. But that's part of what makes our Mondays fun.

Of course this won't work for teams that get over a certain size, but right now, we're scrappy and it's fun. It's also pretty effective.

Like what we're doing? Give us a star on GitHub

We're trying to change the CMS status quo by delivering editors with a great experience, but first and foremost, giving developers a CMS that they don't absolutely despise working with. All of our new features are meant to be extensible and work simply and sanely.
Stop by GitHub and give us a star!

Get up and running with one line

Getting started with Payload is easy—and free forever. Just fire up a new terminal window and run the following command:

npx create-payload-app

It's 2022 - time to stop using headless WordPress and ACF

James Mikrut — Tue, 04 Oct 2022 13:17:43 +0000

In 2016, WordPress + ACF was one of the best ways to build a headless CMS. ACF made it useful. But it was never ideal.

The Payload team has built so many projects with headless WordPress + ACF, and at one time, it was the best tool for the job. ACF features like Conditional Logic and the "Flexible Content" field made it worth the WP headache.

But it always felt so wrong in so many ways. When we started building Payload, part of our goal was to take what ACF did well—but do it even better. We were set on not making any compromises. If ACF had a core feature that we used, Payload should have it as well. The same experience created in WP should be better in every single way if built in Payload.

That means in regards to developer experience, editor experience, speed, maintainability—everything.

Payload is a strong WordPress alternative

Now that Payload 1.0 is released, there's no longer a reason for developers to build on headless WP. We built a repo that shows how you'd typically build something in WP, and how you can replicate the same thing with Payload.

Check out the video below to see what makes headless WP a bad choice for modern devs, and why Payload is so much better suited to the job.

Give us a star on GitHub

If you haven't already, stop by our GitHub page and leave us a star by clicking on the star icon in the top right corner. This helps us grow and gain exposure within the development community.

Join our community on Discord

We've recently started a Discord community for the Payload community to interact in realtime. Often, Discord will be the first to hear about announcements like this move to open-source, and it can prove to be a great resource if you need help building with Payload. Click here to join!

Get up and running with one line

Getting started is easy—and free forever. Just fire up a new terminal window and run the following command:

npx create-payload-app

Speed test - Payload vs. Directus vs. Strapi

James Mikrut — Wed, 28 Sep 2022 13:42:59 +0000

Payload is a powerful alternative to Strapi or Directus, and in this post, we stacked them up in a head-to-head GraphQL performance test.

Disclaimer: clearly we're a bit partial to Payload, but we've done our best to make sure this test is as fair as possible. We are only comparing features that Payload, Directus, and Strapi equally support and are attempting to run the most fair comparison possible. That said, the results are very interesting and the test repos themselves serve as a little bonus showing how vastly different the real-world project workflows are from Payload to Directus to Strapi.

All of the test code is available and open-source. You can take a look at how we seed, what the queries we use look like, and how the test itself is structured. You can also run the tests yourself!

Links to seed scripts:

Links to GraphQL queries used:

And finally, here's a link to the performance testing script itself.

The idea behind the test

With this performance test, we wanted to see how a real-world, complex document query might fare while retrieved from the three different CMS' GraphQL endpoints. Let's consider a complex "mega menu" document, where there may be 30-50 "links" to other pages / posts / etc. and lots of media relations like icons, images, etc. that need to be rendered in a given mega menu. Just with that one mega menu document, we might have to retrieve a ton of "related" documents, with lots of JSON coming back from the response.

In our past experience, this can quickly become problematic (especially if you are server-side rendering) because that mega menu "document" is used by and needs to be retrieved for every single server-rendered view of a given app or website. That means that unless your CMS is heavily optimized, you're going to need to shell out some cash to make sure your server can handle this type of request. To make matters worse, modern frontend frameworks like Gatsby or Next often pre-render views, which means that during the build process, your server could get hammered with requests to your API.

Document structure we're testing against

To reflect a moderately complex real-world query, we designed a document structure that features 60+ relationships as well as complex data structures like groups, arrays, nested arrays, and blocks. The document itself is seeded predictably and exactly in the same manner through all three content management systems, and the GraphQL queries that are run are exactly the same outside of specific CMS syntax differences.

Ensuring environment parity

In all CMS benchmarks, we worked with a local dev environment and used local databases so that latency was eliminated as a factor. The database contents for each CMS benchmark were closely controlled so as to ensure that the number of documents / rows within each CMS database was as similar to one another as possible. The machine we used to test for all three vendors was a 16" Macbook Pro 2021, M1 Max with 32GB of RAM. Node version was 16.13.1.

We then wrote a simple script that could be shared by all three CMS benchmarks to hammer out 100 fetch requests sequentially, each with the same query, to the GraphQL HTTP endpoint. We then report on total test time, min response time, max response time, and average response time.

Seeding data to test with

In Payload

Because everything in Payload is code-based, seeding is super easy. We find that for local development, seeding is an absolute must - because that way you don't need to manually click around and create documents each time to test with, and the codebase can be spun up quickly by as many team members as necessary.

For this reason, in our own projects, we typically set the env variable PAYLOAD_DROP_DATABASE=true so that the database is dropped upon every server start, and then we seed an initial set of documents to test and build with. This is a super awesome dev pattern and really increases our team's velocity. It's also reusable for test suites and can help big time with automated testing.

Bonus: Payload ships with a Local API that can be used to efficiently seed documents with. It runs code directly on your Node server, with no need to go through the REST or GraphQL HTTP layers. This means it's blazingly fast (even faster than our GraphQL performance) and it's perfect for local seeding.

In Directus

With Directus, seeding is quite a bit more challenging because Directus is not code-based. Rather, your field configs are stored in your database itself, so there were a lot of steps for us to get this up and running.

We first had to initialize a project and create a first user to authenticate with. Then we had to design the field schema via pointing and clicking in the Directus UI (not in code). We then wrote a seed script to use the Directus SDK in order to generate some documents via the REST API. The seed script itself was pretty tricky to write because the data that we needed to pass to "relationship" fields is pretty Directus-specific and we had to do some reverse-engineering to figure it out.

Finally, we had to create an SQL dump of the database, which at this point contains all our fields and our first user. We stored the SQL dump in our repo for developers to be able to easily replicate this test without having to manually create a new project and configure all fields. But once you import the database dump, you still need to manually run the seed script to populate the database with documents to test against.

This is all quite a bit more complex than what you have to do in Payload. With Payload, a developer coming into the project for the first time just runs yarn dev. That's it. With Directus, there are quite a few more steps. We spent about a day trying to figure out if there was any way to export collection configs / re-import them into a new project but we gave up because we couldn't find anything in the docs. There are some discussions about adding import / export endpoints which would be a great feature, but as of now the process was a bit difficult for us.

In Strapi

It took us quite a while to figure out some oddities of the way everything works in Strapi. For example, the concept of an "admin user" is completely different than a "regular user", and we needed to write a shell script to create the admin user from the Strapi CLI. We got stuck for a while trying to authenticate via REST with our "admin user", only to find out that those users are different from regular "users". That's certainly a "gotcha".

For seeding, we opted not to use a config/functions/bootstrap.js, which seems to be the recommended solution. We found this to be an incomplete solution as it didn't allow us to create users or modify permissions. We ended up using a combination of scripts to create an admin user as well as authenticated user, load permissions using the Config Sync plugin's import command, then seeding the complex documents using the REST API. Again, with Strapi, there are quite a few hoops to jump through.

Benchmarks

Now that we had documents of the same data shape seeded within each CMS, we set off to run the benchmark tests themselves.

#1. Payload

We knew that Payload was fast and we place a lot of emphasis on ensuring that it performs as quickly as possible, but with our recent addition of the dataloader pattern, our result surprised even us.

Metric	Time
Average response time	15ms
Min response time	8ms
Max response time	43ms
Total test duration	1,513ms

#2. Directus

Directus came in second with some interesting results. The max response time was quite high, but average was not bad. Quite a bit slower than Payload, but still reasonable.

Metric	Time
Average response time	45ms
Min response time	24ms
Max response time	139ms
Total test duration	4,459ms

#3. Strapi

Strapi fell quite a bit behind both Payload and Directus and this is interesting to us because of the SQL-based nature of Strapi 4, and the relational nature of our tests.

Metric	Time
Average response time	102ms
Min response time	77ms
Max response time	353ms
Total test duration	10,172ms

Conclusion

On average, Payload is 3x faster than Directus and almost 7x faster than Strapi in response time.

We're super proud of the efficiency that we've been able to produce with Payload and this is only the beginning. Our team is expanding, and we have lots of plans in store over the next few months including even more UI optimizations, new features, and lots of tutorials / example boilerplates. Oh, and Payload Cloud. Keep an eye out for that one because it's going to be awesome.

Give Payload a star

We're completely free, MIT-licensed and open-source. If you haven't already, stop by our GitHub page and leave us a star by clicking on the star icon in the top right corner. This helps us grow and gain exposure within the development community.

Get started in one line

If you haven't yet given Payload a shot, you can get started with one command:

npx create-payload-app

Announcing the official Payload Cloud Storage Plugin

James Mikrut — Tue, 09 Aug 2022 15:15:00 +0000

Payload just released its official Cloud Storage plugin, with support for file storage in AWS S3 and Azure Blob Storage out of the box.

Payload plugins can be insanely powerful, and they're trivial to write thanks to Payload's code-first, config-based nature. We've already seen an impressive array of community plugins, including a few great cloud storage solutions from Richard VanBergen and Alex Bechmann. We're extremely thankful for our community, especially Richard and Alex in regards to their help with cloud storage.

Over the past few months, enterprise clients of ours have asked quite a few times for us to build an officially supported cloud storage plugin, so that all files uploaded to Payload can be stored in the cloud storage provider of their choice. Of course, by default, Payload allows you to store uploads on your own server, but at scale, you might want to offload them to a CDN-backed service for performance and scalability reasons.

So we built a robust, officially supported cloud storage plugin. It's extremely powerful and comes out-of-the-box with Amazon S3 and Azure Blob Storage support. The plugin is built with an adapter-based approach, so you can use it with any cloud storage provider that you wish.

The plugin is available now - check it out on GitHub or NPM!

Usage

Installing and using the plugin is easy. Here's what it looks like to use the S3 adapter:

import { buildConfig } from "payload/config";
import Media from "./collections/Media";
import { cloudStorage } from "@payloadcms/plugin-cloud-storage";
import { s3Adapter } from "@payloadcms/plugin-cloud-storage/s3";

export default buildConfig({
  collections: [Media],
  plugins: [
    // Pass the plugin to Payload
    cloudStorage({
      collections: {
        // Enable cloud storage for Media collection
        media: {
          // Create the S3 adapter
          adapter: s3Adapter({
            config: {
              endpoint: process.env.S3_ENDPOINT,
              credentials: {
                accessKeyId: process.env.S3_ACCESS_KEY_ID,
                secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
              },
            },
            bucket: process.env.S3_BUCKET,
          }),
        },
      },
    }),
  ],
});

With this plugin, instead of saving files locally to your server, Payload will ship off all uploads, including all automatically resized images, straight to your cloud storage provider.

Adapter-based Implementation

The plugin is built on a simple, yet powerful adapter-based implementation. This means it can be used for any type of cloud storage you need. Out of the box, it comes with Azure Blob Storage and AWS S3 support, but you can easily write your own adapter(s) to host your uploads wherever you need.

Access Control

Payload comes with built-in access control for all uploads. It works seamlessly by re-using your upload-enabled collection's read access control—meaning you can easily restrict who can access which uploads that are stored in your site. This is super powerful in lots of cases, and is a specific strength of Payload over other headless CMS.

Imagine if you had a public Customers collection, and your customers could upload sensitive documents to attach to their profile. You may only want certain users to be able to view these sensitive documents - i.e. admins or the customers themselves. But most people should be denied access.

With Payload, this is built-in. And this plugin fully supports this functionality, so you can build powerful and elegant access control patterns to protect your files—even when they're hosted on your cloud provider.

Get started

Get started by installing the plugin in your Payload app with the following command:

yarn add @payloadcms/plugin-cloud-storage

From there, check out the docs to learn about all the things you can do.

Give Payload a star

If you haven't already, stop by our GitHub page and leave us a star by clicking on the star icon in the top right corner. This helps us grow and gain exposure within the development community.

Request a Plugin

Need a plugin to be created? Join our Discord server and post in the #plugins channel. The Payload team might create it for you, or maybe a community member can help!

Get started in one line

If you haven't yet given the CMS a shot, you can get started free with one command:

npx create-payload-app

Tutorial - Configuring TypeScript, Express, Jest, Payload, and VSCode Debugging from Scratch

James Mikrut — Thu, 02 Jun 2022 12:48:37 +0000

Setting up a TypeScript project from scratch can be intimidating. Bring in Jest, VSCode debugging, and a headless CMS - and there are many moving parts. But it doesn't have to be difficult!

We've been working hard at making Payload + TypeScript a match made in heaven, and over the last few months we've released a suite of features, including automatic type generation, which makes Payload by far the best TypeScript headless CMS available. To celebrate, we're going to show you how to scaffold a TypeScript and Express project from scratch, including testing it with Jest and debugging with VSCode.

By understanding just a few new concepts, you can master your dev environment's setup to maximize productivity and gain a deep understanding how it all works together. Let's get started.

Software Requirements

Before going further, make sure you have the following software:

Yarn or NPM
NodeJS
A Mongo Database

Step 1 - initialize a new project

Create a new folder, cd into it, and initialize:

mkdir ts-payload && cd ts-payload && yarn init

Step 2 - install dependencies

We'll need a few baseline dependencies:

dotenv - to set up our environment easily
express - Payload is built on top of Express
ts-node - to execute our TypeScript project in development mode
typescript - base TS dependency
payload - no description necessary
nodemon - to make sure our project restarts automatically when files change

Install these dependencies by running:

yarn add dotenv express payload

and the rest as devDependencies using:

yarn add --dev nodemon ts-node typescript

Step 3 - create a tsconfig:

In the root of your project folder, create a new file called tsconfig.json and add the following content to it. This file tells the TS compiler how to behave, including where to write its output files.

Example tsconfig.json:

{
  "compilerOptions": {
    "target": "es5",
    "lib": [
      "dom",
      "dom.iterable",
      "esnext"
    ],
    "outDir": "./dist",
    "skipLibCheck": true,
    "strict": false,
    "esModuleInterop": true,
    "module": "commonjs",
    "moduleResolution": "node",
    "allowSyntheticDefaultImports": true,
    "resolveJsonModule": true,
    "isolatedModules": true,
    "jsx": "preserve",
    "sourceMap": true
  },
  "include": [
    "src"
  ],
  "ts-node": {
    "transpileOnly": true
  }
}

In the above example config, we're planning to keep all of our TypeScript files in /src, and then build to /dist.

Step 4 - set up your .env file

We'll be using dotenv to manage our environment and get ourselves set up for deployment to various different environments like staging and production later down the road. The dotenv package will read all values in a .env file within our project root and bind their values to process.env so that you can access them in your code.

Important:

If you're using GitHub, make sure you ignore your environment file from your repository. Often, environment files store sensitive information and should not be included in source code.

Let's create a .env file in the root folder of your project and add the following:

PORT=3000
MONGO_URL=mongodb://localhost/your-project-name
PAYLOAD_SECRET_KEY=alwifhjoq284jgo5w34jgo43f3
PAYLOAD_CONFIG_PATH=src/payload.config.ts
PAYLOAD_PUBLIC_SERVER_URL=http://localhost:3000

Make sure that the MONGO_URL line in your .env matches an available MongoDB instance. If you have Mongo running locally on your computer, the line above should work right out of the box, but if you want to use a hosted MongoDB like Mongo Atlas, make sure you copy and paste the connection string from your database and update your .env accordingly.

For more information on what these values do, take a look at Payload's Getting Started docs.

Step 5 - create your server

Setting up an Express server might be pretty familiar. Create a src/server.ts file in your project and add the following to the file:

import express from 'express';
import payload from 'payload';
import path from 'path';

// Use `dotenv` to import your `.env` file automatically
require('dotenv').config({
  path: path.resolve(__dirname, '../.env'),
});

const app = express();

payload.init({
  secret: process.env.PAYLOAD_SECRET_KEY,
  mongoURL: process.env.MONGO_URL,
  express: app,
})

app.listen(process.env.PORT, async () => {
    console.log(`Express is now listening for incoming connections on port ${process.env.PORT}.`)
});

The file above first imports our server dependencies. Then, we use dotenv to load our .env file at our project root. Next, we initialize Payload by providing it with our secret key, Mongo connection string, and Express app. Finally, we tell our Express app to listen on the port defined in our .env file.

Step 6 - create a nodemon file

We'll use Nodemon to automatically restart our server when any .ts files change within our ./src directory. Nodemon will execute ts-node for us, which will use our server as its entry point. Create a nodemon.json file within the root of your project and add the following content.

Example nodemon.json:

{
  "watch": [
    "./src/**/*.ts"
  ],
  "exec": "ts-node ./src/server.ts"
}

Step 7 - add your Payload config

The Payload config is central to everything Payload does. Add it to your src folder and enter the following baseline code:

./src/payload.config.ts:

import dotenv from 'dotenv';
import path from 'path';
import { buildConfig } from 'payload/config';

dotenv.config({
  path: path.resolve(__dirname, '../.env'),
});

export default buildConfig({
  serverURL: process.env.PAYLOAD_PUBLIC_SERVER_URL,
    typescript: {
        outputFile: path.resolve(__dirname, './generated-types.ts'),
    },
  collections: [
    {
      slug: 'posts',
      admin: {
        useAsTitle: 'title',
      },
      fields: [
        {
          name: 'title',
          type: 'text',
        },
        {
          name: 'author',
          type: 'relationship',
          relationTo: 'users',
        },
      ]
    }
  ]
});

This config is very basic - but check out the Config docs for more on what the Payload config can do. Out of the box, this config will give you a default Users collection, a simple Posts collection with a few fields, and will open up the admin panel to you at http://localhost:3000/admin.

The config also specifies where Payload should output its auto-generated TypeScript types which is super cool (we'll come back to this).

Step 8 - add some NPM scripts

The last step before we can fire up our project is to add some development, build, and production NPM scripts.

Open your package.json and update the scripts property to the following:

{
  "scripts": {
    "generate:types": "PAYLOAD_CONFIG_PATH=src/payload.config.ts payload generate:types",
    "dev": "nodemon",
    "build:payload": "PAYLOAD_CONFIG_PATH=src/payload.config.ts payload build",
    "build:server": "tsc",
    "build": "yarn build:server && yarn build:payload",
    "serve": "PAYLOAD_CONFIG_PATH=dist/payload.config.js NODE_ENV=production node dist/server.js"
  },
}

To support Windows environments consider adding the cross-env package as a devDependency and use it in scripts before setting variables.

The first script uses Payload's generate:types command in order to automatically generate TypeScript types for each of your collections and globals automatically. You can run this command whenever you need to regenerate your types, and then you can use these types in your Payload code directly.

Tip:

Payload's ability to automatically generate types from your configs is super powerful and will benefit you immensely if you are writing custom hooks or custom components.

The next script is to execute nodemon, which will read the nodemon.json config that we've written and execute our /src/server.ts script, which fires up Payload in development mode.

The following three scripts are how we will prepare Payload's admin panel for production as well as how to compile the server's TypeScript code into regular JS to use in production.

Finally, we have a serve script which is used to serve our app in production mode once it's built.

Firing it up

We're ready to go! Run yarn dev in the root of your folder to start up Payload. Then, visit http://localhost:3000/admin to create your first user and sign into the admin panel.

Generating Payload types

Now that we've got a server generated, let's try and generate some types. Run the following command to automatically generate a file that contains an interface for the default Users collection and our simple Posts collection:

yarn generate:types

Then check out the file that was created at /src/generated-types.ts. You can import these types in your own code to do some pretty awesome stuff.

Testing with Jest

Now it's time to get some tests written. There are a ton of different approaches to testing, but we're going to go straight for end-to-end tests. We'll use Jest to write our tests, and set up a fully functional Express server before we start our tests so that we can test against something that's as close to production as possible.

We'll also use mongodb-memory-server to connect to for all tests, so that we don't have to clutter up our development database with testing documents. This is great, because our tests will be totally controlled and isolated, but coverage will be incredibly thorough due to how we'll be testing the full API from top to bottom.

Payload will automatically attempt to use mongodb-memory-server if two conditions are met:

It is locally installed in your project
NODE_ENV is equal to test

Adding and configuring test dependencies

OK. Let's install all the testing dependencies we'll need:

yarn add --dev jest mongodb-memory-server babel-jest @babel/core @babel/preset-env @babel/preset-typescript isomorphic-fetch @types/jest

Now let's add two new config files. First, babel.config.js:

module.exports = {
  presets: [
    ['@babel/preset-env', { targets: { node: 'current' } }],
    '@babel/preset-typescript',
    '@babel/preset-react',
  ],
};

We use Babel so we can write tests in TypeScript, and test full React components.

Next, jest.config.js:

module.exports = {
  verbose: true,
  testEnvironment: 'node',
  globalSetup: '<rootDir>/src/tests/globalSetup.ts',
  roots: ['<rootDir>/src/'],
};

Global Test Setup

A sharp eye might find that we're using a globalSetup file in jest.config.js to scaffold our project before any of the real magic starts. Let's add that file:

src/tests/globalSetup.ts:

import '../server';
import { resolve } from 'path';
import testCredentials from './credentials';

require('dotenv').config({
  path: resolve(__dirname, '../.env'),
});

const { PAYLOAD_PUBLIC_SERVER_URL } = process.env;

const globalSetup = async (): Promise<void> => {
  const response = await fetch(`${PAYLOAD_PUBLIC_SERVER_URL}/api/users/first-register`, {
    body: JSON.stringify({
      email: testCredentials.email,
      password: testCredentials.password,
    }),
    headers: {
      'Content-Type': 'application/json',
    },
    method: 'post',
  });

  const data = await response.json();

  if (!data.user || !data.user.token) {
    throw new Error('Failed to register first user');
  }
};

export default globalSetup;

In this file, we're performing the following actions before our tests are executed:

Importing the server, which will boot up Express and Payload using mongodb-memory-server
Loading our .env file
Registering a first user so that we can authenticate in our tests

You'll notice we are importing testCredentials from next to our globalSetup file. Because our Payload API will require authentication for many of our tests, and we're creating that user in our globalSetup file, we will want to reuse our user credentials in other tests to ensure we can authenticate as the newly created user. Let's create a reusable file to store our user's credentials:

src/tests/credentials.ts:

export default {
  email: 'test@test.com',
  password: 'test',
}

Our first test

Now that we've got our global setup in place, we can write our first test.

Add a file called src/tests/login.spec.ts:

import { User } from '../generated-types';
import testCredentials from './credentials';

require('isomorphic-fetch');

describe('Users', () => {
  it('should allow a user to log in', async () => {
    const result: {
      token: string
      user: User
    }  = await fetch(`${process.env.PAYLOAD_PUBLIC_SERVER_URL}/api/users/login`, {
      method: 'post',
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        email: testCredentials.email,
        password: testCredentials.password,
      }),
    }).then((res) => res.json());

    expect(result.token).toBeDefined();
  });
});

The test above is written in TypeScript and imports our auto-generated User TypeScript interface to properly type the fetch response that is returned from Payload's login REST API.

It will expect that a token is returned from the response.

Running tests

The last step is to add a script to execute our tests. Let's add a new line to our package.json scripts property:

    "test": "jest --forceExit --detectOpenHandles"

Now, we can run yarn test to see a successful test!

Debugging with VSCode

Debugging can be an absolutely invaluable tool to developers working on anything more complex than a simple app. It can be difficult to understand how to set up, but once you have it configured properly, a proper debugging workflow can be significantly more powerful than just relying on console.log all the time.

You can debug your application itself, and you can even debug your tests to troubleshoot any tests that might fail in the future. Let's see how to set up VSCode to debug our new Typescript app and its tests.

First, create a new folder within your project root called .vscode. Then, add a launch.json within that folder, containing the following configuration:

./.vscode/launch.json:

{
  "version": "0.2.0",
  "configurations": [
        {
      "type": "node",
      "request": "launch",
      "name": "Launch Program",
      "env": {
        "PAYLOAD_CONFIG_PATH": "src/payload.config.ts",
      },
      "program": "${workspaceFolder}/src/server.ts",
    },
    {
      "name": "Debug Jest Tests",
      "type": "node",
      "request": "launch",
      "runtimeArgs": [
        "--inspect-brk",
        "${workspaceRoot}/node_modules/.bin/jest",
        "--runInBand"
      ],
      "env": {
        "PAYLOAD_CONFIG_PATH": "src/payload.config.ts"
      },
    }
  ]
}

The file above includes two configurations. The first is to be able to debug your Express + Payload app itself, including any files you've imported within your Payload config. The second debug config is to be able to set breakpoints and debug directly within your Jest testing suite.

To debug, you can set breakpoints right in your code by clicking to the left of the line numbers. A breakpoint will be set and show as a red circle. When VSCode executes your scripts, it will pause at the breakpoint(s) you set and allow you to inspect the value of all variables, where your function(s) have been called from, and much more.

To start the debugger, click the "Run and Debug" sidebar icon in VSCode, choose the debugger you want to start, and click the "Play" button. If you've placed breakpoints, VSCode will automatically pause when it reaches your breakpoint.

Here is an example of a breakpoint being hit within our src/server.ts file:

Here is a screenshot of a breakpoint being hit within our login.spec.ts test:

Debugging can be an invaluable tool to you as a developer. Setting it up early in your project will pay dividends over time as your project gets more complex, and it will help you understand how your project works to an extremely fine degree.

Conclusion

With all of the above pieces in place, you have a modern and well-equipped dev environment that you can use to build out Payload into anything you can think of - be it an API to power a web app, native app, or just a headless CMS to power a website.

You can find the code for this guide here. Let us know what you think!

Star us on GitHub

If you haven't already, stop by GitHub and give us a star by clicking on the "Star" button in the top-right corner of our repo. With our inclusion into YC and our move to open-source, we're looking to dramatically expand our community and we can't do it without you.

Join our community on Discord

Building a Professionally Designed Website Episode 4 - Heroes, Layout Building Blocks, Animations & Design

James Mikrut — Tue, 24 Aug 2021 12:03:09 +0000

This is Part 4 in our deep-dive into how to develop a professionally designed, high-end NextJS + Payload CMS website from start to finish.

In this episode, we'll finish out the entire site. Buckle up—it's a long one. You'll see how to develop a super dynamic layout builder that puts the exact right amount of control into your admins' hands. You'll see how a wide variety of layout blocks are built while all relying on a very solid React + TypeScript codebase. In addition to the layout building components, we'll go through each Hero style that we've designed, show how the Form integration works, and add some miscellaneous polish to the site here and there.

By the end of this video, the site will be fully complete.

0:00 - Intro
2:31 - Payload Roadmap
4:11 - Dynamically generated admin descriptions
5:07 - Plugins infrastructure
6:41 - Remaining work to be done
10:00 - Picking up where we left off
11:17 - Building the Heroes
12:35 - Extending Payload's Rich Text Editor
16:29 - Field Conditional Logic
22:05 - Converting SlateJS Rich Text into JSX
25:35 - Creating custom Rich Text Editor components on the frontend
37:12 - Media component
37:34 - Payload's "depth" function
43:41 - Beginning the layout blocks
47:22 - The Call to Action block
59:13 - The Content block
1:00:12 - More Rich Text Editor customization
1:15:14 - The Media block
1:16:23 - Parallax component
1:22:38 - Slider block
1:31:00 - Study Slider block
1:34:19 - Spacer block
1:35:19 - Statistics block
1:45:05 - Sticky Content block
1:56:03 - Forms & Form Submissions
2:15:02 - Background Noise
2:17:30 - Outro

The design of the site was done by TRBL.

Use Payload for free

Check out Payload for free, forever. The Personal License gives you everything Payload does, all for free. We think you'll find Payload's developer experience second to none. It just feels right to build sites and apps with it.

Feedback welcome

As always, let us know what you think about the video - and about Payload!

The Payload Headless CMS just got more powerful with Plugins

James Mikrut — Wed, 28 Jul 2021 13:58:51 +0000

Payload's new and robust Plugins infrastructure makes reusing and modularizing your Payload code super easy.

Because Payload is built as a code-first, config-based headless CMS, using third-party plugins and even writing your own is extremely straightforward. All that's necessary is a little knowledge of JavaScript, and then from there, you can extend and modify Payload in any way that you can dream up.

With Plugins, Payload can be extended and modified in a ton of ways. Over time, we're going to build up a library of commonly requested and officially supported plugins—but what's really exciting is the fact that anyone can make their own with just a little bit of JS knowledge.

const config = buildConfig({
  // .. your config here
    serverURL: 'http://localhost:3000',

  // plugins are just simple JS functions
  plugins: [
    passwordProtect(['pages']),
    syncToHubSpot(['customers']),
  ],
});

Easy to learn, easy to write

Most CMS plugin systems are upsettingly difficult to learn and require you to devote yourself to learning how to build a plugin for that CMS. You might be able to cobble something together after spending a day or so of time learning how to work in someone else's system, but you'll never get that time back. Worse yet - your learning will only ever apply to that one platform.

Payload is different. Because the core infrastructure of Payload is so open-ended, being based fully on vanilla JS / TS, the knowledge that you need to build Payload plugins is universally applicable in your career as a developer. Writing a Payload plugin is as easy as taking in a config, and then returning an enhanced config. Within, you can rely on Payload's already powerful infrastructure design like Hooks, function-based access control, custom components, open-ended Express usage, and more.

Possibilities

The possibilities run a lot wider than what you might expect at first glance. You can add and enhance just about any part of Payload. Below, we'll write out some examples.

CRM Connector (MailChimp, HubSpot, Salesforce, etc.)

Imagine if your Customers collection could be automatically synced with HubSpot, MailChimp, or similar each time a customer is created or updated.

Required plugin functions:

Add an afterChange hook to relevant collections that takes the incoming data and sends it to your CRM

Easy. The plugin would just take in a user's config, copy it, inject some hooks containing your platform's integration logic, and return the new config.

Field encryption

If you're working with sensitive data, you might want to encrypt certain fields, so that even when their data is stored at rest in the database, it is not in plaintext. Common examples might be passwords, API keys, HIPAA data, credit card data, or other personally identifiable information. A Plugin could be easily written to automatically encrypt/decrypt data as it enters and exits your APIs. The plugin would need to:

Add a beforeChange hook to all fields needing to be encrypted that converts the incoming value to an encrypted counterpart
Add an afterRead hook too all encrypted fields which decrypts them automatically as they are sent out by your APIs

Form builder

A common use case for sites and apps is to expose a fully featured Form Builder to your editors. One way to do this through a Plugin would be as follows:

Inject a new collection called Forms. The Forms collection would come with a blocks field type, where each block would represent a field type (select, text, checkbox, etc.) Editors could then build out their own custom forms and build relationship fields which would reference them.
Inject a new collection called Form Submissions. This new collection would accept JSON submissions and automatically validate their incoming data compared to the corresponding Form.

Password-protection

What if you needed password protection over documents in certain collections? A plugin could do this easily. It would need to:

Automatically inject some new fields into password-enabled collections
Add a new REST endpoint as well as a new GraphQL mutation for end-users to provide a password, and have their password be validated
Add a beforeRead hook to all appropriate collections that shows and hides the document based on if the user has access
Conditionally show and hide existing fields in the admin panel based on if the user has access

Stay tuned for this one.

This is the first officially-supported plugin that Payload will release. You'll be able to read the source code to see how it's done.

Other possibilities

Here are some other quick example use cases:

Add a full ecommerce backend to any Payload app
Add custom reporting views to Payload's admin panel
Integrate all upload-enabled collections with a third-party file host like S3 or Cloudinary
Add custom routes or GraphQL queries / mutations with any type of custom functionality that you can think of

Documentation

Find more information, including a simple example plugin, in our Plugin documentation.

Request a Plugin

Need a plugin to be created? Start a GitHub discussion. If we don't build it, maybe someone else has - or will!

Let us know your thoughts

We've got big things planned for Payload, and we'd love to hear what you think. If you haven't yet given the CMS a shot, you can get started free with one command:

npx create-payload-app

Building a Professionally Designed Website Episode 3 - Header & Footer, Hamburger Menu, Animating SVGs

James Mikrut — Mon, 07 Jun 2021 19:51:12 +0000

This is Part 3 in our deep-dive into how to develop a professionally designed, high-end NextJS + Payload CMS website from start to finish.

In this episode, we're diving deep into React. You'll learn how a design agency organizes its components and you'll start to see a lot of the CSS boilerplate that we defined in Episode 2 start being put to use. We're going to build and animate the Header, Footer, and Mega Menu—all in React and TypeScript.

0:00 - Intro
3:33 - The logo - an SVG React component
12:02- Animating the hamburger button
25:13 - Retrieving menu global data from Payload
32:27- Building the main menu
38:48- Building a CMSLink component
48:25- The animated "Let's Talk" SVG call to action
55:35- Building the Footer
1:14:06- Outro

Use Payload for free

Feedback welcome

As always, let us know what you think about the video - and about Payload!

Building a Professionally Designed Website Episode 2 - CSS Framework, Baseline Grid, NextJS Structure

James Mikrut — Tue, 20 Apr 2021 14:38:04 +0000

This is Episode 2 in a series that takes you through how to build a high-end website from a design agency's perspective using Payload CMS, TypeScript, and statically rendered NextJS.

If you missed the first episode, I'd highly recommend going back and watching that one before jumping in here. In that first episode, we covered a lot of ground, including the setup of the entire Payload CMS needed to drive the content of the website.

In that first episode, we defined all Collections and Globals necessary as well as went in-depth with layout-building Block definitions.

Now, in Episode 2, we're ready to start work on the frontend. In this episode, we dive into how to build a lightweight but powerful CSS framework that we'll rely on through the entire site. We'll define breakpoints, media queries, typography, colors, a global reset, and finally - a really awesome way to build a baseline grid using rems.

0:00 - Intro
2:01 - React JSS + NextJS setup
4:27 - Writing our first global styles
10:27 - The baseline grid - in CSS rems
17:10 - Reusable values (breakpoints, colors)
19:37 - Typography
22:48 - CSS media queries
27:24 - Installing dependencies (CSS grid, Modal, Window Info)
35:15 - First component - GridContainer
42:47 - Testing our grid, discussing NextJS page folder
47:48 - Stubbing out the Template, Header and Footer components
55:22 - Outro

Give Payload a shot

If you're a JS / TS developer and haven't heard of Payload yet, you should give it a shot. Its developer experience is second to none and it just feels right to build sites and apps with it.

Feedback welcome

As always, let us know what you think about the video - and about Payload!

Building a Professionally Designed Website with NextJS, TypeScript, and Payload CMS - Ep. 1

James Mikrut — Tue, 23 Mar 2021 14:51:03 +0000

Have you ever wondered how professional design firms go about building high-end websites? If so, this series is for you.

Hi, my name is James. Outside of co-founding Payload CMS, I also operate a professional design firm called TRBL. There, we specialize in custom UX and full-stack development—usually TypeScript front-to-back. I've been a designer and full-stack developer for around 10 years now and self-employed for six. I absolutely love the work that I do.

I have been working on a new video series that describes, in detail, how professional design firms like TRBL go about building completely custom websites using a headless CMS, React, and TypeScript. This series is going to be a bit different than the typical to-do app style videos and I want it to show a much deeper level of project approach and code mentality—all the way from how to scaffold centralized, reusable TypeScript types to how to build websites with React layout-building components.

0:00 - Intro
3:16 - Project plan & series overview
15:26 - Design review
25:17 - Payload introduction
33:25 - Payload + NextJS boilerplate setup
43:17 - CMS scaffolding (form submissions, studies, shared Payload fields, etc.)
1:00:08 - The first layout building block - Content
1:16:05 - Globals configuration (mega menu, footer, social media)
1:35:53 - All layout blocks - fast-forward and review
1:46:49 - Recap

Series Goals

One of my goals with this series is to provide a window into how professional design firms go about delivering sites with a lot of value to clients that absolutely need that level of quality, and are willing to pay for it. The site that we'll be building is actual, real-world work, and although it's more of a small-to-medium budget for us, the principles involved can be used to deliver websites where clients will happily seek out talented designers / developers and pay $100K+, because they need it.

Another goal of mine is that I want to challenge our industry to find the clients that value the work that we do, and I want developers to stop selling themselves short. Also, don't worry about the Squarespaces of the world. Find the budgets that reward you for your skill. There is a segment of the market that still absolutely requires high-end development standards and approaches, and it's alive and well.

Feedback is Appreciated

This is episode 1 - and it's my first ever YouTube video. I would greatly appreciate any candid feedback that this community has for me, and if you haven't already, please give Payload a shot. We'd love to hear your thoughts.

Let us know what you think, and thank you!

NextJS, Payload, and TypeScript in a Single Express Server Boilerplate

James Mikrut — Tue, 09 Feb 2021 14:49:30 +0000

One of Payload CMS’ core ideologies is that it does not impose any of its own structure on you as a developer. You give it an Express server to use—not the other way around, and this pays dividends in developer freedom and developer experience.

An example of how this approach can be leveraged is by running a full NextJS site on the same Express app as your Payload CMS instance. We’ve built a boilerplate that demonstrates exactly how this works:

Check it out:

https://github.com/payloadcms/nextjs-custom-server

This boilerplate includes the following:

Payload CMS and NextJS up and running on a single Express server
Super fast Local API usage within pages’ getServerSideProps
A demonstration for how to use TypeScript in a Payload and NextJS project
Example code for how Payload’s Blocks field type can be leveraged to produce dynamic, layout-builder style pages
Page meta data using NextJS’ Head component
Payload’s Upload support, including automatic image resizing
How Payload’s Local API can be used to seed initial data into your database
How Payload’s Rich Text field can be used to map 1:1 to React components
TRBL’s ESLint config set up and ready to go
Environment variables properly and securely configured using dotenv

When this type of setup is best used

If you know you need a CMS and will be leveraging NextJS in a server-side rendering capacity, and know that you will not be deploying on Vercel, this boilerplate will be perfect for you. This approach can be super valuable and can get you up and running with a full CMS—complete with everything you need to build a modern, blazingly fast site or app including custom validation, full authentication, access control, and much more.

Configuring TypeScript

Much of the complexity that we handle within this boilerplate comes from using TypeScript to build a custom NextJS server. At Payload, we’re big fans of TypeScript (all of Payload is written in TS). We’re doing our best to adopt and embrace it completely, and we think that it’s only going to get more and more popular.

This boilerplate contains two tsconfig.json files:

The main tsconfig.json, which will be used for the entirety of your NextJS app, including all your React components
The tsconfig.server.json file, which will handle everything in the /server folder

You’ll see that we’ve extended the main tsconfig.json config within the server config and overridden a few properties.

Due to how NextJS relies on dynamic import statements, it requires that its TypeScript projects specify "module": "esnext" in their TS configs. But, Express requires the CommonJS pattern — meaning we have no choice but to require two separate TS configs. No big deal, but this is a common “gotcha” when working with NextJS and TypeScript.

Setting up the server

The Express server itself is pretty simple:


/* eslint-disable global-require */
/* eslint-disable no-console */
import path from 'path';
import next from 'next';
import nextBuild from 'next/dist/build';
import express from 'express';
import payload from 'payload';
import { config as dotenv } from 'dotenv';

dotenv({
  path: path.resolve(__dirname, '../.env'),
});

process.env.PAYLOAD_PUBLIC_SERVER_URL = process.env.SERVER_URL;
process.env.NEXT_PUBLIC_SERVER_URL = process.env.SERVER_URL;

const dev = process.env.NODE_ENV !== 'production';
const server = express();

payload.init({
  license: process.env.PAYLOAD_LICENSE,
  secret: process.env.PAYLOAD_SECRET_KEY,
  mongoURL: process.env.MONGO_URL,
  express: server,
});

if (!process.env.NEXT_BUILD) {
  const nextApp = next({ dev });

  const nextHandler = nextApp.getRequestHandler();

  server.get('*', (req, res) => nextHandler(req, res));

  nextApp.prepare().then(() => {
    console.log('NextJS started');

    server.listen(process.env.PORT, async () => {
      console.log(`Server listening on ${process.env.PORT}...`);
    });
  });
} else {
  server.listen(process.env.PORT, async () => {
    console.log('NextJS is now building...');
    await nextBuild(path.join(__dirname, '../'));
    process.exit();
  });
}

First, we load dotenv and then we expose our SERVER_URL to both NextJS and Payload. Prefixing environment variables with NEXT_PUBLIC_ will ensure that the variable is accessible within NextJS components, and similarly, prefixing a variable with PAYLOAD_PUBLIC_ will expose the variable to Payload’s admin panel.

Important: Only expose environment variables like this if you know that they are 100% safe to be readable by the public. For more information about how Payload allows you to expose environment variables to its Admin panel, click here.

On line 20, we then initialize Payload by passing it a license key (only necessary in production), a long and unguessable secret string used to secure Payload, a URL pointing to our MongoDB instance, and our newly instantiated Express app.

Note: Payload stores your data in MongoDB — so to use Payload, you need to make sure that you have MongoDB up and running either locally or with a third-party platform like MongoDB Atlas.

Serving your app vs. building it

On line 27, we perform different actions based on if the NEXT_BUILD environment variable is set. We do this as a nice-to-have because your Next app is going to be relying on your Payload APIs, especially if it has any static page generation to do. When you go to build your Next app, you probably also need your Payload server to be running.

So, if the NEXT_BUILD variable is set, we start up your Express server for you before allowing Next to build. If it’s unset, we just go ahead and prepare the Next app as usual—then fire up the Express server. Easy peasy.

Layout Building with Blocks

Payload comes with extremely versatile field types that allow you to model any type of data that you need. One of the most capable types is the Block field — and with it, you can allow your content editors to build completely dynamic page layouts with a super streamlined interface right within the Payload admin panel. Admins can then add, remove, and reorder blocks easily based on predefined components that you provide them with.

The beauty of using a JavaScript library like React together with your Payload API means that you can write React components that map 1:1 with your blocks’ data. Your React components can accept the data that your editors author as props, and boom—your layouts are extremely well-organized and extensible well into the future.

In this boilerplate, we’ve pictured how you can even write your Payload block configs directly in the same file as their React component counterparts. You could even go so far as to re-use your frontend website’s React component that shows the data saved within Payload’s admin panel itself to edit that same data. There is a ton of potential here.

For example, check out the Call to Action block in this repo.

In that one file, we define the following:

Reusable TypeScript types that correspond with the data within the Block
A reusable function to be used with Payload’s Field Conditional Logic to dynamically show and hide fields based on what type of button is selected (custom or page)
The Block config itself, describing the fields that are contained within the block. This will be passed to Payload and is the core “definition” of the block
The React component to be used on the frontend NextJS site to render the CallToAction block itself

These things don’t all need to be in the same file, but if you want to, Payload allows for it. Both NextJS and Payload support transpiling JSX within their files. You should be able to write your projects however you want.

Here’s how that CallToAction block looks in the Admin panel:

And here’s how it looks in the minimally styled NextJS frontend:

Dynamically rendering Blocks in React

Actually going to render the blocks themselves in React is also pretty trivial:

/components/RenderBlocks/index.tsx:

import React from 'react';
import { Layout } from '../../collections/Page';
import { components } from '../../blocks';
import classes from './index.module.css';

type Props = {
  layout: Layout[]
  className?: string
}

const RenderBlocks: React.FC<Props> = ({ layout, className }) => (
  <div className={[
    classes.renderBlocks,
    className,
  ].filter(Boolean).join(' ')}
  >
    {layout.map((block, i) => {
      const Block: React.FC<any> = components[block.blockType];

      if (Block) {
        return (
          <section
            key={i}
            className={classes.block}
          >
            <Block {...block} />
          </section>
        );
      }

      return null;
    })}
  </div>
);

export default RenderBlocks;

The above component accepts a layout prop that is typed to an array of Payload blocks. The component then maps over the provided blocks and selects a block from those provided by the blockType of each block in the array. Props are provided, and the block is rendered! Beautiful. So simple, and so much power.

Seeding data using Payload’s Local API

This boilerplate comes with an optional seed script which can be run via yarn seed or npm run seed.

It automatically creates one Media document (which uploads and formats a JPG) and two sample Page documents that demonstrate a few Blocks in action.

Payload’s Local API is extremely powerful. It’s got a ton of use cases—including retrieving documents directly on the server within custom routes or within NextJS’ getServerSideProps as seen in the Page component within this boilerplate. It’s super fast, because there’s no HTTP layer: it’s not a typical REST API call or a GraphQL query. It never leaves your server and returns results in a handful of milliseconds, and it’s even faster if you’re running a local MongoDB instance. You thought NextJS server-rendering was fast? Try it when you don’t even need to leave your server to get your data. That’s fast.

You can also use the Local API completely separately from your running server within separate Node scripts.

By passing local: true to Payload’s init() call, Payload will skip setting up the REST and GraphQL APIs and only expose its Local API operations. Perfect for seed scripts and similar programmatic activities like batch-sending emails to customers, migrating your data from one shape to another, manually syncing Customer records to a CRM, etc.

Here’s the seed script that comes with this boilerplate:

const payload = require('payload');
const path = require('path');

const home = require('./home.json');
const sample = require('./sample.json');

require('dotenv').config();

const { PAYLOAD_SECRET_KEY, MONGO_URL } = process.env;

payload.init({
  secret: PAYLOAD_SECRET_KEY,
  mongoURL: MONGO_URL,
  local: true,
});

const seedInitialData = async () => {
  const createdMedia = await payload.create({
    collection: 'media',
    data: {
      alt: 'Payload',
    },
    filePath: path.resolve(__dirname, './payload.jpg'),
  });

  const createdSamplePage = await payload.create({
    collection: 'pages',
    data: sample,
  });

  const homeString = JSON.stringify(home)
    .replaceAll('{{IMAGE_ID}}', createdMedia.id)
    .replaceAll('{{SAMPLE_PAGE_ID}}', createdSamplePage.id);

  await payload.create({
    collection: 'pages',
    data: JSON.parse(homeString),
  });

  console.log('Seed completed!');
  process.exit(0);
};

seedInitialData();

Pretty awesome stuff.

When this boilerplate should not be used

If you plan to next export a fully static version of your NextJS site, then the value of this boilerplate diminishes a bit and you should likely keep your front + backend entirely separate from each other. In this case, the only strength that this approach offers is that you can host your CMS and your app itself on one server, with one deployment. If possible, in this case, you might want to consider deploying your statically exported site on a CDN-friendly host like Netlify, Vercel, or even an S3 bucket, and host your Payload instance on DigitalOcean, Heroku, or similar.

More examples are on their way

We plan to release many more boilerplates in the future, so if this one doesn’t make sense for your needs, make sure to follow along with us to keep up with everything we’ve got coming out, including examples for how to use Payload alongside of a fully static site exported with Next, built with Gatsby, or other similar tactics.

What’s next?

With this boilerplate, you can build fully featured NextJS sites and apps fully powered by a CMS. So get building! Define your own Collections that describe the shape of your data, make use of Payload’s Globals for items such as header and footer nav structures, or build a full user-authenticated app by relying on Payload’s extensible Authentication support.

If you’d rather start a blank Payload project, you can get started in one line:

npx create-payload-app

From there, you’ll be prompted to choose between a few different starter templates in JS or TS.

It’s also super easy to build out a Payload project yourself from scratch.

Let us know what you think

We want Payload to be the best CMS out there for modern JavaScript developers. Since our launch, we’ve received amazing feedback on our ideas and have had an awesome reception from the community, but we’re only getting started. We’d love to hear what you think. Leave a comment here with your thoughts, submit any issues or feature requests you might come across on our GitHub repo, or send us an email. We happily give out pro-bono licenses to open-source projects and nonprofits, as well as personal projects on a case-by-case basis. If this is you, let us know!

Thank you for reading and keep an eye out for more!

Forem: James Mikrut

SQL vs. NoSQL - cutting through the Tech Twitter noise

Data Definition Language

Migrations

Changing database shape

Transforming existing data

Relational data and performance

Handling complex data structures

Field-based localization

Array-based structures

Takeaways

Planning session recording

Where can you help?

On GitHub Issues and Engineering Efficiency

Earlier this week, Payload hit zero open issues which is pretty wild to me. When is the last time you saw a repo with 9K stars and zero open issues?

GitHub apparently doesn't like this

Issues for Payload are top priority

Our attempt to keep focus - Bugfix Mondays

Like what we're doing? Give us a star on GitHub

Get up and running with one line

It's 2022 - time to stop using headless WordPress and ACF

In 2016, WordPress + ACF was one of the best ways to build a headless CMS. ACF made it useful. But it was never ideal.

Payload is a strong WordPress alternative

Give us a star on GitHub

Join our community on Discord

Get up and running with one line

Speed test - Payload vs. Directus vs. Strapi

Payload is a powerful alternative to Strapi or Directus, and in this post, we stacked them up in a head-to-head GraphQL performance test.

The idea behind the test

Document structure we're testing against

Ensuring environment parity

Seeding data to test with

Benchmarks

#1. Payload

#2. Directus

#3. Strapi

Conclusion

Give Payload a star

Get started in one line

Announcing the official Payload Cloud Storage Plugin

Payload just released its official Cloud Storage plugin, with support for file storage in AWS S3 and Azure Blob Storage out of the box.

Usage

Adapter-based Implementation

Access Control

Get started

Give Payload a star

Request a Plugin

Get started in one line

Tutorial - Configuring TypeScript, Express, Jest, Payload, and VSCode Debugging from Scratch

Setting up a TypeScript project from scratch can be intimidating. Bring in Jest, VSCode debugging, and a headless CMS - and there are many moving parts. But it doesn't have to be difficult!

Software Requirements

Step 1 - initialize a new project

Step 2 - install dependencies

Step 3 - create a tsconfig:

Step 4 - set up your .env file

Step 5 - create your server

Step 6 - create a nodemon file

Step 7 - add your Payload config

Step 8 - add some NPM scripts

Firing it up

Generating Payload types

Testing with Jest

Adding and configuring test dependencies

Global Test Setup

Our first test

Running tests

Debugging with VSCode

Conclusion

Star us on GitHub

Join our community on Discord

Building a Professionally Designed Website Episode 4 - Heroes, Layout Building Blocks, Animations & Design

This is Part 4 in our deep-dive into how to develop a professionally designed, high-end NextJS + Payload CMS website from start to finish.

Table of Contents

Code for this episode

Figma link

The real-world, live website

Use Payload for free

Feedback welcome

The Payload Headless CMS just got more powerful with Plugins

Payload's new and robust Plugins infrastructure makes reusing and modularizing your Payload code super easy.