Forem: Contentful Blog

How to build chatbots with Contentful and Azure functions

Contentful Blog — Wed, 26 Aug 2020 09:00:00 +0000

In 2019, I was boarding a plane for London when I realized that I forgot my shaving kit. I quickly opened the app for the hotel that I would be staying in. Thanks to its chatbot feature, I dropped them a request. They were able to immediately solve my shaving problem and, in addition, booked me a taxi from the airport.

The benefits of chatbots are pretty evident: they make customers happy, they operate as a sales tool and allow you to maintain a conversational relationship. I didn’t get frustrated, I stayed connected with the hotel and now we’ve built a conversational history.

Because conversational bots offer an important channel to reach customers, most brands are developing expert bots that provide specific solutions. Some chatbots provide FAQ solutions, others recommend specific products depending on the customer’s needs, and still others quickly reply to questions about billing and account information. Contentful can handle the content layer of your chatbot, and this post walks you through the architecture, implementation and results.

Specific context and tech stack for chatbot

One of our enterprise customers wanted to build an FAQ chatbot with Contentful after they used the content platform to successfully power their web products. They were already using Microsoft cloud services, so we decided to build a quick proof-of-concept using Microsoft Azure functions and knowledge base. Contentful works with these tools, but can also be used with other cloud and bot services like Amazon Web Services, Google Cloud Platform, or IBM Watson thanks to its APIs. After some initial investigation, we developed the architecture presented here, which uses Contentful webhooks, Azure cloud functions and Microsoft’s knowledge base service.

Architecture

Publishing flow

Editors will publish the FAQ in Contentful
Publishing of the FAQ will trigger a webhook in Contentful
The webhook will call our publish cloud function
The function will push the content to our knowledge base

Unpublishing flow

Editors will unpublish the FAQ in Contentful
Unpublishing of the FAQ will trigger a webhook in Contentful
The webhook will call our unpublish cloud function
The function will remove the FAQ from our knowledge base

Implementation

Contentful

We used Contentful to store questions and answers for the chatbot. In the Contentful web app, editors can easily write, edit and publish that content.

Content Model

We’re keeping the content model very simple for this demo — just a single content type holds our question and answer pairs, as well as an internal title for content editors to have as a reference.

Webhooks

We use webhooks for publish and unpublish events, triggering our cloud functions.

When creating the publish event webhook, make sure to set the trigger of the webhook for the entry publish event as shown below. For the URL, we will need to add the endpoint of our cloud function, as we have not configured any cloud function yet. We can use the DummyWebhook website or ngrok to create dummy endpoints and add them.

When creating the unpublish event webhook, make sure to set the trigger of our webhook for the entry unpublish event as shown below. Just like last time, we’ll populate the URL with a dummy value from DummyWebhook or ngrok.

Once done, you can create an entry and test our workflow by publishing and unpublishing it and checking the webhook payload being sent. You can then view the payloads in our webhook logs.

Publish entry webhook payload:

{
  "sys": {
    "type": "Entry",
    "id": "1ssZF4Lqq1Wpi9KSHvq9jE",
    "space": {
      "sys": {
        "type": "Link",
        "linkType": "Space",
        "id": "bwn0phmhnub6"
      }
    },
    "environment": {
      "sys": {
        "id": "master",
        "type": "Link",
        "linkType": "Environment"
      }
    },
    "contentType": {
      "sys": {
        "type": "Link",
        "linkType": "ContentType",
        "id": "faqQuestion"
      }
    },
    "revision": 2,
    "createdAt": "2020-04-13T23:06:07.225Z",
    "updatedAt": "2020-04-13T23:07:05.635Z"
  },
  "fields": {
    "title": {
      "en-US": "First question"
    },
    "question": {
      "en-US": "What is contentful? \n\n"
    },
    "answer": {
      "en-US": {
        "data": {},
        "content": [
          {
            "data": {},
            "content": [
              {
                "data": {},
                "marks": [],
                "value": "Contentful is Headless CSM",
                "nodeType": "text"
              }
            ],
            "nodeType": "paragraph"
          }
        ],
        "nodeType": "document"
      }
    }
  }
}

Unpublish entry webhook payload:

{
  "sys": {
    "type": "DeletedEntry",
    "id": "1ssZF4Lqq1Wpi9KSHvq9jE",
    "space": {
      "sys": {
        "type": "Link",
        "linkType": "Space",
        "id": "bwn0phmhnub6"
      }
    },
    "environment": {
      "sys": {
        "id": "master",
        "type": "Link",
        "linkType": "Environment"
      }
    },
    "contentType": {
      "sys": {
        "type": "Link",
        "linkType": "ContentType",
        "id": "faqQuestion"
      }
    },
    "revision": 2,
    "createdAt": "2020-04-13T23:27:30.971Z",
    "updatedAt": "2020-04-13T23:27:30.971Z",
    "deletedAt": "2020-04-13T23:27:30.971Z"
  }
}

For now, that’s all we have to do within Contentful.

Creating the FAQ chatbot in Microsoft Knowledge Base

QnA service

1. Go to https://www.qnamaker.ai/ and create a free account.

2. Create a knowledge base.

3. Create a QnA service.

4. Go back to https://www.qnamaker.ai/ and refresh to see your subscription name and Azure QnA service.

5. Complete the rest of the steps to create a knowledge base. Once done, you will have a knowledge base similar to the one shown below.

APIs to integrate

Now that we’re done with creating our knowledge base, we need to populate it with questions and answers coming from Contentful for the chatbot to use. Let’s explore which APIs we can use to add, update or delete information from our knowledge base.

We will be using the update endpoint to add and remove content. We will also use the download endpoint to get all questions and answers and then filter them.

Azure functions

Azure functions are a powerful way to develop and deploy serverless applications. For this use case we chose Azure functions as we can quickly develop and test our proof of concept.

Setup

This video tutorial from Microsoft gives you a guide to set up an Azure function project using Visual Studio Code.

We used the following settings for this chatbot project.

Language: you can select multiple languages, but we used JavaScript
Function trigger: http
Name: the function we used publishWebhookHandler for the first function and unpublishWebhookHandler for the second function
Access Rights: set it to anonymous for now. For production, make sure to implement proper security mechanisms

We created a new Azure function project with two functions: publishWebhookHandler and unpublishWebhookHandler.

publishWebhookHandler

This function will be used to push content into our knowledge base using the webhook coming from Contentful whenever the content is published.

Algorithm

Retrieve the following from the webhook: entry ID, question and answer
Make an API call to our knowledge base to get all the questions and the answers. Once you have all questions, filter the question using metadata and check for the entry ID. We’re doing this to check if we have duplicate questions in our knowledge base, so we update them instead of adding a duplicate entry.
If the question is present we will update that question using the update endpoint. Use the same endpoint to add new questions. The only difference is the body of the payload. You can see in the metadata that we are adding key contentfulid and its value. This helps us to map entries in Contentful with entries in the knowledge base and will help us for deleting and updating entries.

Adding a new question:

{
  "add": {
    "qnaList": [
      {
        "answer": "answer",
        "source": "source",
        "questions": [
          "question"
        ],
        "metadata": [{"name":"contentfulid","value":"1ssZF4Lqq1Wpi9KSHvq9jE"}]
      }
    ]
  }
}

Updating a question:

{
  "update": {
    "name": "QnA Maker FAQ Prompts Bot",
    "qnaList": [
      {
        "id":87,
        "answer": "answer 2",
        "source": "source",
        "questions": {
          "add":["questions 1"],
          "delete":[]
        }
      }
    ]
  }
}

Sample function for adding content to our knowledge base:

module.exports = async function(context, req) {
   if (req.body) {
       const contentfulID = req.body.sys.id;
       const contentfulQuestion = req.body.fields.question['en-US'];
       const contentfulAnswer = req.body.fields.answer['en-US'].content[0].content[0].value;

       const questionFromKB = await getQuestionIdFromKB(contentfulID);
       const response = await  upsertQuestionInKB(context, questionFromKB, contentfulID, contentfulAnswer, contentfulQuestion);

       context.res = {
           status: 200,
           body: JSON.stringify(response),
       };
   } else {
       context.res = {
           status: 400,
           body: 'Please pass a name on the query string or in the request body',
       };
   }
};

unpublishWebhookHandler

This function removes content from our knowledge base using the unpublish webhook, whenever content is unpublished.

module.exports = async function(context, req) {
 if (req.body) {
     const contentfulID = req.body.sys.id;
     const questionId = await getQuestionIdFromKB(contentfulID);
     const response = await removeQuestionFromKB(questionId);
     context.res = {
         body: JSON.stringify(response),
     };
 } else {
     context.res = {
         status: 400,
         body: 'Please pass a name on the query string or in the request body',
     };
 }
};

Algorithm

Retrieve the entry id from the webhook
Make an API call to our knowledge base to get all the questions, and filter the question using the entry id
Make an API call to remove the question from our knowledge base. The payload will look similar to the following:

{
   "delete":{
      "ids":[
         29
      ]
   }
}

Deploy

Once both functions are ready, they can be deployed to Azure Cloud.

When the deployment is complete, get the publishWebhookHandler and unpublishWebhookHandler URLs from Microsoft Azure portal. Add them to the webhooks, which we created in the first step.

Testing our chatbot end to end

Populate content

We published three Q&As for testing in Contentful. Each of the publish events triggered a webhook and they were pushed into our knowledge base. To check, we logged into https://www.qnamaker.ai to see if the questions are published.

On the screenshot below you can see the three published Q&As on Contentful as well as added on our knowledge base.

Now, our knowledge base will train the content. Inside QnA Service, clicking on Test you can test your chatbot.

Results

Now you have an intelligent chatbot customers can talk to. This way customers do not need to go through the extensive list on a website searching for a simple answer. You can also measure the sentiments of the customers and their frequent questions. With this additional information, we are empowered to improve our FAQs.

Future enhancements for our chatbot

Our content model can be extended to add multiple questions for a single answer, and for every entry, additional metadata could be added. For this we have to update the sync functionality within the publishWebhookHandler function.
Contentful also provides different localization patterns, which can be used to create content in multiple languages, allowing us to power multilingual bots.
In this article, we explored using the Microsoft tech stack, but similar things can be achieved using AWS Lambda and AWS Lex. Any chatbot platform that provides API access to their knowledge base can be used with Contentful. Using webhooks and cloud functions allow us to push content into a knowledge base and train it afterward.

If you’re looking to implement your own conversational chatbot, you can sign up and try it for yourself. Feel free to contact us. Our solution services team works with enterprise customers to accelerate their digital journey.

Meet the Example App - your kick-off point for developing with Contentful

Contentful Blog — Tue, 04 Dec 2018 00:00:00 +0000

As developers, we know that selecting the best stack for the job is never easy. That’s why we at Contentful have been working hard to give you The Example App — a series of reference applications that walks you through the basics of Contentful.

The Example App — realize your project with Contentful

We built The Example App to help you answer two questions:

How can I realize my project with Contentful?
How do I get started using my favorite language?

By experimenting with The Example App, you will learn how Contentful’s content infrastructure is ideal for your next project and how you can write an app that works to retrieve content from your space in Contentful.

Use your favorite language to learn about Contentful

The Example App has introductory examples for how to build an application with Contentful using .NET, Java, Node.js, Ruby, Python, PHP, Kotlin, and Swift.

Get up to speed on content modeling

Content modelling is the practice of structuring your content. At Contentful, you do this by creating content types. The Example App features a thorough lesson on how to build a content model to fit your project. You can even get a working example of Example App to-go on your Android device as well as for iOS on the App Store.

Highlights

Learn about our APIs

Contentful follows an API-first approach, which means that all of its functionality is provided by APIs.

This enables you to:

Modify data schemas or configure a webhook through the Content Management API
Deliver cross-channel content through the Content Delivery API
Preview unpublished content through the Content Preview API
Resize, crop, or re-compress images through the Images API

Develop an app for mobile

With our Example App available in so many languages, you can choose to dive into developing a mobile-first solution to retrieve and serve your content. Let’s say you wanted to develop for Android — our GitHub repo provides you with everything you need to load into your Java SDK.

However, Java is legacy for building Android apps at this point and Kotlin is the future. Kotlin comes recommended by Google for writing Android apps since it’s a less verbose, concise language than Java. What do you do now? Lucky for you, we’re always striving ahead to support all sorts of (modern) ways for you to work and we’ve got you covered with a repo for The Example App in Kotlin!

The Android version of The Example App is written in Kotlin and will be what the app would look like out-of-the-box from the GitHub repo; development of a variant of the Example App that uses GraphQL is also underway.

With the app on the store and source code on GitHub, you can very easily learn how to combine new tech such as Kotlin, architecture components, and more to be integrated into your app using Contentful.

Clone The Example App

While theory is great, nothing beats learning from code examples. Head over to Contentful’s GitHub page to find The Example App’s GitHub repo in the language of your choice, and start modifying our examples to your liking.

If you’re looking for a higher-level overview to get your bearings and go in the right direction, we also have this foolproof guide on how to get started with Contentful that you can check out before progressing to the Example App.

A loosely-coupled approach to creating MailChimp campaigns with Contentful webhooks

Contentful Blog — Fri, 30 Nov 2018 00:00:00 +0000

Liquid syntax error: 'raw' tag was never closed

More than just serving JSON – five requirements of your next API-first CMS

Contentful Blog — Thu, 29 Nov 2018 00:00:00 +0000

When your job is to build digital products, sooner or later you will have the urge to provide your co-workers with a way to edit content within your product. In the case of most websites, traditional CMS systems out there tend to do the job just fine. It becomes trickier when you have to ship the same content to different products running on different platforms using different technology stacks.

In case your CMS stores (and serves) content as HTML, you won’t be able to port the content to other technologies easily. Platforms such as iOS and Android can not deal with HTML.

The solution to this problem is to serve structured JSON rather than HTML. After all, JSON is the language of the web, and all the common platforms support it which is a big advantage.

"Cool! It’s just an HTTP GET then? I can create a JSON API based on content in a database, and I’m all good, right?"

Not entirely – it is true that you can start with "just an API" that returns JSON to achieve quick results; but in the long run, management of cross-platform and scalable content means way more than “just serving JSON”. That’s when a headless or API-first CMS comes into play. These systems provide you not just with a scalable JSON API, but also an interface to edit your content and ways to evolve your content – a complete content infrastructure so to say.

In this article, I will share five API-first CMS requirements that you should watch out for if you're going to master the problem of scalable future-proof content.

1. Your CMS provides the content flexibility you need to move fast!

_You should be able to setup content structures that fit your needs in minutes. _ When people think of a CMS, they always think of these massive systems you have to install yourself, that are structured in pages and include giant WYSIWYG editors ("What you see is what you get"). One field or text area doesn’t provide enough structure to use the same content source everywhere and across distribution channels! Your system has to give you the ability to break down all these big content blobs into small reusable pieces to really succeed in multi-channel delivery.

You have to understand that content management is not about pages and certainly not about one WYSIWYG editor of a website, but rather about enabling digital experiences based on small building blocks. It may be that you’re building a small web app and want to include an editable announcements section, or that you want to manage copy for your mobile app – all these content pieces are use cases for a CMS.

You have to be able to create, add and edit reusable content structures quickly – no matter how small or complex they are. You’ll never know which content requirements your next project will bring and your CMS setup should not become the bottleneck.

All these content structures will evolve into a big picture, so they have to be able to be connected, too — all your content has to become a graph to scale!

Let’s take content type for a collection of music as an example. A song probably was created by an artist, and this artist has written several songs throughout their career. To avoid content duplication in every song entry, you should be using two different content types here – a "song" type and an “artist” type.

All entries from both the content types can then be related to one another — so that when you update an artist, your changes will be available to all other entries that reference it.

With a content graph structure, you can request particular nodes but also a sum of entries due to them being connected with one another. It’s up to you, but your CMS should give you the flexibility to fetch whatever data you need.

2. Your CMS treats rich text as JSON!

When you have granular content structures to avoid duplication, the next step is to make sure your content works on every platform. To achieve that, you have to follow the rule of separation of concerns. Separation of concerns defines that you break down pieces of your applications into distinct sections so that each section really only addresses one concern.

Your CMS’ job is to enable you to create and edit content – nothing else! The styling and representation of this content is a different concern and should be dealt with by another part of your application. Whenever platform-specific content like HTML goes into your content to apply styles, you’re breaking the model of separation of concerns — and, trust me, this will give your future-self many headaches when you decide to serve that same piece of content to a different product or platform later!

A JSON object with fields including HTML is not good enough – forget systems that produce HTML!

One possible alternative is Markdown, which achieves the separation of concerns nicely and works well for writing articles on your blog, but it comes with three significant disadvantages when dealing with more complex content structures.

Markdown’s feature set is limited.

Markdown bets strongly on semantics but the content that developers have to deal with today has additional requirements. It misses formatting options such as:

Underline (yes, it’s achievable on a web page via CSS but there is no HTML equivalent for underlining which leads to the lack of an underline option in Markdown — try explaining this to a content creator...)
Ability to embed video and several other formats of content
Options to describe components (for example, an accordion)

You could potentially make all these work in Markdown, but the fact remains that it is not the right tool for the job.

Markdown is not capable of representing relationships

JSON is 100% structured — one object includes another object includes another object. Let’s have a look at a possible API response for an entry:

{
  "fields": {
    "title": “A title”,
    "body": “## A post written by Jane Doe”
    "artist": {
      "fields": {
        "name": “Jane Doe”
      }
    }
  }
}

What you see above is structured JSON describing a song with a title, a body written in Markdown, and a reference to another object which represents the artist of the song.

Cases like this are where relationships between entries shine. Since each song includes a reference to the specific artist who made it and other songs, should you update the name of the artist, that change will be reflected in every entry referencing said artist.

On the other hand, content stored in Markdown serves only as a long string that does not allow references to other entries, which makes maintaining a lot of content very hard and prone to errors.

Markdown allows HTML content

Markdown allowing HTML input might not seem like a big deal, but if people get the chance to customize their content using HTML, they will do it and break your apps. It will happen, and you have to make sure that it won’t.

There should be no way to mess up that cross-platform portability!

Rich content editing has to be JSON based!

To succeed with complex data structures, rich content editing should also be JSON-based. This way, editors can reference other entries right within text areas, and yes, in the best case they can underline words without breaking your systems, too.

3. Your CMS allows you to make your content creators feel safe!

Once you’ve made it this far and all your content is JSON-based, the next question is how to preview the content. An editorial preview mechanism is crucial for editorial teams but how is that accomplished when you have only one JSON API?

To build a proper preview environment, you need the option to see unpublished content changes. It’s time for a second API – a preview API.

With this second API, you can set up different environments in which editors can check content and the changes they make to it before publishing.

Your editors have to feel safe and comfortable working with the system you provide – and a preview mechanism is a big part of that.

4. Your CMS gives you the tools to evolve fearlessly!

Coming from traditional CMSes, developers are used to having an exact local copy of the production environment (that includes the production database) on their development machine. When making a structural change, they tend to write an SQL script, test it locally and apply it to production once they feel good about it – but how can you follow this workflow when your content isn’t stored in your databases but only accessible via an API?

This workflow leads to two main requirements for your content infrastructure:

The ability to make a copy of the production content
The ability to script content changes at scale

Making copies of your production content

At Contentful, you can create copies of your "master content", which you can access under a slightly different URL — we call these copies sandbox environments. There, you can create, change and delete them without affecting any production data; providing you with safety to develop content changes locally.

Using a combination of your environment ID and the space ID, you can access its contents to build production, staging or local environments:

cdn.contentful.com/spaces/{space_id}/environments/{environment_id}/entries

cdn.contentful.com/spaces/.../environments/master/entries
cdn.contentful.com/spaces/.../environments/my-new-feature/entries
~~~

A sandbox environment is an equivalent to the SQL database dump of traditional systems. The next question is – how do you write database scripts in this new world?

### Automated migration tooling

First of all, to enable developers to script content changes, **you need a third API to write data.** Unfortunately, barebones API endpoints with write access are not enough. Developers want to write scripts quickly and not deep-dive into documentation about API endpoints.

![Developers want to write scripts quickly and Contentful lets them do exactly](//images.ctfassets.net/fo9twyrwpveg/5Vs3xFOhwIs2ekM6ksWE0q/77a9b44443dc7e374172e83bddb4d680/image_6.png)

Defining scripts on top of API-endpoints usually will result in a custom solution. At Contentful we decided to make [content migrations a first-class citizen of our ecosystem](https://github.com/contentful/contentful-migration) to give developers a straightforward way to make changes to their content!

### CMS as Code workflow

With the combination of sandbox environments and migration scripts, you can handle your CMS the same way you would [treat the source code and database running in your application](https://www.contentful.com/r/knowledgebase/cms-as-code/?utm_campaign=requirements-api-first-cms-json&utm_medium=referral&utm_source=devto&utm_content=requirements-api-first-cms-json&utm_term=). 

![Combination of environment sandboxes lets you work more effectively](//images.ctfassets.net/fo9twyrwpveg/5Egtp2Qs8ggiK6GQ6e2cmA/f73a584674aa10c7b8f08dedd613f499/image_7.png) 

You can create development workflows to spin up environments for certain features and run tests against them, or simply to have staging or QA environments, and also automate your CI pipeline. There are many instances of large-scale companies [implementing](https://www.contentful.com/blog/2018/09/13/content-model-changes-scale-telus-cms-as-code/?utm_campaign=requirements-api-first-cms-json&utm_medium=referral&utm_source=devto&utm_content=requirements-api-first-cms-json&utm_term=)[ developer workflows successfully with Contentful.](https://www.contentful.com/blog/2018/09/13/content-model-changes-scale-telus-cms-as-code/?utm_campaign=requirements-api-first-cms-json&utm_medium=referral&utm_source=devto&utm_content=requirements-api-first-cms-json&utm_term=)

## 5. Your CMS plays well with other vendors!

The developer ecosystem today consists out of very specialized services. There are solutions for continuous integration, commerce, translations; the list goes on and on. The content that will power your products will most likely have some touchpoints with other services in your stack. 

Your API-first CMS has to be ready to integrate nicely with other software-as-a-service (SaaS) products.

![Integrate your API-first CMS easily with other SaaS solutions with Contentful](//images.ctfassets.net/fo9twyrwpveg/7ePP9kr7eEywcIAsQ0WcgM/7e40552b83714f0a9b7d476e0e169453/image_8.png)

### An extendable editor interface

The possibility to configure the editor interface is highly essential to boost the productivity of content creators. Assuming you use different APIs in your product, you also might want to create short paths to, for example, retrieve [data from your commerce platform](https://www.contentful.com/developers/marketplace/contentful-ui-shopify/?utm_campaign=requirements-api-first-cms-json&utm_medium=referral&utm_source=devto&utm_content=requirements-api-first-cms-json&utm_term=), [control AB-tests](https://blog.optimizely.com/2018/02/13/experimentation-content-creation-process/) or [translate words](https://www.contentful.com/developers/marketplace/translate/?utm_campaign=requirements-api-first-cms-json&utm_medium=referral&utm_source=devto&utm_content=requirements-api-first-cms-json&utm_term=) right within your CMS.

### Flexible and adjustable webhooks

Additionally, you want to be able to notify other parts of your infrastructure whenever someone adds or deletes content. A common way to solve this is webhooks — requests that a cloud service sends to a URL of your choice.

Not every webhook implementation is the same and connecting different services can initially be more difficult than it seems. However, with particular webhook features such as the ability to define custom headers, control webhook actions at a very granular level and adjust the payload of webhooks you will make your API-providers talk to each other and you’ll be able to [connect services directly](https://www.contentful.com/developers/docs/tutorials/general/enhancing-search-experience-with-algolia/?utm_campaign=requirements-api-first-cms-json&utm_medium=referral&utm_source=devto&utm_content=requirements-api-first-cms-json&utm_term=) to save work and additional infrastructure.

## It’s more than one JSON API

As you’ve witnessed, in order to succeed with all the great content shipped in your products, it needs a little bit more than one scalable JSON API on top of a database as your CMS.

On an API-level, here’s what you need at minimum:

- a scalable cached API to serve your sites and apps
- a preview API to allow you to build environments for editors to preview their changes
- a write or management API that lets you perform automated content changes

*Tip: Bonus points if there is a way to automatically resize and crop images, too.*

Additionally, the content structures and editing experience are just as important as the APIs themselves, but if you’re dealing with a lot of content, **_your CMS should provide you with solutions to change content that’s already in your production system_**. There is nothing worse than being stuck and not being able to move forward anymore – this is a point that is often overlooked. 

With that word – choose your API-first CMS wisely. :)

(You can find a more detailed look at what features your CMS should have in the slides to the talk ["When a CMS is not enough – tales from a content infrastructure"](https://speakerdeck.com/stefanjudis/when-a-cms-is-not-enough)).

Writing GraphQL queries in native Ruby = Love ❤️

Contentful Blog — Tue, 20 Nov 2018 00:00:00 +0000

GraphQL has made waves with front end developers, but if you’re a backend developer, chances are it hasn’t infiltrated your world –– you might not even know what it is. And it’s not your fault. There has been a limited amount of blog posts, talks and information around using GraphQL APIs on languages that don’t have an official Apollo client.

How to automatically watermark Contentful assets using a serverless function

Contentful Blog — Wed, 14 Nov 2018 13:57:28 +0000

We made it friends! We finally reached the point where developers can deploy HTTP endpoints without worrying about infrastructure. You might have guessed it, and you’re right – I’m talking about the serverless buzzword! The big infrastructure players; AWS, Microsoft, IBM, and others all provide functions as a service these days. Serverless functions make it possible for you to write 100 lines of code, test it and ship it – 100% scalable and doesn’t cost you a single dollar whenever the deployed code is not running.

Why is an HTTP endpoint such a big deal?

The industry is shifting towards specialized services to solve specific problems in ways that are better and quicker than I (and probably you) could do it when starting to build solutions today. You sign up to a service of choice and can start using it – saving you weeks, if not months, of developing software that doesn’t belong to your core product. How many times have you built a login mechanism in your software career? For me, it’s been many, many times...

The question arises how you can automate certain things when you are not in control of the source code of all the services that can be in your future technology stack. The answer to this is two core components – APIs and webhooks.

In case you haven’t worked with them previously, webhooks are HTTP requests sent by a service to a defined HTTP endpoint. These requests allow you to connect one service with another, and then combine all the small functionality pieces into something bigger, greater, and more powerful.

However, before you go out and subscribe to a gazillion software-as-a-service (SaaS) products, let me give you a bit of advice. Not every cloud service that provides an API is the same and not every webhook implementation is useful. You should check for two main characteristics:

The provided API has to include read, create and update functionalities
Webhooks have to be secure, filterable and adjustable

In this article, I want to describe how you can implement a webhook flow to improve your productivity when working with images in Contentful. Let’s assume that you’re dealing with photos that should be served to your applications as a watermarked version which includes your company logo. Your content creators could manually add watermarks each time on every image, but you could also automate this task away to save time and effort.

The main flow – webhooks triggering serverless functions

Here’s the plan: whenever someone creates a new asset and uploads an image in Contentful, a webhook is sent. This webhook triggers a serverless function that downloads the uploaded image. The composition of this image with, for instance a company logo, will then be re-uploaded to Contentful as the watermarked version.

{: img }

This way you get to both keep the original images and save the work of modifying these yourself. Sounds good? Let’s do it!

Getting started with the serverless framework

To start developing functions that run in the cloud, the serverless framework is always my first choice. It gives you a nice abstraction layer on top of the functionality provided by all the infrastructure providers. You can develop and deploy functions right from your machine, which is way more comfortable than what’s usually provided by online editors.

As a first step, head over to their getting started guide. It explains how you install the serverless executable and what you need to configure. When everything is set it takes you one command to create a project that is ready to deploy to AWS.

$ serverless create --template aws-nodejs
~~~{% endraw %}

This command above creates only two files for you, with {% raw %}`serverless.yml`{% endraw %} being the main configuration file for your serverless projects. Its initial creation includes lots of documentation and comments, but not many configuration parameters and values will be defined in it at this point.{% raw %}

~~~yaml
service: aws-nodejs # NOTE: update this with your service name

# define environment
provider:
  name: aws
  runtime: nodejs8.10
# define available functions
functions:
  hello:
    handler: handler.hello
~~~{% endraw %}

The {% raw %}`serverless.yml`{% endraw %} defines a service name (this is how you’ll identify it in AWS later), you can configure the environment in AWS (Node.js in version 8.10), and it lists the functions that should be available later. Node.js version 8 means that you can use recent additions to the language like {% raw %}`async/await`{% endraw %} which makes source code so much more readable!

The other file {% raw %}`handler.js`{% endraw %} exports a function that is ready to be deployed returning a "hello world" JSON example message.{% raw %}

~~~javascript
'use strict';

module.exports.hello = async (event, context) => {
  return {
    statusCode: 200,
    body: JSON.stringify({
      message: 'Go Serverless v1.0! Your function executed successfully!',
      input: event,
    }),
  };
};
~~~{% endraw %}

### Make the function executable via HTTP

While these two files are enough to deploy your first function to AWS, this function isn’t connected to an HTTP endpoint yet and thus not accessible from the outside. To make that happen, you have to add your first custom configuration to the {% raw %}`serverless.yml`{% endraw %}. 

In AWS, [API gateway](https://aws.amazon.com/api-gateway/) handles the configuration of HTTP endpoints. The cool thing about the serverless framework is that it provides ways to configure endpoints without jumping between all the different pages and settings in the AWS console.

Events triggered by different services of the infrastructure are responsible for function execution in AWS. HTTP, in this case, is only one of them but is the foundation to build serverless function-driven APIs. 

By adding the configuration for {% raw %}`events`{% endraw %} to the particular function, the serverless framework configures API gateway for you and make this function available on the internet.{% raw %}

~~~yaml
# … service configuration
# … provider configuration

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: /hello
          method: get
~~~{% endraw %}

You can now deploy your new function using {% raw %}`serverless deploy`{% endraw %}.{% raw %}

$ serverless deploy
[15:38:58]
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (387 B)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
Serverless: Stack update finished...
Service Information
service: hello-world
stage: dev
region: us-east-1
stack: hello-world-dev
api keys:
None
endpoints:
GET - https://lbrdiu0h92.execute-api.us-east-1.amazonaws.com/dev/hello
functions:
hello: hello-world-dev-hello


The deployment logs give you information of where the endpoint is deployed so that you can open it in your browser and see the JSON response immediately.

{: img }
![Change the method name and HTTP method to POST](//images.ctfassets.net/fo9twyrwpveg/6ozEksTGfeQEmsyCqAWY68/76fe98aa88e169e04ddefea09a35f791/image_1.png)

As a last preparation step, let’s change the method name from {% raw %}`hello` to `watermark` and also use the HTTP method `POST` instead of `GET`.

~~~yaml
# … service configuration
# … provider configuration

functions:
  watermark:
    handler: handler.watermark

    events:
      - http:
          path: /watermark
          method: post
~~~

These two files and a few lines of additional configuration are all it takes to lay the foundation for an endpoint that is capable of handling Contentful webhooks to watermark images and re-upload them.

### Tailor the webhook to your needs

In Contentful you can define a new webhook that will be sent whenever you publish a new asset.

{: img }
![Define a new webhook to be sent after publishing a new asset](//images.ctfassets.net/fo9twyrwpveg/fFnXAMHeRaSk8ecsyUESC/963f5b6b2547bdc49380957affce42d2/image_2.png)

What is cool is that you can define custom payloads in the webhooks configuration. Using JSON pointers, you can slim down the payload and avoid complexity in your functions because it sends only the data you’re interested in.

Adjustable webhooks payloads are also critical when you want to connect two services directly because it might be that a service only accepts requests with particular headers or payloads.

{: img }
![The definition of the entire webhook payload sent to the serverless function](//images.ctfassets.net/fo9twyrwpveg/20MQxKhM4YqI24aS2kyqQK/73c012e719a0cb9d90e51795db31e836/image_3.png)

What you see above is the definition of the entire webhook payload sent to the serverless function. The syntax of JSON pointers (`{ /payload/… }`) makes it possible to access the original payload and restructure everything to your requirements. In this example, you’re going to use the following:

* The URL of the uploaded images to download it in the serverless function
* The filename of this file, to be used as a base for the filename of the watermarked image
* The content type of the uploaded file to only run watermarks for jpegs
* The width and height of the image so to not watermark images that are too small
* The title and description of the uploaded asset to be reuse in the watermarked asset

## Manipulating images in a serverless function using Jimp

Here comes the fun part... image manipulation often requires you to install native dependencies, which can be a little bit tricky when dealing with serverless functions. The [Jimp package](https://github.com/oliver-moran/jimp) is handy in this case because it performs image manipulations entirely in JavaScript. The manipulation is probably a little bit slower than a native implementation, but you don’t have to deal with the troubles of installing native dependencies on AWS.

### Adding npm dependencies to your serverless project

You can add a `package.json` to the project root with `npm init --yes` and then install the package as a dependency with `npm install --save jimp`. When you later deploy the project, the serverless framework scans your project and figures out what dependencies need to be included in the package that goes to AWS. It even excludes development dependencies to keep the package as small as possible. It’s honestly very cool!

You can happily use the Jimp package and manipulate images in your Lambda code.

~~~javascript
const original = await Jimp.read(url).opacity(0.4);
~~~

Your project structure should at this point look as follows:

handler.js
serverless.yml
package.json
package-lock.json
node_modules/


### The composition of two images with Jimp

The watermark image creation can be divided into four parts:

* Read the original image that should be watermarked
* Read the image that acts as the watermark
* Make the watermark image transparent
* Combine both images

Luckily all these steps are reflected in the methods provided by Jimp.{% raw %}

~~~javascript
// step 1
const original = await Jimp.read('https://url/to/image');
// step 2
const mark = await Jimp.read('https://url/or/filepath/to/mark); 
// step 3
mark.opacity(0.5);
// step 4
const watermarkedImage = await original.composite(mark, 50, 50); 
~~~

What’s very handy about `Jimp.read` is that it accepts file paths and URLs which means that you don’t have to deal with file downloads yourself.

### Read the webhook payload and watermark the uploaded image

Remember the payload we defined in Contentful? This payload is available in the `body` property of the function argument `event` inside of the lambda function. You can combine the passed URL with the Jimp functionality and start watermarking images.

~~~javascript
module.exports.watermark = async (event) => {
  try {
    console.log(
      `Received webhook: ${JSON.stringify(JSON.parse(event.body), null, 2)}`
    );
    const {
      url,
      title,
      description,
      fileName,
      contentType,
      width,
      height
    } = JSON.parse(event.body);

    // validate and check if a new watermark version
    // has to be created
    if ( contentType !== ‘image/jpeg’) {
      return {
        statusCode: 200,
        message: ‘Nothing to do...'
      }
    }

    // ... 

    const original = await Jimp.read(url);
    const mark = await Jimp.read('https://path/to/mark); 
    mark.opacity(0.5);
    const watermarkedImage = await original.composite(mark, 50, 50); 
  } catch(e) {
    return {
      statusCode: 500,
      error: e.message
    }
  }
~~~

The code snippet above also includes validation steps to not always create new watermarked images and to avoid neverending loops. It does not include the download of the watermark logo to avoid complexity, but you can find the full version [on GitHub.](https://github.com/stefanjudis/serverless-contentful-watermark/blob/master/handler.js)

### Uploading the new watermarked asset to Contentful

To upload the generated image, you can use the [Contentful Management SDK](https://github.com/contentful/contentful-management.js?files=1) which provides you the `createAssetsFromFiles` function to upload images with [the Content Management API](https://www.contentful.com/developers/docs/references/content-management-api/?utm_campaign=automatically-watermark-contentful-assets-using-serverless&utm_medium=referral&utm_source=devto&utm_content=automatically-watermark-contentful-assets-using-serverless&utm_term=).

~~~javascript
const contentful = require(‘contentful-management’);

const uploadAsset = async ({
  title,
  description,
  fileName,
  contentType,
  stream
}) => {
  const client = contentful.createClient({ accessToken: ‘...’ });
  const space = await client.getSpace(SPACE_ID);
  const env = await space.getEnvironment('master');
  let asset = await env.createAssetFromFiles({
    fields: {
      title: {
        'en-US': title
      },
      description: {
        'en-US': description
      },
      file: {
        'en-US': {
          contentType,
          fileName,
          file: stream
        }
      }
    }
  });
  asset = await asset.processForAllLocales();
  asset = await asset.publish();
};
~~~

### The overall structure

Reading and validating the JSON payload, downloading and processing of the needed images and uploading a new asset to Contentful are the core parts of this function. The overall implementation for the complete functionality is around 200 lines of code and this article only describes some of the details. If you want to learn more or try it out you can go [the repository on GitHub.](https://github.com/stefanjudis/serverless-contentful-watermark/blob/master/handler.js)

The flow codewise should be as follows:

~~~javascript
Module.exports.watermark = async (event) {
  // read webhook payload
  // validate payload and check if you can skip watermark creation
  // read the image of the published asset
  // read the watermark logo
  // composite both images
  // upload it back to Contentful
}
~~~

## Serverless functions combined with webhooks are the SaaS glue of the future

It doesn’t matter if you use serverless functions to enrich the functionality of a single service like in this example the asset creation of Contentful. Functions are a perfect fit to combine different services, too. 

Think for example of Algolia (a Search SaaS provider). To use their search, you have to index data. With webhooks, you can either push data directly into Algolia or, for complex cases, you can spin up a serverless function that controls which data should go in. Head over to [the docs](https://www.contentful.com/developers/docs/tutorials/general/enhancing-search-experience-with-algolia/?utm_campaign=automatically-watermark-contentful-assets-using-serverless&utm_medium=referral&utm_source=devto&utm_content=automatically-watermark-contentful-assets-using-serverless&utm_term=) to learn more.

In my opinion, we’ve entered a very bright future with the ability to deploy these function-based HTTP endpoints, you and I can focus on building great products without reinventing the wheel!

### Additional comments and notes

#### How to develop Lambda functions locally?

To develop serverless functions deployed to AWS locally, there is the excellent [serverless-offline package](https://www.npmjs.com/package/serverless-offline) which emulates AWS on your local machine. I highly recommend checking it out.

#### Environment variables

The described use case of re-uploading assets to Contentful requires you to create a management token which has write access to your Contentful space. Be aware that this token should not make it into your version control and should be handled via environment variables (that’s the solution you’ll find in the [provided GitHub repository](https://github.com/stefanjudis/serverless-contentful-watermark)) or configuration files. Take care!

#### Limitations of API gateway

In this example, API gateway triggers the function that deals with image processing. API Gateway currently has a maximum timeout of 30 seconds whereas Lambda functions [can run for up to 15 minutes](https://aws.amazon.com/about-aws/whats-new/2018/10/aws-lambda-supports-functions-that-can-run-up-to-15-minutes/). For processing tasks taking longer than 30s, it is recommended to split this single function into two functions to avoid hitting that timeout.

How to Build an Ecommerce Static Site with Jekyll, Contentful, and Commerce Layer

Contentful Blog — Wed, 14 Nov 2018 13:48:32 +0000

There are plenty of guides out there showing how to use static site generators (SSGs) for basic ecommerce stores — with most of them using Snipcart to add shopping cart functionalities to a simple product catalog.

The question is: are static sites a good alternative for enterprise-level ecommerce as well?

We definitely think so, and this tutorial aims to prove just that.

About this tutorial

By reading this guide, you will learn how to build a static site ecommerce with enterprise-level features. The resulting demo site will be a multi-country, multi-language online shop with the following characteristics:

Each country will have its own catalogue with custom product selection and sorting
Each country will have its own price list with localized currency
Each country will have its own inventory shipping from multiple warehouses

Instead of diving into the technical details, we will focus on the site architecture and development workflow. After learning the big picture, we recommend you explore the source code and follow the step-by-step guide that is available on GitHub.

Tools

To build our demo site, we selected some of the best tools, one for each piece of the site architecture:

Jekyll as the SSG
Contentful to manage content
Commerce Layer as the e-commerce platform
GitHub for code versioning
Netlify for deployment and content delivery

It's worth mentioning that we are going to manage all content with Contentful, including the product information, categories, and catalogues. Unlike traditional platforms, Commerce Layer is a purely transactional engine that leaves content management to the CMS. This approach lets creatives and content editors design any customer experience, without locking them into any site structure or template system.

Architecture and workflow

The picture below outlines how the tools fit together in the overall architecture. As you can see, each tool is dedicated to a specific step of the pipeline, contributing to the content editing and publishing workflow.

{: img }

Let's have a look at each step of the diagram:

Merchants work on the product offering within Commerce Layer. SKUs - the items that are being sold - are imported into Contentful for content enrichment. Prices and stock information are not imported.
Content editors get the list of available SKUs and create products within Contentful. Their job is also to assign products to categories, categories to catalogues, and catalogues to countries. Published content is built into the Jekyll site on Netlify.
Developers work on the site structure, layout, and client-side functions. The code is built into a static site and deployed to Netlify. Each product variant is tagged with the related SKU code, using the data-sku-code data attribute on any DOM element. This tagging lets the client-side code interact with the Commerce Layer API in real-time. Every time the code changes, developers push the code to Github for versioning and continuous deployment.
The client-side code fetches prices and inventory for each available SKU. It also adds shopping cart functionalities to the page and sends orders to Commerce Layer. Merchants fulfill orders through the OMS. Newly-created SKUs are sent to Contentful for content enrichment, feeding the same workflow.

Content model

The second step of the workflow is where content editors manage the product catalogue in Contentful. This is a key step that determines how the site works. The diagram below outlines the content model that we designed to support the multi-country, multi-language structure:

{: img }

Let's describe each model, their roles and relationships:

Countries have one associated catalogue, and the same catalogue can be associated to many countries. The market ID attribute determines the price list, inventory model, payment methods, shipping methods and promotions that are available for that country, as defined within Commerce Layer.
Catalogues are sorted list of categories. This means that each country can have its own categories with custom sorting.
Categories can have one or more lists of products by country. Each list has its own sorting, which means the same category can have a custom product merchandising for each country. If content editors don't define any specific list for a given country, the default one will be used.
Products can have one or more variants. All content that is common to all their variants can be defined at the product level.
Variants are the items that are being sold. The SKU code attribute links them to the SKUs defined in Commerce Layer and makes the variants shoppable. In our use case, variants can have a size.

Continuous delivery

Our static site ecommerce is made of three types of data that need to be refreshed as they change: code, content, and commerce.

When code changes, developers push the changes to Github, triggering a new build on Netlify. The build command fetches all published content from Contentful (jekyll contentful) and builds the static site (jekyll build).

{: img }

When content changes, content editors hit the publish button on Contentful. To trigger a new build for each publishing, we need to create a build hook on Netlify...

{: img }

...and link it to the publish webhook on Contentful:

{: img }

When commerce changes, we don't need to trigger any new build, since prices and inventory are dynamically fetched from the static site JavaScript.

Understanding the mix of dynamic and static data

At this point what we have is a static site that includes product information based on the content stored in Contentful. To make this site a real shop we bring in the Commerce Layer JavaScript library which enriches the HTML with commerce data.

When customers enter the site and choose a shipping country, this included JavaScript library gets an OAuth2 access token putting the related market ID in the request scope. All the subsequent requests are scoped to that market.

{: img }

The market in scope determines the right price list, inventory model, promotions, payment and shipping methods as defined by the merchant within Commerce Layer. As a developer, you just need to use the right access token to get country-specific commerce data and let customers place orders without errors.

Being a static site, all pages are plain HTML files served by Netlify CDN. The only dynamic information is provided by Commerce Layer API as displayed by the graphic below (red boxes) for each relevant page.

{: img }

As you can see, not all pages require the same kind of data from Commerce Layer:

For each product, the category page fetches the price of the first variant
The product page fetches the selected variant price and availability message
The shopping bag is dynamically built from the current order line items
The shopping bag preview is always populated and the "Proceed to checkout" button is linked to the current order checkout URL, that is part of the order API response.

Considering that prices generally don't change very often - unless you have some adaptive pricing mechanism in place - we could also import prices into Jekyll as YAML files and leverage Commerce Layer webhooks system to trigger a new deploy on each price change.

This would further improve the website speed by reducing the number of client-side requests to the Commerce Layer API. Pages, like the category overview page, could be almost completely static (apart from the shopping bag preview) as we wouldn't need to fetch the prices dynamically.

Placing orders

When customers proceed to checkout, they are redirected to the Commerce Layer hosted checkout application, where they can add all the required information and place the order.

{: img }

This is the easiest solution to reduce the time-to-market and start selling as soon as the content catalog is ready. That said, Commerce Layer also provides APIs to create a custom tailored experience on your own site. Just keep in mind that this would require us to add some server-side components to our static site, at least to manage the authentication and safely store its client_secret.

Sum up

Static sites are a great alternative to database-driven websites. They are fast, scalable, secure and cost-effective. All amazing features that can make your ecommerce shine.

In this article, we described how some of the best tools could be combined to build the perfect stack - a.k.a. JAMstack - for static ecommerce. The separation of concerns between code, content, and commerce is pretty clear. The work of developers, content editors and merchants is perfectly orchestrated.

From a features perspective, we just scratched the surface of what we can do. There are a lot of great services that would perfectly fit in our diagram, making our static site even more dynamic.

Next steps – Improving functionality with more services

Other than the fully branded checkout experience, our static site ecommerce could be improved adding the following features:

Full-text search and filtering, using a tool like Algolia
Customer account, with access to their order history, address books and wallets (credit cards on file) through the Commerce Layer customer APIs.
Last but not least, better content. Despite focusing our demo on publishing a product catalogue, we strongly believe that commerce should be driven by content. A platform like Commerce Layer provides the security, performance and flexibility you need to sell your products with confidence. But this isn’t enough. Content must be creative, emotive, and inspiring. By using Contentful, we can enrich our content model and build a unique customer journey. With our transactional engine sorted, we can focus on elevating our content experience to a whole new level.

Develop, edit & deploy websites entirely in the cloud with the CodeSandbox, Contentful and Netlify trio

Contentful Blog — Tue, 07 Aug 2018 15:16:26 +0000

Setting up a development machine can be a tedious process. I once worked in a company where it was an achievement to complete the setting up of the development environment, for the custom shop system we were working on, in under two days (to be fair though, this was before Vagrant and Docker became a thing). And why’s that?

Building software products relies heavily on things like your favorite editor or IDE to be productive, it depends on installed dependencies like databases, shell programs or servers to actually run and update your software. Is this still a necessity or could we ditch all of that and rely completely on cloud services today?

I recently gave a talk about the Frontend stack 2018 and had a look at how far you can get without placing a single file on your computer. As it turned out; you really can create websites, make them editable and later deploy them (I’m a big fan of the recent static site generators) from any computer using powerful online services today.

CodeSandbox – the new online editor in town

A while back, I noticed CodeSandbox being increasingly used for React prototyping, when people started sharing sandboxes on Twitter with specific React patterns or best practices. "Do we need another online editor?" was my immediate response.

Earlier this year, I wrote an article on how to use render props in React and decided to give CodeSandbox a try. It doesn’t feel like my local editor (I’m using VSCode) – but it’s pretty close.

In CodeSandbox you can start by forking one of the 500,000 (!) available user sandboxes, or choose to start from scratch using starter templates for React, Vue, Angular, and other frameworks. Looking at all the user-created sandboxes, you’ll see that the editor is used primarily for quick prototyping in the React ecosystem today. However, this doesn’t mean that you can’t use it to build something more complex inside or outside the React ecosystem.

Getting started with JavaScript development entirely in the cloud

Getting started with a new JavaScript-based project using a modern framework was very tedious, and far and away from being beginner friendly in the past. The folks working on React discovered that this had to change and came up with create-react-app. This project helps you to bootstrap and start a new React project in a few minutes by taking all the configuration away and providing all the needed defaults (#zeroconfig all the things 🎉).

create-react-app is the base for CodeSandbox to create new React projects. For Preact, Vue and other frameworks similar CLI tools are available, and there’s even a "vanilla" starter template without heavy framework dependencies that uses Parcel (a new zero-config bundler – it’s fantastic, trust me!) under the hood to give you all the freedom you need.

When you decide to go the React route and initialize a new project, you’ll get a codebase that is ready to dive into React development.

Crucial editor features that let you forget that you’re "just" in an online editor

cmd/ctrl+p to quickly access files and commands

There are a few things that I can not live without while doing web development – first, CMD+p and CMD+Shift+p. These two shortcuts let you jump to any file or execute any command available via a quick and easy-to-use fuzzy search. Programming is very often about productivity, and these two shortcuts help you to achieve anything without leaving the keyboard.

Dependency handling and automatic installation

But now you’re in a cloud environment, right? How does it work to install dependencies then? CodeSandbox provides a dialog which lets you choose dependencies from npm easily. When you install packages with this dialog, the package.json will be automatically updated. Sweet!

Prettier included by default

When developing in CodeSandbox, Prettier is enabled by default, is configurable, and also runs very smoothly!

Hot reloading in a separate window

Looking at the screenshot above, the editor provides you with an in-browser preview. The cool thing is that you can open the preview in a separate window, which is perfect for two monitor setups like mine. This way, the code is on one monitor and I can see the changes in near-realtime on the other one.

Autocompletion for projects shipping with TypeScript type definitions

When I discovered that VSCode picks up type definition included in npm packages, I finally decided to go for TypeScript. As a JavaScript developer, I’m very used to working without great autocompletion but seeing my editor picking up TypeScript definitions is excellent. To see that CodeSandbox does the same is nice!

GitHub integration makes CodeSandbox a real tool to work with

The last feature that got me was GitHub integration, which lets you create a project in CodeSandbox, push it to GitHub, and then make commits from CodeSandbox directly to GitHub. Very cool stuff!

The only feature missing

Split-view mode for viewing multiple files at once is the only thing missing to make it my everyday application for development. Let’s hope that it’ll come soon! 🤞🏻

Contentful – the content infrastructure for any project

With CodeSandbox you can quickly create your next JavaScript project and push it to GitHub. Very often when you do website development, the projects are built for people that are not that comfortable with writing code, though.

Take a quick, one-pager portfolio site for a friend as an example. How would you realize this project saving the effort of updating content with a pull request but also without setting up a complete content management system yourself? You can use Contentful’s content infrastructure for that.

With Contentful you can define your needed data models in minutes and get the data back using JSON APIs. For the above example, you need an entity with individual fields for an image, a headline, and a paragraph respectively. This flexibility is where Contentful shines – create a content type portfolio and define the three needed fields without any need to set up a server or something similar.

Your non-techy friend can now make content changes to the JavaScript app you’re building without editing JSON files or React code.

Later, you can use the provided JavaScript SDK to get the Contentful data edited by your friend.

import React from "react";
import ReactDOM from "react-dom";
import { createClient } from "contentful";
import "./styles.css";

// create the SDK client with the needed credentials
// which you can get in the web app
const client = createClient({
  space: "...",
  accessToken: "..."
});

function Portfolio() { /* … */ }

class App extends React.Component {
  constructor() {
    super();
    this.state = { isLoading: true, portfolio: null };
    // fetch the entry of your portfolio entry type
    client
      .getEntries({
        content_type: "portfolio"
      })
      .then(({ items }) => {
        this.setState({ isLoading: false, portfolio: items[0] });
      });
  }

render() {
    return this.state.isLoading ? (
      <div>Loading</div>
    ) : (
      <div className="App">
        <Portfolio {...this.state.portfolio.fields} />
      </div>
    );
  }
}
~~~{% endraw %}

[When you look at the code above](https://codesandbox.io/s/5wxvp413zl), one of my favorite things is that you can connect content entries with your frontend component quite easily ({% raw %}`<Portfolio {...this.state.portfolio.fields} />`).  This connection makes Contentful [a perfect fit for component-driven applications and sites](https://www.contentful.com/blog/2017/10/11/love-letter-to-component-ready-cms/?utm_campaign=deploy-cloud-codesandbox&utm_medium=referral&utm_source=devto&utm_content=deploy-cloud-codesandbox&utm_term=).

### Netlify – the few-clicks static deployment tool
Now, you have the code editable and connected to Github in CodeSandbox, and you can edit the content that powers the application via Contentful. The last step is to deploy this application somewhere. 

[Netlify](https://www.netlify.com/) is a relatively new service out there that specializes in static deployments. The cool thing about Netlify is that they also allow you to define build scripts - they’re your CI service and host, so to say. 

To deploy a site to Netlify you can start by importing a specific GitHub repository:

![Deploy a site by importing a specific GitHub repository](//images.ctfassets.net/fo9twyrwpveg/5k48xcI4xO8KcYmwEwqMYc/5008a772edae369b530bc6b30965929b/image_6.png)

Your next steps are to define a build command and public directory in which the static files will be present. In the case of *create-react-app*, the build command is {% raw %}`npm run build`{% endraw %} and the published directory will be {% raw %}`build`{% endraw %}. After submitting these two configurations, your first deploy will get up and running, and you’ll get a unique subdomain on netlify.com. 

One less obvious thing after importing a GitHub repository in Netlify is that this process also defines webhooks on GitHub. Now every time you push new code to GitHub, Netlify will automatically redeploy your site. Magic? Magic!

![Webhooks to automatically redeploy your site](//images.ctfassets.net/fo9twyrwpveg/JrUjqEneesI2S8YgomK6O/d5cd2a5d6f77ec80b542a66782c2096b/image_7.png)

### Usage of webhooks to connect all the pieces

For this portfolio demo site, there is no additional webhook configuration needed because the React application fetches that data right from the browser. Client-side only applications have the significant downside that they show a loading spinner initially until they fetched all needed API data though. A more performant way to do it is to statically pre-render React on the server and then do something that is called [hydration](https://reactjs.org/docs/react-dom.html#hydrate) when the client-side React code kicks in.

The static pre-render approach means that you would have to rerender the generated HTML files and deploy them whenever your friend updates her portfolio data in Contentful, too. An additional webhook notifying Netlify has to be configured in Contentful in that case.

![image 8](//images.ctfassets.net/fo9twyrwpveg/jPkoNvQOEoW4Ae286OG20/1c285c2794cc523aea5edcef70eefb6b/image_8.jpg)

With a little bit more of webhook configuration, you can then set up a complete deployment pipeline without any local installation and configuration shipping static sites without loading spinners.

### Use the power of web services to not reinvent the wheel

In conclusion, I’m amazed how much the web development field has changed. [Serverless technologies change the way we work](https://www.contentful.com/blog/2018/04/05/graphql-and-serverless-where-cloud-computing-is-heading/?utm_campaign=deploy-cloud-codesandbox&utm_medium=referral&utm_source=devto&utm_content=deploy-cloud-codesandbox&utm_term=) and we no longer have to worry about all the tiny pieces that might be needed in a project. For running Node.js projects we can use [Zeit](https://zeit.co/), for authentication [Auth0](https://auth0.com/), and for a performant search experience [Algolia](https://www.algolia.com/).

![The power of different web services as building blocks for your site](//images.ctfassets.net/fo9twyrwpveg/qWbrjftQbuSsq64Q6wq62/354b97dfc1c04876db1907283427c896/image_9.png)

Webhooks and serverless functions give me the possibility to connect all these services and write quick connectors in the cloud in an easy and speedy manner. That’s very exciting because I can now focus on building things rather than setting things up and... **I can even create and deploy websites on my friend's computer while we’re sitting on his balcony.**

Develop, edit & deploy websites entirely in the cloud with the CodeSandbox, Contentful and Netlify trio

Contentful Blog — Tue, 07 Aug 2018 15:16:26 +0000

CodeSandbox – the new online editor in town

Getting started with JavaScript development entirely in the cloud

When you decide to go the React route and initialize a new project, you’ll get a codebase that is ready to dive into React development.

Crucial editor features that let you forget that you’re "just" in an online editor

cmd/ctrl+p to quickly access files and commands

Dependency handling and automatic installation

Prettier included by default

When developing in CodeSandbox, Prettier is enabled by default, is configurable, and also runs very smoothly!

Hot reloading in a separate window

Autocompletion for projects shipping with TypeScript type definitions

GitHub integration makes CodeSandbox a real tool to work with

The last feature that got me was GitHub integration, which lets you create a project in CodeSandbox, push it to GitHub, and then make commits from CodeSandbox directly to GitHub. Very cool stuff!

The only feature missing

Split-view mode for viewing multiple files at once is the only thing missing to make it my everyday application for development. Let’s hope that it’ll come soon! 🤞🏻

Contentful – the content infrastructure for any project

Your non-techy friend can now make content changes to the JavaScript app you’re building without editing JSON files or React code.

Later, you can use the provided JavaScript SDK to get the Contentful data edited by your friend.

import React from "react";
import ReactDOM from "react-dom";
import { createClient } from "contentful";
import "./styles.css";

// create the SDK client with the needed credentials
// which you can get in the web app
const client = createClient({
  space: "...",
  accessToken: "..."
});

function Portfolio() { /* … */ }

class App extends React.Component {
  constructor() {
    super();
    this.state = { isLoading: true, portfolio: null };
    // fetch the entry of your portfolio entry type
    client
      .getEntries({
        content_type: "portfolio"
      })
      .then(({ items }) => {
        this.setState({ isLoading: false, portfolio: items[0] });
      });
  }

render() {
    return this.state.isLoading ? (
      <div>Loading</div>
    ) : (
      <div className="App">
        <Portfolio {...this.state.portfolio.fields} />
      </div>
    );
  }
}
~~~{% endraw %}

[When you look at the code above](https://codesandbox.io/s/5wxvp413zl), one of my favorite things is that you can connect content entries with your frontend component quite easily ({% raw %}`<Portfolio {...this.state.portfolio.fields} />`).  This connection makes Contentful [a perfect fit for component-driven applications and sites](https://www.contentful.com/blog/2017/10/11/love-letter-to-component-ready-cms/?utm_campaign=deploy-cloud-codesandbox&utm_medium=referral&utm_source=devto&utm_content=deploy-cloud-codesandbox&utm_term=).

### Netlify – the few-clicks static deployment tool
Now, you have the code editable and connected to Github in CodeSandbox, and you can edit the content that powers the application via Contentful. The last step is to deploy this application somewhere. 

[Netlify](https://www.netlify.com/) is a relatively new service out there that specializes in static deployments. The cool thing about Netlify is that they also allow you to define build scripts - they’re your CI service and host, so to say. 

To deploy a site to Netlify you can start by importing a specific GitHub repository:

![Deploy a site by importing a specific GitHub repository](//images.ctfassets.net/fo9twyrwpveg/5k48xcI4xO8KcYmwEwqMYc/5008a772edae369b530bc6b30965929b/image_6.png)

Your next steps are to define a build command and public directory in which the static files will be present. In the case of *create-react-app*, the build command is {% raw %}`npm run build`{% endraw %} and the published directory will be {% raw %}`build`{% endraw %}. After submitting these two configurations, your first deploy will get up and running, and you’ll get a unique subdomain on netlify.com. 

One less obvious thing after importing a GitHub repository in Netlify is that this process also defines webhooks on GitHub. Now every time you push new code to GitHub, Netlify will automatically redeploy your site. Magic? Magic!

![Webhooks to automatically redeploy your site](//images.ctfassets.net/fo9twyrwpveg/JrUjqEneesI2S8YgomK6O/d5cd2a5d6f77ec80b542a66782c2096b/image_7.png)

### Usage of webhooks to connect all the pieces

For this portfolio demo site, there is no additional webhook configuration needed because the React application fetches that data right from the browser. Client-side only applications have the significant downside that they show a loading spinner initially until they fetched all needed API data though. A more performant way to do it is to statically pre-render React on the server and then do something that is called [hydration](https://reactjs.org/docs/react-dom.html#hydrate) when the client-side React code kicks in.

The static pre-render approach means that you would have to rerender the generated HTML files and deploy them whenever your friend updates her portfolio data in Contentful, too. An additional webhook notifying Netlify has to be configured in Contentful in that case.

![image 8](//images.ctfassets.net/fo9twyrwpveg/jPkoNvQOEoW4Ae286OG20/1c285c2794cc523aea5edcef70eefb6b/image_8.jpg)

With a little bit more of webhook configuration, you can then set up a complete deployment pipeline without any local installation and configuration shipping static sites without loading spinners.

### Use the power of web services to not reinvent the wheel

In conclusion, I’m amazed how much the web development field has changed. [Serverless technologies change the way we work](https://www.contentful.com/blog/2018/04/05/graphql-and-serverless-where-cloud-computing-is-heading/?utm_campaign=deploy-cloud-codesandbox&utm_medium=referral&utm_source=devto&utm_content=deploy-cloud-codesandbox&utm_term=) and we no longer have to worry about all the tiny pieces that might be needed in a project. For running Node.js projects we can use [Zeit](https://zeit.co/), for authentication [Auth0](https://auth0.com/), and for a performant search experience [Algolia](https://www.algolia.com/).

![The power of different web services as building blocks for your site](//images.ctfassets.net/fo9twyrwpveg/qWbrjftQbuSsq64Q6wq62/354b97dfc1c04876db1907283427c896/image_9.png)

Webhooks and serverless functions give me the possibility to connect all these services and write quick connectors in the cloud in an easy and speedy manner. That’s very exciting because I can now focus on building things rather than setting things up and... **I can even create and deploy websites on my friend's computer while we’re sitting on his balcony.**

An extremely picky developer's take on static site generators for PHP: Part 2 - Jigsaw

Contentful Blog — Wed, 01 Aug 2018 14:24:04 +0000

In the first article of the series we took a look at Sculpin, the PHP static site generator which is currently the most starred on Github. Today we’re exploring Jigsaw, a tool which promises to bring a Laravel-based approach to the world of PHP static site generators (SSGs).

I’m gonna be honest with you—I’m not the biggest Laravel user. I appreciate what it's done for PHP, and how it has helped push forward the world of PHP development in a time where older frameworks were struggling to keep up with innovations coming from other platforms. It’s also a prime example of the good kind of cross pollination happening in the PHP world: it employs packages from the Symfony world, The PHP League, Doctrine, and more. It’s a good lesson in building something cool on top of great foundations, without having to continuously reinvent the wheel. Overall, I think that Laravel is an excellent framework, but I'm definitely inexperienced with it. If you think I have overlooked or misjudged something (both Laravel and Jigsaw-related), please feel free to reach out to me on Twitter: constructive criticism is always welcome!

Jigsaw (~900★)

Just like I did with Sculpin, I’ll jump straight into the action, so let’s create a sample project and play with it! We’ll open the official docs and start from there.

Unlike Sculpin, Jigsaw doesn’t provide a skeleton project we can use as quickstart. Instead, we must create a normal project, add tightenco/jigsaw as dependency, and use its CLI:

mkdir jigsaw-example && cd jigsaw-example
composer require tightenco/jigsaw
./vendor/bin/jigsaw init

Let’s go through the default directory structure that was just generated. In the top level directory, we have a config.php with this code:

<?php

return [
    'baseUrl' => '',
    'production' => false,
    'collections' => [],
];

I’m not sure yet what this means, but having a configuration file clearly placed seems like a good approach 👍. The second PHP file is called bootstrap.php, and it’s empty (only some commented-out code), but it hints at how we could use the event system at some point.

We also have Javascript dependencies. I see a webpack.mix.js, which lets me know that Laravel Mix is being used. So I run npm install and in the meantime I’ll explore the docs to try to understand a bit more of what I’m doing 😉.

About 10 dependencies and 300MB in node_modules later, we’re back. Now, let’s try to build our site. It appears we have 2 ways of doing so: we can use the Jigsaw CLI to build and then serve, or use npm and its live reload capabilities provided by Browsersync:

# With Jigsaw CLI
./vendor/bin/jigsaw build
./vendor/bin/jigsaw serve # localhost:8000


# With npm
npm run watch # localhost:3000

Regardless, both work and you get this!

Naturally, only John Oliver can really convey my reaction to this page.

The directory structure

Now let’s explore the skeleton project that was generated earlier. The order in which we’re exploring parts of the project might appear to be random because, well, it is. That’s the cool thing about discovering something new, right? The exploratory phase, where you peek at this, take a look at that, and oh, a source directory! What lies therein?

Being a Laravel-based project, I’m not surprised to find an index.blade.php file, which contains the shiny Hello world! from above. The @extends('_layouts.master') directive hints to me that I should go look in the _layouts directory for a master.blade.php template, and lo and behold, there it is. Everything seems nice and tidy with the templates, let’s go check how assets are handled.

Assets and configuration

Laravel Mix provides out of the box support for Sass, so source/_assets/sass/main.scss is your starting point. SCSS might not be the latest and shiniest thing on the CSS landscape, but it’s battle-tested and definitely gets the job done, so I approve of its inclusion. Let’s turn that page into something that will get your attention

/* source/_assets/sass/main.scss */
$bg-color: red;

body {
  background-color: $bg-color;
}

Let’s hit save and webpack should do its magic and turn our background red, right? Yes, it does! I'll spare you the screenshot for your eyes' sake, and now let's continue our exploration.

I’d like to set up a basic blog, with posts written in Markdown files. To do that, let’s introduce the concept of collections, which, as the docs state, give you the ability to access your content at an aggregate level. Sounds about right for a blog. Remember the config.php file from earlier? Let’s tweak it to tell Jigsaw about our collection of blog posts:

<?php

return [
    'baseUrl' => '',
    'production' => false,
    'collections' => [
        'posts' => [
            'path' => 'blog/{date|Y-m-d}/{filename}',
        ],
    ],
];

Jigsaw will look for the corresponding directory for every collection we define. In our example, posts will become source/_posts, so in that directory let’s create a blog post to show our love for one of the greatest songs of all times.

---
extends: _layouts.post
title: "The Fresh Prince"
author: Will Smith
date: 1990-09-10
section: content
---

Now this is a story all about how
My life got flipped-turned upside down
And I'd like to take a minute
Just sit right there
I'll tell you how I became the prince of a town called Bel-Air

Let’s save this file as the-fresh-prince.md, and then let’s create the appropriate template as specified in the extends part of the front matter: source/_layouts/post.blade.php.

@extends('_layouts.master')

@section('body')
<h1></h1>
    <p>By  - </p>

    @yield('content')
@endsection

Let’s run ./vendor/bin/jigsaw build, and we’ll have a nice build_local directory generated with our shiny website, so let’s open the URL (which we still have to build ourselves) and… It works!

(Author’s note: at this point I would like to add more GIFs, but I have to keep it professional. Let’s all just pretend to have a GIF of Jake Peralta saying noice, thank you.)

Is Jigsaw Contentful-ready?

We’ve covered some of the basics, so now the million-dollar question: is Jigsaw Contentful-ready? With this, of course, I mean how easily content stored in Contentful can be integrated in a Jigsaw-powered website. We open the documentation page about event listeners, and with that page open in the tab, we can give this a try. Remember bootstrap.php? It contained some commented-out code which hinted at how events work, so let’s reopen that:

<?php

use TightenCo\Jigsaw\Jigsaw;

/** @var $container \Illuminate\Container\Container */
/** @var $events \TightenCo\Jigsaw\Events\EventBus */

/**
* You can run custom code at different stages of the build process by
* listening to the 'beforeBuild', 'afterCollections', and 'afterBuild' events.
*
* For example:
*
* $events->beforeBuild(function (Jigsaw $jigsaw) {
*     // Your code here
* });
*/

Ok, seems to be reasonably easy. As extra help, after some googling I found an article about creating a sitemap with Jigsaw, and the original pull request for the events feature, so I have enough to figure this out. Let’s composer require contentful/contentful and hack this thing together!



// In bootstrap.php

require_once 'contentful_fetcher.php';
$events->beforeBuild(ContentfulFetcher::class);


// In contentful_fetcher.php

use Contentful\Delivery\Client;
use Contentful\Delivery\Resource\Entry;
use TightenCo\Jigsaw\Jigsaw;

class ContentfulFetcher
{
    /**
     * Jigsaw requires either a callable or the name of
     * a class which implements a method called "handle"
     *
     * @param Jigsaw $jigsaw
     */
    public function handle(Jigsaw $jigsaw): void
    {
       // Let's only build for production
        if ($jigsaw->getEnvironment() === 'local') {
            return;
        }

        $client = $this->getClient($jigsaw);

        foreach ($jigsaw->getCollections() as $collection) {
            $this->cleanSourceDir($jigsaw, $collection);
            $entries = $this->getPlainEntries($client, $collection);

            foreach ($entries as $entry) {
                $this->writeEntryToMarkdownFile($jigsaw, $entry, $collection);
            }
        }
    }

    private function getClient(Jigsaw $jigsaw): Client
    {
        return new Client(
            $jigsaw->getConfig('contentful.accessToken'),
            $jigsaw->getConfig('contentful.spaceId')
        );
     }

    /**
     * Before every run, we remove the previous contents of the source directory.
     */
    private function cleanSourceDir(Jigsaw $jigsaw, string $collection): void
    {
        $dir = $jigsaw->getSourcePath().'/_'.$collection;
        $jigsaw->getFilesystem()->deleteDirectory($dir, true);
    }

    /**
     * We transform objects of type Contentful\Delivery\Resource\Entry
     * into plain arrays for ease of use later on.
     */
    private function getPlainEntries(Client $client, string $collection): array
    {
        // Convention:
        // Every defined collection corresponds to a content type ID in Contentful
        $query = (new Contentful\Delivery\Query())
            ->setContentType($collection)
        ;
        $entries = $client->getEntries($query);

        if (!count($entries)) {
            return [];
        }

        /** @var Contentful\Delivery\Resource\ContentType $contentType */
        $contentType = $entries[0]->getContentType();

        return \array_map(function (Entry $entry) use ($contentType) {
            // Convention:
            // The content types may contain a field called body,
            // which will be used as the markdown content of the generated file,
            // so we give it a default value
            $plainEntry = [
                'id' => $entry->getId(),
                'body' => '',
            ];
            foreach ($contentType->getFields() as $field) {
                $plainEntry[$field->getId()] = $entry->get($field->getId());
            }

            return $plainEntry;
        }, $entries->getItems());
    }

    /**
     * Converts an entry into a markdown file and writes it to disk.
     */
    private function writeEntryToMarkdownFile(Jigsaw $jigsaw, array $entry, string $collection): void
    {
        $contents = '---
extends: _layouts.'.$collection.'
section: content
';
        $body = '';
        foreach ($entry as $field => $value) {
            if ($field === 'body') {
                $body = $value;

                continue;
            }
            $contents .= $field.': '.$value."\n";
        }

        $contents .= '---'."\n\n".$body;

        $path = sprintf('_%s/%s.md', $collection, $entry['id']);
        $jigsaw->writeSourceFile($path, $contents);
    }
}

This code is more of a proof of concept than complete implementation (it relies a lot on conventions, and it lacks handling of special fields such as locations, links, etc); however, it should get the job done for a basic example.

First of all, we pass a fully-qualified class name (FQCN) to $events->beforeBuild(), and Jigsaw will know that it must create an instance of that class and execute the handle method of it. Truth be told, I think there should be an interface in order to have a stronger contract, something like this:

namespace TightenCo\Jigsaw;

use TightenCo\Jigsaw\Jigsaw;

interface EventHandlerInterface
{
    public function handle(Jigsaw $jigsaw);
}

Or perhaps just requiring any valid callable (thus including objects implementing __invoke). It works now in its present state but I like strong contracts, and you should know by now that I'm picky—it's in the title of the series!

This is an overview of what ContentfulFetcher actually does:

If the local environment is detected, the whole process will be stopped. This is done to avoid the performance impact due to having to fetch all data from Contentful on every build (which can likely be very often, if running on watch mode). The idea is to run a ./vendor/bin/jigsaw build production initially to fetch all content, and then operate with what’s been stored locally. Of course, a good idea would be to add the generated files to the local .gitignore, to avoid keeping them in version control (content doesn't belong there).
We create an instance of a client object using the Contentful Delivery SDK for PHP.
We iterate through all the defined collections, and we use a convention to map a local collection to a content type ID in Contentful. We then clean the corresponding directory, where we will dump the generated files.
We fetch entries for the given content type, and we use the Contentful\Delivery\Resource\ContentType object to get a list of fields. With those fields, we build the appropriate YAML front matter section, and then we add the body at the end (string handling here is very basic and could be improved).
We finally dump the contents of the generated file. We use the Contentful ID to save them, as we can define a pretty URL when configuring the collection.

There’s a lot of gluing going on, but it works on my machine™️. This could be done a thousand times better, with more flexible configuration, but as a proof of concept I’m reasonably satisfied with it.

Verdict

Before our final verdict, I think it's only fair to mention that during the process of writing this article, I experienced a couple of issues which have since been fixed. Before publishing, I got in touch with Matt Stauffer (one of the people behind Jigsaw) and explained that I was experiencing some difficulties with the handling of assets and events (misconfiguration for the former, and lack of docs for the latter). Over the course of a weekend, both issues have been fixed 👏

I quite like Jigsaw, but it’s rather barebones in its out-of-the-box configuration. Unlike Sculpin, it makes no assumption over what kind of use you’ll have for it, which is a double-edged sword: it avoids clutter and gives you complete control, but it also creates a higher barrier of entry for simple use cases such as blog.

To solve this issue, Matt revealed to me that they’re building a generator with website skeletons, to address the most common needs (such as a blog, a documentation website, etc), with a Github issue to track progress. He also mentioned that they just launched a new showcase of websites built with Jigsaw, which I encourage you to check out.

As a non-Laravel developer, I was afraid that Jigsaw would be difficult to pick up and use, but it definitely wasn’t the case. It is actively being developed (the events system was added just a few months ago) and is on track to quickly become the most popular PHP static site generator, in terms of stars on Github. It provides very good defaults, and assets are preconfigured to use a modern stack (Sass for CSS, and webpack with Babel for JS).

The only real criticisms I can make are actually the same as when I discussed Sculpin: content and presentation (assets and templates) could be more separate; instead, they belong to the same source directory, and I would love to be able to have a more defined structure for collections, instead of relying exclusively on the YAML bit before the actual Markdown contents.

Next article will be about Couscous, which describes itself as a documentation generator. Stay tuned!

Making the most of Marketo with UI extensions and Lambda functions

Contentful Blog — Fri, 20 Jul 2018 07:43:24 +0000

Marketo offers a suite of marketing automation tools that addresses the different subfields of marketing and hence, can be utilized extensively by the various arms of a full-blown marketing department. We at Contentful use Marketo as one of the solutions to handle duties performed by our marketing folks, such as forms on our landing pages.

Being such a comprehensive platform, Marketo has a bit of a steep learning curve and, at first glance, some of its features can seem tricky to implement. In our case of forms, handled by authors such as our demand generation team, the out-of-the-box process to embed the code of a Marketo form into a page is both tedious and time-consuming.

The process is an involved sequence of grabbing the embed code and pasting it into a HTML block or a markdown field on a page, and then repeating for every page where a form is desired. The stock appearance of the forms is also somewhat generic, so a degree of customization is needed to make them look more appealing.

We found a quick workaround to take manual repetition out of the equation and fill that in with easy-to-use selection dropdowns in our Contentful system using UI extensions (read more in our concept documentation). We also managed to style the forms to align more with Contentful’s own branding.

Prettifying Marketo forms

First and foremost, Marketo forms utilize inline styles on a form’s elements and have a plain default design. These forms are also decidedly proprietary with a very unique markup pattern, leading us to having to rethink our forms in order to work with Marketo.

To make sure we removed all potential conflicting styles from Marketo, we found an existing function, that we modified to be a little cleaner, which uses a div on the page to get the ID of the form (generated by our static site generator with Contentful data from the UI extension). From there, we load the form from Marketo and strip all styles and conflicting CSS added by the JavaScript from Marketo, leaving us with an unstyled HTML form.

With our new clean HTML form and the UI extension, we are able to style the Marketo form using SASS @extend with our own form design’s styles, configured to be Marketo-form-compatible.

.custom_mkto_form {
  // ...
  .mktoLabel {
    @extend .label;
    display: inline-flex;
  }

  .mktoButton {
    @extend .btn;
    @extend .btn--blue;
    @extend .btn--large;
  }

  .mktoErrorMsg {
    @extend .error-message;
  }
  // ...
}

Now, we have a form that is both easy for the author to use on a page and has nice brand-specific styles.

UI Extensions and Lambda save the day

Since it can also be a tedious and time-consuming process to embed forms manually on each page where a form is desired, we addressed this by building a UI extension in Contentful. This offers authors the ability to select the form they wanted to use on a landing page via a list of choices hosted in something like a dropdown menu.

There was a hurdle to leap over first, however. Contentful UI extensions only perform client-side API calls, but calls to the Marketo API had to be done server-side due to Marketo’s CORS and the server endpoint restrictions in place.

To do this, we built a Lambda function, hosted on Netlify to take advantage of their awesome new functions feature. The UI extension would perform a HTTP call to this function, which would return a payload of all of the forms in our Marketo instance, and subsequently parse through the payload and generate a dropdown selection of all the forms available for the editors to use on their landing pages.

You can use this very UI Extension by utilizing our new UI Extension example installer. Simply head to Settings from within the Contentful application, click on Extensions and then Add Extension. From there, click on Install an example, locate Marketo Forms from the list and Install.

The next step would be to fill the required Lambda Function URL installation parameter with the URL endpoint of your Lambda function. You can find the code for this Lambda function in the repository of our extensions.

It is best to create this Lambda function in either Netlify Functions, direct use of AWS Lambda, or any of the other serverless function platforms. Not all Lambda function services are created equal and the code might have to be modified to fit your platform of choice.

Setting it up is relatively easy; follow our tutorial on how you can get your Marketo credentials and the Environment Variables needed by the Lambda function. To use it, select the JSON object Field when you are creating your content model for a new landing page, and proceed to set the Appearance tab to be Marketo forms.

Now when your editors select a form to use on a landing page, your developers will have a nice JSON dump of the form data to work with and integrate with the Marketo Forms JavaScript API.

Try out UI extensions

UI extensions are a great way to customize and extend your editorial experience with Contentful; anyone with some JavaScript knowledge can create their own extension, leaving you with an incredibly accessible, powerful feature with endless possibilities. Start by creating a free Contentful account, if you don't already have one, and discover UI extensions... and also how great our content infrastructure works with your code and static site projects.

The understated innovation of static site generators

Contentful Blog — Tue, 17 Jul 2018 08:18:34 +0000

This article is a reflection on the bigger picture around the new generation of static sites generators that are currently gaining traction and adoption.

Static site generators have been around for a while and an early player in the field was Jekyll, which gained a lot of traction at launch, triggered the birth of GitHub Pages and hence created a developer-friendly solution to content. We’re now seeing a new wave of static site generators like GatsbyJS, that no longer are limited to publishing needs of the developer audience (as in, manage your content completely on GitHub), and instead have the ambition of becoming the new tool for content publishing.

The current wave of static site generators comes with an entire ecosystem that makes the movement wider in scope than site-building tools themselves or even just content. While there’s a lot of attention surrounding what this means for applications that directly touch the browser, there’s much less thought put into behind-the-scenes reasons on why these new stacks like Contentful + GatsbyJS + Netlify are getting attention and being chosen by development teams.

Runtime that reaches out for data and some adventures in game servers

Years ago, I worked at Wooga, building game servers for social games—one problematic performance aspect of games is that the state of the game has to be updated very frequently. This means game servers are write-heavy beasts that hammer your databases in a way where caching, the technique most often used in lifting database load to cope with high traffic, would not be an option.

It was back then when I started reflecting on the implication of choices regarding the position of the runtime. By runtime, I mean the process where the code is running—which is with respect to all databases and services that need to be reached in order for actions, that allow you to fulfill the current request, to be performed.

The solution to the pressure that our applications put on databases was a complete turn-around of all strategies surrounding interaction with the data layer; which meant transitioning from a model, where each and every request had a direct database interaction, to one where a user’s entire game-state would be loaded in the runtime (we chose Erlang) at the beginning of a session. You can still find the slides—from a scary number of years ago—from the beginning of the project and the considerations after running it in production.

In a nutshell, that project led to the decrease number of interactions with databases by two or three orders of magnitude and achieve sub-millisecond response time improvement by two orders of magnitude compared to previous approach. The idea must have also germinated in other brains within the gaming industry, as Microsoft launched Orleans a year later.

What we learned from the push for having data in the runtime

So, how is our work at Wooga on strategies around working with game-state now relevant in the scope of static site generators?

Working on that topic brought me to more general reflections on how stacks are built (data at last)—the runtime is the part that coordinates all actions, the request enters the runtime and from the runtime, you ensure that a number of other systems are touched or triggered.

After a request enters your runtime, you write your code which is a mix of two things:

Interacting with elements external to the process

a. Reading data
b. Writing data
c. Checking authorization
d. Triggering tracking
The actual business logic that must be performed in order to satisfy the request

Now if you think about the two places most of the pain (bugs, errors, outage) comes from; writing and operating application, I’m sure that the former is thought of as the more evil place to be.

I remember clearly what it meant to start programming in a paradigm where writing was mostly out of the picture, which enabled me to realize the idea of ninja state or "flow" that you can find in books like “Getting things done” or “Flow”.

It’s a superior state of mind, where you are able to just think and focus on what you want to achieve and are not stuck with internal dialogs like:

"Where’s the data I need?" - all data is just there; just access and manipulate it as needed
"In what order would these things happen?" - in the exact order you’re writing them, which should be arranged flow naturally according to how one would think
"Is there anything that could fail in this?" - the only problem you might have is with the data at hand; but there’s no other interference or clutter that could influence what you’re writing (no network errors, collisions, race condition).

These considerations have led me to think about the "upside down stack", one where the database or other services come first—not in between or just in time as the logic progresses—and where you enter the runtime with all data and authorization necessary in order to perform the request.

How are static site generators a step towards the "upside-down stack"?

Static site generators are essentially build processes, processes that are triggered because of change in either the code or data underlying the static artifact (this is a novelty of SSGs building both data & code versus building just code). Because these processes are triggered by changes in the data or the code, those changes are what enter computation rather than something that is fetched a runtime.

This is how static site generators are akin to the "upside-down stack"; data comes first and is what you enter the process with, versus being something that is sought after only during an ongoing process.

This vision is already present in a modern build system like Concourse CI with its concept of Pipelines being composed of Job and Resources. This goes in the direction of thinking "Okay, this job is triggered by a change on some resources and, in order to perform the job, other resources will also be needed." The dependency tree of the resources required in order to perform an action is specified beforehand and you can ensure that all resources are in place prior to performing a job.

This is where I see static site generators as early adopters of certain patterns that will gain more generalized adoption. If you have a domain where the data is read more than it is written, a build-driven approach has a number of advantages (among them excellent performance, low cost and a better security model).

The intersection with serverless

As mentioned in my previous post, where I predicted some bigger trends behind the surface of Serverless & GraphQL, serverless has deeper implications than what is usually known for. For example, it can be seen as Serverless is a very compelling technology to deal with workloads that are build driven like static site generators, you need to handle workloads that are event driven (with events generated by changes to the application, the data or the domain model) and then different in scale and distribution over time than workloads to serving constant traffic.

To support build cycles and pipelines, Serverless becomes relevant again as infrastructure used to host these new kinds of architectures. Step functions are something to look at in order to understand how this future might look like.

Conclusion

Before concluding this post, it’s important to look at what static site generators represent and which factors they rely on to achieve the popularity they currently enjoy. The first important realization is that static site generators can be more dynamic than the name suggests and that the concepts they apply might see much wider adoption if accompanied with a significant innovation of the toolchains currently in use. As an example static site generators might be a pointer to the fact that build processes are about to be re-evaluated as a delivery mechanism for more than application logic.

In short, here are trends to watch for, as they might serve as seeds for future looking-thoughts:

Data enters the runtime at the very beginning of the computation, instead of being grabbed during computation
The event that triggers the computation is a change in application logic, in the data of the domain that the application deals with, or changes of the data domain itself
A user request can be satisfied by serving a pre-compiled set of assets (that are built only when the underlying data is updated, and not at request time)
The data is "built", instead of being fetched and integrated at every request; requiring more modern build systems and innovations of the concept of build systems

Next step

It’s important to keep watching how the phenomenon of static sites continues to evolve and seeing the prospects of the new generation of web architectures that are build-based rather than the current request/response-focused patterns.

In order for that trend to grow, other trends need to fall into place; build systems and databases (or systems that front the database) are some areas we need to look at for innovations. This will be something I will elaborate in follow up posts of this series.