Forem: Johnny Tordgeman

Let’s Build a WebSockets Project With Rust and Yew 0.19

Johnny Tordgeman — Mon, 14 Feb 2022 21:26:55 +0000

Cover Photo by Mike Winkler on Unsplash

Let me start by saying I’m a HUGE fan of Yew. Using the power and flexibility of Rust to build front-end components is something that I feel will only get bigger as the adaptation of WebAssembly grows. Recently, one of my side projects needed some WebSockets love. I figured this would be the perfect time to dive into Yew’s recently released 0.19 version, which introduced several significant changes to the framework (the removal of web-sys and its supported services (i.e. ConsoleService) and the introduction of Functional Components, to name a few).

As you might have guessed from the title already, we will build a chat app (such a cliche, I know, I’ll show myself out) using Yew, yew-router, yew-agent, and several other crates.

At least it’s got GIF support!

To give you a sense of what we are going to build, check out this lovely image:

🛑 Before moving forward make sure you have Rust and NodeJS installed.

🗿 The WebSocket Server

To use WebSockets on the client-side you need a server capable of working with WebSockets (a shocker, I know). To keep this post a (somewhat) reasonable length, and for the sake of focusing on a single subject, we won’t be discussing the WebSockets server too much. In a nutshell: other than handling the incoming and outgoing connections, the server saves the incoming connections in an array called users. Every few seconds the server will compare the users array to the server’s current active connection list to verify that all the users in the list are in fact still connected.

While you can go and build/use your own WebSockets server for our lovely chat app, the easier solution here will be to clone the NodeJS WebSocket server I used at https://github.com/jtordgeman/SimpleWebsocketServer. Once cloned, run npm i and then npm start and if all went well, you should now have a cute WebSockets server running locally on port 8080.

Time to Take Off 🚀

Clone the starter project at https://github.com/jtordgeman/YewChat
Install the toolchain dependencies

npm i

The starter project is nothing but an empty project with wasm-pack and webpack already set up, so we can focus our attention to Yew

🛤 ️Routing

Our app has three possible routes:

login— a simple page with a textbox and a button for a user to enter their nickname
chat — The main page — has a list of users, the chat display, and a textbox to write stuff to the chat
404 — If anyone tries to go to a page that doesn't exist — this catch-all page will show up in all its glory

If you are coming from React you probably know (and use) react-router-dom. Well, good news! yew-router is pretty much the exact same idea! yew-router handles the display of different pages (components) depending on the URL.

So enough with the theory, let’s get busy:

Create a new folder named components under the src folder. We will host all of our components here.
Create three files under the newly created components folder: chat.rs , login.rs , mod.rs. We will leave chat and login empty for now, in mod.rs add the following:

At the top of lib.rs , add the following use statements:

Note that Login and Chat will show as unresolved. We will fix that when we start dealing with components later on.

Thanks to yew-router we can use an enum to define our routes. We will add it to our lib.rs as follows:

Pretty neat stuff! We annotate each entry with the URL it handles. Note that the 404 route uses an additional annotation: #[not_found] — this macro comes from the yew-router package and is basically what makes this route a catch-all route.

Next we need a way to translate the enum value of a route to an actual component. Add the following switch function just below the Route enum:

To render our router (and in turn, the components it routes to) we will use a functional component. But what is a functional component you may ask? A functional component is a simplified version of the regular Yew component which can receive properties and determines the rendered content by returning HTML. In short — functional components are basically components that have been reduced to have only the view method.

Last but not least, we need an entry point to our app — a function that the JavaScript can call in order to initiate our Yew app. Add the run_app function to lib.rs as follows:

🔬So what do we have here?

Line 1 : Using the wasm_bindgen macro we expose the run_app function to JavaScript.
Line 3 : We initialize the wasm_logger crate. Using wasm_logger and the log crates we can write debug logs to the browser’s console.
Line 4 : We are passing our Main component (defined in the previous step) to the start_app method of yew, to — as you might have guessed — start the app.

With the routing in place, let’s move on to creating our components.

🪅 ️Components — Phase 1

For simplicity sake, our app will consist of two components:

Login — a simple functional component to get the username and connect to the WebSockets server.
Chat — the main component of the app — shows the chat window, and a text area to write content to the chat.

When users browse to the Login component they will type a username and click connect — that username then needs to pass down to the Chat component and be used there to connect to the WebSocket server. To achieve that we will use a Context (think of it as a global state of the app) that will hold the username for any component to use. The way we define a Context is as follows:

Define a struct for the context and a type alias for it:

In the main component, define a context using the use_state hook:

Lastly, let’s use the context by wrapping the main component HTML with a ContextProvider element which enables the child elements to actually use this context.

A ContextProvider element requires a context struct (User in our case) and a context object (ctx), that is set using the element’s context property.

👁️ Note: the child elements of a ContextProvider will re-render whenever the context changes.

With the context in place, let’s build the Login component!

In components/login.rs add the login functional component:

Declare a use_state hook that will manage the state of the username the user typed in. When declaring a use_state hook we provide the default value (an empty String in our case), similarly to React’s use_state hook:

To get a reference to the context (which as you might remember will hold the global state for the username), use the use_context hook:

The Login component UI will contain an input element and a button. Whenever the input changes we need to update the local username variable. To connect between the UI element and username, we use a callback:

🔬So what do we have here?

Line 7 : We clone the username state handler so we can later update it using its set method.
Line 9 : We create a Callback from the input element’s onchange event.
Line 10 : We get the target element (the element that fired the event) using the InputEvent’s target_unchecked_into method.
Line 11 : We update the username state with the new value that the input element currently holds using the element’s value method.

🎉 If you have ever done any JavaScript work with HTML elements then all of this should look pretty similar.

The next callback we need to create is for the submit button. Once the username is typed, and the user clicks on the submit button — we need to save that username to the global context so that other components can use it if needed. Create the onclick callback as follows:

Lastly, let’s add the actual UI for this component using the html! macro:

👀 A few callouts:

The input element registers to the oninput callback. Every time the input changes — we call oninput and update the username state.
We use the Link element from the yew_router crate to route the browser to the Chat route using the to property.
The Link element contains a button element, which registers to the onclick callback which updates the global context with the value of username. In addition to that, the button also sets the disabled property to prevent the user from clicking it, unless username has a length of 2 characters or more.

And that’s a wrap for the Login component 🎊. We now have a component that uses a Context in order to share its data (username in our case) with other components.

Now we may be tempted to run ahead and create the chat component, but let’s stop for a minute and think about how the chat component even works…

👋 Hello WebSockets!

At the heart of our chat application are WebSockets. WebSockets enable us to asynchronously send and receive messages between the server and the client without the need for constant polling from the client. This feature makes Websockets the perfect centerpiece for our chat app. Let’s go ahead and create a WebSocket service that will handle all the aspects of working with WebSockets in our app!

Create a new folder called services under the src folder.
Add a new file called mod.rs inside services with the following content:

Our Websocket service will handle the following:

Listen to incoming messages from the WebSocket server.
Write messages to the WebSocket server.
Communicate with other components using MPSC (Multi Producers Single Consumer) Channel (more on this soon 🤔)

Add a new file called websocket.rs inside the services folder with the following content:

😱 OMG there’s a lot going on here, let’s bite-size this whole thing:

Lines 6–8: Creates the WebsocketService struct that holds a single property of type Sender. Sender allows us to asynchronously send messages on the channel that the receiver will later receive, in the order they are sent. Sender is cloneable, which means that every component that uses the service can clone it and use it to send messages back to the receiver, which as the name suggests — there can only be a single one.
Lines 11–46 : The new function initialize the service, similar to a constructor in other languages. The function does the following:
Connects to the WebSocket server (line 12)
Create the MPSC channel (line 16)
Spawns a new future (async task) on the current thread that will listen to the receiving end of the MPSC channel and write the received message to the WebSocket server. This is how components communicate with our service — by sending messages across the channel (lines 18–23).
Spawns another new future on the current thread to listen to the incoming messages from the WebSocket server and log them out(lines 25–43).
Finally, the function returns an instance of WebSocketService (Self) that holds the channel transmitter (in_tx).

Here’s a handy diagram visualizing how everything connects:

Awesome, with the WebSocket service behind us, let’s proceed with creating the actual chat component!

🪅 ️Components — Phase 2

For our Chat component, we will use a regular (read not functional) component because we will make good use of its different lifecycle methods.

Open up chat.rs and add the following:

👀 A few callouts:

The Msg enum holds the possible messages (read actions) our component can receive. Chat has two basic functionalities: either it handles a message received from the WebSocket server (HandleMsg), or it submits a message to the server (SubmitMessage) — for example when a user types something into the chat.
MessageData represents a chat message, containing who is it from and what the actual message is.
The MsgTypes enum holds the different types of messages the component can send or receive.
Lastly, WebSocketMessage represents what a message to the WebSocket server looks like: it has a type (message_type) and then either an array of strings (such as in the case of the chat users list) or a single string (such as the case of sending a chat message to the server).

Next, let’s add the Chat component itself:

The Chat components hold the user list (users), the input typed to the text box (chat_input), a reference to the Websocket service (wss), and finally, an array of all the messages typed to the chat (messages).

The first component lifecycle method we add is the create method. create initializes the component state and it’s ComponentLink:

👀 Quite a few things are happening here so let’s break them down:

We get the user object (of type Rc) we saved in the Context (lines 4–7) and then clone its username field (line 9), which is the actual user name the user used to log in.
We create a new WebSocket message to register the current client with the WebSocket server (lines 11–15). Next, we send it to the WebSocket server using the WebSocketService’s channel transmitter (tx) (lines 17–23). If everything goes well we log a message to the console (line 22).
We finish the method off by sending a fresh instance of Chat (lines 25–30).

Next, we add the view method, which is responsible for rendering the component:

That's one long method 😅 Most of the stuff here is just HTML and styling so we won't dive into that, but here are a few interesting pointers:

On line 4 we create a callback of type MouseEvent to use when the submit button is clicked. The callback will send a message of type SubmitMessage whenever the button is clicked. This message will then get handled by the update lifecycle method.
On line 10 we iterate over the users field by using map and collect to render a list of currently connected users (each user is rendered in its own div element).
On line 33 we iterate over the messages field, similar to how we did it previously for users.
We use the ref property on the input element (line 57) so we can later reference it (i.e. to read its value) outside of the view lifecycle method.
We set the Submit button’s onclick property (line 58) to submit the callback defined on line 4.

We are FINALLY ready to run the project for the first time and see what we achieved so far! Inside the project root folder run npm start. This will kick the process of compiling our app into WASM, optimizing it, building the HTML (with Webpack), and finally — running a dev server.

If all goes well you should be greeted with a login page where you can enter a username. Once done the main chat screen appears, let’s inspect the dev tools console:

Three lines pop out almost immediately:

We successfully sent a WebSocket message from our chat component to the WebSocket server (first debug line)
The WebSocket service sent the username to the WebSocket server
The WebSocket server returned a message with a type of users containing an array with the names of the connected users.

Now that begs the question: if our WebSocket service received a message back with all the connected users — why don't we see that on our UI? Did our chat component even get that message at all?

Eagled-eyed readers might notice that when we previously discussed our WebSocket service. We only discussed how components send messages to the service, but nowhere did we mention how the service sends back messages to the components. Let’s take another dive into the rabbit hole that is our WebSocket service…

🔌 WebSockets — Phase 2

We are currently using an MPSC channel to communicate between components and the WebSocket service, but as the name implies — we cannot use the same approach for updating the components since we would need the opposite direction, i.e. multi consumers (components) single producer (service), which doesn't exist.

https://medium.com/media/644b0fe0703d066ae60bd7165831a174/href

We use Yew Agents! Agents can be used to “route messages between components independently of where they sit in the component hierarchy” (taken from Yew’s official docs). That means we can use an agent’s dispatcher to send a message from the WebSocket service down to any listening component. Let’s create an event bus service which we will use to send these messages with.

Create a new file called event_bus.rs under services and paste the following content to it:

https://medium.com/media/5d8876d1d684913d723182682dcef02e/href

🔬So what do we have here?

Lines 10–13 : we create the EventBus struct which holds a link and a list of subscribers.
Lines 15–47 : we implement the Agent trait for our EventBus. There are a bunch of methods we need to implement (connected, disconnected etc), but the one we care about the most is handle_input, which iterates over all of the subscribers and using link, sends them the content of the message.

Now let’s go back to our WebSocket service and use the new event bus:

Open websocket.rs under the services folder. Add the required use statements and the event_bus initializing. We are creating a dispatcher because we need a one-way communication channel between the service and the components:
https://medium.com/media/8ed2ab338cd5d31bbdf04811ba1758bc/href
To send a message on the channel, all we have to do is use send:

https://medium.com/media/40bf2850055509e4dc735edd39e7aed5/href

You can see we are making use of the EventBusMsg struct we defined in event_bus.rs to send a String message down the channel (lines 9 and 15).

Lastly, we need to add the event bus to the Chat component.

Open chat.rs under the components folder, and add _producer to the Chat struct:
https://medium.com/media/35f9ec005dceb966bf32355d07e3360a/href
Head over to the create lifecycle method of Chat and add the default value for _producer to the returning Self statement:

https://medium.com/media/04010ad3c68e103fd8c2ba33ba68b048/href

If you run the app again now you’ll notice nothing really changes. The reason for that is we never implemented the update lifecycle method, which is in charge of handling the different messages our component get (like HandleMsg or SubmitMessage). The method returns a boolean indicating whether the component should be re-rendered or not.

Under the create method, add the following implementation of update:

https://medium.com/media/a758d26b80832373969f8b0a75d5a68d/href

🔬So what do we have here?

The method matches msg to one of the two possible messages the component can get: HandleMsg or SubmitMessage. If the message is of type HandleMsg we perform another match and check whether it is a message of type Users or of type Message. Messages of type Users get sent every time the connected user list changes (a user connect/disconnect) and upon receiving this message — we populate the users array with the name and avatar of each user that is connected. Messages of type Message get sent when a user posts a message to the chat, and pretty similar to how we handled users, we append the new message to the messages array. In both of these cases, we return true since we need the component to rerender itself with the new data.

In the case the message is of type SubmitMessage we grab the message text from the input HTML element (remember that ref property we set earlier?) and using the WebSocket service channel transmitter (tx) we send the message to the WebSocket server. In the case of SubmitMessage we return false as we don't need the component to rerender itself.

And with this change, our app is now ready for prime time!

This post was definitely on the longer side, but I hope you got to experience how it is like working with Yew when building a web app. You can find the full source code at https://github.com/jtordgeman/YewChat. Feel free to hit me up on Twitter or by leaving a comment here — I promise I'll answer nicely ;p

Last but not least, huge thanks to Sara Lumelsky for proofreading and fixing my stupid spelling mistakes! :)

Let’s Build a Salesforce Commerce Cloud Products Search Component with OCAPI, Rust, and Yew — Part…

Johnny Tordgeman — Tue, 23 Nov 2021 17:54:59 +0000

Cover Photo by Michael Fenton on Unsplash

Salesforce Commerce Cloud + Rust = ❤️

Anyone familiar with Salesforce Commerce Cloud (SFCC for short) knows that you usually develop a cartridge to extend and add new features to your SFCC instance. Since a cartridge is basically a module written in JavaScript, how do we go about adding Rust to the cartridge world? Are we really going to write a Rust-based cartridge in this post?

Well, not really. While we can’t write Rust cartridges yet (😭), we can write a Rust-based WebAssembly component and load it in our SFCC instance (💡). You might be thinking: johnny, by definition a WebAssembly cannot directly access the DOM (https://developer.mozilla.org/en-US/docs/WebAssembly/Concept). How are we going to render the component to the page? Is there some kind of magic involved?? well…

Yew 101

Meet Yew. By its own definition, Yew is "a modern Rust framework for creating multi-threaded front-end web apps using WebAssembly." (https://yew.rs/#what-is-yew). It features component-based development (React devs should feel right at home here), access to services (fetch for example) and great JavaScript interoperability.

If I had to summarize Yew in one sentence I would say it's a framework that allows you to write back-to-front web applications in a component mindset, completely in Rust. Now that's some cool sh*t right there! 🤘

Let’s begin our journey with a simple Yew application, just to, you know, get a feeling of how things work. Later on, we will add OCAPI into the mix and use it to search for a product from our SFCC instance.

Creating the Project

Our simple application will bind an HTML input element to a read-only textarea element, copying everything the user writes in the input element to the textarea. I know it's not much (and in fact sounds pretty damn stupid) but think of it as the base for our product searcher, which will also use an input element to get the search term and display the results in a nicely styled div below it.

🏗 Before we begin, make sure you have the following installed:

* Rust 1.50+- https://www.rust-lang.org/learn/get-started

* WASM-Pack — https://github.com/rustwasm/wasm-pack

Create a new Rust library project by running cargo new --lib yew-product-searcher, cd to the new yew-product-searcher folder, and open it in your favorite code editor.
Our project is going to be a dynamic library, so let’s update our Cargo.toml to reflect that. Add the following snippet before the dependencies section:
Next, let’s add the dependencies for our project. Update your Cargo.toml dependencies section as follows:

📦 So why do we need all of these dependencies?

wasm-bindgen — A Rust library we will use to communicate between Rust and JavaScript. Also contains a set of macros that help when working with WASM and Rust.
yew/yewtil — The Yew library and its utility library.
serde/serde_json — A serialization library we will use to serialize and deserialize JSON payloads.
web-sys — Yew uses web-sys under the hood to communicate between Rust and Web APIs. By default, none of the features are added. We specify the features we are going to use with it in our dependencies declaration.

⛳ Round 1: The Entry Point

Like most things in life, our Yew application needs an entry point. The grand entrance. In our case, that point is the src/lib.rs file.

We begin by adding the required use statements:

🙋‍♂ Before you ask — yes I know we don't have an app.rs yet - we will add it shortly.️
Now, let’s add the actual entry point function:

🔬 So what do we have here?

Line 1 — Using the wasm_bindgen(start) attribute, we annotate the app’s entry point. It is required by wasm-pack.
Line 2 — Our main function. Returns an Option of either an empty tuple or a JsValue object.
Lines 3–6 — Using web_sys we stroll through the DOM tree, from the top window object (line 3) all the way to the body (line 5). Once we have the body element we can get the collection of its children elements by calling the much appropriately named method - children(line 6).
Lines 7–9 — Using the named_item method, we can get a reference to the div element which will host our app.
Line 10 — Creates a new yew app and mounts it to an element (in our case, the div with the id of yew-app ).
Line 11 — Returns a unit type.

⛳ Round 2: The App Component

We now have an entry point pointing to nothing (remember app::App from line 10 above?). Now is the perfect time to add the App component - a.k.a our application’s main component - to the mix.

Create a new app.rs file under the src folder with the following content:

Our App is basically a struct containing a link property with ComponentLink as its type and a text property of type String.

😐 Johnny — I know what a String type is, but wtf is aComponentLink?!

Glad you asked! A ComponentLink is basically a link (a.k.a a reference) to the component that enables us, at a later stage, to send messages (read: actions) to the component through callbacks. Imagine ComponentLink as your way of interacting with the components at runtime.
To send a message (a.k.a action) to the component we need to define an enum containing all the possible messages the calling party (the component host, a service, an agent, etc.) can send. Add the Msg enum to app.rs :

We will call our component on every change in the input element (i.e. the oninput event), passing the message of. DuplicateText with an InputData object. The InputData object contains, among other properties, the new value of the element, which is something we will shortly use.
Now it's time to implement Yew’s Component trait for our lovely app. As you’ll soon see — the Component trait basically enables us to control the component’s life cycle and how it will act at each event. We begin with the super basic stuff and set the Message and Properties types:

🔬 So what do we have here?
- Line 1 — We tell Rust we want to implement the Component trait for our App.
- Line 2 — We set the Message type to the Msg enum we created previously to let the component know of the different actions it should expect.
- Line 3 — If you have some React background, you are probably pretty familiar with the concept of properties. In a nutshell, properties are immutable inputs we can add on an element to pass to the component. To visualize the properties concept, check out the following example, in which title is a property we pass to the component:
```
<Page title="Hi properties!"/>
```
Next, we implement the very first life-cycle method: create. The create method fires when the component is created and it is here that we initialize the component properties as well as the ComponentLink:

When the component is created, it receives the properties (if any) from its host (a.k.a parent) as well as establish the link between itself and the host. The create method accepts these two arguments (line 4) that we then use to initialize our component (lines 5–8). Notice that we used _ as the properties variable name since in our simple component we don't them.
As mentioned earlier, we communicate with the component through messages. The life-cycle method that handles these messages is the update method. This method contains the logic to handle the different messages, and ultimately decide if the component should re-render itself:

When the update method is called it will perform a match on the msg object (line 4). We only have one possible message on our Msg enum for this component, so it will match DuplicateText (line 5). If the associated element's updated value matches DuplicateText we update the value of the text property with the value of the message. We end the method by returning true (line 7), indicating that the component needs to re-render.

📣 If we had more than one message, we had to handle cases where nothing matches, which usually results in returning false as the component doesn't need to re-render.
Next up: the change method. Just like the update method, change is also able to re-render the component, but unlike update — change deals with a change in the component properties. change should return true (meaning re-render) only when the properties changed from the last call. In any other case (the component doesn't have properties at all, the properties didn't change etc.) we should return false:
And finally, the crown jewel — view. The view method is where we define how to render our component to the DOM. To prevent things from getting messy here, Yew provides us with a super useful macro — html! — for declaring HTML (and SVG) elements, their attributes, and their events. You can think of the html! macro as the equivalent of JSX in React:

🔬 So what do we have here?

Line 4 — Using the html! macro.
Line 6 / Line 9 — When we want to write some text directly to the DOM we wrap it in curly brackets.
Line 10— We set the oninput callback using the link property, passing it the InputData of the element.
Line 14 — We bind the textarea’s value attribute to the component text property, which will get updated whenever the data changes on the input element.

🗣 Keep the class names used here in the back of your mind. We will meet up with them again soon.

⛳ Round 3: Packing Time

Our component is ready for prime time, but before we are able to load it, we have to pack it. We are going to use wasm-pack as our weapon of choice, mainly for its simplicity, but it's important to know that there are other ways to pack a WebAssembly as well.

Open your shell and cd to the project folder. Inside of it run the following:

wasm-pack build --target web

A wild folder appears!

If you check the project folder now you will notice a new folder — pkg. This new folder contains the compiled WebAssembly (yay 🎉) along with its supporting JavaScript wrappers.

Before we proceed to add OCAPI and SFCC to the mix, let’s quickly test that it's working as expected.

⛳ Final Round: Testing

To quickly test our WASM module, let’s write a short HTML file that will load the module and initiate it using the run_app command we defined way back in the lib.rs file.

In the root folder of the project create an index.html file with the following content:

😱 What do we have here??

Line 6 — Remember all those class names we added to the view method? So they are all Tailwind CSS classes and in order to use them, we need to include the Tailwind CSS file

👀 It is not recommended to include the unoptimized CSS file from a CDN like we are doing here, but to keep things simple and avoid installing the Tailwind tooling, etc, we will just use that file.

Line 10 — The div in which our app will render itself.
Line 11 — We declare a JavaScript module script section.
Line 12 — We import the init and run_app methods from yew_product_searcher.js , the JavaScript wasm wrapper wasm-pack created for us. It is with these functions that we are going to load our module in the browser.
Lines 13–16 — As we cannot have root-level async/await, we create an async function called run (line 13), await on the init function (line 14) and finally run the run_app function which starts our Yew app (line 15).
Line 17 — Calling the async run method.

Use your local webserver of choice (I personally use Caddy) and embrace the beauty that is Yewplicator:

🥂 Summary

So far we:

Built a Yew project from scratch.
Added a root component with a number of HTML elements that interact with one another without any JavaScript.
Packed the WASM module with wasm-pack and
Created a test HTML file that uses a JavaScript module script to load and initialize the WASM.

In our next post, we will take our simple Yewplicator and throw SFCC into the mix by transforming it to an OCAPI based product searcher component.

Building a Simple Bot Protection With NGINX JavaScript Module (NJS) and TypeScript

Johnny Tordgeman — Mon, 24 May 2021 19:57:34 +0000

Cover Photo by Phillip Glickman on Unsplash

I love Lua. I also love NGINX. The three of us get along just great. Like every relationship, we’ve had our highs and lows (yes, I’m looking at you Lua patterns), but overall life was perfect. Then, NGINX JavaScript Module (NJS for short) came along.

I ❤️ JS/TS

NGINX JavaScript module was first introduced in 2015 but recently received a big boost in functionality with the 0.5.x update. Since I'm a sucker for anything JS, I decided to test it out by building a simple (read naive and not production ready ) bot protection module 🤖.

Configuring NGINX

Before diving into bot fight, we have to set up NGINX to support the JavaScript module. The instructions below are for my setup (Ubuntu 20.4/Nginx 1.18), so YMMV, but the general idea should be the same for most setups.

Start by adding the NGINX PPA key by running:

curl -s https://nginx.org/keys/nginx_signing.key | sudo apt-key add -
Setup the repository key by running:

sudo sh -c 'echo "deb http://nginx.org/packages/ubuntu/ focal nginx" >> /etc/apt/sources.list.d/nginx.list'

Update the repository list by runningsudo apt update.
Install NJS by running sudo apt install nginx-module-njs.

If all went well, at this point, you should get this lovely message on your terminal:

Big success 🥂

Enable NJS by adding the following to the top of your main nginx.conf file:

load_module modules/ngx_http_js_module.so;

Restart NGINX to load NJS into the running instance:

sudo nginx -s reload

Now your NGINX is ready for some JS love, so let’s move on and create our first line of defense — IP filtering!

damn right you are!

Opening Act — Creating the Project

Our bot protection project is going to be written in TypeScript. For that, we need to create a project that will transpile TypeScript to ES5 JavaScript, which NJS can understand. As you may have guessed, NodeJS is a must here, so make sure you are all set up before continuing.

Create the new project folder and initialize it:

mkdir njs-bot-protection && cd njs-bot-protection
npm init -y

Install the required packages:

npm i -D @rollup/plugin-typescript @types/node njs-types rollup typescript

Add the build script to the package.json ’s scripts section:

{
    ...
    "scripts": {
        "build": "rollup -c"
    },
    ...
}

To compile the project, you’ll need to tell the TypeScript compiler how to do that with the tsconfig.json file. Create a new tsconfig.json file in the root of the project, and add the following content to it:

Lastly, let’s add the rollup config, which will wrap everything up and produce the endgame js file that NJS will read. Create a new rollup.config.js file in the root of the project, and add the following content to it:

And with that, our boilerplate is all loaded and ready to go. That means it’s time to kick some bots!

Round 1 — IP Filtering

Our first line of bot defense is IP blocking; we compare the IP of an incoming request with a list of known IPs with bad reputations, and if we find a match, we redirect the request to a “block” page.

We’ll begin with creating the JavaScript module:

In the project root folder, create a new folder called src, and then inside of it create a new bot.ts file.
Add the following code snippet to bot.ts :

💡 So what do we have here?

Line 1 : Imports the built-in module for the file system (i.e., fs). This module deals with the file system, allowing us to read and write files, among other activities.
Line 2 : Calls the loadFile function, passing it the name of the file we wish to load.
Lines 4–12 : The implementation of loadFile. First, we initialize the data variable to an empty string array (line 5), then we try to read and parse a text file containing a list of bad IP addresses into the data object (line 7), and finally we return the data object (line 11).
Lines 14–21 : The implementation of verifyIP — the heart of our module (for now). This is the function we will expose to NGINX to verify the IP. We first check if the array of bad reputation IPs contains the current request client IP (line 15). If yes, redirect the request to the block page and end processing (lines 16 and 17). If not , redirect internally to the pages location (line 20).
Line 23 : Exports (read exposes) verifyIPexternally.

Build the module by running npm run build in your terminal. If all goes well, you should find the compiled bot.js file in the dist folder 🎉

With the file in hand, let’s configure NGINX to be able to use it:

In your NGINX folder ( /etc/nginx in my case) create a folder named njs and copy bot.js from the previous section inside it.
Create a new folder called njs under /var/lib , create a file called ips.txt inside it, and populate it with a list of bad reputation IPs (one IP per line). You can either add your own list of IPs or use something like https://github.com/stamparm/ipsum.
In your nginx.conf , under the http section, add the following:

js_path "/etc/nginx/njs/";
js_import bot.js;

💡 So what do we have here?

js_path — Sets the path for the NJS modules folder.
js_import — Imports a module from the NJS modules folder. If not specified, the imported module namespace will be determined by the file name (in our case, bot)

Under the server section (mine is on /etc/nginx/conf.d/default.conf ) modify the / location as follows:

location / {
    js_content bot.verifyIP;
}

By calling verifyIP using the js_content directive we set it as the content handler, which means verifyIP can control the content we send back to the caller (in our case, either show a block page or pass the request to the origin)

Still under the server section, add the block.html location and the pages named location:

location [@pages](http://twitter.com/pages) {
    root /usr/share/nginx/html;
    proxy_pass [http://localhost:8080](http://localhost:8080);
}

location /block.html {
    root /usr/share/nginx/html;
}

(The namedpages location will be used by our NJS module to internally redirect the request if it shouldn't be blocked. You likely have your own logic for this redirection so change this to fit your needs)

At the bottom of the file, add the server block for port 8080:

server {
        listen 8080;
        location / {
        root /usr/share/nginx/html;
        index index.html index.htm;
    }
}

Under the /usr/share/nginx/html folder, add the block.html file as follows:

And with that, our IP protection is ready! Add your own IP to the ips.txt file and restart NGINX (sudo nginx -s reload). Browse to your instance and you should be greeted with the following:

Take that Mr. Evil Bot! 🤖 ⛔

Round 2 — JavaScript Detection

Our second protection layer is JavaScript detection. We use this detection to determine if the visitor coming to our site is running JavaScript (which every normal browser should do) or not (a warning sign that this visitor might not be a legitimate user). We begin with injecting a JavaScript snippet to the pages that will bake a cookie on the root path:

Add the following code snippets to bot.ts :

💡 So what do we have here?

Line 1 : Imports the built-in Crypto module. This module deals with cryptography, and we will soon use it for creating an HMAC.
Lines 5–18 : The implementation of getCookiePayload. The function sets a date object to one hour ahead of the current time (lines 6–8), then uses the date object to HMAC (using the crypto module) the signature we passed to the function (the value object) with the date object (lines 10–14). Lastly, the function returns the cookie information in a string format (name, value, expiration, etc.). You may notice the cookie value contains not only the hashed signature but also the date object we used to HMAC the signature with. You’ll see why we do that soon.
Lines 20–30 : The implementation of addSnippet. The function buffers the request data, and once it finishes (line 23) it:
Creates a signature based on the client IP and the User-Agent header (line 24).
Replaces the closing head tag with a script section that inserts a cookie (from the getCookiePayload function) on the browser side using JavaScript’s document.cookie property. (lines 25–28).
Sends the modified response back to the client (line 29).

Export the new addSnippet function by updating the export statement at the bottom of the file:

export default { verifyIP, addSnippet };

Under the @pages location block, modify the / location as follows:

location [@pages](http://twitter.com/pages) {
    js_body_filter bot.addSnippet;
    proxy_pass [http://localhost:8080](http://localhost:8080);
}

Unlike verifyIP, we don't want addSnippet to manage the content of the response, we want it to inject content (a script tag in our case) to whatever response comes back from the origin. This is where js_body_filter comes into play. Using the js_body_filter directive we tell NJS that the function we provide will modify the original response from the origin and return it once finished.

Restart NGINX and browse to a page on your instance. You should see our new script added just before the closing head tag:

If the client is running JavaScript, a new cookie called njs will be baked. Next, let’s create the validation for this cookie/lack of cookie:

Add the verifyCookie function (and its supportive functions/variables), to bot.ts :

💡 So what do we have here?

Lines 5–11 : The implementation of the updateFile function, which uses the fs module to save an array of strings to a file.
Lines 13–52 : The motherload implementation. When validating the njs cookie, we have a flow of verification and consequences we have to follow:

a. We begin with extracting the njs cookie from the request’s Cookie header (lines 14–20).

b. If we don’t have a cookie (or we do and it’s malformed), we compare the client IP against our list of client IPs that have reached us without a cookie. If we find a match from within the last hour, we fail the request (returning false, lines 26–27). If we don’t, we delete the IP (if it's on the list but past one hour) and pass the request (lines 29–34).

c. If we do have a cookie, we split it into a timestamp and a payload and use the timestamp to create our own HMAC hash based on the request’s User-Agent header and client IP. If our own HMAC matches the HMAC of the njs cookie, we pass the request. Otherwise, we fail it (lines 38–45).

d. If anything goes wrong during the validation, we fail open (meaning pass) the request (lines 48–51).

Add the new verify function, which calls the new verifyCookie function, and act according to its result:

🔥 At the point you might be thinking to yourself at this point that this verify function looks eerily similar to the verifyIP function from the earlier — you are absolutely right, and I will touch on that in a minute!

To test our new cookie validation functionality, open up your configuration file (mine is at /etc/nginx/conf.d/default.conf ) and change the js_content directive from verifyIP to verify:

location / {
    js_content bot.verify;
}

Restart NGINX and try to visit the site twice without the njs cookie — ✋ 🎤- you are blocked!

Not today Mr. Evil Bot! 🤖 ⛔

Final Round — Bringing It All Together

So now we have the cookie verification, but we took off our IP verification because we can only have one js_content directive, how do we go around fixing that?

You may remember that a few minutes ago we created the verify function (which eagle-eyed readers may have noticed is VERY similar to the verifyIP function we used before). If we update our verifyIP function so that it returns a boolean response as verification, and add that verification to verify, we get the best of both worlds with one big function that verifies requests for both IPs and cookies!

Refactor the verifyIP function as follows:

Update the verify function to call verifyIP as follows:

Update the export statement, as we no longer need to expose verifyIP:

export default { addSnippet, verify };

Restart NGINX and enjoy your home-made bot protection using NJS and TypeScript 🎉

Ewwwww, so cute ❤️

🍾 The module source code is available on GitHub!

Late Night Confessions — Building a Website Using Rust, Rocket, Diesel, and Askama — Part 3

Johnny Tordgeman — Thu, 29 Apr 2021 18:52:30 +0000

Cover Photo by Agnis Leznins on Unsplash

With a working API layer under our belt, it's time to put in the final piece of the puzzle: the presentation layer. This layer consists of static resources (HTML, CSS, JavaScript) and API calls to display and add confessions.

✏️ As this tutorial is not frontend focused, I wanted to keep things simple and use vanilla JavaScript, HTML and CSS. I will also not go into too much detail with why or how these are implemented, as we are not really here for frontend stuff, are we? 😎

Working With Templates

When we serve index.html back to the user, we want it to have a confession already so the user won’t load the page and then wait for the first API call to return the confession. That basically means that our application needs to parse the index.html static file, find the right place to inject a confession, and then inject the content in the correct place after fetching a random confession from the database. Now that might work if you have one item to inject, but what if you have two or ten?

That's where templates come into play. Templates allow us to have a static HTML that contains placeholders (aka variables) that the templating engine will replace with the content of our choice. In addition, templates allow us to use tags to control the template logic (if statements, for example). However, we will not be dealing with those in this tutorial.

For Late Night Confessions, I chose Askama as the templating engine because of its easy integration and simple syntax. Let’s create a template for our index.html file (trust me, it's easier than you think):

In the project root folder, create a new folder called templates.
Inside the templates folder, create a new file called index.html with the following content:

The vast majority of this file is plain old HTML syntax, take a glance at lines 32 and 47— These are Askama variables:

confession (line 32) — This variable will be replaced by an actual confession from the database.
total_confessions (line 47) — This variable will be replaced by the total number of confessions in our database.

And that's basically that for making our HTML an Askama template. Let’s quickly add the styles.css and confessions.js files to the site/static folder and get back to Rust to integrate Askama:

A few points of interest in our JavaScript:

The refreshCard function makes a call to our GET API to get a new random confession and update the UI.
The event listener for click validates the text area (so it is not empty, somewhat naive, I know, but 🤷‍♂️), and then if all is well, makes a call to our POST API with the new confession. If everything checks out (line 43), we alert the user everything is fine and update the footer note with the updated saved confessions number.
We listen to the animationend event to update the displayed confession once the timer runs off (lines 53–55).

Working With Askama

Back in our Rust project, let’s add Askama to the Cargo.toml file:

As mentioned earlier, Askama works with templates, so the first thing we need to do is tell Askama where our template is and which variables it's going to have:

🔬 So what do we have here?

Line 3 : We derive the Template trait, which is the main trait Askama uses. It includes template related methods such as render.
Line 4 : We use the template attribute to specify the path to our template. The path is relative to the templates folder in the project root folder. Other than path you can also specify a source (directly set the template, without a template file) or enable debug using the print sub-attribute. Read more about the different uses of the template attribute on the official docs.
Lines 5–8 : We define the fields the struct will hold, which are identical to the variables we defined in the template itself.

We now have two functions that need to get a random confession from the database: get_confession (the handler for the GET API) and root (the handler that renders index.html ). To avoid code duplication, let’s extract get_confession’s confession fetching logic to a new function, get_random_confession:

Notice that our new function takes a conn variable of type &PgConnection as argument and return a Result of either a Confession (if successful) or diesel’s error type (if there was a database-related error).

With the get_random_confession function in place, let’s update the get_confession handler to use the new function as its closure:

Lastly, let’s update the root handler to render the template using Askama and set it as the response:

🔬 So what do we have here?

Line 2 : We changed the return type to Html for success and our CustomError in case of an error.
Line 3 : We get a random confession from the database using the new get_random_confession function, just like we are doing for the API call to /api/confession.
Lines 4–6 : We query Postgres for the number of confessions we have saved on it.
Lines 8–11 : We initiate the HomepageTemplate struct (a.k.a our template definition) with the data from Postgres.
Lines 13–14 : This is where the magic happens. Askama turns our template into an HTML string (line 13), which we return as the handler response (line 14).

And that's basically it! Go ahead and run cargo run and browse to localhost:8000. You should get something similar to the following:

A deep and heartwarming confession indeed…

Summary

Our site creation journey has come to its end, but take a look back at everything you achieved! You now have a web app that is able to handle static files and API requests, uses a Postgres instance for persisting data and takes advantage of templates for quicker and easier HTML manipulation.

I would love to read your comments on this mini-post series over on Twitter, and please join me on my next Rust post series: “Let’s Build a Fitness Tracking Webapp With Rust and Yew”.

Credit Where Credit’s Due

Thank you, Carla Notarobot, for the starry night CSS animation background!
Thank you, Traf, for the progress bar CSS animation!

Late Night Confessions — Building a Website Using Rust, Rocket, Diesel, and Askama — Part 2

Johnny Tordgeman — Thu, 22 Apr 2021 17:56:41 +0000

Cover Photo by David Watkis on Unsplash

We previously left off with our server able to handle static content, but that is about all. In order to store and retrieve confessions, our app needs to interact with a database. That’s where Diesel comes to our aid!

⚠ In order for Diesel to interact with a database, a database instance needs to already exist. Make sure you have access to a Postgres instance (local or cloud based, both work) before moving forward.

Housekeeping

We begin with installing diesel_cli - a tool that helps us manage the database. As we only use Diesel for Postgres, we use the features flag to specify that:

cargo install diesel_cli --no-default-features --features postgres

In the root folder of the project, create a .env file. At the top of the file add the DATABASE_URL property that Diesel will use to get the connection details of your Postgres instance.

In the project root folder run diesel setup. Diesel will create a new database (confessions), as well as a set of empty migrations.

A new database and migrations

Using Migrations

With the database setup, it’s time to create the confessions table. Diesel uses a concept called migrations to track changes done to a database schema. You can think of migrations as a list of actions that you either apply to the database ( up.sql ) or revert ( down.sql ).

Generate a new migration set for the confessions table by running the following command at the root of the project:

diesel migration generate confessions_table

This creates a new folder inside of the migrations folder that holds the new migration set (up/down.sql) for theconfessions table:

To create the new table, cd to the new migration folder and add the following to the up.sql file:

In down.sql we specify how to revert the migration (i.e., dropping the confessions table):

Apply the new migration by running the following command:

diesel migration run

💡To revert the last migration run diesel migration redo .

cd to the src folder to find a new file called schema.rs. This file contains the table definition created by Diesel that enables us to work with the database in a typesafe way.

With the housekeeping behind us, we proceed to establish a connection between our Rocket instance and Diesel.

Getting Connected

The first rule of working with a database is connecting to a database. A common method of connection between a database and an application is Connection Pool — a data structure that maintains active database connections (pool of connections) which the application can use at any point of time it needs.

Rocket, with its rocket_contrib crate (a crate that adds functionality commonly used by Rocket applications), allows us to easily set up a connection pool to our database using an ORM of our choice. In our case, that’s going to be Diesel.

We begin with adding three dependencies to our cargo.toml file: diesel, serde, and rocket_contrib :

🔭 As we only need certain features from the above crates, we specify these features using the features property. In our case we only need Postgres support so we specify that in the features list for diesel and rocket_contrib.

Add the following import statements to your main.rs file:

Next, we configure the connection settings for our database. Create a new file named Rocket.toml in the root folder of the project with the following content:

At this point you are probably thinking to yourself: “Johnny WTF? this is exactly the same connection string we configured earlier in the .env file. Can’t we just use that environmental variable and be done with it?”

it really is!

Of course you can! But I’ll touch on how to do that a little bit later. For now, let’s roll with Rocket.toml.

Open your main.rs file and add a new unit-like struct called DBPool:

The database attribute is used to bind a previously configured database to a poolable type in our application.The database attribute accepts the name of the database to bind as a single string parameter. This must match a database key configured in Rocket.toml. The macro will generate all the code needed on the decorated type to enable us to retrieve a connection from the database pool later on, or fail with an error.

Lastly we need to attach the database to our Rocket instance. We do that using the attach method of our Rocket instance. The attach method takes a fairing (think of that like a middleware) and attaches it to the request flow.

Append the attach method to the Rocket instance as follows:

🍲 Before we proceed, I’d like to take a quick (completely optional) detour and talk about how to use the connection string from the .env file instead of duplicating it with Rocket.toml. If you don’t feel like messing around with creating a database procedurally, feel free to skip over to the next section — Working With Models.

If created earlier, delete the Rocket.toml file from the root folder of the project.
dotenv is a crate that makes it super easy to work with environmental variables from a .env file. Add the dotenv crate as a dependency in your Cargo.toml file:

In main.rs , refactor your rocket function as follows:

🔬So what do we have here?

Line 3 : We call the dotenv function (from the dotenv crate) to load variables found in the root folder .env file into Rust’s environment variables.
Line 5 : Using Rust’s standard library we load up the value of the DATABASE_URL variable.
Lines 6–9 : We define a Map that holds two keys — url for the connection string and pool_size for the size of the connection pool.
Line 10 : We create a Rocket config using Figment and add our database name (confessions_db) as a key to the databases collection. This closely resembles the Rocket.toml file and for a good reason — its basically the same thing just done programmatically instead of a toml formatted file.
Line 12: Instead of initializing the Rocket instance using ignite, we use the custom method, passing on the Figment configuration object created on line 10.

Before moving on to creating the Database models, let’s build the project to make sure everything compiles as expected.

🏗️ When I tried to compile the project on my Macbook, I got a compilation error related to Diesel stating that I’m missing libpq. If you happen to get the same, follow these steps:

Install libpq using homebrew: brew install libpq
In the project root folder create a new folder named .cargo and inside of it create a new file called config with the following content:

Run cargo build and enjoy.

Working With Models

To represent our database table in a type-safe way, we need to create a model (struct) that represents it. Think of a model as the link connecting your database table with your Rust code.

In your src folder create a new file called models.rs. This file will be the home for the models we use in our project.
We begin with the Confession model, which is used when querying the database. Add the Confession struct to models.rs :

Our struct looks identical to the Postgres table schema we created earlier (you can peek into schema.rs for a reminder on how it looks). But what is this Queryable attribute on top of it? It is a Diesel attribute that basically marks this struct as a READABLE result from the database. Under the hood, it will generate the code needed to load a result from a SQL query.

📢 The order of the fields in the model matters! Make sure to define them in the same order as the table definition in schema.rs .

To save a confession to the database, we don’t need to specify the id property since it's auto-incremented on the database side. For this reason, we will create an additional model in our models.rs file called NewConfession:

We annotate this new model with the Insertable attribute so it can be used to INSERT data to our database. In addition, we also add the table_name attribute to specify which table this model is allowed to insert data to.

Lastly, we add the schema and models modules to main.rs :

Handling API Requests

It’s time to add a new route handler to our Rocket instance to handle POST requests containing new confessions.

In main.rs , add a new struct named ConfessionJSON, which represents the JSON data sent to us from the browser:

Add a new struct named NewConfessionResponse which represents the JSON response we send back to the browser upon adding a new confession:

Add a new POST route that will handle requests to /confession :

🔬 So what do we have here?

Line 6 : We define the route using three attributes:
post — The HTTP verb this route is bound to.
format — The required content type of the request. In our case we are going to use application/json. Any POST request to /confession which does not have a content type of application/json will NOT be routed to the post_confession handler.
data — The name of the variable the body will be bound to. In this example, I named the variableconfession(surrounded by < and >, which is a must), but you can name it anything you like, as long as you name it the same in the handler (line 2).
Lines 7–10 : Here we define the handler for the /confession route. We pass confession as an argument (same variable from step 1’s data attribute) and set its type as ConfessionJSON wrapped by serde’s Json attribute. Serde will deserialize the request body’s JSON payload as a Rust struct (ConfessionJSON) giving us a typesafe way to access it. In addition to confession we get access to the database connection pool we created earlier, thanks to the attachment of it to our Rocket instance.
Line 10 : The handler will return a Result containing either a JSON with an HTTP status of 201 (created) or an error (using a custom error that we will write next).

Add the implementation for the post_confession handler:

Now this handler might seem scary (👻) in its current form, so let’s break it into smaller chunks:

Lines 6–14 : We create a new confession by calling the run method on our connection pool using Diesel’s insert_into method. The method takes the table name as the first argument and then using the values method we pass a struct (of type NewConfession as that is our Insertable struct) with the data that needs to be saved. Finally, we call await on the run method as it is an asynchronous function.
Lines 16–18 : We create a new NewConfessionResponse with the result of the insert item query (the new_confession variable).
Line 20 : We return a Result with the newly created confession, wrapped with Created to return a status code of 201.

If post_confession fails for whatever reason, it returns a CustomError error, that we need to create next. Inside the src folder, create a new file called error.rs and add the following content:

I won’t go into much detail on what’s happening here, but the main takeaways from this file are:

We create an enum to hold different error types and decorate it using Failure’s Fail attribute (lines 7–11).
We implement the From trait so we can support the diesel error type (lines 13–17).
Rocket requires the response of a handler (be it an error or a valid response) to implement the Responder trait. We implement this trait on our CustomError to display an error (of diesel::result::Error) to the caller (lines 19–28).

Our post_confession handler is now completed 🎉. Let’s mount it to our Rocket instance’s routes with a new base of /api :

We are finally ready to test our new API! Run the app with cargo run and on a different terminal run the following curl:

If all went well you should get back a JSON response with the confession and its new ID.

big success 🏆

That was quite a ride, wasn’t it? You’d be happy to know (or not) that adding the GET route — for getting a random confession out of Postgres — is a much simpler task:

Add the new get_confession handler to main.rs :

Nothing really exciting happening here. We get a connection from the pool (line 5), use the confessions table (line 7) to query for a single random confession (line 1 defines the SQL’s RANDOM function, lines 8–10 build the query) and eventually returning a JSON of the confession (line 14).

Mount the new get_confession handler to our Rocket instance’s routes:

Launch 🚀 with cargo run and in another terminal window run this lovely curl:

Now, what is it that we got back? A random confession from Postgres that’s what!

✋🎤

And with that our API is completed. We have a Rocket web server running with two API endpoints (post and get confessions) and an additional route to handle static content. Let’s move on to the final task for our website — adding the presentation layer.

Late Night Confessions — Building a Website Using Rust, Rocket, Diesel, and Askama — Part 1

Johnny Tordgeman — Thu, 15 Apr 2021 16:07:37 +0000

Cover Photo by Hanbyul Jeong on Unsplash

It’s 1:00 AM (🎵 soft jazz music). You’re lying in bed trying to get some sleep. The city lights shine into your bedroom, softly illuminating the wall in front of you. You can hear the rain outside and the sound of people going about their late-night business. You can’t fall asleep. Something is bothering you, and you feel like you have to get it off your chest. You have to tell someone, but you don’t want to do it on social media. You want to spill your guts but remain completely anonymous. If only there was a place you could do this…

This is the setting we are going to use for the project of this post — Late Night Confessions : A 🦀 Rust-based website built using Rocket (web framework), Diesel (ORM), and Askama (template rendering engine).

Overview

The site functionality is pretty simple:

Show the user a random confession.
Allow the user to add a new confession.

Since we want to keep our confessions anonymous, we will not create a signup or login form.

Our site has three types of requests to handle:

Static content requests — to get static files (i.e., index.html, styles.css)
API request to get a confession.
API request to post a confession.

All confessions will be saved in a Postgres database instance. We will use Diesel to manage the database activities.

Our finished project will look something like this:

The full source code can be found on Github.

Launching a 🚀

We begin by creating a new crate for our project called late-night-rocket:

cargo new late-night-rocket

Next, let’s add Rocket and Failure as dependencies to our Cargo.toml:

🔭 Why do we need to specifically specify the_ 0.5.0-dev version? Since we want our app to run on a stable Rust release, 0.5.0-dev is (as of writing this post) the only version that supports stable Rust.

Rocket makes heavy use of macros. This means our first order of business is to import all of them to our crate. We do this by adding the macro_use attribute to the top of our main.rs file, pointing to the Rocket crate:

Next, let’s add some use statements to import modules we would need:

Finally, it’s time to create some routes!

1. We begin with the root route responsible for rendering the content of index.html when the / path is called:

🔬 So what do we have here?

Line 1: Say hello to our first Rocket attribute — route. Using this attribute, we define an HTTP method for our route (i.e get, post), a path (either static or dynamic), and an optional data parameter (not needed here since we are defining a GET route).
Lines 3–5: We use the NamedFile struct to try and open the index.html file (we will create it next), and if we are successful, return the content of the file (the file extension determines the content type automatically). If we fail, we map the errors to a NotFound error type and return it. Notice the use of await on line 4. We are asynchronously loading the file so that other operations won't have to wait for the file to load to continue processing.

2. Next, static_files, as the name suggests, will handle requests for static files, such as images:

The route looks very similar to the root route, with one noticeable difference; unlike the static root route, this route is dynamic, meaning it matches anything after the / as the path variable. We then use this path variable and append it to the site folder (line 3) before trying to get the file (using NamedFile, just like in the previous route).

3. Next, we mount the new routes to a Rocket instance using the mount method:

🔬So what do we have here?

We build a new Rocket instance, then we mount the routes using the mount method, which takes two parameters as input:

A base path to use on all the associated routes (/ in our case)
A list of routes via the routes! macro.

4. Finally, we need to launch our rocket for it to serve requests. There are two mechanisms to launch a Rocket instance, but we will focus on the preferred approach for this tutorial's purposes: the launch attribute. Add it above the rocket function defined in the previous step:

5. At the root of our project, create a new folder named site and inside of it, create a folder named static.

6. Inside the static folder, create an index.html file with the following content:

Create an empty styles.css file in the same folder. We will populate it later on.

Go ahead and cargo run the project and then browse to localhost:8000. Congratulations, you have a working Rocket server!

Look ma! I can serve a page! 👶

With our server up and running, we are ready to take on the next task and pour some Diesel (pun intended) into our Rocket to gain database support!

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 11: Playing With Templates

Johnny Tordgeman — Mon, 18 May 2020 19:01:36 +0000

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 11: Playing With Templates

Photo by Mae Mu on Unsplash

Time flies when you’re having fun. Today is our last stop on the SFRA development train, and its an exciting one for sure! In today’s post, we will play around with ISML templates, or in other words, the visual representation layer of our storefront.

Let’s Talk About Cache, Baby

Templates are used a lot, but by default, they are not cached. That being said, there may be times where we want to cache the template for better performance. For cases like these, ISML comes to our aid with the iscache element. The iscache element should be placed at the top of the page and enables us to set cache on a template and configure it, using a number of properties, to suit our needs:

Status : A boolean flag that enables/disables page caching. Since caching is disabled by default, just the presence of the iscache tag implicitly enables page caching.
Type : A required property that sets the type of cache being used. The values for type can be either relative or daily, the former allows us to specify a period of time (in hours and minutes) for the page to be deleted from the cache, and the later allows us to specify a specific time in the day when the page will be deleted from the cache (eg. 3 am).
Hour : When the cache type is set to relative, the hour property represents the duration in hours before the page is cleared from the cache. When the cache type is set to daily, the property represents the hour of the day (in 24h format) when the cache will expire.
Minute : When the cache type is set to relative, the minute property represents the duration in minutes before the page is cleared from the cache. When the cache type is set to daily, the property represents the minute of the hour in the day when the cache will expire.

🐘 The iscache element has two additional properties which are beyond the scope of this post: varyby and if. If you like to learn more about them, head over to the official SFCC documentation.

Go Wild 🌈

Consider the following tag:

<iscache status="on" type="relative" minute="15"/>

This will cache the page for a period of 15 minutes (notice the use of relative and minute here).

Another example would be:

<iscache status="on" type="daily" hour="21" minute="30"/>

This tag uses the daily type, which means that the hour and minute properties represent an hour in the day (9:30pm in this case).

A JSON Response

Not all templates have to be visual. There may come a time when you would like to return a JSON object as your response (for example, an API route). Up until now we only used res.render to render our responses to the client, however res has another card up its sleeve: res.json. Using res.json allows us to render a JSON object directly, without the need to add any ISML templates or set the content type of the response.

On the Magic controller (located in our Magic cartridge controllers folder) add the following route:

The code looks similar to our Show method, but instead of rendering it on the page, we render the response from the icanhazdadjoke service directly on the page using res.json!

Go Wild 🌈

If you navigate to the /Magic-Json route (don't remember the full URL? head back to Day 4 to refresh your memory) and you should see something like this:

Our joke JSON in all its glory! 🎉

Overcoming the Language Barrier

Internationalization is another important concept for our storefront, as it allows us to provide content to visitors in more than one language. ISML templates support this concept using a feature called properties files. So what are these magical files? Well, nothing more than a dictionary of key/value of text really. Let’s create a new template on our Magic cartridge called hello.isml that initially have the following content:

As you can see, it’s a pretty basic (and ugly) template that shows a header and some text. Now, this text only caters to an English speaking audience, but what if we want to show a visitor whose browser language is Spanish the same text in his native language?

If you look closely at the templates folder of our cartridge, you’ll notice a folder called resources. As the name suggests, this folder holds the resources (or language) files for our template. Let’s create a default language (a.k.a properties) file in the folder that will hold the English text. Create a new file called hello.properties inside the resources folder with the following content:

This is the same text as we had hardcoded on the hello.isml template just a moment ago. Let’s update our template to actually use the properties file:

So what do we have here?

Line 9 : We replaced the hardcoded text with the msg method of the Resources object in order to display the text from a resource file. The method takes three arguments:

The key to the text we wish to display.
The name of the properties file to use.
The default value to display. In this example, I left it as null.

Go Wild 🌈

If you browse to a route that uses this template you will see it just like before:

Now, let’s add another file named hello_es.properties in the resources folder:

And that's it! if a Spanish speaking visitor comes to our site, he will see the translated text:

You can have as many properties files as you wish to support different languages. There is virtually no limit to the internalization you can have on your storefront.

Epilogue

This brings our journey with SFRA to its final destination. I hope I managed to show you at least one cool aspect of the SFCC platform you didn't know before, and as always please contact me with any comments or suggestions either here or on Twitter. I look forward to hearing which day was your favorite 😎

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 10: Advanced Jobs

Johnny Tordgeman — Fri, 15 May 2020 16:01:22 +0000

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 10: Advanced Jobs

Photo by Joshua Rodriguez on Unsplash

Yesterday we created a job that keeps our storefront sitemap updated and fresh, and we did that using the out-of-the-box createSiteMap step. Whenever you can use a premade step like that, that’s awesome! But, there are times that you may want to create a step customized to your own needs. Meet the ExecuteScriptModule step — the ⭐️of today’s post!

The One With the Static Assets

Let’s consider the following situation: you are using a JavaScript library in your storefront, which you download from a third-party CDN, for example: React: <script crossorigin src="https://unpkg.com/react@16/umd/react.production.min.js"></script>

If a visitor with an Ad Blocker configured to block third-party scripts comes to our site, his experience on our site may be ruined because he might not be able to load the React JS library. One possible solution could be to host the React library in the assets folder of our storefront. This approach is awesome, and it will solve the third party issue, however, it will also create a new problem - what happens when the React library gets updated? What will keep our local asset up to date with React’s CDN? 🤔

If at this point you are hearing bells in your head and thinking “this is a repeated task I should automate using SFCC Jobs” then you are absolutely right! Let’s create a job that periodically checks and downloads the React library from its CDN to our local assets folder.

Look Ma, I Can Run My Own Code

If you look at the pre-defined steps an SFCC Job can have — you won’t find anything for saving a file as a local asset. However, you would see a step called ExecuteScriptModule. This step, as its name suggests, allows you to run JavaScript code as a step in your Job. Let’s use this step to point to a new file on our cartridge called a_ssetUpdater.js_, which in turn handles downloading and storing the React library as a local asset of our storefront.

Creating the Job

Follow the instructions from yesterday’s post to create a new job. Give the new job the ID AssetUpdaterJob.
Under the Job Steps tab, click on Configure a step.
Select the ExecuteScriptModule step and provide the following details: * ID : LocalAssetUpdaterStep. * ExecuteScriptModule.Module : magicCartridge/cartridge/scripts/assetUpdater.js * ExecuteScriptModule.FunctionName : execute.
Click Assign.

Adding the Service

As we learned on day 6, the Services Framework is our way of communicating with the outside world. The assetUpdater.js needs to get the react.production.min.js file from https://unpkg.com/react@16/umd/react.production.min.js and for that, we need to create a service and credentials that the script will use:

Open your Business Manager and navigate to Administration > Operations > Services.
Click on the credentials tab, and create a new service credential that will point to https://unpkg.com/react@16/umd/react.production.min.js (I named mine MagicCartridge.Job.Credentials)
Click on the Services tab, and create a new service that will use the newly created MagicCartridge.Job.Credentails (I named mine MagicCartridge.Job.Service)
cd to the services folder of MagicCartridge (cd cartridges/magicCartridge/cartridge/services), and create a new service initialization file called reactservice.js.
Add the following content to the newly created reactservice.js:

🐘 If you need a refresher on how to use the Services Framework just head back to Day 6 and Day 7.

Adding assetUpdater.js

With the job in place, its time to add the script that it will call, assetUpdater.js:

cd to the root folder of MagicCartridge (cd cartridges/magicCartridge/cartridge), create a new folder called scripts and inside of it, create a new file called assetUpdater.js.
Add the following content to the newly created assetUpdater.js:

Let’s break down what's happening here:

Lines 1–4 : Imports the needed classes to get the content of the script (line 4) and to write the file back to the shared library (lines 1–2).

Line 9 : Sets the name of the shared library we are going to save the file to. In my case, my site name is Mobile First and my shared library name is MobileFirstSharedLibrary.

🐘 You can find the names of all the shared libraries on your pod under Administration -> Sites -> Content Libraries.

Lines 10–12 : Sets the destination path for the asset file. Notice the use of File.LIBRARIES constant — this constant represents the storefront libraries root directory. From here we add the name of our shared library and append the default folder to it.

Line 13 : Creates a new fileWriter instance based on the destination file we defined earlier. We will use this instance’s writeLine method later in the code to actually write the content to the destination file.

Lines 16–19 : These lines deal with fetching the content of the script using the Services Framework, and then, if successful save the content to the file using the fileWriter instance.

Lines 23–24 : These lines call the flush() method of fileWriter to flush the buffer used for writing the content, and then the close() method to close the writer instance. You must call these methods when working with the fileWriter class in order to properly close the file you are writing.

Line 27 : Tells the Job framework that our module is done and it should move on to the next Pipelet in the chain.

🤖 It is highly recommended that you go over this script more than once as it introduces a number of new concepts.

Deploy MagicCartridge to your instance and let’s take it for a spin!

Go Wild 🌈

With the service, the job script, and the cartridge in place, let’s test our newly developed job. Navigate back to the Jobs section in Business Manager, and click on our newly created AssetUpdaterJob. In the top right corner, click on Run Now, and then head over to the Schedule and History tab. If all went well, you should see the beloved OK badge next to your run!

Successful run — happy life

But this is not all, you may be asking yourself “this is great, but how do I see the actual file?” By using URLUtils of course! On your storefront ‘s scripts.isml file, add the following script element:

<script src=_"${URLUtils.staticURL(URLUtils.CONTEXT\_LIBRARY,'','/react.min.js')}"_ type=_"text/javascript"_ async></script>

URLUtils.CONTEXT_LIBRARY will point to the site’s shared library so appending the react.min.js file name will actually call the file we just got using our job:

This wraps up today’s post, and I hope that I managed to show you the amazing value a job can add to your site. Another plus of using such an approach of loading static content on your site is that these assets are also browser cached — meaning users will not hit your backend every time they visit your site but will be able to use a cached version of the library.

Tomorrow we will wrap up this post series with a really cool and useful subject — templates.

As always, looking forward to your comments either here or on Twitter 😎

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 9: Jobs

Johnny Tordgeman — Thu, 14 May 2020 17:03:57 +0000

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 9: Jobs

Photo by Anne Nygård on Unsplash

There may come a time where we want to run code automatically and repeatedly on our storefront (think Linux cron job). Such a job could be to update a local database with values, import or export product catalogs or even update a local resource on our site. SFCC provides us with a wonderful and accurately named framework — the Jobs framework — that enables us to handle these tasks.

Automate All the Things!

So what actually is an SFCC job? Well my young Padawan, an SFCC job is really a set of steps (actions) that are executed in order (flow) on a fixed time interval. Executing behind the scenes, jobs do not affect our storefront performance when running and are invisible to the user. Think of a job as a house-elf that does work for you while you sleep.

Jobs are managed in Business Manager under Administration -> Jobs. This is the central point for anything related to Jobs and also the prime focus of today’s post 🎉

Creating a Job

A common (and repeated) task for every website is maintaining a site map that provides information about the site structure so search engines will know what to index. Such a task is probably not something you want to deal with manually on a regular basis, which makes it the perfect candidate for our first job:

Open your Business Manager and navigate to Administration > Operations > Jobs.
Click on New Job (located at the top right corner).
Fill in the following details and click Create :

ID : SiteMapJob
Description : A job to update the site map file.

As soon as the job is created you’ll be greeted with the job configuration page. This is where you can configure (among other things) the two most important properties of a job: when it runs ( Schedule and History ) and what it does ( Job Steps ):

Yay, Job configuration page 🎉

Creating the Job Steps

Steps are the heart of the Job creation process. It’s here that we tell the job what to do, in which order to run its steps and what is the step context (global, specific site, etc.). Let’s add the site map creation step:

Click on the Job Steps tab.
Click on Configure a step.
Choose CreateSiteMap, enter the following details, and click Assign :

ID : SiteMapCreationStep
Description : A job step to create the site map.

At this point, you might notice that while our new step is created, it is highlighted in red 😱. Do not fear, this is normal! A site map is, as its name suggests, scoped to a site. Our step, however, is currently scoped as organization.

Click on the blue Organization button on the top left corner of our step and change the scope dropdown to Specific Sites. Check the site(s) you wish the job to run on and click Assign.

MobileFirst will be the only site my job runs on.

Scheduling The Job

Now that we configured the “what” let's define the “when”:

Click on the Schedule and History tab.
Check the Enabled checkbox. This defines our job as a scheduled one.
Change the Trigger drop-down from Once to Recurring Interval.
Under Every set Amount to 1 and Interval to Days. This triggers our new job once a day.

🐘 There are many other scheduling options you can set in this tab. You can have the job run only on specific days, you can have the interval to be minutes, hours years etc. and you can even set a start and end date for your job if you want to have it run in a specific time frame.

Go Wild 🌈

With everything in place, let’s run our job and see its status!

On the top right corner, you should see a blue Run Now button. Click it and wait a few seconds. On the bottom of the page, under the Job History section you should see the result of running our job:

The sweet taste of success 🍾

And there you have it! An automated job that will keep your sitemap fresh and updated, while letting you deal with much more important stuff.

If you liked this out-of-the-box job we created here, then you are in for a treat tomorrow as we are going to deep dive into the world of custom code job steps.

As always, looking forward to your comments either here or on Twitter 😎

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 8: A Tale of Two Events

Johnny Tordgeman — Wed, 13 May 2020 16:13:51 +0000

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 8: A Tale of Two Events

Photo by Adam Solomon on Unsplash

🎼 Now this is a story all about how
My code got triggered only once
And I’d like to take a minute - just sit right there
I’ll tell you how I spent my day registering to events 🎼

So far in our SFCC journey, we saw how to write code that gets triggered by an action (for example, browsing to a route); but what if we want to run code that corresponds to an event? SFRA allows us to hook to all kind of events, such as payment events or analytics events, but on this post, we are going to focus on two very special events: onSession and onRequest.

Two Events Walk Into a Bar…

SFRA allows us to hook into all kinds of events, for example, app.communication.order.confirmation that notifies on an order confirmation or app.template.htmlHead, which, in turn, allows you to add content to the HTML head element — each handles a specific event in the flow of the storefront life.

onSession and onRequest represents different type of events as they enable us to hook into the life cycle of a storefront request and receive notifications on two different states:

onSession is called at the beginning of a new storefront session.
onRequest is called every single time a storefront request is received from the client.

Hook Me Up!

Hooking to events is done via a special file called hooks.json. This file contains hook entries for any number of events we wish to hook to and has to be declared inside of the cartridge’s package.json file. Each hook entry must contain the following two properties:

Name: The extension point (hook name)
Script: The script file to call when the event is fired.

🐘 Make sure that the script file you set for the hook entry contains a function which will run once the event is fired

For today’s project, we are going to hook to the onRequest event, check every storefront request for a query param called magic, and if found, and its value is true, redirect the user to our Magic-Show route.

We begin by declaring a hook entry for onRequest:

cd to the services folder of MagicCartridge (cd cartridges/magicCartridge/cartridge/services), and create a new file named hooks.json.
Add the following content to the newly created hooks.json:

Let’s break this JSON down:

Line 2 : Declares a key called hooks and its value as an array of hook entries.

Lines 3–6: A hook entry.

Line 4: Sets the name of the event we wish to hook to. In our case we wish to validate every request to our storefront, so we will hook to the onRequest event.

Line 5: Sets the path and name of the script that will be called once the event fires. We will create the MagicValidator.js file in the same folder as hooks.json hence the use of ./ to specify the file path.

With the new hooks.js file in place, it’s time to register it:

cd to the root folder of MagicCartridge (cd cartridges/magicCartridge), and create a new file named package.json.
Add the following content to the newly created file:

Answer the Call!

With hooks.json registered, SFCC will fire an event for every incoming storefront request and call MagicValidator.js each time to handle the event. In turn, MagicValidator.js needs to export a method called onRequest to handle the event.

Let’s add MagicValidator.js to our cartridge and handle the event:

cd to the services folder of MagicCartridge (cd cartridges/magicCartridge/cartridge/services), and create a new file named MagicValidator.js.
Add the following content to the newly created MagicValidator.js file:

So let’s see what we have here:

Line 5 : Verifies the request we are processing is both an HTTP request and a top-level one (we don't want to start validating requests from remote sources on the page).

Line 6 : Checks the request’s query params string for a parameter named magic with the value of true.

🐘 This query param check is rather naive and is only used in this case to demonstate the onRequest handler. If you are working with query params in your code — please use something better.

Line 7 : Redirects the response to the Magic-Show route using the URLUtils class.

Line 11 : If the request does not meet all of the above conditions, then pass the request back to SFCC’s request flow by setting its status to OK.

Lines 14–16 : Exports the onRequest method.

Go Wild 🌈

Armed with the new event handler let’s test our new feature! Browse to any RL in your site front (for example /home). You should notice nothing changes and you are right on the home page as you should be. Now try to add the magic query params (/home?magic=true) and watch what happens:

No matter where we browse in our storefront now, we can go straight to the Magic route just by using a query param.

🐘 What if we want to be able to do this trick only once per session? Which event should we listen to in our hooks.json? (hint: it rhymes with passion)

And there you have it — a way to invoke code based on events and not by user interaction. Just make sure to use this wisely otherwise you will create some heavy load on the request flow, resulting in slow responses from the server.

Tomorrow we are going to deal with a really fun concept of SFCC — Jobs.

As always, looking forward to your comments either here or on Twitter 😎

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 7: More Services Framework

Johnny Tordgeman — Tue, 12 May 2020 16:30:32 +0000

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 7: More Services Framework

Photo by Lefteris kallergis on Unsplash

Yesterday we built our first service using the Services framework, and as much joy as we got from the dad jokes it returned, there was something much less funny about the whole thing: it was static 😱. We had one URL we were calling and that is. No paths, no on-demand headers, no query params — nothing. Well, today we are going to change all that! We are going to build the ultimate dad jokes widget that allows you not only to display a random dad joke (as we did yesterday) but also search a dad joke on a specific term 🤓

Searching for the Perfect Dad Joke

icanhazdadjoke.com allows us to search for a dad joke using its search endpoint: https://icanhazdadjoke.com/search and a query param: term, for the search term. Let’s extend the Services Framework entity we created yesterday to be able to handle the new /Magic-Search route.

Open the Magic.js controller we created yesterday in your IDE.
Below the Show handle function, add the following function:

Let’s go over what we have here:

Line 3 :Sets magicSearch as the template that the controller will render (we will create it in the next section).

Line 4 : Gets the term query parameter from the req object’s querystring object. This querystring object holds all the query parameters passed to the SFRA controller, and each query parameter can be accessed by specifying its name.

Line 6 : The getURL() method returns the URL set in the Services Framework entity credentials. We can use the result of this method to append any path we wish for the service, and in our case, the search path.

Line 7 : This line introduces two new methods: setURL() and addParam():

* setURL() is used to set the URL for the service call, overriding the static one defined in the Services Framework entity’s credentials. In case the entity does not use any credentials, setURL() will define the URL the service will call.

* addParams() is used to add parameters to the service call. If your service is using the GET method, the parameters will be added as query params. If your service is using the POST/PUT method, the parameters will be added as body parameters.

In our implementation, we use setURL() to set the service call to the URL we created in line 6, and addParams() to add the term parameter needed by icanhazdadjoke.com’s search endpoint.

Finally, we use the service object’s call method to make the actual call to the icanhazdadjoke.com service and store the result with the svcResult variable.

Lines 8–11 : If the service call was successful, the svcResult’s object property will hold the JSON response from the service. This JSON response contains a results property that holds an array of joke objects. We will save results to a new property on our properties object called jokes, and the search term to a new property called term.

Line 13 : Calls the render method of the res object, passing it the properties object we created in line 2. This object can later be used in an ISML template when it renders.

Line 14 : Calls the next() method passed as an argument to the function, which calls the next middleware in the chain.

Rendering the Laughs

With the new route handler under our belt, let’s create the ISML template that it will render:

cd to the templates/default folder of our cartridge and create a new file called magicSearch.isml.
Add the following to the new template:

Most of the code in this template is nothing new, so let’s focus on the interesting part — the loop of jokes:

Lines 14–17 : These lines introduce the isloop element. This element enables us to iterate through an array of items, similar to other JavaScript iterators such as forEach. Here we loop through the jokes collection of the pdict object (go back and read yesterday’s post if you don’t remember what the pdict is) while naming each iteration as jokeObject (as it contains a joke object, similar to the one we saw yesterday), and add it’s joke property as a li element to the unordered list (ul element) on the page.

Go Wild 🌈

In order to test our new route and template we need to upload the cartridge back to SFCC. Using our favorite command npm run uploadCartridge on the root of our cartridge folder will take care of that.

To test our search route, use its base url (should be something like https://.demandware.net/on/demandware.store/Sites-RefArch-Site/default/Magic-Search) and append a query parameter named term with the value you want to search for. The end result looks like this: https://.demandware.net/on/demandware.store/Sites-RefArch-Site/default/Magic-Search?term=.

If all went well you should get a page similar to this:

Guess which term I searched for here?

That wraps up our Services Framework fun. Of course, there are other properties you can use with your service, such as .addHeader(name, value) to add a header to the request, or setRequestMethod(method) to change the HTTP method (such as POST or PUT) or even use the setOutFile(file) method to set an output file to store the service response, however, we will save working with these for later.

Tomorrow we will have fun with a very important aspect of working with cartridges — Events (and specifically the onRequest event).

As always, looking forward to your comments either here or on Twitter 😎

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 6: Using Services Framework

Johnny Tordgeman — Mon, 11 May 2020 15:49:46 +0000

11 Days of Salesforce Storefront Reference Architecture (SFRA) — Day 6: Using Services Framework

Photo by Kate Townsend on Unsplash

A common use case for cartridges is to retrieve data from outside sources, and our cartridge is no different. Today, we are going to add a random dad joke below our magic gif because, well, nothing captures the essence of magic like a good dad joke, does it?

Creating a Service

The Services Framework helps us manage calls to external web services and analyze their performance. It is a site-wide setting and is accessible from the Administration section of the Business Manager. A service definition, which is how we describe and set a service within the Services framework, usually consists of 3 parts: Credentials, Profile, and Service, however in some use cases, the credentials and/or profile parts can be omitted (we will see such a scenario on day 7).

Let’s go ahead and create our dad joke service:

Open your Business Manager and navigate to Administration > Operations > Services.
Click on the Credentials tab and then New.
Credentials enable us to set the URL (and, if needed, authentication details) for our service. Set the URL to https://icanhazdadjoke.com/ and name our credentials MagicCatridge.Dad.Credentials. Click Apply to save .

icanhazdadjoke FTW!

Next, we will create a profile for our service. The profile enables us to configure connection related settings to our service, such as timeout and rate-limiting.

Click on the Profiles tab and then New.
Name the new profile MagicCartridge.Dad.Profile and set its connection timeout to 500. This will ensure that if we cannot connect to the service within 500ms, the connection will be dropped. Click Apply.

Finally, let’s create the service itself. Service is where we combine all of the settings for a service, such as its type, status (live/disabled), mode, etc.

Click on the Services tab and then New.
Set the following:

Name the service MagicCartridge.Dad.Service.
Check the Enabled checkbox.
Set the profile to MagicCartridge.Dad.Profile.
Set the credentials to MagicCartridge.Dad.Credentials.

Click Apply to save.

Our new Dad service! ❤️

This concludes our dad joke service configuration. But since nothing uses it yet, it’s sitting all alone in the dark waiting for a purpose in life. We will fix this issue next.

Initializing the Dad Joke Service

Just like many things in life, before we can use our new dad joke service in the cartridge, we have to initialize it. All of our initialization logic will be handled in a single file, dadjokeservice.js, created under a services folder within our cartridge:

cd to the root of the MagicCartridge folder (cd cartridges/magicCartridge/cartridge), and create a new folder named services.
Inside the new services folder, create a new file called dadjokeservice.js and add the following content:

So what are we seeing here?

Line 1 : Imports the LocalServiceRegistery module, which is the base class for creating services.

Line 3 : Creates an instance of the MagicCartridge.Dad.Service we previously defined and saves it to the dadJokeAPIService variable.

Lines 4–8 : Defines a createRequest method, which allows you to set different properties related to the service. In our example, we set the service method always to be GET (line 5) and to always have an Accept header of application/json (line 6). We end the method by returning the params object (line 7).

🐘 There are plenty of properties we can set on the svc object other then headers and method. Refer to Salesforce Commerce Cloud docs for more information.

Lines 9–18 : Defines a parseResponse method, which allows you to get the raw response of the service call, and parse or manipulate it to your needs. In our example, we try to parse the service call response (httpClient.text) to JSON (line 13) and save that to the result variable. If we fail (for example, we got a bad response, etc.) we will save the raw service call response to the result variable (line 15). We end the method by returning the result variable (line 17).

Lines 21–23 : Exports the dadJokeAPIService object.

Using the Dad Joke Service

To display the result of our service call using the Magic template, we have to call the service on the Magic controller and pass the result to the Magic template on the pipeline dictionary (pdict).

First, let’s modify our Magic.js controller to use the dad joke service:

Can you spot the changes from yesterday’s version?

Line 2 : Imports the Dad joke service and sets the service variable to reference it.

Line 5 : Defines an empty properties object.

Line 8 : Uses the service object’s call method to make the actual call to the icanhazdadjoke.com service and store its result with the svcResult variable.

Lines 9–11 : If the service call was successful, the svcResult’s object property will hold the JSON response from the service. This JSON response contains a joke property that holds the actual joke. We will save that property to a new property on our properties object called joke.

Line 13 : Calls the render method of the res object, passing it the properties object we created on line 5. This object can later be used in an ISML template when it renders. To call this object in an ISML template we will use the pdict global object. We will see how to use pdict next.

Updating the Template

With all the backend stuff behind us, its time to update the Magic cartridge’s ISML template to actually display the joke.

cd to the templates/default folder of our cartridge and open the magic.isml file.
Change the code as follows:

So what's changed from yesterday?

Lines 16–25 : Adds 3 new CSS classes to be used for the dad joke display.

Lines 31–34 : Adds the container and content of the joke. Notice the use of pdict at line 33 - it is the equivalent to the properties object we set on the render method above. That object had a joke key and, as such, so does pdict.

Go Wild 🌈

So now we have all the pieces in place let’s see how it renders:

ROTFL 🤣🤣🤣🤣

Every time you refresh the page, a call will be made to the dad joke service, grabbing a fresh (and hilarious) joke. That joke will then be stored in an object we pass to the render function, which is then used by the ISML template (via the pdict object) to render the joke!

With this, we can wrap up the Services framework for today. Tomorrow we will continue to tinker with Services framework and see how we can make it more dynamic.

As always, looking forward to your comments either here or on Twitter 😎