Forem: Nick Parsons

Clerk Update — July 2024

Nick Parsons — Fri, 12 Jul 2024 18:59:00 +0000

The Clerk team has been hard at work shipping new features to help you build secure applications faster. Here’s a rundown of the highlights:

Clerk Elements (Beta)

Clerk Elements is currently in beta and introduces an entirely new set of unstyled UI primitives that make it easy to build completely custom authentication and user management UIs on top of Clerk's APIs and business logic.

Customize with CSS Frameworks: Because everything is unstyled by default, Clerk Elements gives you complete control over the markup rendered in your authentication flows. Rendered markup accepts a className prop for easy styling with CSS frameworks such as Tailwind.
Extend with Component Libraries: Clerk Elements also support the asChild prop, popularized by component libraries like Radix. Bring your existing component library and it'll take care of the rest.

To learn more, visit the Clerk Changelog and Clerk Elements docs →

Google One Tap

Google One Tap support introduces seamless, one-click user sign-ins and sign-ups. This allows your users to effortlessly access your services without having to remember passwords or undergo lengthy registration processes.

New <GoogleOneTap /> Component: If you have already configured Google OAuth with custom credentials, adding support for Google One Tap is as easy as including the new <GoogleOneTap /> component.
Support for Custom Flows & Non-React Environments: For those building applications that require custom flows or do not use React, Google One Tap is supported via clerk-js.

To learn more, visit the Clerk Changelog and <GoogleOneTap /> component docs →

Other Features, Fixes & Improvements

Improved Clerk + Expo Docs: We have fully refactored our Expo docs with:
- New quickstart guide and accompanying starter repo.
- Updated examples for Impersonation, MFA, and OAuth.
- Added section that addresses common Expo deployment questions.
Clerk + Next.js 15: Clerk is now fully compatible with Next.js 15, with support for the React 19 RC. To use Clerk with Next.js 15, upgrade @clerk/nextjs to v5.1.2 or later.
Neon + Clerk Integration Guide: We've published a new integration guide demonstrating how to use Neon with Clerk in a Next.js application, utilizing drizzle-orm for database interactions.

Events & Community Updates

Clerk OSS Fellowship

We are thrilled to announce the launch of Clerk's Open Source Fellowship program, created to foster continued innovation in the open source software community. Our inaugural recipient is Colin McDonnell, the creator of Zod, who will receive funding while he works on bringing Zod to its next version.

// Detect dark theme var iframe = document.getElementById('tweet-1800947431489798163-488'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1800947431489798163&theme=dark" }

Read the announcement →

Clerk + Backdrop Build

We are excited to announce our partnership with Backdrop Build, a hackathon-centric accelerator that enables thousands of builders to take on the challenge of building and launching an idea in just 4 weeks.

Apply to Build →

Summer Hackathon with Xata, Clerk, Inngest & Prisma

We've partnered with Xata, Prisma, and Inngest to bring you a two-week summer hackathon challenge. Winners will be announced on the Xata Community Discord via live stream on July 19, 2024, at 12:00 PM EST.

Read the announcement →

Resources

If you have feedback or suggestions, we want to hear them! Let us know at feedback.clerk.com. For the latest on our product releases, follow @ClerkDev on 𝕏 or join the Clerk Community on Discord.

Ultimate Guide to Magic Link Authentication

Nick Parsons — Sat, 24 Feb 2024 04:05:00 +0000

Data breaches and password overload have made companies and their users wary of using a traditional username/password authentication system. Companies know that handling user passwords is both technically challenging and costly, as it requires stringent security measures, robust infrastructure for storage, and continuous monitoring to prevent unauthorized access. Users find it cumbersome to manage multiple complex passwords for various accounts, especially with the prevalence of methods like SSO, and often end up reusing passwords across different platforms, as a matter of convenience; this not only leaves the individual vulnerable, but also makes companies jobs of securing data that much harder, highlighting the critical requirements for improved security measures.

Luckily, email magic links have emerged as an elegant solution to secure user sessions in a passwordless context. Their rising popularity is underpinned by their dual advantages:

An augmented security posture through the minimization of phishing opportunities and password theft.
An optimized user experience devoid of the need to memorize and manage many login credentials.

What are Magic Links?

Magic links are a token-based authentication (TBA) strategy that uses a unique, time-sensitive URL, which leverages a securely-generated token to serve as a credential for user authentication.

The links are sent directly to the user's registered email or phone number, providing a straightforward, secure authentication method. When a user clicks on a magic link, the embedded token is validated against the server to authenticate the user's identity. This process, by design, eliminates the traditional risks associated with password-based systems and simplifies the login experience for the user.

In this guide, we will show you why you should consider authentication with magic links and how they work at a high level, before going through a Next.js App Router implementation to show how you can add magic links to your application.

The Benefits of Magic Links

Magic links bolster the security architecture of authentication systems by adopting a token-based, stateless interaction model. Each link is cryptographically unique and typically accompanied by an expiration timestamp, making it resilient against replay attacks. Given their ephemeral nature, even if a magic link were to be intercepted or exposed, its short-lived validity constrains the window of opportunity for malicious exploitation.

But there are a few other benefits to magic links for companies and users:

Streamlined user experience. Authentication with magic links eliminate the cognitive burden of remembering and managing many complex passwords, by providing a single-click authentication pathway. This frictionless access modality can enhance user engagement and satisfaction, reducing abandonment rates during the sign-in or sign-up processes.
Compliance and data protection. By using magic links to minimize passwords, organizations can reduce the risk of breaches involving personal data and avoid the consequences of compromised credentials, which can include heavy fines and reputational damage.
Reduction in credential exposure. With magic links, the threat of credential stuffing attacks—where compromised credentials are used to gain unauthorized access to multiple user accounts—is mitigated. Since magic links do not rely on reusable passwords, the standard vector of credential exposure is eliminated, enhancing overall system security.
Improved accessibility. For users with disabilities or those less familiar with technology, magic links offer a more accessible authentication mode. The simplification of the login process—replacing typing and memory demands with a single action—can be particularly advantageous for individuals facing physical or cognitive challenges.

How Magic Links Work

A magic link is structurally composed of two critical elements: the URL, which provides the link for the user's web interaction, and the embedded token, a cryptographically-generated string serving as the temporary credential.

In essence, a typical magic link may resemble the following structure:

https://example.com/authenticate?token=a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6

In this example, the token query parameter carries the weight of authentication, substantiating the user's claim without revealing identity until verified by the server.

Here’s an example of how the process works:

Token Validity

Magic links are designed to become invalid under certain conditions for security purposes:

Expiration: The token embedded in the link has a short lifespan and is often configurable based on application security requirements.
Usage Limitation: Once a magic link is used, it becomes invalid, preventing multiple uses, which could lead to unauthorized access.
Revocation: The system can programmatically revoke a token, thus invalidating the magic link in response to specific triggers or anomalies.

Token Generation

The lifecycle of a magic link commences with the generation of a unique token. This process employs cryptographic algorithms to ensure each token is a random, high-entropy string, making it virtually impossible to predict or reproduce through brute force or other cryptographic attacks. Typically, the token generation utilizes HMAC or AES combined with a CSPRNG to guarantee the robustness of the token against collision and preimage attacks.

Once generated, the token is stored on the server along with metadata that includes the user's identifier, the token's expiration time, and any other relevant session data. The magic link is then composed by appending the token to a predetermined URL structure, forming a complete, ready-to-use hyperlink.

This link is dispatched to the user's email address or phone number via SMTP or SMS protocols. The communication channel must be secure, leveraging TLS for email and similarly secure protocols for SMS to safeguard the link during transit.

User Interaction and Verification

pically interacts with the magic link by clicking on it, which initiates a secure request to the service's endpoint. The service extracts the token from the URL and verifies it against the stored data. This verification process involves several checks:

Authenticity: Validates that the token matches a recently generated and stored token.
Integrity: Ensures the token has not been tampered with during transit.
Timeliness: Confirms the token has not expired based on the predefined validity period.
Non-reuse: Ensures the token has not been used before, adhering to the principle of one-time use.

The server considers the authentication request legitimate if the token passes these checks.

Session Initialization

Post-verification, the server establishes a session for the user. This session is typically stateless, with a new session token or cookie generated to maintain the user's authenticated state in the application. This session token is separate from the magic link token. It has its own security considerations, such as being HttpOnly and Secure, to prevent access via client-side scripts and ensure transmission over HTTPS only.

Building a Magic Link System

Let’s create our own email with magic link authentication. In this example, we’ll use Next.js 13 with the App Router. We’ll also use Supabase for our backend database to store users and tokens.

First, let’s create a new Next.js project:

npx create-next-app@latest

Follow the prompts to select how you want to configure your app, but be sure to select “Yes” for “Would you like to
use App Router? (recommended)”.

Once you have created and configured your project, cd into the created directory and run npm run dev to start it. You’ll see just the default Next.js homepage when you load localhost:3000.

Before we start building out our project, we need to install a few dependencies that we’ll use. Install them using:

npm install nodemailer jsonwebtoken @supabase/supabase-js

What do these do?

nodemailer: A library that allows for easy email sending.
jsonwebtoken: A library to implement JSON Web Tokens for secure data transmission.
@supabase/supabase-js: The Supabase client library for Javascript.

With those installed, let’s open up the project in an IDE. In total, we’re going to have two pages, two API routes, and two helper libraries:

page.js will be our homepage. It will have a simple email field, and will call our requestMagicLink API reroute.
requestMagicLink.js will be the API route that will create our magic link, send it to the email address passed from page.js, and save the token on our Supabase database.
verify.js will be the API route called when the user clicks on the magic link in their email. It will verify the token and redirect the user to the protected dashboard page.
dashboard/page.js will be a simple mock “protected page” (that would require the user to be logged in to view it).
lib/database.js will be a number of database helper functions to save, load, and delete Supabase data.
lib/supabaseClient.js will set up our Supabase client.

page.js

Let’s start with what the user will see, page.js:

// app/page.js
'use client'
import { useState } from 'react'

export default function RequestMagicLink() {
  const [email, setEmail] = useState('')
  const [message, setMessage] = useState('')
  const [isLoading, setIsLoading] = useState(false)

  const handleSubmit = async (event) => {
    event.preventDefault()
    setIsLoading(true)
    setMessage('')

    try {
      const response = await fetch('/api/auth/requestMagicLink', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ email }),
      })

      const data = await response.json()

      if (response.ok) {
        setMessage('Magic link sent! Check your email to log in.')
      } else {
        setMessage(data.error || 'An error occurred. Please try again.')
      }
    } catch (error) {
      setMessage('An error occurred. Please try again.')
    } finally {
      setIsLoading(false)
    }
  }

  return (
    <form onSubmit={handleSubmit}>
      <input
        type="email"
        value={email}
        onChange={(e) => setEmail(e.target.value)}
        placeholder="Enter your email"
        required
      />
      <button type="submit" disabled={isLoading}>
        {isLoading ? 'Sending...' : 'Send Magic Link'}
      </button>
      {message && <p>{message}</p>}
    </form>
  )
}

In the code example above, we are requesting the RequestMagicLink component, which is responsible for handling the functionality of requesting a magic link via an email address. This component provides a UI for users to request a magic link. Users enter their email and submit the form, triggering a request to the server. Feedback is given to the user through messages and button state changes during the process.

First, we set up the state variables for the page. In the Next.js App Router, you can only use useState in client-side components. As all components default to server-side, we have to use the “use client” directive at the top of the file to show this page has to be rendered on the client. We have three state variables:

email: Stores the user's email address. It's updated every time the user types into the email input field.
message: Used to display messages to the user, like confirmation or error messages.
isLoading: Indicates whether the request is being processed. It's used to disable the submit button and change the button text while the request is in progress.

After that, we have the handleSubmit function. This is triggered when the form is submitted, and initially prevents the default form submission action with event.preventDefault(). It then sets isLoading to true to indicate the start of an asynchronous operation and clears any previous messages stored in message.

Then, comes the core part of the component. We make an async POST request to the /api/auth/requestMagicLink endpoint with the user's email in the request body. We then update the message state based on the success or failure of the request:

If the request is successful (response.ok is true), it sets a success message.
If the request fails, it displays an error message, either from the response or a generic error message.

We have some basic error catching, and then set isLoading to false.

The actual form presented to the user is basic, with just an email input field that updates the email state variable on change and a submit button that is disabled and changes its text based on the isLoading state. We have an onSubmit event handler linked to handleSubmit to send the form details and a paragraph that displays any messages stored in the message state.

This page.js file calls requestMagicLink.js; let’s dig into that, next.

requestMagicLink.js

// app/api/auth/requestMagicLink.js
import jwt from "jsonwebtoken";
import nodemailer from "nodemailer";
import { headers } from "next/headers";

import { saveToken, getUserByEmail } from "../../../../lib/database";

export async function POST(req, res) {
  const { email } = await req.json();

  const user = await getUserByEmail(email);

  if (!user) {
    return Response.json({ message: "User note found" });
  }

  // Create a magic link token
  const token = jwt.sign({ email }, process.env.JWT_SECRET, {
    expiresIn: "1h",
  });
  const headersList = headers();
  const host = headersList.get("host");
  const magicLink = `http://${host}/api/auth/verify?token=${token}`;

  // Save token in your database with an expiration time
  await saveToken(user.id, token);

  // Set up email transporter and send the magic link
  const transporter = nodemailer.createTransport({
    host: <email-host>,
    port: 587,
    secure: false, // upgrade later with STARTTLS
    auth: {
      user: <email-username>,
      pass: <email-password>,
    },
  });

  await transporter.sendMail({
    from: <from-address>,
    to: email,
    subject: "Your Magic Link",
    text: `Click here to log in: ${magicLink}`,
  });

  return Response.json({ message: "Magic link sent!" });
}

This example code handles the API requests related to generating and sending the magic link for user authentication. Let's go through the major components and functions of the code.

The function takes req (request) and res (response) objects as parameters, and then extracts the email from the request's JSON body. We then use getUserByEmail from lib/database.js to search for a user in the database using the provided email. If the user doesn’t exist we send a JSON response indicating the user was not found.

The function then creates the token using jsonwebtoken to create a JWT with the user's email, signing it with a secret from environment variables and setting an expiration of 1 hour. With that token we can create our magic link. We retrieve the host from the request headers and construct a URL with the generated token as a query parameter.

The generated token is then saved in the database with an associated user ID using saveToken.

After that, we configure a nodemailer transporter with SMTP settings (host, port, security, and authentication credentials) and send an email to the user with the magic link authentication.

Finally, we send back a JSON response indicating that the magic link was sent.

Here, we using one of our helper libraries, database.js. Let’s go through that next.

database.js

// lib/database.js

import { supabase } from './supabaseClient'

export const saveToken = async (userId, token) => {
  const { data, error } = await supabase.from('magic_tokens').insert([
    {
      user_id: userId,
      token: token,
      expires_at: new Date(Date.now() + 3600000),
    }, // expires in 1 hour
  ])

  if (error) throw new Error(error.message)
  return data
}

export const getUserByEmail = async (email) => {
  const { data, error } = await supabase.from('users').select('*').eq('email', email).single()

  console.log(data)
  if (error) throw new Error(error.message)
  return data
}

export const getTokenData = async (token) => {
  const { data, error } = await supabase.from('magic_tokens').select('*').eq('token', token).single()

  if (error) throw new Error(error.message)
  if (new Date(data.expires_at) < new Date()) {
    throw new Error('Token expired')
  }
  return data
}

export const deleteUserToken = async (token) => {
  const { data, error } = await supabase.from('magic_tokens').delete().match({ token: token })

  if (error) throw new Error(error.message)
  return data
}

This is a collection of utility functions designed to interact with Supabase. Each function is designed to interact with specific tables in a Supabase database, which each handle different aspects like token generation, user lookup, token validation, and cleanup. Let's break down each function:

saveToken

Purpose: To save a newly generated token in the database, fulfilling the requirements for secure token management.
Parameters: Accepts userId (the user's ID) and token (the magic link token).
Functionality:
- Inserts a new record into the magic_tokens table in Supabase with the user's ID, the token, and an expiration time (set to 1 hour ahead of the current time).
- If an error occurs during the database operation, it throws an error with the message received from Supabase.
- Returns the data received from Supabase if the operation is successful.

getUserByEmail

Purpose: To fetch a user's data from the database using their email address.
Parameters: Accepts email, which is the email address of the user.
Functionality:
- Queries the users table in Supabase for a record matching the provided email address.
- Logs the email and the data received for debugging purposes.
- If an error occurs, it throws an error with the message from Supabase.
- Returns the user data if a matching record is found.

getTokenData

Purpose: To retrieve token data from the database.
Parameters: Accepts token, the magic link token.
Functionality:
- Queries the magic_tokens table for a record with a token matching the provided one.
- Checks if the token has expired by comparing the expires_at field with the current time. If the token is expired, it throws an error.
- If an error occurs during the query, it throws an error with the message from Supabase.
- Returns the token data if a matching and non-expired token is found.

deleteUserToken

Purpose: To delete a token from the database, complying with the requirements for token lifecycle management.
Parameters: Accepts token, the magic link token to be deleted.
Functionality:
- Deletes the record from the magic_tokens table that matches the provided token.
- If an error occurs during this operation, it throws an error with the message from Supabase.
- Returns the data received from Supabase upon successful deletion.

This file, in turn, is calling on the other helper library supabaseClient.js.

supabaseClient.js

// lib/supabaseClient.js

import { createClient } from '@supabase/supabase-js'

const supabaseUrl = process.env.SUPABASE_URL
const supabaseAnonKey = process.env.SUPABASE_ANON_KEY

export const supabase = createClient(supabaseUrl, supabaseAnonKey)

This is a straightforward setup for initializing a client instance of Supabase in a Javascript application.

export const supabase = createClient(supabaseUrl, supabaseAnonKey);creates and exports an instance of the Supabase client, which is used to interact with your Supabase project. The createClient function takes the Supabase URL and the anonymous key as arguments and returns the initialized client.

SUPABASE_URL and SUPABASE_ANON_KEY (along with JWT_SECRET) are pulled from our environment variables file, .env.local.

.env.local

SUPABASE_URL=<supabase-url>
SUPABASE_ANON_KEY=<supabase-anon-key>
JWT_SECRET=<jwt-secret>

You get SUPABASE_URL and SUPABASE_ANON_KEY from your Supabase dashboard.

You’ll see from above, we also need to set up two tables in Supabase. We need a users table with an email field, and a magic_tokens table with these fields:

user_id, which comes from the users table.
token, which is the generated token.
expires_at, to add an expiry time to the token.

Let’s get back to the main code. If we fill out the email field on the homepage and click “Send Magic Link,” requestMagicLink will be called and an email sent to the entered email address. Clicking on that link calls the verify.js endpoint.

verify.js

// app/api/auth/verify.js
import jwt from 'jsonwebtoken'

import { getTokenData, deleteUserToken } from '../../../../lib/database'

export async function GET(req) {
  const token = req.url.split('=')[1]

  console.log(token) // Logs the token value
  const tokenData = await getTokenData(token)

  if (!tokenData) {
    return Response.json({ error: 'Invalid or expired token' })
  }

  const { email } = jwt.verify(token, process.env.JWT_SECRET)

  // Delete or invalidate the token
  await deleteUserToken(token)

  return Response.redirect('/dashboard') // Or wherever you want to redirect the user after login
}

This defines the API route for verifying the magic link token as part of an authentication process.

The code is designed to:

Extract a token from the request URL.
Verify the token's validity.
Perform actions based on the token's validity (such as user redirection or error response).

The code retrieves the token from the URL query string by splitting the URL at the = character and taking the second part (req.url.split("=")[1]).

The getTokenData function is called to fetch the token data from the database. If no data is found (implying the token is invalid or expired), it returns a JSON response with an error message. Using jwt.verify, we validate the token against the secret key stored in process.env.JWT_SECRET. This also extracts the payload (email) from the token.

We then call deleteUserToken to remove the token from the database, ensuring it cannot be reused. Finally, it redirects the user to the /dashboard route upon successful token verification.

dashboard/page.js

There isn’t much to dashboard/page.js:

// app/dashboard/page.js
'use client'

export default function Dashboard() {
  return <h1>A verified page</h1>
}

In a production application, this is the page you would build out in your application.

Final Thoughts

There is a lot to think about with email magic links. The above example doesn’t go into robust authentication checking with the user, nor does it add rate-limiting or have significant error handling. An unfortunate truth of authentication is that there is no silver bullet. While magic links take away the headache of managing user credentials, you still have to manage token generation, storage and expiry. Plus, you are still managing and storing user data.

Any good developer is going to take the time to understand, at least, the basic strategies leveraged by the tools they use to speed up their processes (good work understanding magic links!). With that being said, a simpler and more secure alternative to all the code above is to use Clerk. We built Clerk to make it quick and easy to add advanced authentication techniques into your application. To learn more about magic links, visit our documentation or this article on implementing magic links with Next.js and Clerk.

Exploring the Intricacies of OTP Authentication in Next.js

Nick Parsons — Mon, 24 Jul 2023 01:35:51 +0000

Passwords aren’t great. They’re often weak, reused, and need to be stored indefinitely making them susceptible to attacks and leaks. have i been pwned? tells me passwords associated with just one of my email addresses have been leaked 18 times, and I have over 800 individual passwords stored in my password manager.

This is why magic links, SSO, and one-time passwords (OTPs) are becoming standard authentication methods. They provide better or additional security for your users.

Here we’re going to look at OTPs. One-time passwords are a great option for improving the security of your application. Let’s go through exactly what one-time passwords are, how they can improve the security of your application, the best practices for using them in authentication, and how you can implement OTPs in Next.js.

What is a one-time password?

OTP or One-Time Password authentication is a method in which a unique code is sent to a user's device and the user enters this code into an application to verify their identity. It’s valid for only one login session and usually time-limited.

There are two ways OTPs are implemented:

As the main factor for logging in a user. If you don’t want to use any username/password combinations in your authentication flow, you can send a one-time password for log in. You can also use it as an alternative to username/password. This is not particularly common, but can be seen on sites such as local social networking site Nextdoor.
As an additional step in multi-factor authentication (MFA). After a user has logged in with their username/password or another authentication provider (such as Google, Twitter, or GitHub), they are then sent an OTP to a registered email address or phone number as a secondary layer of security for applications. This second option is much more common.

You’d think that one-time passwords would be long and complicated like the suggested passwords from a password manager. But because they are transient in nature and rarely subject to dictionary or brute force attacks, they can be much simpler. OTPs can have different formats, but they usually consist of a series of alphanumeric characters or purely numeric characters. The length of OTPs can vary, but a common format is a 6 or 8-digit numeric code.

The choice between numeric and alphanumeric OTPs depends on the context and requirements. Numeric OTPs can be easier to enter, especially on mobile devices, which makes them a popular choice for SMS-based OTPs. However, alphanumeric OTPs provide a larger possible combination set for the same number of characters, which can be more secure against brute-force attacks.

There are six main steps in the OTP flow:

Trigger. The user initiates the OTP process. This could be a login attempt, a transaction verification, or any other scenario where identity needs to be confirmed.
OTP generation. The server generates an OTP. This is typically a random string or number that is time-limited.
OTP delivery. The generated OTP is sent to the user through a predetermined method. This could be an SMS to their phone, an email to their registered email address, or generated in a hardware token or software app, such as Google Authenticator.
User input. The user receives the OTP and enters it into the application.
OTP verification. The server then verifies the OTP. The server also verifies that the OTP is used within the allowed time period, and hasn't been used before.
Authentication: If the OTP is verified, the server authenticates the user and allows them to proceed. If the OTP is incorrect or expired, the server rejects the request.

Though this is only six steps, there are a lot of moving parts in OTP use. You need extra frontend UI design and logic to handle the OTP inputs; you need to integrate a delivery mechanism such as SMS, email, or an authenticator app; you need extra authentication logic to handle OTP errors; and you need the OTP generation and verification logic as well.

There are several ways to generate and verify an OTP:

Time-Synchronized: In this method, the OTP is generated by applying a cryptographic hash function to a shared secret and the current time, typically measured in intervals of 30 seconds. The Google Authenticator app uses this method.
Counter-Based: Here, the OTP is generated by hashing a shared secret with a counter which increments with each new OTP.
Algorithm-Based: In this method, a mathematical algorithm is applied to the previous password to generate the new one.

If a time-based or counter-based OTP was used, the server repeats the same OTP generation process during verification and checks if the received OTP matches the one it generated.

These algorithms aren’t easy to implement. As you are dealing with an authentication factor, there are specific designs that you must use and RFCs that you must follow, such as this one for Time-Based OTPs. This is the biggest challenge around building your own one-time password (or any authentication) system–implementation.

The security benefits (and challenges) of OTPs

You can quickly see the complexity involved in setting up OTP authentication. But doing so is worth it as they provide two key security benefits.

Risk mitigation

The mitigation of risk with OTPs comes from two different avenues.

The first is mitigating the risk associated with static passwords. Traditional static passwords, if stolen, provide ongoing access until they are changed. OTPs are dynamic and expire after a single use or after a short period of time, limiting the potential damage if they are intercepted or stolen. Another problem is users reusing the same password across multiple services. If one service is compromised, all accounts using the same password are at risk. OTPs eliminate this risk because they are unique for each login session.

The second is the reduction of attack vectors. Because OTPs are typically time-limited, it makes brute-force attacks infeasible. An attacker doesn't have the time to try all possible combinations before the password expires. They also help with phishing. Even if a user is tricked into entering their OTP into a phishing site, the attacker can't reuse that OTP to gain future access to the account.

Additionally, since an OTP is valid for only one login session or transaction, it cannot be reused, preventing replay attacks. In a replay attack, an attacker tries to reuse a password that was intercepted in a previous session. However, if there's a flaw in the system's design where the OTP doesn't expire immediately after use or isn't time-bound, there's a possibility for replay attacks.

Identity security and verification

OTPs also play a significant role in enhancing identity security and verification processes. They provide an additional layer of protection beyond traditional static passwords, aiding in the confirmation of a user's identity in several ways:

OTPs are often used as part of a two-factor authentication process. In addition to a traditional username and password (something the user knows), the user must enter an OTP (something the user has, typically their mobile device). This confirms that the user has access to a specific device (like a phone) that is associated with the account, verifying the identity of the user.
OTPs are commonly used in financial transactions or account changes to verify the identity of the user making the transaction. For example, when conducting a bank transfer or changing account details, an OTP might be sent to the registered mobile number or email address. The user enters the OTP to confirm the transaction, verifying that they are the account holder.
OTPs are often used in password recovery processes to verify the user's identity. The service sends an OTP to the user's registered email address or mobile number, which the user then enters to verify their identity and proceed with resetting their password.
When a user logs in from a new device, an OTP might be sent to their registered contact information. The user must enter the OTP to verify they have access to the registered device, confirming their identity and that the new device is trusted.

By integrating OTPs into authentication and verification processes, services can add an extra level of security and significantly reduce the risk of unauthorized access or identity theft.

Best practices for using one-time passwords

OTPs aren’t infallible, though. But the associated risks are generally around poor implementation rather than inherent to the method. To ensure OTP effectiveness and avoid some of the above potential vulnerabilities, you can follow some best practices.

Basic security practices

These practices are the principles of any good system:

Secure delivery channel. Use a secure delivery channel for sending the OTP. If you're sending the OTP via email or SMS, ensure the communication channel is secure.
Encrypt communication. Ensure all communications between the client and the server are done over HTTPS to prevent any interception of the OTP.
Use secure storage. When storing OTPs on the server side, consider hashing them. This ensures that even if someone gains access to your storage, they cannot obtain the actual OTPs.

Good OTP design

These relate to how well you design your OTP algorithm and logic:

Use a strong OTP. Use an OTP that is long and complex enough to resist brute-force attacks. An OTP with at least 6 digits is usually recommended.
Time-Bound OTP. Make sure the OTP is valid only for a short period of time. This reduces the window an attacker has to use a stolen OTP. Usually, OTPs are valid for about 2-10 minutes.
Limit OTP attempts. Implement rate limiting on your OTP endpoints to protect against brute force attacks. After a certain number of incorrect attempts, either block the user or implement a cool-down period.
Expiry after use. The OTP should expire immediately after it has been used once, to prevent replay attacks.

Good UX

These help users use your OTP and make sure they don’t turn it off:

Backup codes. For applications using OTP as a second factor, provide backup codes that the user can write down and use if they lose access to their OTP delivery method (like losing their phone).
Fallback Options. In case the primary delivery channel fails (e.g., SMS not being delivered), have a secondary option like email or voice call.

Also consider educating users about using OTPs. One of the main threats to OTP use are physical–if an attacker steals a user's phone then they’ll have access to the SMS or email used with OTPs. Or a fraudster can fake a user’s identity to trick a telecoms company into assigning a new SIM with the user’s phone number to them (known as SIM swapping).

While OTPs can greatly enhance security, they are not foolproof. Implementation within a broader security strategy is key.

Setting up OTP Authentication in Next.js

Let’s walk through setting up a one time password system within Next.js. If you already have a Next.js app up and running you can add this code directly. Otherwise create a new app using:

npx create-next-app@latest

We’ll also need to use a few modules to help us with our OTPs, namely:

Twilio. Twilio allows us to send SMS messages programmatically. Here we’re going to use it to send the OTP to the user's phone number via SMS.
bcryptjs. bcryptjs is a JavaScript library for hashing and comparing passwords. We’re going to use it to hash the OTP before storing it in the database for added security.
MongoDB. MongoDB is a NoSQL database that we’ll use to store hashed OTPs along with the associated phone numbers and expiry times.
Upstash. Upstash is a serverless data platform. We’re going to use it’s rate limiter and Redis functionality.

Install these with:

npm install twilio bcryptjs mongodb @upstash/ratelimit @upstash/redis

For Twilio and MongoDB, you’ll also need to sign up for accounts and then need your TWILIO_ACCOUNT_SID, your TWILIO_AUTH_TOKEN, and your MONGODB_URI. For Twilio, you’ll also need to buy a TWILIO_PHONE_NUMBER that will be used to send your SMS messages.

You’ll also need an account with Upstash, and then your UPSTASH_REDIS_REST_URL and UPSTASH_REDIS_REST_TOKEN variables.

With that done, we’ll first create the API route that will generate our OTP:

// pages/api/generateOTP.js
import crypto from "crypto";
import twilio from "twilio";
import bcrypt from "bcryptjs";
import { MongoClient } from "mongodb";

export default async function handler(req, res) {
  if (req.method !== "POST") {
    return res.status(405).end(); // Method Not Allowed
  }

  // Generate a six digit number using the crypto module
  const otp = crypto.randomInt(100000, 999999);

  // Hash the OTP
  const hashedOtp = await bcrypt.hash(otp.toString(), 10);

  // Initialize the Twilio client
  const client = twilio(
    process.env.TWILIO_ACCOUNT_SID,
    process.env.TWILIO_AUTH_TOKEN
  );

  try {
    // Send the OTP via SMS
    await client.messages.create({
      body: `Your OTP is: ${otp}`,
      from: process.env.TWILIO_PHONE_NUMBER, // your Twilio number
      to: req.body.phone, // your user's phone number
    });

    // Store the hashed OTP in the database along with the phone number and expiry time
    const mongoClient = new MongoClient(process.env.MONGODB_URI);
    await mongoClient.connect();
    const otps = mongoClient.db().collection("otps");
    await otps.insertOne({
      phone: req.body.phone,
      otp: hashedOtp,
      expiry: Date.now() + 10 * 60 * 1000, // OTP expires after 10 minutes
    });
    await mongoClient.close();

    // Respond with a success status
    res.status(200).json({ success: true });
  } catch (err) {
    console.error(err);
    res.status(500).json({ error: "Could not send OTP" });
  }
}

With this code we initially import all our dependencies, then create a handler function for our POST endpoint. The body of the POST request will contain the phone number of the user that we’ll get from the frontend. Within the endpoint, we’re doing a few things:

Creating a six-digit random OTP
Hashing that OTP with bcryptjs for storage
Creating a Twilio client and then sending the OTP to the user’s phone number
Creating a MongoDB client and storing the OTP along with the user’s phone number and an expiry time for the password.

Is this best practice? Absolutely not. We are doing a few things right, such as setting an expiry time on the OTP and hashing them. But our OTP generating ‘algorithm’ is laughably simple.

Let’s quickly create a frontend for this now:

import { useState } from "react";

const OTPGenerator = () => {
  const [phone, setPhone] = useState("");
  const [otp, setOTP] = useState("");
  const [isLoading, setIsLoading] = useState(false);
  const [message, setMessage] = useState("");
  const [otpSent, setOtpSent] = useState(false);

  const handleSendOTP = async (event) => {
    event.preventDefault();
    setIsLoading(true);
    setMessage(""); // reset message

    try {
      const response = await fetch("/api/generateOTP", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ phone }),
      });

      if (response.ok) {
        setMessage("OTP has been sent to your phone.");
        setOtpSent(true);
      } else {
        const data = await response.json();
        setMessage(data.error);
      }
    } catch (error) {
      setMessage("An error occurred. Please try again.");
      console.error(error);
    } finally {
      setIsLoading(false);
    }
  };

  const handleVerifyOTP = async (event) => {
    event.preventDefault();
    setIsLoading(true);
    setMessage(""); // reset message

    try {
      const response = await fetch("/api/verifyOTP", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ phone, otp }),
      });

      if (response.ok) {
        setMessage("OTP verification successful!");
        setOtpSent(false);
        setPhone("");
        setOTP("");
      } else {
        const data = await response.json();
        setMessage(data.error);
      }
    } catch (error) {
      setMessage(error);
      console.error(error);
    } finally {
      setIsLoading(false);
    }
  };

  return (
    <div>
      {!otpSent ? (
        <form onSubmit={handleSendOTP}>
          <label>
            Phone Number:
            <input
              type="tel"
              value={phone}
              onChange={(e) => setPhone(e.target.value)}
              required
            />
          </label>
          <button type="submit" disabled={isLoading}>
            {isLoading ? "Sending..." : "Send OTP"}
          </button>
        </form>
      ) : (
        <form onSubmit={handleVerifyOTP}>
          <label>
            Enter OTP:
            <input
              type="text"
              value={otp}
              onChange={(e) => setOTP(e.target.value)}
              required
            />
          </label>
          <button type="submit" disabled={isLoading}>
            {isLoading ? "Verifying..." : "Verify OTP"}
          </button>
        </form>
      )}
      {message && <p>{message}</p>}
    </div>
  );
};

export default OTPGenerator;

All this code will just show a single form on the page. On first load this form will ask for the user’s phone number.

When the user enters their phone number and hits submit, the above generateOTP endpoint will be called. This will send the OTP to the user’s phone number:

Hitting submit will also change the form to accept the OTP as the input. The user can then check their phone and enter the six-digit code: and hit submit again to send the OTP and phone number to a verifyOTP endpoint for verification:

// pages/api/verifyOTP.js
import bcrypt from "bcryptjs";
import { MongoClient } from "mongodb";
import { Ratelimit } from "@upstash/ratelimit";
import { Redis } from "@upstash/redis";

const rateLimiter = new Ratelimit({
redis: Redis.fromEnv(),
limiter: Ratelimit.slidingWindow(2, "3 s"),

export default async function handler(req, res) {
  const user_ip = req.headers["x-forwarded-for"];
  const { success } = await rateLimiter.limit(user_ip);

  if (!success) {
    return res.status(429).json({ error: "Too Many Requests" });
  }

  if (req.method !== "POST") {
    return res.status(405).end(); // Method Not Allowed
  }

  const mongoClient = new MongoClient(process.env.MONGODB_URI);
  await mongoClient.connect();
  const otps = mongoClient.db().collection("otps");

  try {
    // Fetch the OTP record from the database
    const otpRecord = await otps.findOne({ phone: req.body.phone });

    if (!otpRecord) {
      return res.status(400).json({ error: "Invalid phone number or OTP" });
    }

    // Check if the OTP has expired
    if (Date.now() > otpRecord.expiry) {
      return res.status(400).json({ error: "OTP has expired" });
    }

    // Check if the OTPs match
    const otpMatch = await bcrypt.compare(
      req.body.otp.toString(),
      otpRecord.otp
    );
    if (!otpMatch) {
      return res.status(400).json({ error: "Invalid phone number or OTP" });
    }

    // OTP is valid and has not expired, so we can delete it now
    await otps.deleteOne({ phone: req.body.phone });

    // Respond with a success status
    res.status(200).json({ success: true });
  } catch (err) {
    console.error(err);
    res.status(500).json({ error: "Could not verify OTP" });
  } finally {
    await mongoClient.close();
  }
}

When called, this API loads the OTP MongoDB database and finds the one associated with the phone number. It checks whether it has expired, and if not, matches the hashed OTPs. If it's valid, the OTP gets deleted from the database and a 200 code is returned to the frontend.

We could then work that response into any other authentication flow we had set up. We also have a basic rate limiter set into that will make sure a user can’t input more than 2 codes in 3 seconds, to try and prevent brute force attacks:

And that’s it. You have one-time passwords working in Next.js.

OTPs are intricate but worth implementing

This code gives you a one-time password option for Next.js authentication. But we’ve only scratched the surface. We haven’t implemented this within any other authentication flow–this just generates, sends, and verifies the OTP, it doesn’t use it to authenticate the user.

We’ve also generated the OTP in the most basic manner, not following the RFCs and guidelines. We do have some nice best practices–rate-limiting, time-bounding, and expiry–but again these are all basic implementations. We also had to buy a new phone number!

This is the intricacy of OTPs. They are easy to set up, but difficult to get right. Like most authentication methods, it is better to use a provider than trying to create your own. Check out Clerk’s OTP solution here to have these intricacies taken care of for you.

Secure Authentication in Next.js with Email Magic Links

Nick Parsons — Tue, 06 Jun 2023 23:13:36 +0000

Secure Authentication in Next.js with Email Magic Links

Traditional username and password systems are ubiquitous but mostly suck. Every user needs a complicated random 8-20 character password that they can’t possibly remember for every site. Password managers make this easier, but they are ultimately just papering over the cracks.

Added to this cognitive load for users is the cognitive load for developers, who have to implement these systems securely or constantly worry about their sites becoming targets for malicious activity.

Imagine if you could simplify the user experience, decrease your system’s attack surface, and reduce your workload when it comes to user authentication. This is the promise of Magic Links, a powerful approach to passwordless authentication that is rapidly gaining popularity in the world of web development.

Here we want to show you not just the benefits of magic links but how exactly they work, going through an implementation process in Next.js. We’ll also show you why, like most authentication patterns, you don’t want to do this all yourself and how Clerk can help you, just like magic!

Magical magic links

So, why use magic links? Magic links can greatly improve user experience in several ways:

Ease of use: Magic links simplify the login process, as users just need to click a link sent to their email to authenticate themselves. They don’t have to remember any usernames or passwords.
Security: Since there are no passwords to guess, magic links can increase security. This can be especially beneficial for users who often reuse passwords or use weak passwords.
Speed: Magic links streamline the registration and login process. Rather than having to fill out forms, users just need to enter their email address, then check their inbox and click a link.
Reduced friction: Magic links eliminate the common frustration of forgotten passwords and the need for password reset processes. 5.** Mobile-friendly**: Typing passwords on mobile devices can be challenging, especially for complex or long passwords. With magic links, users simply click a link in their email, which is much easier on a mobile device.
Trust: When used properly, magic links can build trust, as they demonstrate a commitment to both user convenience and security.

There are downsides. For one, the user obviously needs to have access to their email. Plus, there is the slight friction element of the user having to leap from the app to their email and back again. In theory, if the user’s email is compromised, an attacker can gain easy access to the app (you can get past this by using two-factor authentication in your authentication flow).

Creating and validating time-sensitive tokens, like those used in magic links, typically involves the following steps:

User Identification: When the user requests a magic link, your application should first ensure that the email provided is valid and linked to a user account in your system. If the account exists, the server generates a unique, temporary token.
Token Generation: Generate a unique token using a secure method. This might involve using a secure random number generator or a library or function designed for generating secure tokens, such as JWT (JSON Web Tokens). The token should be associated with the user’s account in your database, along with a timestamp indicating when it was created.
Time-Sensitive Mechanism: Attach an expiry time to the token. This could be a specific expiry date/time or a duration after which the token will expire. This is usually stored along with the token in the database.
Email Delivery: Send an email to the user containing the magic link. The link should point to your website or app and include the token as a parameter.
Token Validation: When the user clicks the magic link, your application should validate the token. This involves checking that the token exists in your database, is linked to a user, and has not expired. If all these checks pass, the user is authenticated.
Token Deletion/Invalidation: Once a magic link has been used or has expired, it’s important to invalidate it so that it cannot be used again. This can be done by deleting the token from your database or marking it as invalid.

OK, so that’s magic links at a high level. How do you implement them in an application? Let’s go through two ways to do this. Firstly, we’ll show how you can do this with minimal additional libraries in Next.js to show what is required to generate, send, and validate time-sensitive tokens such as magic links. Then we’ll go through how you can implement them quicker and more securely with Clerk.

Build your own magic links in Next.js

We need to expand the high-level flow above into more granular detail for what we need from our application:

The user enters their email: The user will enter their email address, which you’ll send to your backend.
Generate a unique token: On your server, generate a unique token tied to the user’s email. This can be a JWT, a UUID, or some other kind of unique identifier.
Send an email with the magic link: Include the unique token in the magic link and email it to the user.
User clicks the link: The user will click the link, which sends the token back to your server.
Verify the token: On your server, verify that the token is valid and matches the user’s email.
Create a session: If the token is valid, create a new session for the user and send it back to the client.
Store the session on the client: On the client, store the session (usually in a cookie or local storage) so that the user remains logged in.
Authorize the user: Whenever the user makes a request, check that they have a valid session.

So beyond Next.js, for this to work, we need:

A way to generate and verify our tokens. In this instance, we’re going to use JWT for our tokens and use jsonwebtoken to generate and verify them.
A way to send emails. We’ll use nodemailer. You’ll also need SMTP information for your email provider.
A way to create cookies to store session data. We’ll use the cookie library here.

That’s all you need. Let’s first spin up a Next.js app called ‘magic-links’ as the bare bones of what we’ll create:

npx create-next-app@latest magic-links

You’ll be asked a bunch of questions. Here we’re not using TypeScript and neither are we using the App directory (which is the new, better way of using Next.js that we’ll use later. But it doesn’t play as nicely with some of the API routes we’re creating here).

After that we’ll install the necessary packages:

npm install jsonwebtoken nodemailer cookie

You can then run npm run dev to start the development server. At the moment all you’ll get at http://localhost:3000 is the regular Next.js start page:

Obviously we need a login component as the first step. We’ll create a /components directory and add a login.js file:

// components/login.js

import { useState } from "react";

function Login() {
  const [email, setEmail] = useState("");

  const handleSubmit = async (e) => {
    e.preventDefault();
    const res = await fetch("/api/login", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ email }),
    });
  };

  return (
    <form onSubmit={handleSubmit}>
      <input
        type="email"
        value={email}
        onChange={(e) => setEmail(e.target.value)}
        required
      />
      <button type="submit">Send Magic Link</button>
    </form>
  );
}

export default Login;

So there are two main parts to this component. At the bottom we have the actual form a user will fill in with their email address. As they type their email in, the state of the email variable will change to contain their email address.

When the press ‘Send Magic Link,’ The handleSubmit function will be called. This function will call a backend api route, /api/login. We’re going to send a POST request to this endpoint with the email address in the body.

Let’s add this to our index page. Delete all the boilerplate code within that file and replace it with:

// pages/index.js

import { Inter } from "next/font/google";
import Login from "../components/login";

const inter = Inter({ subsets: ["latin"] });

export default function Home() {
  return (
    <main
      className={`flex min-h-screen flex-col items-center justify-between p-24 ${inter.className}`}
    >
      <div className="z-10 w-full max-w-5xl items-center justify-between font-mono text-sm lg:flex">
        <Login />
      </div>
    </main>
  );
}

We’re importing the Login component and adding it to this page. Now a user will see this at http://localhost:3000:

If they were to add their email into that field and press ‘Send Magic Link,’ guess what would happen?

Absolutely nothing! Let’s make something happen. What we need is that /api/login the Login component calls. We’ll create a login.js file in the api directory. Though this is within the pages directory, it won’t be treated as a page by Next.js, it will act like an API endpoint:

// pages/api/login.js

import jwt from 'jsonwebtoken';
import nodemailer from 'nodemailer';

export default async function handler(req, res) {
  const { email } = req.body;

  // Create a magic link token
  const token = jwt.sign({ email }, process.env.JWT_SECRET, { expiresIn: '15m' });

  const transporter = nodemailer.createTransport({
    host: "mail.example.com",
    port: 587,
    secure: false
    auth: {
      user: "username",
      pass: "password",
    },
  });

  // Generate magic link
  const magicLink = `${req.headers.origin}/api/verify?token=${token}`;

  await transporter.sendMail({
    from: '"Your Name" <your-email@example.com>', // sender address
    to: email, // list of receivers
    subject: 'Your Magic Link', // Subject line
    text: `Click on this link to log in: ${magicLink}`, // plain text body
  });

  res.status(200).json({ success: true });
}

This is where the first part of the magic of magic links happens. Let’s step through this to see what’s happening:

First we’re importing the libraries we need, jsonwebtoken and nodemailer.
Within our handler function, we’ll get the email from the body of the request.
With that email as the payload, we’ll use the sign method from jsonwebtoken to create our token. There are two extra parameters we need to pass to sign:
- A JWT_SECRET. You can create a secret (for these purposes, not for production) by running openssl rand -base64 32 in the terminal to generate a key. You then store that key in a .env file in the root of the project for the project to read.
- An expiresIn time. Here, we’ve set this to 15 minutes.
Now we have our token, we need to send it to the user’s email address. Here we’re using the createTransport method from nodemailer to create an object called a transporter that contains all our (i.e. the sender) SMTP email information.
We’ll then create the actual magic link, which will be the origin URL (our URL) with the token appended as a query.
Then we’ll call sendMail on the transporter object to send the email to the recipient

Now if the user presses ‘Send Magic Link,’ they should get an email like this:

Now if they click on that guess what would happen?

Absolutely nothing!

We need an endpoint to verify the token and set a cookie on the client with the user's email. First, let’s create another file in the api directory, this one called verify.js:

// pages/api/verify.js

import jwt from "jsonwebtoken";
import { serialize } from "cookie";

export default async function handler(req, res) {
  const { token } = req.query;

  try {
    // Verify the token - this throws if the token is invalid
    const { email } = jwt.verify(token, process.env.JWT_SECRET);

    // The token is valid, so we create a session
    const sessionToken = jwt.sign({ email }, process.env.JWT_SECRET, {
      expiresIn: "1h",
    });

    res.setHeader(
      "Set-Cookie",
      serialize("auth", sessionToken, {
        httpOnly: true,
        secure: process.env.NODE_ENV !== "development", // Use secure cookies in production
        sameSite: "strict",
        maxAge: 3600, // Expires after 1 hour
        path: "/",
      })
    );

    // Redirect the user to the homepage
    res.writeHead(302, { Location: "/secrets" });
    res.end();
  } catch (err) {
    // The token was invalid, return an error
    res.status(401).json({ error: "Invalid token" });
  }
}

The top part of this is similar to the API route, but let’s step through the entire thing for clarity:

First we’re importing the libraries we need, jsonwebtoken again and cookie for session management.
Within our handler function, we’ll get the token from the query of the request.
With that token as the payload, we’ll use the verify method from jsonwebtoken to verify our token. This again uses our JWT_SECRET that we signed it with to verify.
If that’s a valid token, we’ll then create a session for the user, again using the sign method from jsonwebtoken. We set an expiresIn time of an hour. After that, our user will be logged out.
We add the token to our cookie using serialize from the cookie library, then add the cookie to the headers of our result.
We’ll then redirect the user to a logged-in-only page

That page, /secrets doesn’t yet exist. Let’s create it in the /pages directory:

// pages/secrets.js

import { useEffect, useState } from "react";
import { Inter } from "next/font/google";

const inter = Inter({ subsets: ["latin"] });

export default function Secrets() {
  const [data, setData] = useState(null);

  useEffect(() => {
    // Fetch data from our API route
    fetch("/api/secure-endpoint")
      .then((res) => {
        // If the response was not ok, throw an error
        if (!res.ok) {
          throw new Error("Failed to fetch");
        }
        return res.json();
      })
      .then((data) => {
        setData(data);
      })
      .catch((err) => {
        console.error("An error occurred: ", err.message);
      });
  }, []);

  // Render data or loading message
  return (
    <main
      className={`flex min-h-screen flex-col items-center justify-between p-24 ${inter.className}`}
    >
      <div className="z-10 w-full max-w-5xl items-center justify-between font-mono text-sm lg:flex">
        {data ? (
          <>
            <h1>{data.secret}</h1>
            <h2>{data.email}</h2>
          </>
        ) : (
          <p>This isn&apos;t a secret </p>
        )}
      </div>
    </main>
  );
}

This will conditionally render either a data object with the fields secret and email if the user is authenticated, or the text ‘This isn’t a secret’ if they aren’t. The data object comes from a call to our final api endpoint that we need to create, secure-endpoint.js:

// pages/api/secure-endpoint.js

import jwt from "jsonwebtoken";

export default function handler(req, res) {
  const { auth } = req.cookies;

  if (!auth) {
    return res.status(401).json({ message: "Unauthorized" });
  }

  try {
    const { email } = jwt.verify(auth, process.env.JWT_SECRET);

    // Now you have the authenticated user's email
    // Do your secure stuff here...

    res.json({ secret: "This is a secret!", email });
  } catch (error) {
    res.status(401).json({ message: "Unauthorized" });
  }
}

This authorizes the user by verifying with verify from jsonwebtoken the session token and, if the user is authenticated, passing back an object with a secret (‘This is a secret!’) and an email. Now if a user fills in their email and clicks on the link, they’ll be sent to the secret page with the secret and their email showing:

That's it! With these routes, you now have a basic magic link authentication system.

Using Clerk for your magic links

The above code works. It’s also a terrible idea.

We aren’t doing any error handling. We aren’t checking for any edge cases. We’re not overly protective of our JWT token. If an attacker learns your JWT secret, they can create valid tokens and impersonate any user. There’s no rate limiting. The cookie storage isn’t ideal.

Plus we have to have our own email server that is configured to send out a lot of transactional emails (which regular email providers don’t really like as they get punished for spam). And, yeah, we know, our login and authenticated pages aren’t exactly winning any design awards.

Basically, there are a number of issues with rolling your own magic links.

That’s why services like Clerk exist–to deal with all of the above and make it much easier for developers to implement strong authentication frameworks such as magic links. Let’s go through this again, but with Clerk as the magic link provider.

We’ll spin up a new Next.js app:

npx create-next-app@latest magic-links

Again, you’ll get the questions. Clerk is written for the latest developments in Next.js, so you can use the App Router.

After that all we need to install is Clerk:

npm install @clerk/nextjs

If we npm run dev now and go to http://localhost:3000 we again get a regular Next.js start page:

Before we get to the code, we’ll also want to do some other set up. First, we’ll need our NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY and our CLERK_SECRET_KEY. You can get both of these from your Dashboard.

Add these to an .env file in the root of your project:

NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY = pk_test_xxxx;
CLERK_SECRET_KEY = sk_test_xxxx;

We’ll also want to configure our Clerk set up to use magic links. We need to change two settings from the defaults in our dashboard. The first is to choose Email verification link as the authentication factor. To do that, go to User & Authentication > Email, Phone, Username in your dashboard:

Then go to Contact information in that subsection and toggle on Email address and check Email Verification Link:

Now we can start with the code. The first step is to wrap our entire Next.js application in the <ClerkProvider>:

// app/layout.tsx
import "./globals.css";
import { Inter } from "next/font/google";
import { ClerkProvider } from "@clerk/nextjs";

const inter = Inter({ subsets: ["latin"] });

export const metadata = {
  title: "Create Next App",
  description: "Generated by create next app",
};

export default function RootLayout({
  children,
}: {
  children: React.ReactNode,
}) {
  return (
    <ClerkProvider>
      <html lang="en">
        <body className={inter.className}>{children}</body>
      </html>
    </ClerkProvider>
  );
}

This is going to provide all we need to pass session and user information to Clerk for the authentication logic. This step is the same for any authentication pattern you are going to use with Clerk in Next.js.

The next step is to use Clerk to protect pages within our application. To do this, we’ll create a file in the /src directory. This will use regular expressions to pattern match against the pages you want to protect. Here, we’re going to protect all our pages:

//middleware.ts

import { authMiddleware } from "@clerk/nextjs";
export default authMiddleware();
export const config = {
  matcher: ["/((?!.*\\..*|_next).*)", "/", "/(api|trpc)(.*)"],
};

Now if you load up http://localhost:3000, you’ll be redirected to a login/signup page:

A user enters their email address here and gets sent a link to click:

Clicking on that link sends them back to http://localhost:3000, but now they can access the site again:

Much easier (and much better looking signup pages and emails as well!)

Better security and a user experience with magic links

Magic links can really help you streamline your sign up and sign in processes. They lessen the burden on your users while providing strong security for them and your application.

But implementing them manually puts the burden on your developers. Managing creation and validation of the tokens, sending emails (and maintaining the system to do so), then coding up the right modals, pages, error states, and edge cases is a huge hassle.

It’s worth developers going through and trying this manually just to see how magic links are implemented. But if you are looking for a production-ready option that you can incorporate into your app today, you can use Clerk.

Social SSO in Next.js

Nick Parsons — Tue, 06 Jun 2023 23:12:00 +0000

Authentication has always been a critical aspect of web development, with user data security being paramount.

As the digital world continues to evolve, so does the need for more efficient and secure authentication methods. However, users do not necessarily want to remember secure logins for every app they use.

This is where OAuth Single Sign-On (SSO) comes into play. OAuth simplifies the user authentication process by allowing users to log in with accounts they already have with other services like Google, Facebook, or Microsoft.

In this article, we will explore how to incorporate OAuth SSO into a Next.js project with JSON Web Tokens (JWTs) and Next.js’s new app router, and later see how Clerk, an authentication service, can help simplify this and make it more secure. We will focus on GitHub SSO for simplicity, but most other OAuth providers can be implemented similarly.

Before we get started, if you ever need to look at the full source, code, the example code is on GitHub.

Set up the project

Before we start, make sure you are on Node v18+. You can check this by running node -v in a terminal.

Scaffolding

We will use create-next-app to scaffold our project. First, you will want to open a terminal in the directory you want to put the project in and run

npx create-next-app@latest

If you want to use a different package manager, use their own counterpart to npx, like pnpx. create-next-app will ask you some questions about your project. Name the project whatever you want; it doesn’t matter. For TypeScript, ESLint, Tailwind, and using /src, select no. For the app router, select yes. Finally, leave the import aliases unmodified. After, the output should look like this:

Now, open that directory in a code editor of your choice. You will need to install jsonwebtoken, which we will use for verifying JWTs.

npm i jsonwebtoken

Create a GitHub OAuth App

Next, we will need to create the OAuth app in GitHub. Go to github.com/settings/applications/new and fill out the information. Set the homepage URL to http://localhost:3000 and the callback URL to http://localhost:3000/callback (if you are running the Next.js server on a port other than 3000, you can replace 3000 with that port). You can leave device flow unchecked. Click register and you will be sent to the new application page.

Now, let’s go back to the code for a minute. Create a new file called .env.local. This is where you will store your environment variables, which are variables that specify service keys or other configuration that is specific to each deployment and are specified by writing VARIABLE_NAME=variable_value, with each variable being on one line.

In the .env.local file, create a variable named NEXT_PUBLIC_GITHUB_OAUTH_ID. This will contain your GitHub OAuth application’s id. Note that the NEXT_PUBLIC prefix tells Next.js to allow the env variable to be accessed by replacing all calls to process.env.NEXT_PUBLIC_GITHUB_OAUTH_ID with the value of the variable at build time. If you did not include this prefix, only the server would be able to access the value, which is good for secrets. With that aside, we need to get the GitHub OAuth application id, which you can find on the application dashboard you were sent to earlier. Copy it and paste it into the .env file.

We also need to get a secret, which we will put with the variable GITHUB_OAUTH_SECRET. On the dashboard, you can click “Generate a client secret” to get a new secret. Copy it and paste it in the .env file under the new variable.

Finally, we need JWT_SECRET, which should be a random password we use for encrypting JWTs. You can use a password generator for this.

That is it for set up! Remember not to give anyone any of the secrets unless you trust them, as they can be used to impersonate users and gain unauthorized access to the application.

Implement sign in

Our next step is to allow people to sign in by adding a sign in button on the homepage that links to the GitHub OAuth verification page. Open app/page.js and insert the following below the Next.js logo (You should be able to find it by looking for an <Image> with src=src="/next.svg").

<a
  href={`https://github.com/login/oauth/authorize?scope=user:email&client_id=${process.env.NEXT_PUBLIC_GITHUB_OAUTH_ID}`}
  className={buttonStyle.button}
>
  Sign in with GitHub
</a>

The link will send people to a page where they can authorize the OAuth app’s access to their account information to allow the app to verify their identity. You might notice className contains a style that does not currently exist. We will fix this next. In app, create a file named button.module.css and paste the following in.

.button {
  z-index: 2;
  margin: 10px;
  transition-duration: 0.2s;
  display: inline-block;
  font-weight: 600;
  border: 1px solid transparent;
  border-radius: 6px;
  white-space: nowrap;
  padding: 5px 16px;
  font-size: 14px;
  line-height: 20px;
  vertical-align: middle;
  cursor: pointer;
  user-select: none;
  color: #24292e;
  background-color: #fafbfc;
}

.button:hover {
  background-color: #f3f4f6;
}

.button:focus {
  outline: none;
  border: 1px solid #c0d3eb;
  box-shadow: 0 0 0 1px black;
}

.button:active {
  background-color: #edeff2;
}

This just provides some simple styling for the button. In order to use it, we need to import it in page.js. Add this line to the top of page.js.

import buttonStyle from "./button.module.css";

To get the layout correct, we also have to add this to .center in page.module.css

flex-direction: column;

Now, run npm run dev and go to http://localhost:3000. You should see this:

If you click “Sign in with GitHub,” you should be redirected to a GitHub page asking you to authorize your OAuth app. However, if you click authorize, you will get a 404 error. That is because we have not implemented a callback page yet, which we will do next.

Creating a callback

Before we start this step, a bit of explaining is required. You might have seen OAuth flows implemented differently, and this is perfectly fine. There are multiple ways to handle OAuth codes and retrieve access tokens. In this case, we are using Authorization Code Grant, which is where the OAuth provider (GitHub in this case) sends a request to the specified callback URL (which we are implementing now) with a code. Then, the server sends a request to GitHub with the code, and GitHub responds with the access token. This is generally the most secure method, but for mobile apps or SPAs, it does not always work. With that explained, we will get started on implementing the callback.

You first want to create a folder named callback in app and a file named route.js inside callback. This tells Next.js’s filesystem router that route.js should handle API requests sent to /callback. Now, paste this into route.js:

import jwt from "jsonwebtoken";
import { cookies } from "next/headers";
import { redirect } from "next/navigation";

export async function GET(req) {
  const code = new URL(req.url).searchParams.get("code");

  if (!code) {
    return new Response("No code provided", { status: 400 });
  }

  try {
    const tokenResponse = await fetch(
      "https://github.com/login/oauth/access_token",
      {
        method: "POST",
        headers: {
          accept: "application/json",
          "Content-Type": "application/json",
        },
        body: JSON.stringify({
          client_id: process.env.NEXT_PUBLIC_GITHUB_OAUTH_ID,
          client_secret: process.env.GITHUB_OAUTH_SECRET,
          code,
        }),
      }
    );

    const tokenData = await tokenResponse.json();

    const { access_token } = tokenData;

    if (!access_token) {
      return new Response("GitHub login failed", { status: 400 });
    }

    const jwtToken = jwt.sign({ access_token }, process.env.JWT_SECRET, {
      expiresIn: "1h",
    });
    cookies().set("auth", jwtToken, {
      httpOnly: true,
      secure: process.env.NODE_ENV !== "development",
      sameSite: "lax",
      maxAge: 3600, // 1 hour
      path: "/",
    });
  } catch (error) {
    console.error(error);
    return new Response("Internal server error", { status: 500 });
  }
  redirect("/dashboard");
}

We will go through this step by step. First, we import a few different things:

jsonwebtoken is for creating JWTs, which we use to store the userdata and allow us to verify that bad actors did not change it (we will do this later)
cookies from next/headers is for setting cookies on the client. This function is an abstraction over the Set-Cookie HTTP header, hence the import location.
redirect from next/navigation is for redirecting users. Similar to the cookies function, this function abstracts over HTTP responses by automatically responding with an HTTP 3xx status code (read: a redirect) with the Location header set to the value passed.

After that, we export a new function named GET, which takes req as a parameter. This function handles all HTTP requests to /callback with the HTTP method GET, which is the method used for typical requests that do not contain a body. req contains all of the information about the request.

The next part retrieves the code from the code query parameter in the URL. You can refer to the explanation at the start of this section to learn how this is used. If there is no code, the function returns a 400 error.

Now that we have the code, we send it back to GitHub to get the access token. After getting the response, we parse the JSON body and extract the access token. If there is no access token, another error is returned.

We sign the token to allow us to store it in a cookie with confidence that it cannot change without the JWT secret. Using the signed token, we set the auth cookie for the user. The parameters we passed are very important. httpOnly makes sure JavaScript cannot access the cookie’s value, secure makes sure that when deployed in production, the cookie cannot be sent over unencrypted HTTP, and sameSite prevents the cookie from being sent to third party websites. sameSite=lax is the default in modern browsers, but it is still a good idea to make it explicit.

Finally, we redirect the user to /dashboard. If you run this code, you should end up at another 404 at /dashboard. We will create the dashboard next.

Creating a protected dashboard

Once again, we need to create a file named dashboard and a file inside it named page.js. Because this is a page rather than an API route, the naming is different. Copy the following into page.js

import { cookies } from "next/headers";
import jwt from "jsonwebtoken";
import { redirect } from "next/navigation";
import buttonStyle from "../button.module.css";

export default async function Page() {
  const auth = cookies().get("auth");
  if (!auth) {
    redirect(
      `https://github.com/login/oauth/authorize?scope=user:email&client_id=${process.env.NEXT_PUBLIC_GITHUB_OAUTH_ID}`
    );
    return;
  }
  const user = jwt.verify(auth.value, process.env.JWT_SECRET);
  const response = await fetch("https://api.github.com/user", {
    headers: {
      Authorization: `token ${user.access_token}`,
    },
    cache: "no-store",
  });
  const userdata = await response.json();
  return (
    <div>
      <h1
        style={{
          "text-align": "center",
          "margin-top": "100px",
        }}
      >
        Welcome {userdata.name} to the dashboard!
      </h1>
      <a href="/logout" className={buttonStyle.button}>
        Logout
      </a>
    </div>
  );
}

Once again, we will go through this step by step. The imports are largely the same as the callback. The only difference is the addition of ../button.module.css, which is the CSS file we created earlier for button styling.

This time, instead of a function named GET, because this is a page, we export a function named Page. Of course, because it is a default export in this case, the naming doesn’t really matter.

Next, we get the auth cookie. If no auth cookie is found, the user is not signed in, so we redirect the user to the sign in.

If there is an auth cookie, we verify the JWT contained in it. This is how we make sure the value was not tampered with. JWTs have a header, payload, and signature. The payload’s value can be accessed without the JWT secret. However, the JWT cannot be modified, as the signature is generated using the JWT secret and payload value. Therefore you need both of those to generate a valid signature. What we are doing here is both decrypting the payload and making sure the signature is valid.

Now that we have verified the JWT and extracted the payload, we use the access token in the JWT to request the user’s data from the GitHub API. The cache parameter disables Next.js’s AOT caching/static generation. Normally we would do this inside a useEffect() call, but because we are using React Server Components with Next.js, this is the idiomatic way of using fetch() (remember that this renders once on the server and does not render on the client period). We then parse the response body to get the user data.

After that, we just have some markup that renders a welcome message with the user’s name and a logout button. We use inline styles for the header because creating another CSS file just for two rules didn’t make sense. You might notice that the logout button links to /logout, which does not exist yet. We will fix that.

Tip: If you have lots of protected routes and want to avoid code repetition, you could implement the authentication as middleware or just a function imported in the relevant routes

Adding a logout route

This is pretty simple. The first thing to do is to create another route, just like what we did for /callback. Create a new folder in app called logout and a file called route.js in the new folder. Insert the following in the new file:

import { cookies } from "next/headers";
import { redirect } from "next/navigation";

export async function GET(req) {
  cookies().delete("auth");
  redirect("/");
}

As you can see, this script is pretty simple. We just intercept GET requests to /logout, delete the cookie named auth, and redirect the user to the homepage. Note that if the user tries to log in again, it might seem as if they are still signed in, as GitHub will silently send the access token and redirect the user when they click login. However, if you implement a more complete sign in solution, this will allow users to sign in to another account with a different method.

That is all! You now have a basic system for OAuth authentication. However, there are a few issues

Problems with this Implementation

While this works, there are some problems:

There is no protection against CSRF (Cross Site Request Forgery)
There are no refresh tokens, and access tokens can last indefinitely (this is a limitation of GitHub OAuth apps)
GitHub is currently the only sign in method (non OAuth sign requires many more security measures due to the need to store information in a database)
There is support if issues arise with authentication

We could solve most of these problems, but it would take a significant amount of time (this tutorial is already almost 3,000 words) and increase maintenance. However, there is a better solution: you can use a service like Clerk. Clerk is a user management platform that allows you to easily and securely implement most of the popular OAuth providers along with standard email/password sign in, magic link/passwordless, and even Metamask.

Using Clerk for Authentication

Once again, run:

npx create-next-app@latest

and select the same choices as previously (any name, everything “no” except for the app router). Then, run:

npm install @clerk/nextjs

This installs the Clerk SDK for Next.js. Now, we need to set up a Clerk app. Go to Clerk’s app creation page (create an account if you did not already) and go through the configuration. For authentication providers, you can stick to just GitHub again or add as many providers as you want. It doesn’t change the process, and you can change it later. After that, you should be redirected to a page where you see a snippet containing NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY and CLERK_SECRET_KEY. Copy this and paste it into a new file named .env.local. This includes the environment variables that store your public and private Clerk keys.

To allow us to use Clerk React components in our project, we first need to insert the <ClerkProvider> component into layout.js. Replace all content in layout.js with the following:

import "./globals.css";
import { Inter } from "next/font/google";
import { ClerkProvider } from "@clerk/nextjs";
const inter = Inter({ subsets: ["latin"] });

export const metadata = {
  title: "Create Next App",
  description: "Generated by create next app",
};

export default function RootLayout({ children }) {
  return (
    <html lang="en">
      <body className={inter.className}>
        <ClerkProvider>{children}</ClerkProvider>
      </body>
    </html>
  );
}

<ClerkProvider> provides the context necessary to allow Clerk components to function properly, and due to it being layout.js, it is inserted into every page.

Next, we need to create the sign in button, just like in the first version. Open page.js in app and insert this below the Next.js logo (the image with src="/next.svg"):

{
  auth().userId ? (
    <a href="/dashboard" className={buttonStyle.button}>
      Dashboard
    </a>
  ) : (
    <SignInButton mode="modal" redirectUrl="/dashboard">
      <button className={buttonStyle.button}>Sign In</button>
    </SignInButton>
  );
}

This checks if the user is signed in. If they are, it shows a link to the dashboard, and if not, it shows a button to sign in. To make this work, you will also need to import two things at the top of the file:

import { SignInButton, auth } from "@clerk/nextjs";
import buttonStyle from "./button.module.css";

The first line contains the functions and components needed from the Clerk SDK, and the second is for the styling. You will want the same styling as in the previous version, so create a new file named button.module.css and insert this into it:

.button {
  z-index: 2;
  margin: 10px;
  transition-duration: 0.2s;
  display: inline-block;
  font-weight: 600;
  border: 1px solid transparent;
  border-radius: 6px;
  white-space: nowrap;
  padding: 5px 16px;
  font-size: 14px;
  line-height: 20px;
  vertical-align: middle;
  cursor: pointer;
  user-select: none;
  color: #24292e;
  background-color: #fafbfc;
}

.button:hover {
  background-color: #f3f4f6;
}

.button:focus {
  outline: none;
  border: 1px solid #c0d3eb;
  box-shadow: 0 0 0 1px black;
}

.button:active {
  background-color: #edeff2;
}

Then, add this to .center in page.module.css

flex-direction: column;

Now sign in should work, but we still need to implement the dashboard. First, we need to create a JavaScript file at the project root named middleware.js (make sure to put this at the project root, not /app). Paste the following in the new file:

import { authMiddleware } from "@clerk/nextjs";
export default authMiddleware();

export const config = {
    matcher: ["/((?!.*\\..*|_next).*)", "/", "/(api|trpc)(.*)"]
};

This is a basic setup for the authentication middleware. You can configure it more to exclude specific routes, but this should work in most cases.

The next step is to create a folder called dashboard and a file named page.js inside it, just like with the implementation we already made. Inside the file, paste this:

import { currentUser, RedirectToSignIn, SignOutButton } from "@clerk/nextjs";
import buttonStyle from "../button.module.css";

export default async function Page() {
  const userdata = await currentUser();
  if (!userdata) {
    return <RedirectToSignIn />;
  }
  return (
    <div>
      <h1
        style={{
          "text-align": "center",
          "margin-top": "100px",
        }}
      >
        Welcome {userdata.emailAddresses[0].emailAddress} to the dashboard!
      </h1>
      <SignOutButton>
        <button className={buttonStyle.button}>Logout</button>
      </SignOutButton>
    </div>
  );
}

We will go through this step by step.

First, we import some packages.

We import multiple different functions and components from the Clerk SDK. The first, currentUser(), helps us get the current user’s information. The second, RedirectToSignIn, not to be confused with the function redirectToSignIn, which is also exported, is a component that redirects the user to a sign in page. Finally, we have SignOutButton, which creates a button to allow the user to log out.
buttonStyle is just CSS from our button.module.css file, which we use for the log out button.

Next, we export a function called Page, which has been explained previously.

After that, we use currentUser() to get the user’s information. If it does not exist (meaning the user is signed out), we redirect to a sign in page using a client side redirect.

Finally, using the userdata, we return markup containing a header with the user’s email and a button to log out.

Now, sign in, the dashboard, and sign out should all work.

Easy SSO

That’s it! You now should have two implementations of OAuth authentication, one that is hand built and more limited and one that is built with Clerk and more feature rich. If you want to learn more about Clerk, check out the Clerk Docs. I hoped you learned something, and thanks for reading!

Feature Engineering for Fraud Detection

Nick Parsons — Wed, 07 Dec 2022 17:40:00 +0000

Introduction

Fraud detection is critical in keeping remediating fraud and services safe and functional. First and foremost, it helps to protect businesses and individuals from financial loss. By identifying potential instances of fraud, companies can take steps to prevent fraudulent activity from occurring, which can save them a significant amount of money. Fraud detection also saves users a lot of headaches and instills trust in them when they know these protections are in place.

There is a wide variety of applications under the broader umbrella of fraud/risk detection — credit card transaction fraud, payment fraud, identity fraud, account takeover, etc. All of these types of fraud or risk can be broadly categorized into two types — first-party fraud (i.e., fraudulent users trying to trick a bank) or third-party fraud (i.e., a bad actor who is trying to compromise the account of a legitimate user). Most of these applications have a similar structure when it comes to ML and feature engineering, which we will explore in this post. First, we will briefly describe how the modeling problem is formulated and how the labels are obtained. And finally, we will do a deep dive into feature engineering for fraud.

Modeling

The fraud detection problem is often modeled as a supervised learning-based binary classification, and Gradient Boosting Decision Trees (GBDTs) are often used as the model. GBDTs work by training a service of decision trees on the data, where each tree is built to correct the mistakes made by the previous trees in the sequence.

In the case of fraud detection, the features input into the GBDTs might include information about the transaction, the customer involved, and any other relevant financial or behavioral data, and whether training data includes labels that indicate if a transaction is fraudulent or not.

The GBDT algorithm uses this information to identify patterns and relationships, even complex non-linear relationships, that are indicative of fraudulent activity. By training the model on a large dataset of known fraudulent and non-fraudulent transactions, the GBDT algorithm can learn to make highly accurate predictions about the likelihood of a given transaction being fraudulent.

Fraud detection often involves working with large and highly unbalanced datasets, where the number of fraudulent transactions is relatively small compared to the number of non-fraudulent transactions. This can make it difficult to identify the features that are most relevant and useful for training the model. To mitigate this, there are a few approaches:

Oversampling the fraudulent transactions or undersampling the legitimate transactions - this can balance out the dataset, but can also lead the model to believe that fraud is more frequent than it is.
Changing the metrics used to evaluate the model – you can use metrics like precision, rather than accuracy, or use the F1 score to evaluate how well the model is performing.
Using algorithms like ensemble methods or cost-sensitive learning – Both of these algorithms can help improve the performance of the machine learning model on the minority class. GBDT (which itself is an ensemble) is a great algorithm for fraud detection because it is powerful and flexible, and relatively easy to use and interpret, but some other ensemble methods are particularly effective at handling imbalanced datasets, and cost-sensitive learning is particularly adept at considering the costs associated with false positives and false negatives.

Labels

Since the problem is formulated as supervised learning, ground truth labels are needed to train the model — in this case, whether transactions as fraudulent or non-fraudulent.

In some problems, e.g., credit card transaction fraud, if a transaction was incorrectly classified as non-fraudulent and permitted to happen, the end user may dispute the charge. That can serve as good quality labels. But in general, obtaining high-quality labels in a timely fashion is a huge problem for fraud detection, unlike recommendation systems where what the user clicked on can easily serve as good-quality labels.

As a result, typically, humans often need to be involved to manually provide the ground truth labels along with a bunch of automated augmentation/correction mechanisms.

Another challenge with labels is that there is a delay in receiving labels. There is typically a 2-3 week delay in processing fraudulent activity on an account. In some problems, like account takeovers, malicious actors may wait months before starting to use the accounts to say post spam or conduct transactions. In such cases, the system may not find out the correct label of a decision for account fraud prediction for many months. In general, it can take around 6 months to get 95% of all labels which means models have to be trained on several months old data at any point.

Feature Engineering

In order to create effective features for fraud detection, it is necessary to understand the underlying data and the problem that the model is trying to solve (first or third-party fraud, as well as the types of fraud therein). This often involves a combination of domain expertise, data exploration, and experimentation to identify the most relevant and informative features to use as input (often accomplished through exploratory data analysis, or “EDA”). Once these features have been identified, they may need to be transformed or combined in order to make them more useful for training a machine learning model (also included in EDA). This process of feature engineering is crucial for achieving good performance in fraud detection and other machine learning tasks.

One of the challenges with feature engineering for fraud in particular is that, like with labels, much of the most useful features need to be handwritten. There are some features that can be used from the raw data, but, more often than not, the most useful features are ones that combine several different pieces of data, like the amount and the location and time of day (e.g., someone is likely to spend more money on dinner in a high cost-of-living area than they are to spend at a coffee shop in a low cost-of-living area) and/or splitting the values for a specific piece of data (like splitting the date from the time or the dollars from the cents of a transaction amount).

Some of the most common patterns of features for fraud detection include:

Affinity features built on top of rolling counter-based features — how many times an event of some kind happened in rolling windows (e.g., how many times a user has made an ATM withdrawal in this city in the last 6 months or the average transaction amount for a user with a particular credit card).
Velocity features — how quickly events happen (e.g., the ratio of the purchases the user made in the last hour to the average number of transactions they made per hour in the last 30 days). These are also ratios of some counters, even though the focus is on capturing the velocity of the actions taken vs. the affinity between two entities.
Reputation features — some “reputation” score for various things like the email domain of the user, the IP address the activity is coming from, the vendor the purchase was made from, etc. Some of these are, again, using counters of past behavior.
External API-based features — e.g., the credit score of the user or location of an IP, etc.
Relatively static profile features — e.g., the zip code from which the request originated or the age of the user’s account, etc.

As mentioned above, determining which of these features is most useful involves a lot of industry knowledge and experimentation (since the normal log and wait process would require you to wait months for results, due to the nature of the data being received and the rate a which it is received). When doing experimentation via exploratory data analysis (EDA), the engineer uses different statistics and visualizations of the data to find the best data points and combinations by looking for patterns, anomalies, and relationships between data (which can also inform the engineer that only one piece of data from the relationship needs to be used).

Note that even though fraud models are trained on very old training data, realtime ML still gives you a massive leg up in fraud detection because of its adeptness with long-tail, as referenced in this post.

Unique Feature Engineering Challenges

Feature engineering for fraud detection can be challenging for a number of reasons. As was mentioned before, fraud is a complex and dynamic problem that can take many different forms, which makes it difficult to identify the specific patterns and relationships that are indicative of fraudulent activity.

Fraud detection often requires working with sensitive financial data, which may have strict privacy and security requirements. This can make it difficult to obtain the data that is needed for feature engineering and can also limit the types of transformations and manipulations that can be performed on the data. When working with sensitive data, your best chances of success are to:

Ensure data is properly secured – to protect against unauthorized access, implement security measures such as encryption, access controls, and monitoring.
Consider which data is actually necessary – during the feature engineering stage, using tactics such as exploratory data analysis (EDA) helps you to see if any of the features in consideration have relationships with others, which can allow you to select the less sensitive metric for the model that will be put into production.
Anonymize the data as much as possible – your use of sensitive data is more likely to be allowed to continue if proper protections are put in place. The first layer of ensuring this is making sure only the people who should have access to the data do, as mentioned in the first bullet. It is also a good practice to anonymize data within your system, in case of situations such as compromised credentials being utilized by bad actors.

Another challenge is that, as previously mentioned, labels are often delayed, so training data consists of very old examples (sometimes several months old). This means that, if you were to employ the popular practice of logging and waiting, you would need to wait months to test a new feature. This is why a practice such as EDA is often used. Another “issue” with labels being delayed is that point-in-time data feature reconstruction needs to cover a long window (often times a year) to get the necessary data volume. This also increases the risk of feature code changing, which further necessitates immutability or versioning of features to prevent confusion and ensure that you’re working with the correct data, in the correct format.

Earlier, we mentioned that it is especially common to create features for fraud detection models by combining or splitting the features you receive from the raw data. This often means that you need more data sources to get enough data of the types you need, which increases the need to hit external APIs. Sometimes, this comes in the form of batched data dumps (like the credit scores for a large number of people as of a certain date) that need to be “merged” with live requests (like to a credit bureau) in a lambda-like architecture. With live requests, it is also important to consider the latency constraints mandated by the system or use case. In most cases with fraud detection, it is very sensitive to lag, since a bad actor can do a lot of damage in even a few seconds. This means that features must be only a few seconds old, if that. In addition, serving latencies of the models using those features need to be incredibly low; often, a serving latency of around 200ms end-to-end (including feature extraction and model scoring) is required to make a decision as to whether a transaction is fraud in an acceptable amount of time.

Another reason that working with external APIs (e.g., hitting credit bureau APIs for credit scores) is hard is that it can be difficult to create correct point-in-time values; since the data is external, you don’t always have easy access to historical data or the details you need from it, meaning you need to ensure you record all of the right information when you first access the data. For example, we mentioned that the IP address is a feature that can be used to predict fraud; one issue with this is that the location of an IP changes over time, so you need to know the location of the IP at the time of training, which mandates careful managing of the data obtained from external APIs, so that you have the details you need when you need them in the future.

Final Thoughts

Feature engineering is a complex artform, and the precision and thoughtfulness that needs to go into it when working with sensitive data and high stakes, like in the case of fraud detection, further complicates its planning and implementation. While there are a good number of things to account for when performing feature engineering for fraud detection, this article helps to highlight many of the challenges, technologies, and strategies that can be employed throughout this process to give you a leg up when going through the process.

If you would like to see an example of how these strategies can be employed, NVIDIA did an awesome writeup of the winning solution from Kaggle’s IEEE CIS Fraud Detection competition.

For more high-quality technical content on all things machine learning, visit the Fennel Blog.

Build an iOS 1-on-1 Video Chat App with SwiftUI and Dolby.io

Nick Parsons — Wed, 27 May 2020 18:59:03 +0000

In this tutorial, we'll integrate video chat into an iOS application. To do this, we integrate Dolby.io's Interactivity APIs, formally known as Voxeet, into our application. Video chat can easily be integrated with Stream Chat for a seamless communication experience.

Note: the library is still named Voxeet.

For this part, the application will support 1-on-1 private chat. Since Dolby is a pure client-side library, we only configure our ios application. However, to facilitate the UI for indicating whether a user has a call waiting, we use a few endpoints in the backend.

Note: Because these are minor and largely stub implementations, we don't go into them in this tutorial. Please refer to the source code on GitHub if you're curious. Also, ensure the backend is running if following along with this tutorial.

The app performs these steps:

Initialize Voxeet libraries
When a user navigates to the "People" screen, show a video icon next to a user's name.
When a user clicks this icon, start and join a Voxeet conference with a unique alias. The user waits for the other party to join. The application informs the backend of the new call.
When the other user enters following the previous steps, they'll be placed in a 1-on-1 conference.
When either user leaves, the call is ended in the application. The backend is informed of the call ending.

Voxeet's UXKit takes care of the connection and presentation for this 1-on-1 call. Our application needs to create and join the call, and the UXKit will overlay the video call UI.

Let's dive in!

Step 1: Create a Dolby Account and Install Voxeet Dependencies 🔗

Go to dolby.io and create an account. Once registered, navigate to the Dashboard (you can click in the top-right if you're not there). Find the list of applications:

If you already have an app ("my first app"), click into it. If you don't, create an app by hitting "Add New App". Now grab the API key and secret for our application. The keys we need are under the "Interactivity APIs" section. :

Click on "Show Secret" to view the secret.

Next, add VoxeetUIKit to our Podfile and pod install:

Note: Since the WebRTC dependency is larger than GitHub's file limit, we did not include the Pods in the repo. Please ensure you run pod install before attempting to run!

Step 2: Configure the Voxeet UXKit Library 🔖

Since the Voxeet library is not tied to a backend user account, we can configure it on the application load. We do this in the AppDelegate:

Change <VOXEET_CONSUMER_KEY> and <VOXEET_CONSUMER_SECRET> to the values you retrieved in Step 1. We configure Voxeet not to have any push notifications, turn on the speaker and video by default, and appear maximized. We also set telecom to true. When this is set, the conference will behave like a cellular call, meaning when either party hangs up or declines the request (decline not implemented in this tutorial), it will end the call.

If you'd like to use push notifications to notify a user when a call is incoming, check out CallKit combined with VoxeetSDK.shared.notification.push.type = .callKit. This is out of scope for this tutorial.

Step 3: Starting a Call 📱

Next, we add a video action to the list to people, in between the start chat and follow icons from previous parts of this series. Here's what our view will look like:

To do this, we just add another image view in our ListView in PeopleView:

This is a simple system icon that we load via systemName. With the icon in place, we can add a click handler via onTapGesture to start our 1-on-1 call. Ignore the icon foregroundColor call for now. We'll get to that in a bit.

Let's look at startConferenceCall:

Here we create our conference call using Voxeet with an alias. We use this alias as an identifier, so the other user's application knows how to join the same call. The call to .create yields us a conference object. First, we call our backend via startCall to register the call, so the other user knows there's a call waiting. This is simply a POST request:

Note: We use Voxeet's conference implementation as it's perfect for facilitating a video chat between two people. The conference object is more powerful than this (multiple users, broadcast streams, screen sharing, etc.), but here only use it for a 1-on-1 call. The terms conference and calls are used interchangeably in this tutorial, given the scope of our application.

Once we've notified the backend of the call, we join the conference via .join. Since we're using Voxeet's UXKit a video chat UI slides up from the bottom automatically:

Step 4: Joining a Call 🤳

Now that the user has started a call with someone, we want the other user to see a call started so they can join. To keep things simple, we just turn the video icon red if there's a call active. Recall from above that we are changing the video icon color via foregroundColor via a call to .videoIconColor:

Here we'll check a @State var calls for a call from the other user. If we do find one, we color the icon red. The calls var gets initialized when PeopleView appears via fetch:

We call to account.fetchCalls to retrieve a list of active calls for the current user:

This is simply a GET request against our mock backend (refer to source). The response object will have a from field, indicating who the user started the call.

Now that we know a user is calling the user will see a screen like this:

If the user joins the call, the UXKit UI will change to show the call has started:

Step 5: Leaving a Call ☎️

When a user is done, they hang up using the end call icon. We don't need to do anything special on our side to end the call, Voxeet takes care of that. We need to listen for the conference call ending so we can notify the backend of the change.

Note: In a production application, you'd likely want to use Voxeet's push notifications and CallKit bindings or look at binding Dolby's Interactivity API WebSockets to the backend. While the approach below works, it is less robust than using the full Notification system built into Dolby's Interactivity APIs.

We'll bind to the notification center and listen for the conference call to end. Upon ending, we'll notify the backend the call has finished:

We use SwiftUI's convenient .onReceive method to bind to the NotificationCenter event .VTConferenceDestroyed. When this happens, we know the call has finished. Since we configured VoxeetUXKit.shared.telecom = true this coincides on both sides when either party hangs up. When we get the notification, we simply call to the backend to stop the call via account.stopCall and fetch the new set to refresh the view (effectively removing the red video indicator). The stopCall is a simple DESTROY call to the backend:

And we're done! We now have a 1-on-1 chat between two parties integrated into our application using Dolby's Voxeet offering. For more information on how to integrate Stream Chat within your application, have a look at the GitHub repo.

Happy coding! 🤓

Encrypted Chat on iOS (Swift)

Nick Parsons — Fri, 08 May 2020 18:52:22 +0000

In this tutorial, we'll build encrypted chat on iOS using Swift. We'll combine Stream Chat and Virgil Security. Both Stream Chat and Virgil make it easy to create a solution with high security with all the features you expect. These two services allow developers to integrate chat that is zero-knowledge. The application embeds Virgil Security's E3Kit with Stream Chat's Swift components.

Note: All source code for this application is available on GitHub.

What is end-to-end encryption?

End-to-end encryption means that two people can have trade messages via the internet without anyone else being able to read them, even if the transmission or storage is compromised. To do this, the app encrypts the message before it leaves a user's device, and only the intended recipient can decrypt the message.

Virgil Security is a vendor that allows us to create end-to-end encryption via public/private key technology. Virgil provides a platform that allows us to create, store, and offer robust end-to-end encryption securely.

During this tutorial, we will create a Stream Chat app that uses Virgil's encryption to prevent anyone except the intended parties from reading messages. No one in your company, nor any cloud provider you use, can read these messages. Even if a malicious person gained access to the database containing them, all they would see is encrypted text, called ciphertext.

Building an Encrypted Chat Application

To build this application, we'll mostly rely on a few libraries from Stream Chat and Virgil (please check out the dependencies in the source to see what versions). Our final product will encrypt text on the device before sending a message. Decryption and verification will both happen in the receiver's device. Stream's Chat API will only see ciphertext, ensuring our user's data is never seen by anyone else, including you.

To accomplish this, the app performs the following process:

A user authenticates with your backend.
The iOS app requests a Stream auth token and API key from the backend. The Swift app creates a Stream Chat Client for that user.
The mobile app requests a Virgil auth token from the backend and registers with Virgil, which generates their private and public key. The app stores the private key locally, and the public key in Virgil Cloud.
Once the user decides who they want to chat with, the app creates and joins a Stream Chat Channel.
The app asks Virgil's API, via E3Kit, for the receiver's public key.
The user types a message and encrypts it with the receiver's public key using E3Kit, then sends it to Stream. After that, Stream Chat relays the message to the receiver. Stream only receives ciphertext, meaning they will never see the original message.
When the message is received, the app decrypts and verifies it is using E3Kit and passes it along to Stream Chat's iOS components for display.

While this looks complicated, Stream and Virgil do most of the work for us. We'll use Stream's out-of-the-box UI components to render the chat UI and Virgil to do all of the cryptography and key management. We simply combine these services.

The code is split between the iOS frontend contained in the ios folder, and the Express (Node.js) backend is found in the backend folder. See the README.md in each folder to see installing and running instructions. If you'd like to follow along with running code, make sure you get both the backend and ios running before continuing.

Let's walk through and look at the critical code needed for each step.

Prerequisites

Basic knowledge of iOS (Swift) and Node.js is expected. This code is intended to run locally on your machine.

You will need an account with Stream and Virgil. Once you've created your accounts, you can place your credentials in backend/.env if you'd like to run the code. You can use backend/.env.example as a reference for the required credentials. Please see the README in the backend directory for more information.

Step 0. Set up the Backend

For our Swift app to securely interact with Stream and Virgil, the backend provides four endpoints:

POST /v1/authenticate: This endpoint generates an auth token that allows the iOS application to communicate with the other endpoints. To keep things simple, this endpoint enables the client to be any user. The frontend tells the backend who it wants to authenticate as. In your application, this should be replaced with real authentication appropriate for your app.
POST /v1/stream-credentials: This returns the data required for the iOS app to establish a session with Stream. In order return this info we need to tell Stream this user exists and ask them to create a valid auth token:

The response payload has this shape:

apiKey is the Stream account identifier for your Stream instance. It is needed to identify what account your frontend is trying to connect with.
token JWT token to authorize the frontend with Stream.
user: This object contains the data that the frontend needs to connect and render the user's view.
- POST /v1/virgil-credentials: This returns the authentication token used to connect the frontend to Virgil. We use the Virgil Crypto SDK to generate a valid auth token for us:

In this case, the frontend only needs the auth token.

GET /v1/users: Endpoint for returning all users, which exists just to get a list of people to chat with. Please refer to the source if you're curious. Please note the backend uses in-memory storage, so if you restart, it will forget all of the users.

Step 1. User Authenticates With Backend

The first step is to authenticate a user and get our Stream and Virgil credentials. To keep thing simple, we have an insecure form that allows you to log in as anyone:

This is a simple form that takes any arbitrary name, effectively allowing us to log in as anyone (please use an appropriate authentication method for your application). First, let's add to Main.storyboard. We add a "Login View Controller" scene that's backed by a custom controller LoginViewController (to be defined). This controller is embedded in a Navigation Controller. Your storyboard should look something like this:

The form is a simple Stack View with a username field and a submit button. Let's look at our custom LoginViewController:

The usernameField is bound to the storyboard's Username Field, and the login method is bound to the Login button. When a user clicks login, we check if there's a username and if so, we log in via Account.shared.login. Account is a shared object that will store our credentials for future backend interactions. Once the user logs in, we initiate the UsersSegue, which boots our next scene. We'll see how this is done in a second, but first, let's see how the Account object logs in.

Here's how we define Account:

First, we set up our shared object that will store our login state in the authToken and userId properties. Note, apiRoot is the IP address where our backend is running. Please follow the instructions in the backend to run it. Our login method uses Alamofire (AF) to make a post request to our backend with the user to log in. Upon success, we store the authToken and userId, and then we call setupStream.

The method setupStream initializes our Stream Chat client. Here's the implementation:

We call to the backend with our credentials from login. We get back a token, which is a Stream Chat token. This token allows our mobile application to communicate directly with Stream without going through our backend. We also get an apiKey, which identifies the Stream account we're using. We need this data to initialize our Stream Client instance and set the user. Last, we initialize Virgil via setupVirgil:

This method requests our Virgil credentials and configures a VirgilClient class we defined, which wraps Virgil's E3Kit library. Once that's done, we call the completion to indicate success and allow the application to move on.

Let's see what the beginning of our VirgilClient implementation looks like:

VirgilClient is a singleton that is set up via configure. We take the identity (which is our username) and token, and then we generate a tokenCallback. The EThree client uses this callback to get a new JWT token. In our case, we've kept it simple by just returning the same token, but in a real application, you'd likely want to replace this with the rest call to the backend.

We use this token callback and identity to initialize eThree. We then use this instance to register the user.

Now we're set up to start chatting!

Step 2: Listing Users

Next, we'll create a view to list users. In the Main.storyboard we add a UITableView and back it by our custom UsersViewController:

And here's the first few lines of UsersViewController:

First, we fetch the users when the view loads. We do this via the Account instance configured during login. This action simply hits the /v1/users endpoint. Refer to the source if you're curious. We store the users in a users property and reload the table view.

Let's see how we configure the table cells:

To render the table correctly we indicate the number of rows via users.count and set the text of the cell to the user at that index. With this list of users, we can set up a click action on a user's row to start a private 1-on-1 chat with them.

Step 3: Starting an Encrypted Chat Channel

First, add a new blank view to Main.storyboard backed by a new custom class EncryptedChatViewController (we'll see its implementation in a minute):

Add a segue between from the user's table row to show the new controller:

With that setup, we can hook into the segue via the UITableViewController's prepare method:

Before transitioning to the new view, we need to set the view controller up. We generate a unique channel id using the user ids. We initialize a ChannelPresenter from the Stream library, set the type to messaging, and restrict the users. We grab the view controller from the segue and back it with the ChannelPresenter. This will tell the controller which channel to use. We also tell it what users are communicating.

Let's see how the controller creates this view:

Luckily, Stream comes with excellent UI components out of the box. We'll simply inherit from ChatViewController to do the hard work of displaying a chat. The presenter we set up tells the Stream UI component how to render. All we need to do now is hook into the message lifecycle to encrypt and decrypt messages on the fly.

Step 4: Sending an Encrypted Message

Now we're ready to send our first encrypted message. Since we're using Stream's built-in UI, all we need to do is hook into the message sending cycle. We'll do this via the messagePreparationCallback on the ChannelPresenter:

Upon loading the view, we set up the callback, which is a hook provided by Stream, to make any modifications we'd like to the message before sending it over the wire. Since we want to encrypt the message fully, we grab it and modify the text with the VirgilClient object.

For this to work, we need to look up the public key of the other user. Since this action requires a call to the Virgil API, it's asynchronous. We don't want to do this during message preparation, so we do it ahead of time via VirgilClient.shared.prepareUser(otherUser!). Let's see how that method is implemented:

This is relatively simple with Virgil's E3Kit. We find the user's Card which stores all of the information we need to encrypt and decrypt messages. We'll use this Card in the encrypt method during message preparation:

Once again, this is made easy by Virgil. We simply pass the text and the correct user card to authEncrypt, and we're done! Our message is now ciphertext ready to go over the wire. Stream's library will take care of the rest.

Step #5: Decrypting a Message

Since we're using the ChatViewController to render the view, we only need to hook into the cell rendering. We need to decrypt the message text and pass it along. We override the messageCell method:

We make a copy of the message and set the message's text to the decrypted value. In this case, we need to know if it's our message or theirs. First, we'll look at how to decrypt ours:

Since the message was encrypted initially on the device, we have everything we need. We simply ask Virgil E3Kit to decrypt it via authDecrypt. Decrypting the other user's messages is a bit more work:

In this case, we use the same method, but we pass the user card that we retrieved earlier, which verifies the message's authenticity. Now we can see our full chat:

And that's all! We now have an application that uses end-to-end encryption to protect a user's conversation. Happy coding!

Learn How Dubsmash Powers Millions of Users with Stream

Nick Parsons — Thu, 16 Apr 2020 16:37:31 +0000

Our team had a chance to sit down with Tim Specht, Co-Founder and CTO at Dubsmash. If you’re not familiar with the organization, Dubsmash is a company that provides a social platform for users to share videos via their mobile applications. Users can choose an audio recording or soundbite from TV shows, movies, music, and other internet trends and record a video of themselves dubbing over that clip of audio, which can then be posted and shared.

Market Differentiators

Dubsmash’s differentiating factor from other platforms on the market is its focus on creators. The format, features, and functionality of the app keep the creators of the content in mind instead of curating everything toward the consumer due to Dubsmash’s prioritized belief that creators deserve attention. The technology is geared toward helping creators make compelling content so that they can keep engagement up within their followership. Dubsmash hosts a program for artists and creators so that they can boost their careers following their passions- this initiative proves that Dubsmash is dedicated to giving back to its community.

Additionally, Dubsmash has always and will always value the diversity of all kinds. It has been revealed that one of Dubsmash’s main competitors, TikTok, has been explicitly modifying its content not to promote posts featuring unattractive people or poor neighborhoods in their ‘For You’ tab of ‘Recommended Videos.’ TikTok revolves its algorithms around elements that will generate the most amount of engagement, like ranking videos featuring the traditional standard of beauty or clean, modern homes as backdrops to grab society’s attention in the fastest, easiest way possible.

At Dubsmash, there is not nearly as much censorship so that everyone has a fair shot at getting their ideas, art, and content out there, regardless of how someone looks or where they're filming their videos. One advantage of not being TikTok is that the app feels less crowded by semi-pro creators and influencers. That gives users the vibe that they’re more likely to hit the Trending or Explore page on Dubsmash. The Trending page is dominated by hot new songs and flashy dances, even if they’re shot with a lower production quality that feels accessible. Dubsmash gives space for more opportunity by making its 'Explore' page about discovering accounts and all the content they’ve made rather than specific videos. On Tiktok, popular clips may have multi-million views while on Dubsmash, there are only tens of thousands. However, there is still enough visibility to make shooting on Dubsmash worth it, especially for lesser-known artists who want to be discovered.

Dubsmash Market Share

Dubsmash has captured 27% of the U.S. short-form video market share by installs, second to TikTok, which pulls 59%. Dubsmash holds 73% of the U.S. market outside of TikTok as far as active users go, compared to just 23% on Triller, 3.6% on Firework, and 0% on Facebook’s Lasso. Dubsmash has three times as many active users and saw 38% more first-time downloads in 2018 than 2019. 30% of Dubsmash’s daily users are creating content, resulting in 30% month-after-month retention.

Where is Dubsmash Hosted?

Dubsmash uses a hybrid approach to hosting their application. Because the app is fundamentally a mobile application for iOS and Android devices, only the backend, and backend related services need to be hosted. For this, Dubsmash relies heavily on Heroku’s Container Service – all Docker images are stored in Quay, allowing for near real-time rollbacks in the event that something goes wrong.

Heroku allows Dubsmash to scale both horizontally and vertically in an infinite way without having to worry about employing a large number of backend infrastructure engineers. Simply build, push, and deploy.

Aside from Heroku, Dubsmash makes heavy use of AWS as their primary service provider with GCP coming in second due to their robust cloud data warehouse – BigQuery. According to Tim Specht, Co-Founder and CTO at Dubsmash, “If AWS offers a service, Dubsmash likely uses it to some extent.”

GCP’s robust cloud datastore BigQuery offers insight into user data in a fraction of the time that their previously homegrown solution which utilized AWS Redshift, among other tooling. “What would normally take hours of query time can now be done effectively and cost-efficiently in seconds using GCP’s BigQuery, allowing the team at Dubsmash to increase overall productivity and user experience.”

What Type of Infrastructure Does Dubsmash Utilize?

Daily, Dubmash utilizes dozens of third-party services from various vendors to ensure optimal uptime and visibility into user-based metrics.

For example, Celery is heavily used as a distributed task queue for video decoding/encoding and used to process millions of tasks daily. Memcached and Redis are used for caching, allowing for snappy load times; whereas Airflow is used for complex workflows such as analytical reporting and notifications, etc.

The most important question that we asked is what database of choice Dubamash is using and why? The answer was sweet and simple – PostgreSQL. The reasoning behind this is due to the many 1:1 relationships that Dubsmash implements. In contrast, a database such as MongoDB wouldn’t make as much sense due to the lack of 1:N relationships and primary/secondary setup. The reads and writes to Postgres are efficient, as are queries. Postgres works excellent for our use-case, and we (Dubsmash) stand by our decision to use it as our core database.

While this is not an exhaustive list of the various infrastructure that Dubsmash uses daily, it should give you a good idea of how Dubsmash operates.

Primary Coding Languages Used In Production

Python is the primary language of choice at Dubsmash; however, mobile applications are written in native languages. Currently, iOS is utilizing Swift, whereas the Android app is undergoing a migration from Java to Kotlin.

Why Docker?

Docker has been a fantastic product that we heavily utilize here at Dubsmash. Docker allows our developers to not only onboard themselves quickly and efficiently but not have to have in-depth knowledge and understanding of every part of our infrastructure’s stack. For example, our infrastructure relies on Python3, Django, PostgreSQL, Memcached, Celery, and Redis, to name a few.

Without having Docker and Docker Compose at our disposal, our engineers would be wasting precious time and valuable resources bringing themselves up to speed with the various products we utilize, best practices, setup, and cleanup scripts, etc.

“Docker has been a game-changer for our team here at Dubsmash.”, according to Tim.

Third-Party Services Used

Analytics

Dubsmash rarely builds in-house products as they prefer to buy vs. build. However, under certain circumstances, requirements come into play, and you have to make business decisions that ultimately move your business and engineering team forward.

With this in mind, Dubsmash played around with AWS Kinesis, only to find that the product failed to meet their expectations from an analytics and cost perspective.

Instead, Dubsmash dumps all of their data into GCP’s BigQuery, allowing the team to exercise automated and ad-hoc queries on the fly in a matter of seconds with a simple and easy to use query language. Depending on the type of query, and need for the query, the resulting data pulled from BigQuery will end up in Google’s Data Studio for reporting purposes, or tossed into a workflow that Airflow will take over (e.g., user push notifications, etc.).

Realtime Search

During the early stages of Dubsmash, the engineering team played around with several pieces of technology to build a home-grown search engine for real-time user search capabilities. After many wasted hours and a lot of money spent on hosting Elasticsearch, the team decided to switch their mindset and leave it to the experts at Algolia.

Algolia focuses 100% on scalable search, offering lightning-fast query response times compared to that of Elasticsearch.

According to Dubsmash, Elasticsearch had a complicated and proprietary query language, offered poor performance when it came to indexing and reindexing data, and was an overall pain to keep up and running efficiently. With Algolia, Dubsmash was able to cut their time spend on search by a fraction and focus on what matters most – user experience.

Feeds as a Service

At the core of Dubsmash lies Stream, the primary driving force for the Feeds infrastructure users rely on to discover content within the application.

The Co-Founder and CTO admitted that they had been looking at Stream for a while, and even attempted to build an in-house feed service, but failed when it came to overall cost and performance.

“Building a custom in-house infrastructure to power feeds, accompanied with ranking (weights), speed, reliability, and something cost-effective is a true challenge, and we thank Stream for everything that they do.” - Tim Specht

By utilizing Stream's Feeds Infrastructure, Dubsmash can scale their service to an infinite number of users while having the ability to make real-time tweaks to sorting and weighting, time decay, etc., while having peace of mind that their application won’t come to a halt due to hiccups that may have come up in a home-grown solution.

Transactional Email

Emails aren’t used all that often with Dubsmash as they rely heavily on SMS and Push for notifications. However, when emails are sent, AWS Simple Email Services (AWS SES) is used.

SMS Notifications

SMS is the primary form of communication that Dubsmash utilizes to interact with its users (aside from Push Notifications). To establish a reliable SMS workflow, Dubsmash uses AWS Simple Notification Service (AWS SNS) to send messages to users.

Push Notifications

Push notifications are how Dubsmash connects with their uses. For example, if a user hasn’t logged into their application in a couple of days, Dubsmash will trigger a push notification to the user to re-engage the user and bring them back into the application.

At times, Dubsmash has had to send several million push notifications in a single go, sometimes peaking 16+ million push notifications a day. This is hard on any type of infrastructure due to the hardware and network requirements behind the scene. To accomplish this, Dubsmash utilizes a combination of AWS Lambda and AWS SNS to send notifications to users in an efficient method.

To accomplish this scale, the engineers at Dubsmash have devised a method to send batched requests to an AWS Lambda instance, which then divides the total number of messages into batches of 500. From those batches of 500 messages, AWS Lambda instances are provisioned on the fly (coded in Python for little overhead) and sent via AWS SNS.

Image Hosting

Similar to many other applications, Dubsmash uses AWS S3 to store images in the cloud. From there, images are delivered to the client via AWS CloudFront (Amazon's Web Services CDN).

Video Streaming

Streaming video has always been a challenge for many applications. Luckily, Dubmash knows this problem all too well as they specialize in streaming videos to user's devices.

If you have any background in streaming video, you’ll know that it’s a hard problem to solve. There are many constraints on file sizes, file types (e.g., HLS, MPEG-dash), and even enforcements that are put in place by the user's end browser. To get around these constraints, Dubsmash stores all videos at MP4 files on AWS S3 and streams them via AWS Cloudfront to the user's devices and force MP4 (rather than utilizing adaptive streaming).

By doing so, this ensures that there is no downgrading of the video, enforcing the best user experience possible– a core value of Dubsmash.

Future Vision

Dubsmash is always looking ahead in terms of how to grow and evolve as a social media platform. Here are some of their goals for the future:

Build more tools dedicated to the content creators. Dubsmash wants to create more opportunities and flexibility for artists so that they can make high-engagement products for their audiences in creative ways.
Grow user-base inclusively and fairly.
Increase personalization and customization. Dubsmash wants to curate the most relevant content for each user, making proper and attractive recommendations based on app-activity.
Continue to make Dubsmash a fun experience for both creators and consumers
Future features: Filters, speed controls, time stickers

Final Thoughts

A lot of complex, technical, detailed work goes into this app behind the scenes to create a simple, clean, straightforward user experience. Dubsmash will continue to differentiate themselves by positioning their app as a legitimate tastemaking and community platform, as long as a utility for content-creators. Dubsmash is now up to one billion video views per month, and this number will likely keep growing as the team keeps innovating and building.

Migrate from Pusher Chatkit to Stream Chat

Nick Parsons — Thu, 02 Apr 2020 16:23:22 +0000

As previously mentioned in a blog post by Stream, Pusher recently announced its intention to shut down their real-time messaging service, Chatkit, effective April 23rd, 2020, to narrow its product focus to Channels and Beams.

Although Pusher Chatkit fulfilled the basic premise of providing a real-time chat solution, their feature set and capabilities were way behind what the competition and Stream Chat in particular offers, which is why they were ultimately unable to compete.

The abrupt decision has disrupted critical services for all customers on the Chatkit platform. Unfortunately, it's now up to those customers to scramble to migrate off the Pusher Chatkit platform to another provider.

The good news is that it’s possible and easy to move your data off Pusher Chatkit and over to Stream Chat, and many Chatkit users have been onboarded successfully to the Stream platform.

If you’re curious about how Pusher Chatkit stacks up to Stream Chat, check out this detailed comparison post. Our team has also compiled a detailed migration guide which is hosted on GitHub for your convenience.

How Stream Enhances Your Chatkit App

Both Stream and Chatkit provide you with the basic features of chat: one or one or group messaging, user accounts, typing indicators, unread message counts, and more, as well as backend infrastructure. But that’s about where the similarities end. Stream Chat is far more extensible and easier to use compared to Chatkit, as you’ll see.

Ready-Made Frontend Components

With Chatkit, you are required to roll a custom interface and implement all of the features that you required within your application. As you may know or are imagining as a developer, this is much work. The process is resource and time-intensive, which results in a large sum of money required to build on top of Chatkit. Stream Chat, on the other hand, provides ready to go components that allow you to build a fully functional real-time messaging application in a matter of hours.

By using Stream components, you get many basic and advanced features in your app with minimal effort. Features include mentions, slash commands, reactions, threads, gifs, emoticons, file attachments, rich link previews, and much more. You can also decide to roll out a custom interface using the Stream Chat JS client.

Reliable Infrastructure & Scalability

While Pusher's Chatkit product only supports up to 500 members in a single channel, Stream Chat has zero limitations and supports any number of participants in a single channel. Stream Chat has been proven to be highly scalable, with over 500 million users on both its chat and feed APIs. And with an industry-leading 99.99% uptime record and response times as low as 5ms, you are guaranteed stability and robustness when you build on the Stream platform.

Enterprise Ready

Compared to Chatkit (or anyone else in the industry for that matter), Stream is better equipped to support enterprise customers who have unique requirements due to their scale and risk profiles.

Steps to Migrate from Chatkit to Stream

Regardless of your applications scale or active user base, you can migrate to Stream seamlessly in a matter of days and suffer zero data loss. Stream is well experienced with migration procedures haven handled countless migrations for small and large scale customers alike, including when Layer shutdown some time ago.

A detailed migration guide is now available on GitHub and provides all the steps you need to take to move your app’s data over to the Stream platform.

The migration will start with a data export of your Pusher Chatkit data which is outlined on here on GitHub.

Instantiating Your Chat Client

Similar to Pusher's Chatkit, Stream Chat requires that you instantiate a chat client instance for browser/mobile usage. You can also provide additional options, such as API base URL and request timeouts when instantiating the client.

Chatkit:

const chatkit = new Chatkit.default({
  instanceLocator: YOUR_INSTANCE_LOCATOR,
  key: YOUR_KEY,
});

Stream:

const client = new StreamChat('YOUR_API_KEY', {
    timeout: 6000,
});

Users

Users in Stream are similar to users in Chatkit and are identified using custom IDs. While Chatkit uses an authorization URL based approach when authenticating users, Stream Chat employs a token-based system where a token is generated on the server-side and sent to the client to provide access to the chat for that user.

Stream also supports custom metadata for users, so it’s easy to migrate your Chatkit user customData property to the corresponding Stream user.

Chatkit:

const chatManager = new ChatManager({
  instanceLocator: 'your_instance_locator',
  userId: 'user_id',
  tokenProvider: new TokenProvider({ url: 'your.auth.url' })
});

const currentUser = await chatManager.connect();

Stream:

const currentUser = await client.setUser(
    {
        id: 'john',
        name: 'John Doe',
        image: 'https://getstream.io/random_svg/?name=John',
    },
    'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoiamxhaGV5In0.OkDbpbujWJ-XIVHaf00Dnqt3v8Yp_nQ6CGzm-Z4QUVc', // authentication token
);

Rooms & channels

Chatkit has the concept of public and private rooms and supports one-on-one chat as well as group chats (up to 500 members). Stream Chat, on the other hand, provides channels that are highly customizable and can be specialized for a large variety of use cases. Further, group chats can support an unlimited amount of users.

Stream provides 5 built-in channel types by default:

Livestream: Sensible defaults in case you want to build chat like YouTube or Twitch.
Messaging: Configured for one-on-one chat solutions. This roughly maps to Chatkit’s private rooms.
Team: If you want to build your version of Slack or something similar, start here. This is similar to Chatkit’s public rooms.
Gaming: Configured for in-game chat.
Commerce: Good defaults for building something with similar features to that of Intercom or Drift.

If you want something custom, you can create a channel type that meets your needs and configure permissions for its members as necessary.

Chatkit:

const room = await currentUser.createRoom({
  id: '#general',
  name: 'General',
  addUserIds: ['craig', 'kate'],
  customData: { foo: 42 },
});

Stream:

const channel = client.channel('team', '#general', {
    name: 'General',
    created_by: { id: 'userId' },
    members: ['craig', 'kate'],
    custom_data: { foo: 42 },
});

await channel.create();

Messages

Stream supports all the properties of Chatkit messages and more. The difference is, unlike Chatkit, Stream Chat does not break a message down into multiple parts, so you need to account for this when migrating your messages.

Sending a message in Stream is as simple as using the sendMessage method on a channel, and you can add text, attachments, or even mentioned users to a message object (something that Chatkit lacks). Note that messages and attachments also support custom data, so you can easily migrate this off your Chatkit messages as well.

Chatkit:

await currentUser.sendMultipartMessage({
    roomId: myRoom.id,
    parts: [
    { type: "text/plain", content: "Hello world!" },
    {
        type: "image/gif",
        url: "https://gfycat.com/failingforkedheterodontosaurus",
    },
    {
        file: document.querySelector("#attach").files[0],
        customData: { metadata: 42 },
    }
    ],
});

Stream:

await channel.sendMessage({
    text: 'Hello @Josh',
    attachments: [
        {
            type: 'image',
            asset_url: 'https://bit.ly/2K74TaG',
            thumb_url: 'https://bit.ly/2Uumxti',
            myCustomField: 123
        }
    ],
    mentioned_users: [josh.id],
    anotherCustomField: 234
});

Further Resources for Migration to Stream Chat

Follow our Chatkit Migration Guide on GitHub.
Check out our demos that we have made available to see the various types of experiences you can build with the Stream Chat API.
Read tutorials on The Stream Blog.
Check out the Stream Chat docs to view all the features and integrations available to you.

Please feel free to contact support@getstream.io if you are ready to migrate to Stream Chat or if you have any questions.

How to Make an Encrypted Messaging App

Nick Parsons — Thu, 26 Mar 2020 20:54:21 +0000

Hey Android Devs!

Looking to implement encrypted chat/messaging into your Android app?

There is a great post over on The Stream Blog on How to Build an Encrypted Chat/Messaging App for Android.

Everything is written in Kotlin and well detailed. It comes with a complimentary repository if you want to jump right into the code. That can be found here on GitHub.

Enjoy and happy coding! 🤓

Android Chat Bubbles: Building iOS Style Chat in Android

Nick Parsons — Tue, 10 Mar 2020 18:11:43 +0000

In this post, we'll explore how to do two things: 1) create live chat message bubbles in Android that are similar to WhatsApp and iMessage and 2) customize Stream Chat's UI Components.

We'll customize Stream Chat Android's built-in UI components by plugging in a custom message view. This allows us to focus on the text rendering, while Stream does everything else.

The source code is available here. Once we're finished, we'll have a chat experience that looks like this:

Prerequisites

This post assumes a working knowledge of Android and Kotlin. If you're unfamiliar with either or both, it may be useful to check out a getting started guide. If you'd like to run the code, you'll also need a Stream account. Please register here. Once you're registered, you'll see a Stream app with an App ID, API Key, and API Secret:

We won't be going through how to create the backend to receive our tokens in this tutorial; please refer to this repo for token generation. You can also use Stream's CLI; if you look in MainActivity.kt, you'll see placeholders to fill in run the application.

Let's build!

Listing Channels

First, we'll create a view which displays a list of channels for a user to select:

This view is mostly taken care of by Stream's UI Components. To list our channels, the MainActivity.kt will leverage Stream's ChannelList, and we'll configure it to load all of the channels for our user. Here's the code:

First, we initialize our StreamChat instance with our API key. We configure it with the user who'll be using our application. To keep things simple, we'll just declare who's logged in and their frontend token. In a real application, you'd want to perform authentication with a backend that generates this token.

We also give the user an id, a name, and an image(profile image). Once we've done this, we can declare the layout as "activity_main".

Now, the view needs a view model; in this case, we'll simply use the default, Stream-provided "ChannelListViewModel". This is great, as it does all the work to interact with Stream's API. We simply need to configure it to filter for our user's channels.

The last thing we do is set a "click listener" on each channel. We boot a ChannelActivity, which is where we'll customize the chat messages. Before we look at the code for ChannelActivity let's look at the layout:

https://gist.github.com/nparsons08/e7436c12791e8a123cb1e409628c1597

We use a ConstraintLayout to hold our ChannelListView and associated ProgressBars. Since the view is mostly taken care of by Stream, we just need to configure when it is that our ProgressBars show and what our padding and margin are.

Now, we're ready to view a specific channel!

Viewing a Channel

Once a user clicks a channel, our ChannelActivity starts. Here is the code:

First, we get our channel information off of the Intent from our ChannelActivity.newIntent call, in MainActivity.kt, and, then, below, we set our content view activity_channel and viewModel:

You'll notice this was another ConstraintLayout with some views and progress bars; we use Stream's UI Components to build the majority of the view. ChannelHeaderView gives us a nice header with a channel image and channel name (from the parent). MessageListView displays our messages, and MessageInputView gives us a helpful default message input.

We use the built-in Stream view model and feed that to each view. However, we customize the MessageListView view by providing a custom message view "factory", BubbleMessageViewHolderFactory. This "factory" hooks into Stream's code to provide a custom view for each message. This class is straightforward since all we're going to do is instantiate our bubble message class, BubbleMessageViewHolder:

https://gist.github.com/nparsons08/16ead4c7aa2bf6fe6851646eb9c29179

Now that we're hooked into the view rendering, we can customize our messages to look like iMessage or WhatsApp!

Rendering Custom Bubble Messages

We're ready to declare our custom bubble view. To hook into Stream's components, we need to be a subclass of BaseMessageListItemViewHolder. Here is the basic class definition:

We're passed a viewGroup resource from the factory (we will be using "bubble_message"). Then, this view is referred to by itemView. From this, we grab our message header (which will just be spacing, in our case), our text view that will hold the message, and our username view. We also have a MessageListItem and ChannelState, which contain data for our Stream message and channel, respectively. Let's check out our view:

This view is quite simple, as it defines the three pieces we referenced before: the header spacing, the username, and the text. Notice there's no bubble logic here. To get the correct colors and shapes, we'll use drawables and some logic in our class to define how the message will look; this layout is simply the scaffolding for us to fill in.

Implementing this abstract class requires us to implement setStyle and bind. We'll ignore setStyle for this tutorial since we'll be defining all of our styles outright.

You can check out MessageViewListStyle to see all the ways you can customize the components without building your own class; with that said, in this post, we want to explore how to do it ourselves!

Let's dive in! The bind method is where we configure how the message looks:

We're given a ChannelState and MessageListItem, which has everything we need to figure out our display. First, we check the type of message. Stream has a few different message types, which represent different things. For example, you may get a message type that indicates a user is typing or a date separator. In our case, we'll focus on two, the actual messages and date separators; we'll ignore any other message type and remove them from rendering, entirely.

Let's focus on the actual messages to start; we'll look at how to render the date separators after. Here is our configMessage method:

We do three things: 1) configure the actual text display, 2) decide if we need to show a username 3) determine how much spacing we have between messages.

First, let's see how we configure the actual text and render our bubble message:

We set the TextView's text, then decide if it's "our" message or "theirs". If it's "ours", we display the message on the right, in blue. If it's "theirs", it goes on the left, in grey. To set the justification of the message, we use horizontalBias; setting it to 1 sets the message to the right, and 0 sets the message to the left. We'll also set the text color based on the background since blue needs white text and grey needs black. Padding is also set differently for the two cases, since the message will have the "tail" on the right or left, depending on who's message it is.

Since we're replicating iMessage's style, they don't render the tail unless it's the bottom message. Conveniently, Stream gives us position information and makes it easy to check message placement within a group of messages:

If the message is the bottom message, it means it's the only message or the last in a group of messages by the same user. If it's either of those, we add a tail; if it's not, we just do a simple bubble. We'll also set the background of the text view to a "drawable". Let's look at "our" message drawables; here's the drawable with a tail:

We're doing a bit of a smart trick by defining layer-list with two items. One is a regular rectangle with rounding, the other is a rotated rectangle that is covered by the standard rectangle. We pad the regular rectangle out to give a small gutter, so the edge of the rotated rectangle sticks out. This gives us a beautiful "tail" for the message bubble.

While we chose to keep it simple here with just rectangles, if you'd like a more stylized tail, such as the one you see in iMessage, you can use vector drawables to convert an SVG to a drawable.

For messages that don't need a tail, we define a simple shape with some spacing for the tail gutter:

Messages for "theirs" is the same pattern but mirrored and grey:

Nice! Now we have some good looking bubbles:

Now we can add the username, if necessary, and space the messages correctly. First, we'll look at configUsername:

First, we decide if we show the username at all. There are three cases where we want it gone. If it's a 1-on-1 chat, if the message is "ours", or if it's not the top of a group of messages (otherwise, we'd have usernames on every message within a grouping). This logic means we only label groups of messages from other people in group chats.

If we have that case, we set the text of the username TextView to the user's name. We set the bias and margin depending on if it's on the right or the left.

Now we can space our messages depending on their placement in a group of messages:

This is simple since our default spacing is in our layout resource. We simply shrink the height if we're not the top message. Now you'll see the nice spacing and appropriate tails with usernames in our group chats:

All that's left is to deal with our date separator messages, and we're done! These MessageListItems come with no user and just a date. These are used to provide a horizontal separator between messages that are spaced out in time. These messages are automatically created by Stream when it decides the time between messages has been too long, and it'd be nice to group them. For example, if you message back a day later, you can break up the view with a date marker to inform the user of roughly when messages have come in. Here's configDate:

This is straightforward since our default text view has no styling and the correct spacing. We just need to turn off the username view and add the date. Now we have a convenient date separator:

And we're done! We now have custom bubble messages hooked into Stream.

Final Thoughts

Replicating iOS style messages on Android is relatively simple with drawables. Stream makes it even simpler with it's out of the box UI Components. You can start with Stream default views and customize them or build your own. If you want full control, you can ignore the UI Components and use the lower level Stream libraries directly. It's easy to peel back the layers when you need it!

For in-depth details on how to use the Stream Chat SDK for Android, check out the interactive tutorial here. You'll learn how to build everything that is required for real-time chat – typing indicators, reactions, threads, etc.

Happy coding! 💬

Nick Parsons 🚀

@nickparsons

Android Chat Bubbles: Building iOS Style Chat in Android via The Stream Blog (@getstream_io) getstream.io/blog/android-b…

18:12 PM - 10 Mar 2020