Forem: Miro @ Inuko.net

Salesforce.com REST API for data manipulation

Miro @ Inuko.net — Sat, 17 Aug 2024 13:57:51 +0000

Need to manipulate Salesforce.com data quickly?

Would you rather use the tools and language you know?

We too, let's get it done.

The problem

One of the most infuriating things about Salesforce.com is the inability to filter based on child records.
Our client needed to add Contacts to Mailchimp. To get started only a subset - those who made donations via the Darujme platform. Donations are tracked as a custom Salesforce object that is connected to the contact object - the person who made the donation.

The Salesforce - Mailchimp integration, only allows the standard list-like filters on the primary object and parent references (for contact that would account for example).

So we need to find all the types of the donations and add them onto the contact. If you imagine the donations object as a child node in a tree below the contact, you can see why this would be called “Rollup” the type.

Solution #1 Rollup field

We can add a rollup field to the contact object, one per donation type.
To do that, we go into the Fields & Relationships and add a new rollup field.
Then we select the donations object (called transaction here).
Finally we choose the COUNT Roll-Up Type and setup filter criteria on the donations object, to count only donations of one type.

Another interesting alternative is to choose the creation date and MAX Rollup-type. This way we can check if a donation of this type was made and also when. If we need to find recent donors, this information will be handy.

Solution #2 Use the API Luke

If we have many donation types, one field per type might be impractical. Or if we need to do the rollup operation only once, rollups might be an overkill, since we don't need them to be recomputed.

The Salesforce.com REST API is perfectly suited for data manipulation and transformation. Compared to Apex - the native SF programming language - the setup is trivial and we can use whatever language we want!

Before we can read and write data via the Rest API, we have to first authenticate and authorise our app.
This can be either quite complicated (for proper OAuth) or you could open a security hole into your system (with legacy password auth).

So for a quick and dirty solution we shall use Postman for doing the authentication for us and we’ll copy the instance_url and Access Token.
In your SF, go to setting and enable cors for postman.

Then follow the Postman setup here.

Armed with the access token - our key - we can now read and write SF data.

We shall do this with javascript and nodejs.

First let’s read all donations of the ‘Darujme’ type and collect all the contacts that have a donation of this type.
Note that salesforce will not return all donations in one request, instead we need to “page” through the results. We will accomplish this with the nextRecordsUrl and the done field.
The done field is set to true, if there are no more results. Otherwise we make another request to the nextRecordsUrl (nicely provided to us by SF) and continue in the loop.

const getContacts = async () => {

    const myHeaders = new Headers();
    myHeaders.append("Content-Type", "application/json");
    myHeaders.append("Authorization", "Bearer " + ACCESS_TOKEN);

    const requestOptions = {
        method: "GET",
        headers: myHeaders,
        redirect: "follow"
    };

    const soql = "SELECT id, name, donation_type__c, contact__c, contact__r.name FROM donation__c where donation_type__c = 'Darujme'";

    let url = ENDPOINT + "/services/data/v61.0/query/?q=" + soql;

    const contacts = {};
    while(true) {
        const resp = await fetch(url, requestOptions)
        const r = await resp.json();
        if (r.nextRecordsUrl)
            url = URL + r.nextRecordsUrl;

        for (const rec of r.records) {
            if (!contacts[rec.contact__c]) {
                contacts[rec.contact__c] = 1;
                //console.log("ADDED: "+ rec.contact__r.Name)
            }
        }

        if (r.done)
            break;
    }
    return contacts;

Now that we collected all the contacts that donated through the Darujme platform, we can proceed to update the contact records.

The most straightforward way is to send an update request for each contact. And this will work (we checked), but it will be sloooow.

const updateContactsOneByOne = async (contacts) => {

    const myHeaders = new Headers();
    myHeaders.append("Content-Type", "application/json");
    myHeaders.append("Authorization", "Bearer " + ACCESS_TOKEN);

    // One by one -> super slow
    for (const cid of Object.keys(contacts)) {
        const raw = JSON.stringify({
            "donation_type__c": "Darujme"
        });

        const requestOptions = {
            method: "PATCH",
            headers: myHeaders,
            body: raw,
            redirect: "follow"
        };

        const r = await fetch(URL + "/services/data/v61.0/sobjects/Contact/" + cid, requestOptions);
        console.log(cid + "=>" + r.status);
    }
}

Instead, when doing large batch changes we need to use the batch (composite) api (duh:).

const updateContactsBatch = async (contacts) => {

    const myHeaders = new Headers();
    myHeaders.append("Content-Type", "application/json");
    myHeaders.append("Authorization", "Bearer " + ACCESS_TOKEN);

    // 200 at the time -> super fast (super less slow:( )
    const ids = Object.keys(contacts).slice();
    let b = 1;
    while (ids.length > 0) {
        const batch = ids.splice(0, 200);

        const raw = {
            "allOrNone": false,
            "records": batch.map(x => ({
                "attributes": { "type": "Contact" },
                id: x,
                "donation_type__c": "Darujme"
            }))
        };
        const requestOptions = {
            method: "PATCH",
            headers: myHeaders,
            body: raw,
            redirect: "follow"
        };

        const r = await fetch(ENDPOINT + "/services/data/v61.0/composite/sobjects/", requestOptions);
        const j = await r.json();
        console.log("##" + b); b++;
        for (const rr of j) {
            console.log(rr.id + "=>" + rr.success + (rr.errors && rr.errors.length ? " ERR:" + JSON.stringify(rr.error || "") : ""));
        }
    }
}

The code is a little more complicated. First we splice the first 200 (at most) contacts into a batch. Then we send the batch in an mass update request.
We continue the loop while there are contacts left to update (the splice operation removes the elements from the array).

That’s it! (Of course we have previously added a custom field onto the Contact).

We have manually “rolled up” the donation type onto the parent contact.

REST assured

Like with everything else in life, information has to be regularly maintained, otherwise it will turn into a big mess.
Having a powerful API that can help us read and write data with ease, with our choice of language, environment and tools is a must.

Thankfully the Salesforce.com REST API fits the bill nicely.

We hope the code above was useful in your data endeavours.

Don’t hesitate to get in touch if you spotted some bugs, or you would like our help with data-massages.

Happy hacking!

External content for GatsbyJS

Miro @ Inuko.net — Fri, 09 Aug 2024 15:22:54 +0000

Let's build a static website with external content.
In this post I will show you how you can static-render data from any data source with GatsbyJS.

GatsbyJS

If you enjoy React and want a standard-compliant and high performance web, you should look at GatsbyJS.

What does it do?

It will compile your React code into a collection of static HTML files.

Why should you care?

Maximum performance - when a user looks at your web, no code is run client or server side!
Fastest images - progressive loading of images in the right resolution for the user's device.
SEO - static HTML is google (robots) friendly. Slow websites are penalized by google search!
React - productive front end development.
Typescript - catch errors, before your users see them (optional, but highly recommended)!
Content - connect and use many content sources out-of-the-box or add your own!

Gatsby JS Content

GatsbyJS represents content as a tree of nodes. A node can be an image a block of text.
For example a blog post is a collection of text and image nodes.

You can create nodes by hand in your gatsby-node.js file. But there are easier ways.

Plugins

Nodes are created by plugins. Which plugin you need, depends on the CMS of your choice.
The most straightforward option is the filesystem plugin, which turns files into nodes.
To find the right plugin for you take a look here

Plugin not found

If you have a data source that is not covered by the existing plugins, let's build our own.

The process is fairly straigthforward, the only complex part are images.

Loading nodes

Create the gatsby-node.ts (or js) file in your gatsby project root folder.

Add this code to get started. The sourceNodes method is called automatically when gatsby project is build.

import { GatsbyNode } from "gatsby"
import { createRemoteFileNode } from "gatsby-source-filesystem"

export const sourceNodes: GatsbyNode["sourceNodes"] = async ({
    actions: { createNode },
    createNodeId,
    createContentDigest,
    store,
    cache,
}) => {
}

Now let's fetch our data. Here I use https://inuko.net app platform, but the process is similar for any source.

interface IPost {
    id: string;
    title: string;
    content: string;
    image_id: string;
}
const fetchPosts = async () => {
    const postQuery = {
        entity: {
            name: "cms_content",
            allattrs: true,
            links: [{
                name: "cms_site", from: "siteid", to: "id", alias: "cs",
                filter: { conditions: [{ attribute: "name", operator: "eq", value: "NAME_OF_MY_WEB" }] }
            }]
        }
    };
    const posts = await fetchJson("/api/fetch", postQuery) as IPost[];
    return posts;
}

And let's also fetch images we will need.

interface IImage {
    id: string;
    name: string; // sunset.jpg
    image_url: string; // https://sample.com/54565735235464322
}
const fetchImages = async () {
    const imageQuery = {
        entity: {
            name: "cms_image",
            allattrs: true,
            links: [{
                name: "cms_site", from: "siteid", to: "id", alias: "cs",
                filter: { conditions: [{ attribute: "name", operator: "eq", value: "NAME_OF_MY_WEB" }] }
            }]
        }
    };
    const images = await fetchJson("/api/fetch", imageQuery) as IImage[];
    return images;
}

We now have a list of (blog) posts and a list of images (links).
In this example we have a simple structure where each post has some text content and an id of a single image.

The next step is to convert the data we got from our server to data that gatsby can use.
Data in gatsby is represented by *node*s, so let's look at how to convert our server data into a node.

export const sourceNodes: GatsbyNode["sourceNodes"] = async ({
    actions: { createNode },
    createNodeId,
    createContentDigest,
    store,
    cache,
}) => {

    const posts = await fetchPosts();
    const images = await fetchImages();
    // create an object for image by id lookup
    const imageDict = images.reduce((d, c) => (d[c.id] = c, d), {} as { [name: string]: IImage });

    for (const post of posts) {
        // create a node from post
        const postNodeId = createNodeId(`XPost-${post.id}`)

        if (post.image_id && imageDict[post.image_id]) {
            const image = imageDict[post.image_id];
            const name = image.name;
            const url = image.image_url;
            const fileNode = await createRemoteFileNode({
                url: url,
                //store,
                cache,
                createNode,
                createNodeId,
                // !!! important !!!
                // If your image url does not have a valid image extension, this will tell the system what type of image we are adding
                ext: name.substring(name.lastIndexOf(".")),
            });

            post.mediaFile___NODE = fileNode.id

            post.internalId = post.id; // copy our internal post. id is replaced by gatsbyjs

            const nodeMeta = {
                id: postNodeId,
                parent: null,
                children: [],
                internal: {
                    type: `XPosts`,
                    mediaType: `text/json`,
                    content: JSON.stringify(post),
                    contentDigest: createContentDigest(post),
                },
            }
            createNode(Object.assign({}, post, nodeMeta))
        }
    }
}

We iterate over all posts and create a corresponding node for each.
If a post has an image post.image_id we also create an remoteFileNode node and
append it to the post node
post.mediaFile___NODE = fileNode.id

Important note: gatsby will automatically create image nodes from our file nodes, but it needs a way to detect it is an image.
If your urls contain file extensions or your server will reply with an image content type, you are all set.
If that is not the case, you can set an explicit extension on the file node (png, jpg) to trigger image node creation.
ext: name.substring(name.lastIndexOf("."))

Image nodes

Maybe you are wondering why we go to such trouble of loading images as nodes. We could just use the image urls directly.
Well sometimes the images might be behind an authenticated service for example.
But the real reason is we want to use the fantastic image plugin that gatsby provides.
It will automatically convert the images into the best format and size for any device that will look at our web.
This means the images will load faster and will look better (and get a better score by google:).

Consuming nodes in pages

We are now ready to consume the nodes we've created.
There are many ways in which you can do this, for this example we will a few posts and render them on a features page.

First we need to load the data nodes we are interested in. We will use the useStaticQuery.
Then we will pass the data to our reusable component called SectionGrid that will render all loaded posts.

const FeaturePage = (props:{}) => {
    const data = useStaticQuery(graphql`
        query featuresQueryEn {
            allXPosts(
                filter: {language: {eq: "en"}, pageid: {label: {eq: "features"}}}
                sort: {position: ASC}
                ) {
              edges {
                node {
                  id
                  content
                  title
                  mediaFile {
                    childImageSharp {
                      gatsbyImageData
                    }
                  }
                }
              }
            }
          }
    `);

    return <SectionGrid data={data} title={<h1>Features</h1>} />
}

Render time!

import { Link } from "gatsby";
import { GatsbyImage } from "gatsby-plugin-image";
import React from "react";

export const SectionGrid = (props: {data: any, title: string}) => {
    const edges = props.data.allXPosts.edges as any[];

    return <div className="sectionGrid">
        <div className="hero">
            {props.title}
        </div>
        {edges.map(edge => {
            const node = edge.node;
            return <div className="section">
                <div>
                    <GatsbyImage image={node.mediaFile.childImageSharp.gatsbyImageData} alt={edge.name} />
                </div>
                <div className="sectionText">
                    <h2>{node.title}</h2>
                    <div>{node.content}</div>
                </div>
            </div>
        })}
    </div>
}

This component will iterate over all nodes we got from out query. And it will render two divs one for the post image and one for the text content.

That's a wrap

Building a great looking and performant website has never been easier.
Tools like Gatsbyjs (and its plugins) will do most of the heavy lifting for us.
Thus we can dedicate 100% of our time to content and design.

I hope that armed with this post, you'll be able to further automate your workflow.
Now you can bring your product catalog, knowledge base or other valuable content to your website.
Without copy-pasta and with fully automated image processing.

Happy hacking!

Business Travel Log

Miro @ Inuko.net — Thu, 01 Aug 2024 06:26:41 +0000

How best to start the holiday season?

With a new project, of course!

Do you often go on business trips and find filling out travel expense forms or processing them tedious?

We have a solution for you.

Who are the users?

This application was created based on ideas from the people at Cesta von.
Mentors and other employees visit communities where they help people find their way out of poverty. Since people in need cannot travel for advice—and often don't even know where to find help—Cesta von workers go to their communities. To use their time efficiently, one 'business' trip includes several meetings and often several towns/communities.

Since (social) workers often use their own cars for travel, the organization wants to reimburse them for travel expenses. And this is where things get complicated. The calculation itself isn't difficult, but it requires collecting information about all trips for the month, their segments, the average fuel price, and then putting it all into a formula.

Then there are the financial gals in the back-office, who have to double check the data, gather signatures, archive the documents and eventually transfer the funds.

To make the business-trip reporting a little easier on both the workers and the back-office employees, regular business trips are rolled into a montly allowance. The regularity of course has to be first monitored (for a couple of months).

Finally internal organization reports and tax office reports have to be generated and submitted regularly.

What are they using right now?

If Excel comes to mind, yes, this was the original solution. The advantage of Excel is that the formula is ready in a few minutes.

But then there are significant disadvantages:

Users must search for the distances traveled on the business trip.
They must correctly enter only the required information.
They must fill out Excel on a computer because, although Excel on a mobile phone works, it's more of a punishment.
They must correctly insert rows and not mess up the formulas.
They must correctly enter numbers and dates (decimal commas, slashes in dates, and other creative innovations).
Additional documents (e.g., travel tickets) cannot be "attached" to Excel.
A new file must be created for each month.
Combining data from multiple files is a manual, error-prone process.

User needs

So, to sum it all up, the new solution should:

Work on a mobile phone.
Maintain a travel database so that users can enter trips continuously and only check and send the summary at the end of the month.
Automatically calculate everything needed (distances between places, estimated travel durations, etc.).
Prevent incorrect information entry—only cities and towns in Slovakia, dates in a uniform format, etc.
Automatically generate a document in PDF format for the accounting department.
Maintain information about cars (technical documents).
Upload documents—travel tickets, fuel receipts, etc.
Allow the conversion of regular business trips into a travel allowance.
Have an easy way to generate aggregate statistics.

Our App

Armed with a list of functionalities, we got to work. We focused primarily on:

Intuitiveness — The app is used only a few times a month, so we can't rely on users remembering a complicated procedure.
Automation — So that it's not necessary to ask for information we can get automatically.
Speed — The app must be usable on the go. During business trips, stops and documents can be added gradually—photographed directly on the phone.

Trip Module

The user creates a new business trip and gradually adds the places visited. The app automatically assumes that the trip starts and ends at the user's residence. The default start and end can, of course, be changed or don't have to be the same. With each added place, the distances are recalculated.

For each segment of the trip, it's possible to indicate whether the user's own vehicle or public transport—train, bus, etc.—was used.

For each segment, it's also possible to select the departure time; the app will calculate arrivals on its own.

For each segment where the user used a train or bus, a snapshot of the travel ticket can be attached.

Report Module

The user simply selects the month (or closes the previous one).
Enters the price at which they refueled (and attaches a photo of the receipt).
Chooses whether they want reimbursement for expenses in their account or in cash.
Everything else is calculated by the app—it remembers the user's current car and its consumption.

After review, the user marks the report as sent for processing. Subsequently, the report is processed by the cashier, and expenses are reimbursed.

The report module can be used by the cashier to calculate flat-rate allowances for regular business trips. Recurring business trips over the monitored period (4, or 2 months) are calculated as a flat rate for the user.

Currently, it's still necessary to print and sign the report. We are working on eliminating this last bit of bureaucracy ;)

Statistics & Dashboards

The managers and back-office accountants use the interactive charts to get a quick overview.

Or to drill-down to inspect some particular discrepancies.
Or to calculate financial aggregates.

Conclusion

The Cesťák app simplifies business trip expense management by automating data collection, calculating travel costs, and generating reports.

It is mobile-friendly, allowing users to log trips continuously, capture travel receipts, and upload documents directly from their phones.

Key benefits include ease of use, significant time savings, error reduction, and seamless integration with accounting processes. Managers and accountants can utilize interactive charts for quick overviews and detailed inspections.

Apps, where we can practically hear our users sighing in relief because the Excel frustration is finally over are our favorites.

Get paid in QR

Miro @ Inuko.net — Fri, 26 Jul 2024 07:17:59 +0000

No, not in pixels, in euros.

No, not with credit card.

Yes with phone.

Motivation

If you sell product or services getting paid is an obvious requirement.

These days the credit-card is the king, well online anyway.

You have an abundance of payment gateways, that will handle credit cards, google or apple pay for you.

For a fee, of course.

In a physical store, you would get a credit-card terminal.

Some banks can sell you a much cheaper Android app that turns you phone into a CC terminal.

It is worth considering, especially if you process only a few payments a day.

These options cover almost all situations in typical consumer scenarios.

Where a person goes online or into a brick&mortar shop and buys a bunch of products or gets a haircut.

But, there is much more.

What if you are a non-profit or citizen-organization that needs to process small payments.
What if you are a B2B company and your customers want ot pay with direct transfers.
Or what if you just don't want to pay the ~2-3% overhead for credit card payments.

In all of these cases the answer, in the European Union, is SEPA.

More specifically SEPA Credit Transfer.

Or its younger, but much faster sibling SEPA Instant Credit Transfer.

SEPA Credit Transfer

If you ar enot familiar with the name, don't worry, it is the same old same old wire-transfer.

Eg. you tell your bank to take money from your account and to put in on somebody elses.

What was historically a paper form to fill out, is these days a couple of clicks in your mobile banking app.

But, we now have standardized international bank account numbers - IBANs.

Making it super easy to send money anywhere in the EU or even world-wide.

And with SEPA instant, your money arrives in seconds.

But, but the IBANs are so long, they are almost frightful.

And people are notoriously bad at entering stuff, so you can be happy if you get the right amount, forget about additional notes.

All of these problems are easily solved with QR codes.

Your customers can scan them with their phones, check the amount and click pay in their banking app.

With the growing list of banks in the SEPA Instant Credit Transfer scheme, chances are high, your money arrives in an eyeblink.

In a future blog, we will take a look at how to watch for incoming payments on your bank account.

To automate invoice generation and sending for example.

Let me show you how to generate QR codes for Slovakia and Czech Republic.

I want to make this list longer, so if you know about how QR codes are made in your country, please let me know.

QR Slovakia

Slovak banks have agreed on a common standard called Pay by Square.

There is even a url schema that can be used in online scenarios.

Eg. the user clicks a link and their banking app is launched with all the payment information filled out.

Sadly, these are not interoperable (in 2024).

Let me show you how to generate the QR code in NodeJS.


app.get("/api/paybysquare", mustAuth, async (req, res, next) => {
    try {
        const body = req.query.content as string;
        const model = JSON.parse(body);
        const content = await generate(model);
        const qrStream = new PassThrough();
        const result = await toFileStream(qrStream, content,
            {
                type: 'png',
                width: 200,
                errorCorrectionLevel: 'H'
            }
        );

        qrStream.pipe(res);
    } catch (ex) {
        next(ex);
    }
});

The generate method is from the bysquare library.

You can install it with npm as usual npm i bysquare

To get an PNG image we can use in the browser or in a invoice document, we call the method like this.


interface IBySquareModel {
    IBAN: string;
    Amount: number;
    CurrencyCode: string; // must be "EUR",
    VariableSymbol: string;
    Payments: number; // must be 1,
    PaymentOptions: number; // must be 1,
    BankAccounts: number; // must be 1,
    PaymentNote?: string; // optional note
}

const BySquareQR = (payModel: IBySquareModel) => {
    return <img style={{ width: "120px", height: "120px" }} src={"/api/paybysquare?content=" + encodeURIComponent(JSON.stringify(payModel))} />
}

It is often very helpfull to add a descriptive message to the payment.

Here is some simple code that will normalize client names.

This will strip diacritics, turing Ján Kováč into Jan Kovac.

Should be still perfectly readable and will make sure the bank systems don't mess it up.

    const paymentMessage = customerName?.normalize("NFKD").replace(/[\u0300-\u036f]/g, "").substring(0, 100) || "";

QR Czech Republic

Even though Czech Republic or Czechia has so far (2024) resisted the euro, they too have QR code scheme.

And they even have a royalty free code generator we can use.
Now, for security reasons, you still might want to generate it yourself.

const czQrLink = "https://api.paylibo.com/paylibo/generator/image?iban=" + iban + "&amount=" + 
czkPrice + "&currency=CZK&vs=" + vsym + 
"&message=" + message;

const CzQrImage = (czQrLink: string) => {
    return <img style={{ width: "120px", height: "120px" }} src={czQrLink} />
}

International

Sadly, there is not yet an international or european QR standard.

So you'll have to to show the right QR code based on your user's country.

If you expect payments from Czech Republic, keep in mind the Czech QR codes only work in Czech crowns (czech currency CZK).

Or if you are based in Czech Republic and you expect payments from Slovakia, the QR only works in Euros.

The code below can help you calculate the right currency amount.

First we get the current reference exchange rate from the central bank.

Then we calculate the euro and czk prices, depending on the product price and currency.


const getExchangeRate = async () => {
    const now = new Date();
    const dt = new Date(now.getFullYear(), now.getMonth(), now.getDate() - 1).toISOString().substring(0, 10);

    const url = "https://nbs.sk/export/sk/exchange-rate/" + dt + "/xml";
    const dat  = { url: url }
    const opts = {
        headers: {
            'Content-Type': 'application/json'
        },
        method: "POST",
        body: JSON.stringify(dat)
    }

    const resp = await fetch("/api/httpfetch", opts);
    const xml = await resp.text();

    const parser = new DOMParser();
    const xmlDoc = parser.parseFromString(xml, "text/xml");
    const node = xmlDoc.querySelector("Cube [currency='CZK']");
    if (node) {
        const attr = node.getAttribute("rate");
        if (attr) {
            const num = attr.replace(/\s/, "").replace(",", ".");
            return +num;
        }
    }
    return undefined;
}

    // basePrice is the amount to be paid, for the products or services
    // currencyCode is the currency the products or services are sold in.
    const czkRate = await getExchangeRate();

    const eurPrice = currencyCode === "EUR" ? basePrice : (Math.ceil(basePrice / czkRate * 100) / 100).toFixed(2);
    const czkPrice = currencyCode === "CZK" ? basePrice : (Math.ceil(basePrice * czkRate * 100) / 100).toFixed(2);

Get Paid

Trying to read badly printed IBANs from invoices or even typing in the horribly long IBANs is super annoying.

That you have to be very careful and check three times, because it is money, makes it even more so.

We make software to make our lives easier, and QR codes for payments fit this goal nicely.

I hope you found the information useful, and if you have some pointers about your national QR code schemes, please shoot them my way.

Happy hacking!

Sign in with Apple for Node.js

Miro @ Inuko.net — Fri, 19 Jul 2024 12:54:42 +0000

A frictionless sign-up and sign-in experience has many benefits.

Once a user is ready to try our app, don't lose them in the sign up procedure.

Once a user starts using our app, don't lose them because they forgot their login.

Multi-factor and not reusing passwords keeps user's data safe.

Intro

We already wrote about sign-in & sign-up with Google and Microsoft here.

In this blog we want to look at another provider of authentication services - Apple.

Motivation

There are 1.5 billion active iPhone users.

All of them will be able to sign-in and sign-up with TouchID or FaceID, never-ever needing a password.

We believe this is a massive benefit for the users. It will also make your app stand out.

The technology is called Sign in with Apple and by now you have surely seen it on the web.

While researching this topic, we haven't found a good blog or npm package. There is some info scattered around the web.

And there is even a PassportJS package that is close, but is missing critical features (like getting the user's email).

Then, there is the absence of any testing environment by Apple, which makes it really hard to develop.

Long story short, it took us hours to piece the whole picture together. And several more hours of deploying test versions until it all worked.

Apple account setup

Before we delve into the code, we have to provide information to Apple.

You will also have to sign-up for a developer account with Apple.

Keep in mind that this is a paid service.

App ID

First we will create an new App ID (or use an existing app if you have one).

Go to your developer account
Under Certificates, IDs & Profiles click on Identifiers here.
Click the big blue PLUS and create a new AppID, select type App.
Note down the App ID Prefix (TeamID) in the top right corner, we will need it later.
Scroll down in the long list of capabilities and choose Sign in with Apple
In the edit dialog select Enable as a primary App ID
Choose a name and identifier for your app and finish the setup (click continue).

Services ID

Now that we have the App ID, we can create the Services ID.

Go back to the list of identifiers and click the big blue PLUS.
Select Services ID and click continue.
Choose a name and identifier. Write down the identifier this is the client_id and we will need it later.
Mark Sign in with Apple in the list of options below the description.
In the Configure dialog choose the Primary App ID (the one we created above).
In the domains field, write your domain fe: inuko.net
In the Return URLs field, write your callback url, fe: https://yourdomain/auth/apple/callback. Sadly we can't use localhost for testing.
Finish the setup (click continue).

Keys

To recap, by now you should have an App ID and a Services ID ready and configured.

Now we have to make a key to secure the communication between our app and Apple.

In Certificates, Identifiers & Profiles in the left hand menu, choose Keys
Click the big blue PLUS.
Choose a key name
Mark Sign in with Apple in the list of options below the key name.
In the Configure dialog choose the Primary App ID (the one we created above).
Finish the setup (click continue).
Download the key. Make sure you keep it safe.
Open the key again and write down the Key ID, we'll need it later.

Recap

You should now have an App ID, Services ID and a Key.

Check if you have written down the required ids (team_id in the app, client_id, and key_id).

Make sure you have the key downloaded and safe.

Implementation

Sign in with Apple follows the OpenID or OAuth flows. If you've built or seen an implementation, it will look familiar.

However Apple decided to significantly up the game, so there is also a lot of new and unique things to implement.

The implementation below is for NodeJS backend service.

OAuth recap

The protocol is conceptually straightforward. In case you need a refresher, here are all the steps in detail:

The users clicks on the "Sign in with ABC" button in our web app.
This will call our backend service.
The backend service will build a URL to the providers sign-in page (with the required identifiers) and redirects the browser.
Now the user is on the ABC provider login page (ABC can be Google, Microsoft, Apple, Facebook, etc.)
The user signs-in (potentially with MFA or FaceID etc.).
If the login is correct the ABC provider redirect back to our callback URL (that's why we have to set it up in the Services ID section).
We are back in our backend service. The ABC provider redirected us back and passed us the CODE
Now that we have the CODE we can ask the ABC provider about user details.

Let's see how this work for Apple:

Button

Please follow the guidelines for the Sign in with Apple button design.

It is always a good idea to provide familiar UI, even more when dealing with security.

For our implementation the button is a link to /auth/apple

Prepare and redirect to Apple

The NodeJS handler will prepare the authentication URL and redirect to apple.

Please note:

We are sending the response_type "code id_token".
We are sending the scope email, so that we can get the user's email later from Apple
We are sending the response_mode as form_post. Without this, Apple will respond with an error (because we want the email).
Whatever you pass in state you'll get back from Apple. Use it to store extra info or as a nonce (you need to keep it also in session then).

app.get("/auth/apple", (req, res) => {
    const scope = 'email'; // we want to know the user's email
    const state = crypto.randomBytes(19).toString("hex");;
    const redirectUri = "https://inuko.net/auth/apple/callback";
    const client_id = APPLE_CLIENT_ID; // something like com.ourdomain.app

    const authorizationUri = `https://appleid.apple.com/auth/authorize?response_type=code id_token&client_id=${client_id}&redirect_uri=${redirectUri}&state=${state}&scope=${scope}&response_mode=form_post`;

    res.redirect(authorizationUri);
});

Handle return

For the following code to work we need to add 2 npm packages.

node-rsa
jsonwebtoken

Install both with npm install node-rsa jsonwebtoken.

Once Apple redirects back to our backend service we get 3 pieces of info.

The CODE, the id_token (identifying who is calling our web api) and state, which is just echoed back.

app.post('/auth/apple/callback', async (req, res, next) => {
    const { code, id_token, state } = req.body;
    //...
}

Verify caller

The first step is verifing that we are actually called by Apple.

Now this is not stricly required and the rest of the code will work without this step.

But, it is good practice. Please check the comments in the code for what's going on.

const verify_apple_id_token = async (id_token: string) => {

    // 1. Decode the id_token into parts. We need the header, so we pass the options complete: true.
    const parts = jwt.decode(id_token, { complete: true });

    // 2. Fetch apple public keys. There will be several. Cache if needed.
    const keyResp = await fetch(`https://appleid.apple.com/auth/keys`);
    const keyCollection = await keyResp.json();

    // 3. Find the right key in the key collection, that matches the token's key id (kid) and the algorithm used (alg).
    const publicKey64 = keyCollection.keys.find((x: any) => x.kid === parts.header.kid && x.alg === parts.header.alg);

    // 4. Transform the key information into a usable format.
    const NodeRSA = require("node-rsa")
    const pubKey = new NodeRSA();
    pubKey.importKey({ n: Buffer.from(publicKey64.n, 'base64'), e: Buffer.from(publicKey64.e, 'base64') }, 'components-public');
    const pk = pubKey.exportKey(['public']);

    // 5. Verify the token was made by Apple. This will THROW an exception, if validation fails!
    jwt.verify(id_token, pk, { algorithms: [parts.header.alg] });
}

Create client_secret

When we are sure, that it is trully Apple calling our backend service, we can fetch the user's details.

Hold your horses, not so fast, first we need the secret. The client_secret to be more precise.

Don't worry, we have all the ingredients, we just have to mix them in the right way.

Please check your notes to get them:

APPLE_TEAM_ID - your apple developer team id (you can get it from the App ID page or from your apple developer profile).
APPLE_CLIENT_ID - your App ID identifier (something like com.domain.app).
APPLE_PRIVATE_KEY - the text from the Key file you have downloaded in the Key section.
APPLE_KEY_ID - the key ID (you can get it when you click on the key in the apple developer identifiers page.)

const make_appleid_client_secret = () => {
    const payload = {
        iss: APPLE_TEAM_ID,
        iat: new Date().valueOf() / 1000,
        exp: new Date(new Date().valueOf() + 86400 * 1000 * 100).valueOf() / 1000,
        aud: "https://appleid.apple.com",
        sub: APPLE_CLIENT_ID
    };
    const private_key = APPLE_PRIVATE_KEY;
    const clientSecret = jwt.sign(payload, key, { algorithm: 'ES256', header: { "kid": APPLE_KEY_ID } });
    return clientSecret;
}

The secret we created above is valid for 100 days (for any user). The maximum allowed by Apple is 180 days.

If you decide you want to cache it, make sure you keep it safe!

Get the user's email

Finally we can call Apple and get the user's email, so that we can check they are indeed our user and log them in.

const fetch_appleid_user_info = async (code, clientSecret, state) =>{
    const d = new URLSearchParams();
    d.append('grant_type', 'authorization_code');
    d.append('code', code);
    d.append('client_id', APPLE_CLIENT_ID);
    d.append('client_secret', clientSecret);
    d.append('redirect_uri', "https://inuko.net/auth/apple/callback");
    d.append('scope', "email");
    d.append('state', state);

    const resp = await fetch("https://appleid.apple.com/auth/token", { method: "POST", body: d });
    const jresp = await resp.json();
    if (!jresp.id_token) {
        throw Error("Apple Login Error - no token - 0x2");
    }

    const token = jwt.decode(jresp.id_token);
    //const appleUserId = token.sub; // UNIQUE User identifier.
    const appleUserEmail = token.email;
    if (!appleUserEmail) {
        throw Error("Apple Login Error - no email - 0x3");
    }
    return appleUserEmail;
}

Putting it all together

Let's put all the pieces together now.

const jwt = require("jsonwebtoken");

app.post('/auth/apple/callback', async (req, res, next) => {
    try {
        const { code, id_token, state } = req.body;

        await verify_apple_id_token(id_token);

        const clientSecret = make_appleid_client_secret();

        const userEmail = await fetch_appleid_user_info(code, clientSecret, state);

        const user = await get_user_from_database_by_email(userEmail); // Replace with your actual code...
        if (!user)
            throw Error("User not found!");

        req.login(user, err => {
            if (err) {
                next(err);
            } else {
                // SUCCESS - go to app
                res.redirect("/");
            }
        })
    }
    catch (err) {
        next(err);
    }
});

That was fun...

Putting this together felt a lot like building a large Lego set without any instructions.

You know, like figuring our which parts have similar colors and fit together.

I hope you find the explanation and the code above helpful.

Whether you enjoy this kind of work or you just like to have this feature in your app:)

Happy hacking!