Forem: Joe Holmes

GPT-4 Spaced Repetition with "The Making of Modern Ukraine"

Joe Holmes — Mon, 26 Jun 2023 16:46:16 +0000

tl;dr

I developed a series of Colab notebooks that use the transcripts of Timothy Snyder's "Making of Modern Ukraine" course from Yale University to create spaced repetition flashcards for Anki using GPT-4.

They're intended to supplement listening to the lectures themselves. After finishing each lecture, you can review the key concepts from the lecture with Anki each day. In my experience, this resulted in super effective digestion/retention of the material--in this case, a sweeping and fascinating history from ~988 AD to the present.

You can check out the two Colab notebooks here:

And you can download the whole course's Anki flashcards, as well as its collection of generated markdown notes, here.

Finally, if you use these resources or want to express support for this project, please donate what you can spare to Razom.

Background history

I first discovered the awesome combination of spaced repetition and high quality online courses while working through The Teaching Company's course Everyday Engineering in 2021-2. For each of those lessons, I took a first pass of notes and screenshots, then followed it up with a second pass in which I converted my notes into flashcards. I did so using Obsidian and one of several community plugins for Anki upload, Flashcards.

While this manual process of card creation was a powerful way to retain the material, it took a lot of work over a long period of time (over a year!) So when ChatGPT/GPT-3.5 released, I knew I wanted to try to use it to streamline the process.

My first naive attempt was to upload chapters from US History's PDF guidebook and ask 3.5 to convert the passage to a few cards in a single prompt. It did not do a good job. But when GPT-4 was released and I was fortunate enough to get API access, I decided to dive in further and try developing something more robust.

For my first attempt, I used The Teaching Company's Fall and Rise of China course. The lessons were remarkably clear and vivid and GPT-4 did a great job generating cards, but the ToS prohibits duplicates or derivative works. I knew I wanted to share this process with the wider world somehow, so for my next experiment I opted for Timothy Snyder's Making of Modern Ukraine course, which is offered for free on Yale University's YouTube channel.

How it was built

In the first notebook, the Course importer, creates a .json file with all the course data, using YouTube, Substack, and the OpenAI embeddings API. In the second notebook, all I had to do was upload the JSON file to Colab every time I wanted to generate new cards.

The notebook uses the course playlist's YouTube ID to fetch the information from each video from using the YouTube v3 API. If you want to follow along, you can find your own version of this API key in the Google Cloud Platform under "APIs and Services."

You'll also need your secret key from OpenAI, along with GPT-4 access.

For each lecture, Professor Snyder wrote a blog post on his Substack that provides a brief summary along with a list of some key terms from the lesson. I ended up making a list of these, too, so I could experiment with them later.

The YouTube API gets each video's title, then the youtube-transcript-api pip package gets each one's transcript. As you've likely noticed if you've ever messed with YouTube's generated transcripts, the transcript is chunked into short, irregular strings of words that are each a few seconds long--not very useful for our purposes.

I found through experimentation that one minute was a good chunk length (it translated to a paragraph of text, more or less.) After splitting the transcript into single minute chunks, I used the nltk.tokenize module of the Natural Language Toolkit to ensure each chunk started and ending with a complete sentence.

Next, to scrape the Substack posts, I used the readabilipy library, a Python wrapper of the awesome readability lib from Mozilla. You can see the code that accomplishes all of this here.

At this point, the data object of the JSON file looks like this:

const course_data = {title: 'The Making of Modern Ukraine',
 teacher: 'Timothy Snyder',
 tag: 'MMU',
 lectures_url: 'https://www.youtube.com/playlist?list=PLh9mgdi4rNewfxO7LhBoz_1Mx1MaO6sw_',
 lectures: [
{title: 'Timothy Snyder: The Making of Modern Ukraine. Class 1: Ukrainian Questions Posed by Russian Invasion',
   id: 'bJczLlwp-d8',
   number: 1,
   blog_url: 'https://snyder.substack.com/p/making-of-modern-ukraine-lecture',
   lecture_transcript: [{start: '00:00', 'end': '01:00',
   text: "one minute of talking here"}
// and so on until the end of the lecture
blog_text: 'scraped blog text']},
// and so on until the end of the playlist
}]

With all the content imported, I moved on to creating embeddings of each chunk of transcript text. Embeddings are arrays of ML-generated numbers used to semantically code strings of text and rank their similarities. They were used in this project to look up the relevant sections of the transcript when improving the question and answer pairs.

If you want a great technical introduction to embeddings, Jay Alammar's blog post on the subject is sublime.

The code below adds an embedding to every chunk of the lecture transcript for every lecture in the course. Pretty simple.

import openai
openai.api_key = openai_sk

def get_embeddings(text):
    response = openai.Embedding.create(
        model="text-embedding-ada-002",
        encoding="cl100k_base",
        input=text,
        max_tokens=8191,
    )
    embeddings = response["data"][0]["embedding"]
    return embeddings

for lecture in course_data['lectures']:
  for chunk in lecture['lecture_transcript']:
    chunk['embedding'] = get_embeddings(chunk['text'])

All that was left after this was to export it.

It's interesting to think of AI-friendly data formats like this functioning as a kind of portable memory for LLMs as time goes on. If I ever want to do anything else with this course, I'll now have all of its content in a single JSON file, ready for similarity search and prompt injection with minimal fuss. I foresee creating more files like this in the future!

Flashcard generation

In the second Colab, the JSON file is imported, then the relevant lecture is selected. (As mentioned above, the notebook is designed to generate flashcards "on demand" as lessons are completed. This has the benefit of allowing tweaks to the prompts as I got a better grasp of what was effective.)

I set up a simple OpenAI API wrapper with the following settings:

def get_completion( user_prompt, model="gpt-3.5-turbo", system_prompt="You are a helpful history professor's assistant. You love capturing the details of history, distilling complex ideas into clear language, and effectively teaching for long-term retention.",):
  completion = openai.ChatCompletion.create(model=model, temperature=0.2, messages=[{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}])
  return completion['choices'][0]['message']['content']

GPT-3.5 is default simply to encourage saving money, but throughout the notebook I found myself falling back on 4's superior reasoning capabilities. The temperature is low to encourage facticity and the system prompt tries putting the LLM in a professorly state of mind.

I then used FAISS to create a simple function for cosine similarity search.

Put briefly, cosine similarity is an efficient way of measuring the similarity of two vectors (i.e. embeddings), which we'll use to find the most similar transcript chunks to an embedding of each flashcard Q&A (a "query embedding.")

If you'd like to know more about how cosine similarity works, I'd recommend Jeremy Howard's lesson on recommendation systems from Practical Deep Learning for Coders. Here's the relevant section. You can see the code I wrote for this function here.

Now that the set up is done, we're ready to move onto the prompting. I've sketched up the basic structure in a flowchart, which you can click on below to look through:

In brief, I wrote an "orientation and rules" prompt that explains what the project is and what effective summarization/question answering looks like (lots of detail, always defining proper nouns, etc.) This prompt is inserted into pretty much every subsequent prompt.

Then the lecture transcript is summarized in a "rolling" way, similar to the refine summary type in Langchain, where each micro-summary is inserted into the subsequent summary prompt (called "the story so far") until the summary is done.

The vocabulary terms from the blog post are all defined using the same process. (I tried a few experimental things with the vocab lists, such as inserting them into later prompt injections and developing Cloze deletion Anki cards out of them, but ended up keeping it simple and not using them.)

Then the summary is used to generate 5-7 great questions. I posed this to GPT-4 as a challenge of asking questions whose answers would display a total, complete, expert-level mastery of the content.

I found this more effective than generating the questions and answers at the same time. It seemed like the model had more focus when the tasks were split. However, decisions like these can cause increased expenses down the road, so you have to weigh your priorities.

The vocab definitions and summary are used together to answer the questions, and the timestamps from the transcript have been included in every output so far such that a "SOURCES" section of the flashcard answer is included in each one. This is an important way to mitigate hallucinations and make these cards more valuable learning tools.

Once the questions and answers are generated, they're converted into JSON. (If the GPT-4 Functions update was released when I worked on this, I would've likely simplified this code quite a bit.) After the JSON is imported in the following section, I was able to loop through each question and answer string and give each one its own prompt and output.

In this loop, the prompt first obtains the embedding of the question and answer string using the same get_embeddings function from earlier. It then uses the FAISS cosine similarity search to retrieve the 2 most similar chunks of transcript text to the flashcard. These chunks are inserted into the prompt as background information which GPT-4 is expected to use to improve the quality of the original question and answer.

Once that's done, each refined Q&A is concatenated and added to a big markdown file along with a tiny summary, a haiku about the lesson's subject, the blog summary and defined terms list, and the completed flashcards.

Every time I finished a lecture I'd download this markdown file to my Obsidian vault, review the notes and make whatever small tweaks I wanted to, and then uploaded them into Anki for review.

Results

Relative to the amount of time and work it took, the system performed quite well. If I intended to use these cards and notes for more applications, they'd all probably need a second pass. I'm sure I'd find some important themes to draw out and drill if I listened to the lectures a second time after this rapid fire study session. But as a simple prototype, these notebooks succeeded in retaining the broad strokes of Snyder's challenging and fascinating lectures. Comparing the time it took to write the notebooks (about a week and a half of focused side project time) with the time it took to manually create flashcards for the engineering course I spoke of earlier (one year+), it boggles the mind how much GPT-4 increased my productivity.

Here's an example flashcard from lecture 7, on the rise of Muscovite power:

What are the two key factors that set East Slavic Rus' apart from other post-Viking states, and how do they relate to the rise of Mongol Rus' and its connection to Kyivan Rus'? #card #MMU #MMU7

East Slavic Rus' stands apart from other post-Viking states due to its Eastern Christian influence, which connects it religiously and culturally to the Byzantine Empire and other regions. The other post-Viking states were Western Christian.
Unlike other post-Viking states like England or Denmark, Rus' no longer exists as an independent entity, raising questions about its successors and their roles in shaping regional politics. Many states come to claim Rus' in the ensuing years.
One such successor is Mongol Rus', which is distinct from other Rus' territories like Galicia-Volhynia due to its centuries-long Mongol rule, positioning it within the broader Mongol or post-Mongol states across Asia and eastern Europe. Mongol Rus' is what will eventually become modern Russia.
Mongol Rus' is a fresh start after the Mongols, in which Eastern Christianity exists (without alternatives) but the laws and "civilizational package" from Kyivan Rus are largely absent.

SOURCES: Eastern Christian influence 4:30-6:30, Rus' no longer exists 11:00-12:00, Mongol Rus' and its connection to Kyivan Rus' 20:00-21:00

The hashtags at the end of the question are translated by the plugin into tags usable in Anki, such that I can easily find all of a lesson's cards. Note also the sources section at the bottom. One thing I'd do next time for a publicly available course is design these to work as hyperlinks that take the user to the YouTube lecture's timestamp.

Another nice result is that the proper names are almost always spelled correctly, which can be challenging with old Slavic spellings particularly. This was one of my motivations for creating the vocab terms list I described earlier.

While this card's answer content is on the long side, it's still manageable. You can see in the notebook's prompts that I encourage it to be as brief as possible at this stage, but it can still bring in some fluff. In other cards, the answers become way too long, such that midway through the course I was regularly in "Anki hell," taking 30+ minutes each morning to review my cards. A good brain workout, to be sure! But a bit too time-consuming.

Take this one, on the Ukrainian famine of 1932-3:

What were the seven policies that led to the political famine in Soviet Ukraine, and how did they impact food distribution and the lives of Ukrainian peasants? #card #MMU #MMU15

The seven policies were:
1. The return of grain advances
2. The meat penalty
3. The blacklists
4. The national interpretation of the famine
5. The affirmation of the existing grain quota
6. The ban on peasants going to cities to beg for food
7. The separation of the Ukrainian Republic from its neighbors
These policies impacted food distribution by prioritizing grain requisition targets over feeding the population, leading to widespread starvation and death among Ukrainian peasants.
The famine was a result of political choices about how to treat particular people, not a lack of food.
The Soviets had food reserves and were exporting food from ports in Soviet Ukraine.
The ban on peasants going to cities to beg for food forced peasants to stay in the countryside, where the state had taken total control and extracted food.
Blacklisted collective farms were cut off from the rest of the Soviet economy, making it illegal for them to exchange goods or services.
The national interpretation of the famine blamed Ukrainization for the famine and labeled those in favor of Ukrainization as dangerous, putting pressure on party members to continue requisitioning grain or face punishment.

SOURCES: explanation of seven policies 34:00-42:00, Soviets had food reserves and were exporting food 34:00-35:00, unusual situation of peasants fleeing to cities 40:00-41:00.

In this case, I needed to split the card into two sections--one testing for simple recall of the policies, another for the more detailed information in the second section. This was fine, though, because the system already appeared to function best with a "human in the loop" approach, reviewing each lesson's notes and editing its cards before sending it on to Anki.

Where the prompts did not perform as well was in Snyder's more abstract lectures, which took the scenic route through some elaborate and deeply nuanced arguments. The model had a hard time following Snyder's train of thought, but to be fair to GPT-4 I think most humans (myself included) would have a hard time as well. These lectures took place mostly at the start and end of the course, and these flashcards required more human attention after the fact. The shared version at the top of this article contains the edited versions of those lessons, which are well worth the added difficulty as Snyder's sweeping historical statements are brilliant--one gets the sense of watching a groundbreaking historian creating new interpretations of current events before one's very eyes.

Making matters more challenging was that the course assigned readings, so there was important background information we missed out on. Here, more consumer-oriented products like the Great Courses series interface much better with the language model.

Take-aways and next steps

One set of ideas that kept coming to mind during this project was the notion of LLM interaction as a form of "cyborgism" (to be contrasted with RLHF-enabled personification of the language model) as described by Janus in a popular LessWrong article. In it, Janus counsels LLM users to try thinking of the AI as an extension of your own mind--like a special pair of glasses, or a robotic hand--instead of as an Other you're expected to converse with. This can reveal creative possibilities for application, as this project seemed to be at the time. It felt like the AI compressed my own thinking and note-taking process into a tiny fraction of the time it'd otherwise take to learn the subject, like Neo learning kung fu in The Matrix.

Another thought was that history is uniquely well-suited to edtech experiments with AI at this stage in its development, because it's almost all textual. Other disciplines may require more images or fancy script (such as chemistry notation, LaTEX, etc.)

Next time I think I'll try codifying the "human in the loop" aspect of generation more, perhaps splitting the flashcard making process into two: first, generating notes with GPT-4 on the lecture and summarizing it from a variety of angles, then prompting the human to select from these notes which ideas he/she finds most valuable for emphasis. Then the amended note could be uploaded to a new notebook, and that human feedback could be incorporated into the generation of the cards.

Thanks for reading. If you'd like to see more of my work, check out my portfolio site. And if you enjoyed this article, please consider donating what you can to Ukrainian relief efforts. Thank you.

Templater Treats for Obsidian October

Joe Holmes — Sat, 09 Oct 2021 23:40:48 +0000

I wrote two small Templater scripts for Obsidian October*, both with their own Python micro-APIs. One transcribes YouTube videos and one fetches the day's quote from DailyZen. Neither are anywhere as complicated as the plugins people are submitting, but it feels good to participate all the same.

*Obsidian is a note-taking app that has lots of novel ways to link your notes together and visualize them. It's totally free, so if it sounds intriguing to you you should try it.

Daily Zen Quote in Your Journal

First comes the API, hosted on Replit and written in Python and Flask. I find Beautiful Soup, the Python library for web scraping, to be easier to work with than the web scrapers popular for JavaScript.

import requests
from bs4 import BeautifulSoup
from markdownify import markdownify as markdownify
from flask import Flask
from flask_cors import CORS
import re
import prettierfier

app = Flask("app")
CORS(app)

@app.route("/")
def output():
  URL = "https://dailyzen.com"
  page = requests.get(URL)
  soup = BeautifulSoup(page.content, "html.parser")
  html = f"{soup('blockquote')[0]}{soup('cite')[0]}"
  pretty_html = prettierfier.prettify_html(html)
  markdown = markdownify(pretty_html, convert=['blockquote', 'cite'])
  quote = re.sub(">", "", markdown)
  print(quote) 
  return f">  {quote.lstrip().rstrip()}"

app.run(host="0.0.0.0", port=8080)

I pay a couple bucks a month to have the Replit hacker plan, which keeps certain servers "always on" without having to configure anything.

Since nothing needs to be passed into this one, it's pretty simple.

async function dailyzen() {
  const result = await fetch("https://DailyZen.bathrobe.repl.co");
  const reader = result.body.getReader();
  const { value } = await reader.read();
  const quote = new TextDecoder().decode(value);
  return quote;
}
module.exports = dailyzen;

Replit sends stuff over in ReadableStream, so you usually need to import it into your file in bite-size chunks. The quotes are so short that they come in entirely on the first chunk.

Then in Templater just call it with tp.user.dailyzen(). Voila.

Fetch YouTube Transcripts in Your Notes

This one's a little trickier.
First, a Templater system prompt gets a YouTube URL. Then the URL is verified, we make sure it's actually a link to a video, and the ID of the video is sent to the server.

Behind the scenes, Replit sends a big ReadableStream back with the transcript in chunks. Those chunks are gathered up until the stream completes, then they're concatenated.

async function transcript(str) {
  let url = new URL(str);
  let videoID;
  //test and make sure it's a youtube URL
  if (url.host == "www.youtube.com" && url.pathname == "/watch") {
    videoID = url.search.slice(3);
    //send videoID as POST request to replit
    const res = await fetch("https://YTTranscript.bathrobe.repl.co", {
      method: "post",
      headers: {
        Accept: "application/json",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        ID: videoID,
      }),
    });
    //replit returns a readablestream
    const reader = res.body.getReader();
    let result = "";
    let transcriptOutput = "";
    await reader.read().then(function processText({ done, value }) {
      if (done) {
        transcriptOutput = result;
        return transcriptOutput;
      }
      let stringValue = new TextDecoder("utf-8").decode(value);
      result += stringValue;
      return reader.read().then(processText);
    });
    return transcriptOutput;
  } else {
    console.log("nope");
    return "I can't recognize this as a YouTube link, sorry!";
  }
  //get the markdown transcript back and return it
}
module.exports = transcript;

Behind the scenes, here's Replit getting us the transcript. There's a pip package available that makes it really easy:

from flask import Flask
from flask import request
from flask_cors import CORS
from youtube_transcript_api import YouTubeTranscriptApi

app = Flask("app")
CORS(app)

@app.route("/", methods=["POST"])
def output():
  ytID = request.json.get("ID")
  transcriptData = YouTubeTranscriptApi.get_transcript(ytID)
  transcript = ""
  for dict in transcriptData:
    transcript += dict["text"] + " \n"
  return transcript

app.run(host="0.0.0.0", port=8080)

Notice how there's a line break at the end of every string returned by the transcript package.

There are several ways to do this, but I found it easiest to prompt the user for a URL within the Templater markdown file itself.

<% await tp.user.transcript(tp.system.prompt("YouTube URL Here"))%>.

Try them out!

If you want to use my servers, help yourself! Just code the JavaScript portions of this tutorial. Of course, they may disappear at any time--these were some just for fun projects.

How to Create a Todo List with React and Sanity

Joe Holmes — Tue, 22 Jun 2021 22:11:41 +0000

Sanity is often described as a "headless CMS"—a content management system decoupled from its presentation, allowing a developer to create any kind of frontend they desire.

But most "headless CMS" platforms are rather limited in what they can accomplish. They're usually designed to be blogs, e-commerce stores, or marketing pages—try and do something more avant-garde and you may soon find yourself in the weeds.

Not so with Sanity! With its thoughtful developer experience and lightning-fast APIs, Sanity can also compare favorably to "managed databases" such as Firebase and Supabase. Systems like these are designed for more complex, technical applications that require user authentication and dynamically presented data. Sanity can hold its own here!

Don't believe me? Let's build an app.

We'll be coding a user authenticated to-do list app with Next.js, Magic.link, and Sanity on the backend.

I've deployed a demo version if you want to check it out yourself.

Requirements

Intermediate knowledge of React, including state, context and effect hooks
Familiarity with TailwindCSS for styling
Sanity CLI installed
Magic.link account created

Clone starter

I created a simple starter project so we wouldn't lose time writing boilerplate. You can clone it by navigating to your preferred directory and running the following command.

git clone https://github.com/bathrobe/magic-next-sanity-todo-starter.git

The starter comes with Tailwind CSS for styling and has a basic user authentication flow already implemented, so we won't need to dive into those challenging waters in this post. The auth code is all taken from a very helpful tutorial on Vercel's website, so consult that if you'd like to know the details.

Let's first install our project's dependencies by running cd magic-next-sanity-todo-starter and then npm install .

When that's done, create a .env.local file at the root of your project. We'll add the keys one by one.

Register for Magic.link

Magic.link uses cutting-edge cryptography to take the headache out of user authentication, which is usually one of the most dreaded developer tasks. They utilize some cookie-based blockchain wizardry to remove the need for passwords or outside auth. You do not need to understand how its innards work in order to use it. Just enjoy!

Sign up for the free plan at https://dashboard.magic.link/signup. At the time of writing, you're permitted 100 users on the free plan, so it's perfect for our purposes.

Get your public and secret test keys. You'll need an encryption secret too. You can go to this Encryption Key Generator and choose any option above 256 bit.

// .env.local

NEXT_PUBLIC_MAGIC_PUB_KEY=pk_test_12345
MAGIC_SECRET_KEY=sk_test_12345
ENCRYPTION_SECRET=random_encryption_string

Deploy a todo schema from Sanity

If you have not yet done so, remember to download the Sanity CLI. Open your terminal and enter the following code.

npm install -g @sanity/cli

If you don't already have a studio you'd like to work with, run sanity init from the command line and create a new one.

The command line will ask you for the name of your studio and where you want to store the studio files. In this example I named it "your-studio" and installed it in the same folder where I stored the starter repo.

When it asks you what schema you'd like to start with, choose the option to install a clean schema with no pre-existing data.

Take note of what you named the dataset, as you'll add it to the .env file later.

Now we'll write a todo schema in our schemas folder.

// your-studio/schemas/todo.js

export default {
    name: "todo",
    title: "Todo",
    type: "document",
    fields: [
      {
        name: "text",
        title: "Todo Text",
        type: "string",
      },
      {
        name: "createdAt",
        title: "Created at",
        type: "datetime",
      },
      {
        name: "dueDate",
        title: "Due date",
        type: "datetime",
      },
      {
        name: "isCompleted",
        title: "Is completed?",
        type: "boolean",
      },
      {
        name: "completedAt",
        title: "Completed At",
        type: "datetime",
      },
      {
        name: "userEmail",
        title: "User Email",
        type: "string",
      },
    ],
  };

Add the schema to your schema.js file.

// your-studio/schemas/schema.js

import createSchema from "part:@sanity/base/schema-creator";
import schemaTypes from "all:part:@sanity/base/schema-type";
//...
import todo from "./todo"

export default createSchema({
  name: "default",
  types: schemaTypes.concat([
        //...
    todo
  ]),
});

Run sanity deploy, choose a unique name for your deployed studio, and soon it should be live.

Alert: Technically, you don't need to deploy your studio—simply accessing the document store and using the APIs is enough for these purposes. But assuming you'd

Get Sanity permissions

Go to sanity.io/manage and head to Settings > API > Tokens. Add a new token with write permissions and save the key somewhere safe.

Directly above the tokens field, add http://localhost:3000 to your CORS Origins. (If you deploy the app, you'll need to add your live URL to this list as well).

You can find your project ID at the top of this page, right under the title of your studio. Add all your Sanity variables to your .env.local.

// .env.local

NEXT_PUBLIC_MAGIC_PUB_KEY=pk_test_12345
MAGIC_SECRET_KEY=sk_test_12345
ENCRYPTION_SECRET=random_encryption_string_from_earlier
NEXT_PUBLIC_SANITY_ID=your_sanity_id
NEXT_PUBLIC_SANITY_DATASET=your_sanity_dataset
SANITY_WRITE_KEY=your_sanity_write_key

All right, we're configured and ready to start coding! Run npm run dev in your project's root and test it out.

Creating the input form

Let's start with the submit form that sends our todos to the studio. Later we'll fetch the data and display it on our frontend.

We can look at the finished product to see what we'll be adding.

First we'll use the useState() hook to store the values of our submit form and todo list.

Creating states to store form values

Let's import our hook as well as the react-date-picker library. Then we'll add our states.

// src/pages/todos.js

import { useState } from "react";
//we must import the datepicker's css modules manually
//so it plays nice with Next.
import DatePicker from "react-date-picker/dist/entry.nostyle";
import "react-date-picker/dist/DatePicker.css";
import "react-calendar/dist/Calendar.css";
import useAuth from "../hooks/useAuth";
import Logout from "../components/Logout";

export default function Todos() {
  const { user, loading } = useAuth();
    //create a state to store todoList array
  const [todoList, setTodoList] = useState([]);
  //create a state for the text in the todo input form
  const [userInput, setUserInput] = useState("");
  //create a state for the due date chosen in the datepicker
  const [dueDate, setDueDate] = useState("");
  //set an error message if either input is missing
  const [errMessage, setErrMessage] = useState("");
    //...

Note the existence of our custom useAuth() hook at the top. More information about custom hooks (as well as a helpful refresher on all things React hooks!) can be found in this Fireship video.

Next we'll define some handlers for our form input and submission button.

Create `handleSubmit` and `handleChange` functions


// src/pages/todos.js

//... right after the useState hooks

//FOR THE INPUT FORM:
const handleChange = (e) => {
    e.preventDefault();
    setUserInput(e.target.value);
  };

    //FOR THE SUBMIT BUTTON:
  const handleSubmit = async (e) => {
    e.preventDefault();
    //if either part of the form isn't filled out
    //set an error message and exit
    if (userInput.length == 0 || dueDate == "") {
      setErrMessage("Todo text and due date must be filled out.");
    } else {
      //otherwise send the todo to our api
            // (we'll make this next!)
      await fetch("/api/todo", {
        method: "POST",
        body: JSON.stringify({
          text: userInput,
          dueDate: dueDate,
          user: user.email,
        }),
      });
      // await fetchTodos(); //(we'll add this later)
      // Clear all inputs after the todo is sent to Sanity
      setUserInput("");
      setErrMessage("");
      setDueDate("");
    }
  };

handleChange stores our text input in a state. handleSubmit first makes sure both of our fields have been filled out, then posts the todo to our serverless API route and clears out the inputs.

You'll notice two things we reference that don't exist yet: the api/todo route and a fetchTodos() function. Fret not! We'll get to these in the next section. For now, we'll finish rendering our form.

Creating the input form

Replace <p>Todo app will go right here!</p> with the form element below.

// src/pages/todos.js

                {/*...*/}
                <form>
                    {/*we flex the text input and datepicker
                    so they display inline. */}
          <div className="flex justify-center items-center">
                    <label for="todo" className="invisible">Your Todo</label>
            <input
              className="w-72 h-12 border p-4 border-blue-100"
              type="text"
                            //our state
              value={userInput}
              placeholder="Make coffee."
                            //our function
              onChange={handleChange}
            />
            <div className="my-8">
              <DatePicker
                className="p-4"
                                //makes it so we cannot set due date in past
                minDate={new Date()}
                                //our dueDate state
                onChange={setDueDate}
                value={dueDate}
              />
            </div>
          </div>{" "}
          <button
            className="focus:outline-none focus:ring focus:border-blue-800
                        px-6 py-2 rounded-xl bg-blue-500 text-blue-50 hover:bg-blue-800 
                        font-semibold"
                        //our function
            onClick={handleSubmit}
          >
            Submit
          </button>
                    {/*error set in handleSubmit*/}
          <p>{errMessage}</p>
        </form>
                {/*...*/}

You should now see a lovely todo input form, with a stylish button lifted almost entirely from Tailwind's helpful docs.

Remember the serverless API route from earlier? Let's build that now.

Make our API route

Create src/pages/api/todo.js. Since we'll want full CRUD functionality in our app, we'll need PUT and DELETE requests later. To keep our code clean, we'll use the switch syntax for the different request types.

// src/pages/api/todo.js

import client from "../../lib/sanity/client";

export default async function handler(req, res) {
  switch (req.method) {
    case "POST":
      //this JSON arrives as a string,
      //so we turn it into a JS object with JSON.parse()
      const newTodo = await JSON.parse(req.body);
      //then use the Sanity client to create a new todo doc
      try {
        await client
          .create({
            _type: "todo",
            text: newTodo.text,
            isCompleted: false,
            createdAt: new Date().toISOString(),
            dueDate: newTodo.dueDate,
            userEmail: newTodo.user,
          })
          .then((res) => {
            console.log(`Todo was created, document ID is ${res._id}`);
          });
        res
          .status(200)
          .json({ msg: `Todo was created, document ID is ${res._id}` });
      } catch (err) {
        console.error(err);
        res.status(500).json({ msg: "Error, check console" });
      }

      break;
  }
}

With that, we should be able to submit a todo from our app and have it stored in our Sanity studio. Let's try it out.

Run npm run dev once more and open up your Sanity studio. If all went well, you should see your new todo inside of Sanity.

Success!

Great job! Now that Sanity is storing our todo values, we can fetch them and display them on our frontend.

Display our todos

We'll define a function that will fetch the latest todos from Sanity and store them in the todoList state. We want the function to run when the page loads, whenever user data changes, and any time we alter the data (by adding, updating, or deleting todos). To manage this logic, we'll begin by adding a useEffect hook.

Head back to pages/todos.js and importuseEffect, as well as our Sanity client.

//src/pages/todos.js

import { useState, useEffect } from "react";
// ...
import client from "../lib/sanity/client";

Right below your useState hooks add the fetchTodos function and tack it onto a useEffect hook.

//src/pages/todos.js

//after the useState hooks
const fetchTodos = async () => {
    let fetchedTodos;
        //make sure the user is loaded
    if (!loading) {
            //pass userEmail as a query parameter
      fetchedTodos = await client.fetch(
            `*[_type=="todo" && userEmail==$userEmail] | order(dueDate asc)
                {_id, text, createdAt, dueDate, isCompleted, completedAt, userEmail}`,
        {
          userEmail: user.email,
        });
            //insert our response in the todoList state
      setTodoList(fetchedTodos);
    }
  };

  useEffect(
    () => {
            //now it will fetch todos on page load...
      fetchTodos();
    },
    //this dependecy array tells React to run the
        //hook again whenever the user loads or changes
    [loading, user]
  );

Uncomment fetchTodos() in the handleSubmit function. Next polyfills the fetch API for all browsers beforehand, so no need to worry about that!

// src/pages/todos.js
// in handleSubmit function...
      await fetch("/api/todo", {
        method: "POST",
        body: JSON.stringify({
          text: userInput,
          dueDate: dueDate,
          user: user.email,
        }),
      });
      //***uncomment this line now***
            //after submitting, our TodoList will now refresh
      await fetchTodos();
            setUserInput("");
      setErrMessage("");
      setDueDate("");
    }
  };
//...

Immediately after handleSubmit, let's also add a handleDelete function that we can pass to our <Todo/> component. Since we're passing this function straight from the page to <Todo/> (skipping the <TodoList/> component), we should use React's useContext hook. Context allows us to avoid passing props unnecessarily.

We'll also pass fetchTodos() to our context hook, so that we can get fresh data when we toggle a todo's status in its component.

// src/pages/todos.js

import { useState, useEffect, createContext } from "react";
//... before the Page component
export const TodoContext = createContext()

export default function Todos() { //...

// then, below the handleSubmit function...

const handleDelete = async (selectedTodo) => {
    await fetch("/api/todo", {
      method: "DELETE",
      body: selectedTodo._id,
    });
        //todos will refresh after delete, too
    await fetchTodos();
  };

    console.log(todoList)

  return (
    <TodoContext.Provider value={{handleDelete, fetchTodos}>
        {/* all your rendered JSX */}
        </TodoContext.Provider>

Check your console to make sure we fetched it all correctly. After you've confirmed it works, feel free to delete the console log.

Very good! It's right there waiting for us. Now that we've fetched the data, we need components to feed it to. That's what we'll build next.

Creating todo components

First create a components folder in src. In it, create a new file called TodoList.js. This will be a simple component that mainly exists to keep our todos.js page a little cleaner.

// src/components/TodoList.js

import Todo from "./Todo";

export default function TodoList({ todoList, user }) {
  return (
    <section>
      <ul>
            {/*if there are todos in the list...*/}
        {todoList.length >= 1
          ? todoList.map((todo, idx) => {
                            //map only the user's todos
              return user.email == todo.userEmail ? (
                <Todo key={todo._id} todo={todo} />
              ) : (
                ""
              );
            })
          : "Enter a todo item"}
      </ul>
    </section>
  );
}

We pass the todoList state in our page component to <TodoList/>, which maps each item in the array to a <Todo/> component.

The ternary operators may look tricky, but they're only saying, "If the todo list has one or more items, go through each item, double-check that they belong to the current user, then render them with our custom Todo component."

The unique ID from Sanity is passed as the key prop in the Todo because React requires it. React has more information about the key prop in their docs.

Now we need to make Todo.js. In the same folder, create that file. Remember the context we created in todos.js? We can now put it into action.

// src/components/Todo.js

import { useState, useContext } from "react";
// import a simple date formatting library
import dayjs from "dayjs";
// import a trashcan icon for our delete button
import { RiDeleteBin5Line } from "react-icons/ri";
import { TodoContext } from "../pages/todos"

export default function Todo({ todo }) {
//with useContext we do not need to pass extra props to <TodoList/>
const { handleDelete, fetchTodos } = useContext(TodoContext)
    //setting states for the isCompleted boolean and a date completed
  const [isCompleted, setIsCompleted] = useState(todo.isCompleted);
  const [completedTime, setCompletedTime] = useState(todo.completedAt);

    //function that syncs the completed checkbox with Sanity
  const handleToggle = async (e) => {
    e.preventDefault();
    const result = await fetch("/api/todo", {
      method: "PUT",
      headers: {
        Accept: "application/json",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        id: todo._id,
                //passes isCompleted React state to Sanity
        isCompleted: isCompleted,
        completedAt: todo.completedAt,
      }),
    });

    const { status, completedAt } = await result.json();
        // refresh our data
        await fetchTodos();
        //pass our Sanity results back into React
    setIsCompleted(status);
    setCompletedTime(completedAt);
  };
  return (
    <li
      className="bg-gray-50 my-6 border shadow-md rounded-xl 
            p-4 border-gray-200 flex justify-center items-center"
      key={todo._id}
    >
      <input
        className="mx-2 cursor-pointer"
        type="checkbox"
        checked={todo.isCompleted}
        onChange={handleToggle}
      />
            {/*if todo is done, cross it out and turn it gray*/}
      <p
        className={`text-lg mx-2 ${
          todo.isCompleted ? "line-through text-gray-500" : ""
        }`}
      >
        {todo.text}
      </p>
      <p className={`text-gray-400 mr-2`}>
                {/*if todo is done, show completedTime
                    if not done, show due date */}
        {todo.isCompleted
          ? `Done ${dayjs(completedTime).format("MMM D, YYYY")}`
          : `Due ${dayjs(todo.dueDate).format("MMM D, YYYY")}`}
      </p>
      <button
        className="mx-2"
        onClick={(e) => {
          e.preventDefault();
          handleDelete(todo);
        }}
      >
        <RiDeleteBin5Line />
      </button>
    </li>
  );
}

Excellent work! All we need to do is add the components to our page and they should render.

Updating `todos.js`

// src/pages/todos.js

// ... at the bottom of the imports
import TodoList from "../components/TodoList"

// ... then directly under the form
                <form> {/*...*/> </form>
                <div className="my-12">
          <h1 className="text-xl font-bold tracking-tight 
                    my-8">Your Todos</h1>
          {loading ? (
            "loading..."
          ) : (
            <TodoList
              user={user}
              todoList={todoList}
            />
          )}
        </div>
//...

Now run npm run dev and you should see your todo items appear.

👨‍🍳 😗

If you try to toggle or delete them you'll throw an error, because we haven't added the PUT and DELETE API routes yet. Let's do those now.

Add the new routes to our API

// src/pages/api/todo.js

//...
//after the POST request

case "PUT":
      const result = await client
        .patch(req.body.id)
        .set({
          isCompleted: !req.body.isCompleted,
                    //create new complete date if Todo is marked as done
          completedAt: !!req.body.isCompleted ? "" : new Date().toISOString(),
        })
        .commit();
      res.status(200).json({
        status: result.isCompleted,
        completedAt: result.completedAt,
      });

      break;
case "DELETE":
      await client
        .delete(req.body)
        .then((res) => {
          res.body;
        })
        .then((res) => console.log(`Todo was deleted`));
      res.status(200).json({ msg: "Success" });
      break;
//...

If your dev environment is still running, stop and start it again with ctrl/command C and npm run dev. You should now be able to mark todos complete and delete them.

The most user-friendly database in the world

If all went well, take a look at your Sanity studio. Your content should update in real-time when the frontend input changes and vice versa.

Now you've got the developer experience of a managed database with the friendly, customizable UI of a content manager. Pretty sweet.

I've made the completed repo public if you want to give it a look.

Next Steps

Hopefully this project can serve as a starting point for more ambitious uses of Sanity. There are plenty of ways to improve the app.

Right now the only way todos relate to their users is by passing in the user email as a field in individual todos. We should create a user.js document in schemas that allows each person to create their own profile. Then that document could be related to each todo of a given user via a reference field.
Completed todos could be sent to their own list in an archive, instead of being displayed in the same spot on the Todo List component. In other words, instead of all todos appearing under "Your Todos", you could have separate lists for "Pending Todos" and "Completed Todos". This would be great practice if you're trying to develop more advanced React skills.
For a fun challenge, could you add a pomodoro timer to specified todo lists and display the countdown in the studio? I've found dividing my work periods into smaller, timed chunks has been a big help in staying productive.

As always, feel free to say hi on Twitter. Happy coding!

How to Create a Telegram Bot with Node.js and Sanity

Joe Holmes — Tue, 22 Jun 2021 22:10:25 +0000

Of the many platforms that have seen a surge in use over the past year, privacy-focused chat app Telegram is among the most successful. In April 2020, Telegram announced it hit 400 million monthly users. It now plans to reach a billion by 2022.

If you're a privacy advocate, beginner developer, or simply like chatting with your friends, creating Telegram chatbots is a great way to practice coding and build something cool. The platform is only getting more popular, so knowledge of the Telegram API is a great skill to have!

In this tutorial, we'll build a simple chatbot in Telegram using NodeJS and Replit. It will send selected memes from our group chat back to an instance of Sanity studio, where we can save them and publish them to a blog.

There are plenty of options for an aspiring chatbot builder— Discord, Facebook, Slack... So why are we going with Telegram?

For one, its API is beginner friendly, so if it's your first time coding a bot you should feel comfortable.

Secondly, Telegram is open source and can be rebuilt from the ground up using the source code (check out the React version). So if you have fun with our project, there's no limit to how far you can extend it in the ecosystem!

Requirements

Basic knowledge of NodeJS
Sanity CLI installed
Telegram account created
Replit account created

Initializing the Telegram bot with Botfather

To begin let's use Botfather, Telegram's in-house bot wizard, to initialize our bot and grab its API keys.

Download Telegram desktop. Visit Botfather to immediately open a chat and begin creating your bot.

Running /start tells Botfather to boot up and show you all of its available commands.

As you can see, there are many options available.

We'll run /newbot. Give your bot a display name, then a username that will be registered with Telegram. The latter has to end with "bot." I chose sanity_gc_memebot for the purposes of this tutorial.

When you give Botfather your bot's username, he'll respond with an API key you can use to program your bot. Copy it and store it somewhere safe. This is all you need to get started!

Next, we'll head to Replit and start hacking on our bot's code.

Starting a new Replit with NodeJS

Replit is an amazing tool for cloud coding. It allows us to write and run our code directly in the browser, free from the tedium of setting up a development environment.

With its pre-built Node.js template we can install Node packages without downloading anything to our hard drive. When we finish our bot, we can leave the Replit running and treat it just like a free, zero-config server.

If you haven't already, create a Replit account. Then select "New Repl" on the left sidebar and "Node.js" from the dropdown.

You should now be greeted with an empty index.js file. Let's get to work!

Setting up our environment variables

At this point, maybe you're a little suspicious. "I'm supposed to keep my API key secret at all times. Am I expected to just paste the key into an in-browser sandbox?" Good instincts!

Thankfully, Replit comes with support for environment variables. Your REPL is likely public, since private REPLs are a paid feature, but that is no cause for alarm. Only the creator and team members of the REPL will be able to view the secret keys.

Click the lock icon on the left sidebar to access the environment variable screen. Once you get there, input the key TELEGRAM_BOT_KEY and paste the value you received from Botfather.

Now we're ready to initialize our bot.

Booting up our bot

In your index.js file write the following code:

// index.js

// imports the node package for us
const TelegramBot = require('node-telegram-bot-api');

// grabs the environment variable
const token = process.env.TELEGRAM_BOT_KEY;

// starts a new Telegram bot instance that "polls" for updates
const bot = new TelegramBot(token, {polling: true});

Click the big green button 'Run' at the top of the Replit window. You should see two things happen automatically: your package.json initializes, and the node-telegram-bot-api package on npm is installed. Usually you have to do both of these things by hand. Pretty sweet!

Giving our bot some basic functionality

Now let's make the bot do a couple tricks and confirm that it's working.

The node package we're working with, [node-telegram-bot-api](https://github.com/yagop/node-telegram-bot-api/blob/master/doc/tutorials.md), comes with great documentation. Let's start with one of the methods from the README. We'll tell our bot to send an introduction anytime a user runs /start.

// index.js

// ... after everything we wrote already

// we're using the API's in-built "onText" method
// it takes in a regex and the message
bot.onText(/\/start/, (msg) => {
// listens for "/start" and responds with the greeting below.
bot.sendMessage(msg.chat.id,
"Hey, I'm a brand new Telegram bot. I live inside a Sanity tutorial.");
});

Be sure to click the "Run" button in Replit again, then head over to Telegram and test it out. (If you don't have a dialogue open with your bot yet, head back to your chat with the Botfather and follow the URL he sent you. It should look like t.me/your_bot.)

Success!

Now let's add a message listener. We'll use this later to upload our photos to Sanity.

// index.js

// ... at the bottom
bot.on('message', (msg) => {
  console.log(msg);
    bot.sendMessage(msg.chat.id, "Message logged to console!")
})

Testing this out will show us what the message object looks like in the Telegram API.

Meanwhile, in Replit:

There's our msg object logged in the right hand side. This is helpful for debugging when our bot becomes more complex.

Initializing Sanity

Now let's get our Studio ready to accept incoming memes. Create a new blog by navigating to your preferred file path and running sanity init. Name it anything you want and select the blog schema.

Let's throw in a quick meme schema to our studio.

In the schemas folder, add meme.js.

// your-studio-folder/schemas/meme.js

export default {
  name: "meme",
  title: "Meme",
  type: "document",
  fields: [
    {
      name: "mainImage",
      title: "Main image",
      type: "image",
    },
    {
      name: "publishedAt",
      title: "Published at",
      type: "datetime",
    },
    {
      name: "author",
      title: "Author",
      type: "string",
    },
  ],
};

Then, in the schema.js file (also in your schemas directory), import the meme.js file and export it along with your blog information.

// your-studio-folder/schemas/schema.js

import meme from "./meme.js"
//and export it with the rest of the schema
//...
meme,
//...

That was fast, huh? Now run sanity deploy and take our studio live.

Secure the write token

Usually, we create new documents in Sanity with the user interface of the Studio app. In this case, we'll be using the Javascript client to create documents using our Replit code. To do that, we need a special key that gives our application write access.

This is a powerful key, so be sure to keep it safe!

First, let's head to sanity.io/manage. It got a new design recently, so go check out some pretty UI goodness if you haven't yet.

Find the project you just initialized, click 'Settings' on the top bar, then 'API settings' on the left, and scroll down to 'Tokens'. Add a new API token named 'My Telegram Bot' and give it the 'Editor' permissions.

A key will appear on the screen. Save it somewhere safe!

Back in Replit, let's add a new environmental variable named SANITY_WRITE_TOKEN. Paste your very powerful, very secret value in the field provided and we'll be ready to test it out.

Testing the Sanity client

When building small projects like this one, it can be tempting to put all our code into one file.

This is usually a bad idea, because it makes our code harder to read. Any time we can abstract out some simpler operations into another file, we should. It helps keep us organized and makes it easier for newcomers to grok what we've built.

Let's do that here. First create a new file in the Replit called helpers.js.

Add the Sanity client to the helpers file and export it.

// helpers.js

const sanityClient = require('@sanity/client')

const client = sanityClient({
  projectId: 'your_project_id',
  dataset: 'production',
//define your apiVersion as the current date
  apiVersion: '2021-04-05', 
  token: process.env.SANITY_WRITE_TOKEN,
  useCdn: false,
})

module.exports = { client }

Note we add the date per Sanity's new Content Lake API upgrade.

Remember we are using Replit and don't need to npm install the Sanity client.

Let's create a new "post" doc to test the Sanity client.

// index.js

// ...
const { client } = require('./helpers')
// ...

bot.on('message', (msg) => {
//here, an in-built method from Sanity creates a new doc for us
  client.create({
    _type: 'post',
    title: msg.text
    })
//when it's done, the bot sends a success message
    .then(res=>bot.sendMessage(msg.chat.id, "Doc was published!"))
  .catch(error => console.error(error));
})

Try it out by pressing the green start button on the Replit again and heading to Telegram. When you send the bot a message, it should create a new Sanity post with the title as your message's text.

It worked!

Send memes from groupchat to Sanity

With our tests passed, we can move on to more complex functions. Now we will try sending images from Telegram to Sanity.

This is a little tricky. We'll need to use Node to download our images from the Telegram API, where we'll store them in a temp folder on our Replit. Then we can grab the image in that folder and upload it to the Sanity media library. Once the image is safely in Sanity's cloud, we'll reference its ID when we create a new meme document.

This may sound difficult now, but I believe it will become much clearer when you see the code.

Create helper function and temp folder

We'll start by adding a helper function to download the image from Telegram to our Replit.

// helpers.js

const fs = require('fs')
const request = require('request')
const sanityClient = require('@sanity/client')

const client = sanityClient({
// ...
})

const download = (url, path, callback) => {
//the "request" library helps us make HTTP requests in node
//though it is deprecated, it works for our purposes.
//more here: https://www.npmjs.com/package/request
  request.head(url, (err, res, body) => {
    request(url)
//using Node's .pipe() method, we connect the
//readable stream to a writeable stream.
      .pipe(fs.createWriteStream(path))
      .on('close', callback)
  })
}

module.exports = { client, download }

We use the request library to fetch the URL of the file we want to download. Then we tell Node to create a new file from the data we've fetched and place it at the path we specify.

We'll give this function the URL of the Telegram endpoint that downloads our image for us. Replit lets us store files on its server, so we'll save it to a temporary folder called images and make that the file path.

Before we go further let's make that folder. Create temp/image.png in Replit. You can create new files and folders using the icons in the upper right of the picture below.

Testing photo functionality

We only want this bot to send over photos. Telegram's API makes it trivial for us to find out if there's a photo in an incoming message.

// index.js

// ...

bot.on('message', (msg) => {
  // this will give us easier access to the chat ID
  const chatId = msg.chat.id;
  // checking if the msg has a photo
  if (msg.photo) {
    bot.sendMessage(chatId, "Hey, that's a picture.")
  }
  else {bot.sendMessage(chatId, "If it's not a picture, I'm not concerned.")}

})

Try it out by attaching a photo to your message.

Importing more modules

Since NodeJS doesn't come with support for ES6's fetch method, we need to add it ourselves. We'll also add Node's file system module so we can manipulate image files on Replit.

// index.js

// ...
const fetch = require("node-fetch");
const fs = require('fs')
// ...

Use Node to download our images from the Telegram server to Replit

Replace the bot.on(...) code with the following:

bot.on('message', (msg) => {
  // this will give us easier access to the chat ID
  const chatId = msg.chat.id;
  // checking if the msg has a photo
  if (msg.photo) {
    bot.sendMessage(
        chatId, 
        "Hey, that's a picture. I'm gonna download it to Replit...")
    //retrieves the file ID from the photo
    const imageID = msg.photo[0].file_id;
    //finds the file in the Telegram servers
    const imageSender = 
        fetch(
            `https://api.telegram.org/bot${token}/getFile?file_id=${imageID}`
                )
      .then(res => res.json())
      .then(data => {
        //sets the file path on Telegram servers
        const filePath = data.result.file_path;
        //finds the URL to download the image from Telegram servers
        const url = 
                `https://api.telegram.org/file/bot${token}/${filePath}`;
        //defines where we want Node to download the image
        const path = `./temp/image.png`
        //uses our helper function to download file
        download(url, path, () => {
          bot.sendMessage(chatId, '✅ Downloaded image to Replit!')
        })
      })
  }
})

Now try sending a photo to the bot in Telegram.

Go back to Replit and check out the temp folder.

Impressive! We're now halfway there.

Sending image from `temp` folder to Sanity studio

Alright, the first part is done. Now we'll tell Replit what to do with the image in its temp folder. The Sanity client makes asset uploading relatively painless, so this is simpler than it could be!

Where your download helper function is, add this code.

// index.js

// ...
// inside bot.on('message'...

download(url, path, () => {
          bot.sendMessage(chatId, '✅ Downloaded image to Replit!');
          //using Sanity's upload methods
          client.assets
          .upload('image',
          //tell Node to read the image and create a new file in Sanity
          fs.createReadStream(path), {filename: `img-${imageID}.png`})
          .then((document) => {
          bot.sendMessage(chatId, 'Creating meme document...', document)
          //destructure the image ID from document object
          const {_id} = document;
          //create the meme document!
          client.create({
            _type: 'meme',
            mainImage: {
            asset: {
              _type: 'reference',
              //feed the image ID into the mainImage field as its reference
              _ref: _id,
              }
            },
            //fill out the author and publishedAt fields
            author: `${msg.from.first_name} ${msg.from.last_name}`,
            publishedAt: new Date().toISOString()
          })
                    //let us know when it's received
          .then(res=>bot.sendMessage(chatId, `✅ Doc was published to Sanity! 
                        ID: ${res._id}`))
                    //catch any errors
          .catch(error => {
            bot.sendMessage(chatId, `❌ Upload failed, check console.`);
            console.error(error);
            })
        })
      .catch((error) => {
      bot.sendMessage(chatId, `❌ Upload failed, check console.`);
      console.error('Upload failed:', error.message)
      })
    })

// ... end of bot.on('message'..) function

In the above code, we do two things with Sanity. First, we upload our asset to the Sanity media library. That way, when it's time to create our document, we only need to reference the ID of the image in the mainImage field and it is attached to our new meme document. Then we create the meme, adding the name of the Telegram user as the author and the current datetime as our publishedAt field.

When we're done, we send a message to the Telegram room letting the user know what happened.

Let's try it out!

Does it appear in Sanity?

Head over to your Sanity studio and see if it appeared.

Yes! With name and date fields. Beautiful! We can now export the dankest memes from our groupchat to the world's most advanced publishing platform.

Robot complete!

I've shared the complete code for this bot on Replit. Just switch the toggle in the top left from "Output" to "Code".

To use it, fork the REPL and add a .env with your API keys, then add your Sanity project ID to the helpers.js file.

What should you do from here?

Naturally, you'll need a front end. If you're going to keep your blog fresh with the day's latest memes, you'll need some fast build times. Maybe consider Hugo?
I've left you code that could use a little cleaning up. Try running /start in the completed bot. See what happens? How do we get the "... I'm not concerned" message to not show up when the start command is issued? A conditional combined with a regex, maybe?
How do you call this bot when you're actually chatting in a group? You could add "@your_bot" to your Telegram group, but how do you get it to retrieve messages? For now, you need to DM your bot, but maybe you can figure out how to talk to it "in public."

Congratulations making it this far! I had a lot of fun and I hope you did too. If you have questions, feedback, or just want to show off the cool stuff you made, reach out on Twitter.

Build an e-commerce site with the Stripe API

Joe Holmes — Wed, 21 Apr 2021 22:24:36 +0000

As a new developer, building your first e-commerce site can be intimidating. There's a lot to worry about when dealing with other people's money: security, authentication, validation... if you're still beginning your coding journey, it's enough to make your head spin. But fear not! Modern development tools—especially the wondrous Stripe API—make accepting online payments simpler and safer than ever before.

Stripe describes itself as "the payments infrastructure of the internet," and is generally recognized as the gold standard of commerce platforms online. Next.js is a fast, popular framework for ReactJS. Sanity, our cutting-edge content platform, is designed to integrate seamlessly with tools like these.

By outsourcing the most sensitive and difficult aspects of ecommerce to Stripe, we'll only have to create our frontend code and a very small API. As a result, this approach is ideal for a beginner looking to code their first ecommerce store.

Requirements

NodeJS installed
Sanity CLI installed
Beginner's familiarity with ReactJS
Beginner's familiarity with API routes and POST requests

Get your Stripe API keys

Before we dive into the code, let's get our Stripe keys. Head over to stripe.com and register. Under "API keys" in the "Developers" tab you should see your publishable key and secret key in test mode.

Clone our starter repo

We'll begin with an absolutely bare-bones starter for a Next.js-Sanity blog. From the command line run:

git clone https://github.com/bathrobe/sanity-next-stripe-starter

Before we forget, let's install our Node dependencies with cd sanity-next-stripe-starter/web and npm install.

When that's done, open the folder in your IDE. I'd recommend looking around the files if you're new to Next.js or Sanity and want to wrap your mind around how they interact.

In the root of the web folder, create a .env.local file and add your Stripe keys.

// web/.env.local

NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=pk_test_your_public_key
STRIPE_SECRET_KEY=sk_test_your_secret_key

Integrate Stripe with Next.js

Now that we've got our keys, let's begin our Stripe integration. In our web folder, install the store's dependencies:

npm i stripe @stripe/stripe-js swr use-shopping-cart --save

We'll be using Vercel's SWR library for fetching, as well as the excellent [use-shopping-cart](https://useshoppingcart.com/) React integration.

Make a new folder named /lib/stripe and create getStripe.js

// web/lib/stripe/getStripe.js

import { loadStripe } from "@stripe/stripe-js";

let stripePromise;
const getStripe = () => {
  if (!stripePromise) {
    stripePromise = loadStripe(process.env.NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY);
  }
  return stripePromise;
};

export default getStripe;

Your lib folder should now look like this:

Making the cart component for the merch page

Make folder components in web and first add Cart.js:

// web/components/Cart.js

import { CartProvider } from "use-shopping-cart";
import getStripe from "../lib/stripe/getStripe";

export default function Cart({ children }) {
  return (
    <CartProvider mode="checkout-session" stripe={getStripe()} currency={"usd"}>
      {children}
    </CartProvider>
  );
}

For security reasons, CartProvider should wrap every page that involves your store. You can also wrap CartProvider around the Component in _app.js. Note that we're hardcoding U.S. dollars as our currency to keep it simple.

Making the product display component

Then make components/Products.js:

// web/components/Products.js

import { useShoppingCart, formatCurrencyString } from "use-shopping-cart";
import urlFor from "../lib/sanity/urlFor";
const Products = ({ products }) => {
  const { addItem, removeItem } = useShoppingCart();
  return (
    <section>
      {products.map((product) => (
        <div key={product.id}>
          <img src={urlFor(product.image).width(200)} alt={product.name} />
          <h2>{product.name}</h2>
          <p>
            {formatCurrencyString({
              value: product.price,
              currency: "usd",
            })}
          </p>
          <button onClick={() => addItem(product)}>Add to cart</button>
          <button onClick={() => removeItem(product.id)}>Remove</button>
        </div>
      ))}
    </section>
  );
};

export default Products;

Initialize your Sanity Studio

Next, let's create our Sanity project. In this tutorial, we're going to build a "monorepo" where our Sanity Studio and Next.js frontend will exist side-by-side. They'll be in the studio and web folders, respectively.

//our folder structure will look like this:
sanity-next-stripe-starter
|- web //contains all our Next.js code
|- studio //contains all our Sanity code

From your command line, make sure you're in sanity-next-stripe-starter (not in /web) and run sanity init to boot up a new Sanity project. Sanity will ask you a list of questions to initialize your project. When asked for the project output path, make it your-file-path/sanity-next-stripe-starter/studio. That way, your frontend and Sanity studio will live side by side in the web and studio folders, respectively. When asked if you'd like a starting schema, choose the blog schema.

Check out your new blog with cd studio and sanity start. If you head over to [localhost:3333](http://localhost:3333) you'll see your new studio in all its splendor.

We'll need some sample data to work with, so add a few blog posts before we go further. (I've included Sanity's lovely image helper in the repo, so upload some pictures too! We'll use images in our merch store as well).

Add your Sanity project ID to the .env.local file.

// web/.env.local

NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=pk_test_your_public_key
STRIPE_SECRET_KEY=sk_test_your_secret_key
NEXT_PUBLIC_SANITY_ID=your_project_id_here

You can find your project id as projectId in studio/sanity.json.

With our sample posts created, cd web and npm run dev to check out the blog.

Create products schema in Sanity

Now let's create a schema for our products. In /studio/schemas/merch.js:

///studio/schemas/merch.js

export default {
  name: "merch",
  title: "Merch",
  type: "document",
  fields: [
    {
      name: "name",
      title: "Name",
      type: "string",
    },
    {
      name: "description",
      title: "Description",
      type: "text",
    },
    {
      name: "price",
      title: "Price",
      description: "For now, add cents as zeroes, ie 500 = $5",
      type: "number",
    },
        {
      name: "currency",
      title: "Currency",
      description: "Keep this 'usd' for the purposes of this tutorial",
      type: "string",
    },
    {
      name: "image",
      title: "Image",
      type: "image",
      options: {
        hotspot: true,
      },
    },
  ],
    initialValue: {
    currency: "usd",
  },
};

Then import merch into schema.js:

///studio/schemas/schema.js

import merch from "./merch"

exports //...
//...
merch,
//...

Fire up sanity start and add some products.

Create a GROQ query for Sanity

Now we have a Products component that will render our merch data. We need a way to pull in our products from Sanity and display them in the frontend. In lib/sanity create merchQuery.js:

// web/lib/sanity/merchQuery.js

//protip: we install sanity's groq package
//for syntax highlighting and the ability to run
//groq queries within VScode
//read more at https://www.npmjs.com/package/groq
import groq from "groq";

export const merchQuery = groq`
*[_type=="merch"]{
    name,
    description,
    price,
    "id": _id,
    "image": image.asset->url,
    currency
  }`;

Create merch page

With our query ready, we can use getStaticProps to pull in our Sanity data with very few lines of code. In pages, create a folder merch and within it add an index.js.

Why create a folder with an index file, instead of just creating pages/merch.js? With this folder structure, in the future it will take less time to create single product pages using Next's dynamic routing. You can see an example of this approach in the post folder's [slug].js file.

Let's create the page that will list our products. In pages/merch/index.js:

// pages/merch/index.js

import Link from "next/link";
import Cart from "../../components/Cart";
// import CartSummary from "../../components/CartSummary";
import Products from "../../components/Products";
import { client } from "../../lib/sanity/client";
import { merchQuery } from "../../lib/sanity/merchQuery";

const Merch = ({ products }) => {
  return (
    <main>
      <h1>My Merch Store</h1>
      <p>
        Powered by the{" "}
        <a href="https://useshoppingcart.com">use-shopping-cart</a> React hooks
        library.
      </p>
      <Cart>
        <Products products={products} />
        {/* <CartSummary /> */}
      </Cart>
      <Link href="/">
        <a>Back Home</a>
      </Link>
    </main>
  );
};

export default Merch;

export async function getStaticProps({ params }) {
  const products = await client.fetch(merchQuery);

  return {
    props: {
      products,
    },
  };
}

We're keeping the CartSummary component commented out for now and will return to it in the next section. Link to your merch store from your home page and we'll be ready to take a look:

// pages/index.js

//...
export default function Home({ posts }) {
  return (
    <div>
      <main>
        <h1>My Blog and <Link href="/merch"><a>Merch Store</a></Link></h1>
        <hr />
//...

Run npm run dev and check it out!

Preparing to write our API

Congratulations! We're now ready to add our checkout functionality. Stripe will handle the processing of our customer's bank account and credit card so we can focus on our site.

We'll begin by abstracting away some helpers for our API in a utilities folder. Create /web/utils/apiHelpers.js and in it add the following functions to simplify our API calls.

// web/utils/apiHelpers.js

export async function fetchGetJSON(url) {
  try {
    const data = await fetch(url).then((res) => res.json());
    return data;
  } catch (err) {
    throw new Error(err.message);
  }
}

export async function fetchPostJSON(url, data) {
  try {
    // Default options are marked with *
    const response = await fetch(url, {
      method: "POST", // *GET, POST, PUT, DELETE, etc.
      mode: "cors", // no-cors, *cors, same-origin
      cache: "no-cache", // *default, no-cache, reload, force-cache, only-if-cached
      credentials: "same-origin", // include, *same-origin, omit
      headers: {
        "Content-Type": "application/json",
        // 'Content-Type': 'application/x-www-form-urlencoded',
      },
      redirect: "follow", // manual, *follow, error
      referrerPolicy: "no-referrer", // no-referrer, *client
      body: JSON.stringify(data || {}), // body data type must match "Content-Type" header
    });
    return await response.json(); // parses JSON response into native JavaScript objects
  } catch (err) {
    throw new Error(err.message);
  }
}

Creating the checkout page component

Let's make our frontend display our shopping cart and checkout button. Create components/CartSummary.js:

// components/CartSummary.js

import { useState, useEffect } from "react";
import { useShoppingCart } from "use-shopping-cart";
import { fetchPostJSON } from "../utils/apiHelpers";

export default function CartSummary() {
  const [loading, setLoading] = useState(false);
  const [cartEmpty, setCartEmpty] = useState(true);
  const {
    formattedTotalPrice,
    cartCount,
    clearCart,
    cartDetails,
    redirectToCheckout,
  } = useShoppingCart();

  useEffect(() => setCartEmpty(!cartCount), [cartCount]);

  const handleCheckout = async (event) => {
    event.preventDefault();
    setLoading(true);

    const response = await fetchPostJSON(
      "/api/checkout_sessions/cart",
      cartDetails
    );

    if (response.statusCode === 500) {
      console.error(response.message);
      return;
    }

    redirectToCheckout({ sessionId: response.id });
  };

  return (
    <form onSubmit={handleCheckout}>
      <h2>Cart summary</h2>
      {/* This is where we'll render our cart;
            The item count changes quickly and may
            be mismatched between client and server.
            To avoid annoying error messages,
            we use 'supressHydrationWarning'.
            https://reactjs.org/docs/dom-elements.html#suppresshydrationwarning*/}
      <p suppressHydrationWarning>
        <strong>Number of Items:</strong> {cartCount}
      </p>
      <p suppressHydrationWarning>
        <strong>Total:</strong> {formattedTotalPrice}
      </p>

            <p>Use 4242 4242 4242 4242 as the card number.</p>
      <button
        className="cart-style-background"
        type="submit"
        disabled={cartEmpty || loading}
      >
        Checkout{" "}
        <div className="card-number">

        </div>
      </button>
      <button
        className="cart-style-background"
        type="button"
        onClick={clearCart}
      >
        Clear Cart
      </button>
    </form>
  );
}

Nice work! In the code above, you'll notice we send API requests to api/checkout_sessions/cart in the fetchPostJSON(). This is the serverless API that will manage our interactions with Stripe. Let's make it.

Building our serverless API

Ready to write an API? In pages create api/checkout_sessions/cart.js. This endpoint will send the POST request to Stripe's servers.

// pages/api/checkout_sessions/cart.js

import { validateCartItems } from "use-shopping-cart/src/serverUtil";
import Stripe from "stripe";
import { client } from "../../../lib/sanity/client";
import { merchQuery } from "../../../lib/sanity/merchQuery";
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY, {
  // https://github.com/stripe/stripe-node#configuration
  apiVersion: "2020-03-02",
});

export default async function handler(req, res) {
  if (req.method === "POST") {
    try {
      // Validate the cart details that were sent from the client.
      const cartItems = req.body;
      //Sanity client performs merchQuery
      let sanityData = await client.fetch(merchQuery);
            // The POST request is then validated against the data from Sanity.
      const line_items = validateCartItems(sanityData, cartItems);
      // Create Checkout Sessions from body params.
      const params = {
        submit_type: "pay",
        mode: "payment",
        payment_method_types: ["card"],
        billing_address_collection: "auto",
        shipping_address_collection: {
          allowed_countries: ["US", "CA"],
        },
                //The validated cart items are inserted.
        line_items,
        success_url: `${req.headers.origin}/result?session_id={CHECKOUT_SESSION_ID}`,
        cancel_url: `${req.headers.origin}`,
      };
      const checkoutSession = await stripe.checkout.sessions.create(params);

      res.status(200).json(checkoutSession);
    } catch (err) {
      res.status(500).json({ statusCode: 500, message: err.message });
    }
  } else {
    res.setHeader("Allow", "POST");
    res.status(405).end("Method Not Allowed");
  }
}

Head back to our pages/merch/index.js file and uncomment the import of CartSummary and the instance <CartSummary/> inside the page component itself.

//uncomment these in pages/merch/index.js
//...
import CartSummary from "../../components/CartSummary";
//...
<CartSummary />

Run npm run dev again and the buttons should work. Woohoo! From here we can be redirected to Stripe's checkout service.

If you click Checkout, you should see your products displayed in Stripe's payment form.

Beautiful! Stripe accepts the Sanity products without us needing to configure anything on the Stripe dashboard. Our blog content can live with our merch content.

Be sure to use the test debit card number 4242 4242 4242 4242 in the card details. The rest of the card information can be anything.

If you click the payment button, you should be redirected to a result page that 404s. Take a look at the URL of that page: [http://localhost:3000/result?session_id=cs_test_a-long-string](http://localhost:3000/result?session_id=cs_test_b1bKM5vNNw5iJjr85s6o70PXI5kpmYMGmUsfHILYoYuI36exjFauvz19CB). This indicates Next is looking for a page named result in our pages directory, which it queries with our "checkout_session" information. Let's add that page, so we can display information about our customer's new order.

Displaying order confirmation from Stripe

Make result.js in pages, and PrintObject.js in components. Our result page will pull in the response from the Stripe API and display it onscreen. We will use PrintObject to show the entire JSON body, so you can pick and choose what data you'd like to display.

// web/pages/result.js

import { useRouter } from "next/router";
import Link from "next/link";
import useSWR from "swr";
import PrintObject from "../components/PrintObject";
import { fetchGetJSON } from "../utils/apiHelpers";

const ResultPage = () => {
  const router = useRouter();
  // Fetch CheckoutSession from static page via
  // https://nextjs.org/docs/basic-features/data-fetching#static-generation
  const { data, error } = useSWR(
    router.query.session_id
      ? `/api/checkout_sessions/${router.query.session_id}`
      : null,
    fetchGetJSON
  );

  if (error) {
        return <div>failed to load</div>;
    }

  return (
    <div className="page-container">
      Congrats
      <h1>Checkout Payment Result</h1>
      <p>
        With the data below, you can display a custom confirmation message to
        your customer.
      </p>
      <p>For example:</p>
      <hr />
      <h3>
        Thank you, {data?.payment_intent.charges.data[0].billing_details.name}.
      </h3>
      <p>
        Confirmation email sent to{" "}
        {data?.payment_intent.charges.data[0].billing_details.email}.
      </p>
      <hr />
      <h2>Status: {data?.payment_intent?.status ?? "loading..."}</h2>
      <h3>CheckoutSession response:</h3>
      <PrintObject content={data ?? "loading..."} />
      <Link href="/">
        <a>Back home</a>
      </Link>
    </div>
  );
};

export default ResultPage;

And in the components folder:

// web/components/PrintObject.js

//this is just a helpful little component that displays
//all the data from our customer's session
const PrintObject = ({ content }) => {
  const formattedContent = JSON.stringify(content, null, 2);
  return <pre>{formattedContent}</pre>;
};

export default PrintObject;

One final step! Notice this snippet in the result.js code:

// web/pages/result.js

//...
const { data, error } = useSWR(
    router.query.session_id
      ? `/api/checkout_sessions/${router.query.session_id}`
      : null,
    fetchGetJSON
  );
//...

SWR is expecting a route for a session ID that it then takes us to when the result is displayed. This means that if we make this file, we can use dynamic routes within the API to return the details of a given Stripe session.

Make pages/api/checkout_sessions/[id].js

// pages/api/checkout_sessions/[id].js

import Stripe from "stripe";
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY, {
  // https://github.com/stripe/stripe-node#configuration
  apiVersion: "2020-03-02",
});

export default async function handler(req, res) {
  const id = req.query.id;
  try {
    if (!id.startsWith("cs_")) {
      throw Error("Incorrect CheckoutSession ID.");
    }
    const checkout_session = await stripe.checkout.sessions.retrieve(id, {
      expand: ["payment_intent"],
    });

    res.status(200).json(checkout_session);
  } catch (err) {
    res.status(500).json({ statusCode: 500, message: err.message });
  }
}

With that, we should be able to redirect the customer to a confirmation message when their order is successfully sent.

Congratulations! If you've made it this far, you've just built your very own ecommerce store. Great job!

Confirming the order on the Stripe dashboard

If we head back to Stripe, we can confirm that the order was successfully sent. Here you will be able to receive payouts from your card payments. The new order should appear in the "Payments" tab.

Clicking on the new payment, we'll see all its information displayed in Stripe—even the images and descriptions were passed along. Sanity manages all the content for our merch by itself, which is awesome.

Next steps

First of all, high-fives for making it through the whole tutorial. This was a challenge, and I hope you're proud! If you want to make this project a live store, there are a few more things you'll need to do:

Deploy your Sanity studio by navigating to the studio folder and running sanity deploy
Push your test products live by clicking "Activate your account" on the Stripe dashboard
Deploy your Next.js site to production and add your live environment variables (I recommend Vercel or Netlify)

And if you'd like to take this project further, here are a few ideas:

Use the Stripe Elements of react-stripe-js to display all checkout UI within your own frontend
Try styling your site with Sanity's totally underrated UI kit
Automatically send a confirmation email from the result.js page using a service like Mailgun

Happy coding! If you have a question or want to show off what you've made, feel free to reach out on Twitter.

Bounty Hunting with Elixir

Joe Holmes — Thu, 01 Apr 2021 17:22:23 +0000

Does 'crypto bounty hunter' sound like your dream job title? Yes? Same here! And I've got good news for both of us.

Futurist crypto blog 1729 offers Bitcoin bounties for learning code, design, and content creation.

I've been following the project for the past few weeks and was pleased to find them in my inbox the other day, where they offered up a challenge in the Elixir programming language. Solve three Elixir problems in Exercism.io, they said, and you might win $100 in BTC.

Parallel Cores, Engage!

Elixir is a functional programming language that excels at concurrent processing (i.e., handling many operations at once). The "free lunch is over," they say, and in the near future it will only be languages capable of concurrency that will fully take advantage of the advances in processor speed. As computer scientist Herb Sutter has written,

applications will increasingly need to be concurrent if they want to fully exploit CPU throughput gains.

For its ability to run concurrent code, and its speed in handling massive computations, Elixir is a great choice for a budding crypto developer.

Discord uses Elixir to handle billions of messages on their backend. When Bleacher Report migrated, they moved from 150 Ruby-on-Rails servers to just 5 Elixir servers. You can read more evidence of Elixir's magic here.

I didn't know anything about the language when I started, but I'm
excited to learn more. Maybe my next project will be a Twitter clone, or perhaps a custom Elixir blockchain.

In this post I'm going to walk through the three challenges I completed for the 1729 bounty: a simple DateTime challenge, the Diffie-Hellman crypto algorithm, and a parallel-processed letter frequency calculator.

Installing Elixir and Exercism

If you're unaware, Exercism is an incredible platform for learning code. As a self-taught developer, I'm used to simply taking notes on Youtube videos (mostly by the King, Brad Traversy). I then build my own test projects out of what I've learned. Most challenge-based apps cost money and seem to limit this project-based self-pacing. I usually avoid them.

Exercism is different. In addition to being free, each Exercism module is supervised by a set of mentors who offer feedback on your code. There's an incredible diversity of languages to learn-- I think I'll finally get around to learning Python next.

Instructions for installing Elixir can be found here. As a Windows user, Chocolately is my go-to: cinst elixir and we were off to the races.

Setting up the Exercism CLI is as simple as following these instructions.

To get acquainted with the basics of the language, I recommend these free resources.

Gigasecond Challenge

The first 1729 challenge was to write a function that would add a billion seconds to a given date.

We were given two tuples containing Date and Time information, respectively. After poking around a bit and attempting to manually convert the integer, I had my first encounter with Elixir's useful in-house functions. The DateTime module contains an add function. It was easy as passing a billion as the add parameter and reformatting the tuple pair for the solution.

defmodule Gigasecond do
  @doc """
  Calculate a date one billion seconds after an input date.
  """
  @spec from({{pos_integer, pos_integer, pos_integer}, {pos_integer, pos_integer, pos_integer}}) ::
          :calendar.datetime()

  def from({{year, month, day}, {hours, minutes, seconds}}) do
  # piecing the incoming tuples together to get a DateTime object
  gd = elem(Date.new(year,month,day), 1)
  gt = elem(Time.new(hours,minutes,seconds),1)
  gdt = elem(DateTime.new(gd,gt),1)
  # using the DateTime add method
  gigatime = DateTime.add(gdt, 1000000000)
  # formatting the gigatime into tuple pair
  {{gigatime.year, gigatime.month, gigatime.day}, {gigatime.hour, gigatime.minute, gigatime.second}}
  end
end

Diffie-Hellman Key Exchange

After that encouraging warm-up, we moved into trickier territory. For the next challenge we needed to implement popular cryptography algorithm Diffie-Hellman in Elixir. Not knowing anything about DH, I started by studying the algorithm and making sure I understood the math of it. Computerphile came to the rescue, as did this Diffie-Hellman calculator.

Protip: Invest in a whiteboard.

After being certain of how the algo worked, the solution was only three in-built Elixir/Erlang modules away.

defmodule DiffieHellman do
  @moduledoc """
  Diffie-Hellman is a method of securely exchanging keys in a public-key
  cryptosystem.
  """

  @doc """
  Given a prime integer `prime_p`, return a random integer between 1 and `prime_p` - 1
  """
  @spec generate_private_key(prime_p :: integer) :: integer
  def generate_private_key(prime_p) do
  :crypto.rand_uniform(1, prime_p - 1)
  end

  @doc """
  Given two prime integers as generators (`prime_p` and `prime_g`), and a private key,
  generate a public key using the mathematical formula:

  (prime_g **  private_key) % prime_p
  """
  @spec generate_public_key(prime_p :: integer, prime_g :: integer, private_key :: integer) ::
          integer
  def generate_public_key(prime_p, prime_g, private_key) do
    :binary.decode_unsigned(:crypto.mod_pow(prime_g, private_key, prime_p))
  end

  @doc """
  Given a prime integer `prime_p`, user B's public key, and user A's private key,
  generate a shared secret using the mathematical formula:

  (public_key_b ** private_key_a) % prime_p
  """
  @spec generate_shared_secret(
          prime_p :: integer,
          public_key_b :: integer,
          private_key_a :: integer
        ) :: integer
  def generate_shared_secret(prime_p, public_key_b, private_key_a) do
    # secret they share is g^a*b mod p
    # public key b is the result of the preceding function for user b
    # pub_key_a ^ priv_key_a % prime_p independently verified in iex, so i know this works :)
    :binary.decode_unsigned(:crypto.mod_pow(public_key_b, private_key_a, prime_p))


  end
end

Parallel Letter Frequency

The final challenge was far-and-away the hardest. Given a list of strings--some in German, some in Dutch, some in English-- calculate the letter frequencies in each string concurrently, then combine them all into a single map.

The special German and Dutch characters highlighted a strength of Elixir, which is that its strings are read as low-level binaries. You can easily convert strings into their corresponding integers in binary or Unicode (though if your computer stumbles on rendering special characters ::cough Powershell:: it can be frustrating!)

I spent a solid day writing a program that:

lowercased each string and removed numbers, punctuation, and spaces
reduced each string to a list of its unique characters
counted the occurrence of each unique character in the total list of graphemes
processed each string concurrently using an async stream
formatted the result and repeated the same workflow for the total list

Then this morning I found out I didn't need to do a whole lot of that, as Elixir has an in-built function for calculating letter frequencies! Oof.

Enum.frequency() allowed me to drastically simplify my code.

The Task module is similar to async/await in Javascript. The code below creates a "stream" of code waiting to be executed, then runs said code in parallel when the function is passed to an enumerator. That may sound complex, but after grokking a little Elixir, I believe you will find it simple! Simpler, in any case, than hacking through Promises in Javascript.

My refactored solution is below. Note the gorgeously simple method used to reduce all enumerated string-counters at the bottom. List.foldl compresses all items in a list according to an accumulator (the empty map) and a function (the similarly-elegant Map.merge function).

defmodule Frequency do
  @doc """
  Count letter frequency in parallel.

  Returns a map of characters to frequencies.

  """
  @spec frequency([String.t()], pos_integer) :: map
  def frequency(texts, workers) do

# first create a stream that will process each text in parallel
stream = Task.async_stream(texts, fn txt -> 
  if txt != nil do
    # convert each formatted text to graphemes
    grp = String.graphemes(
      String.downcase(String.replace(txt, ~r/[1234567890!#$%&()*+,.:;<=>?@\^_`{|}~-]/, ""))
    )
    # filter out empty spaces and count the frequencies 
     Enum.filter(grp, fn x -> x != " " end) |> Enum.frequencies()
  else %{} end
end, max_concurrency: workers)

# remove :ok atom from result and run all processes
total = Enum.map(stream, fn x -> elem(x,1) end)
# reduce all maps in total list by merging
List.foldl(total, %{}, fn x, a -> Map.merge(x, a, fn _k, v1, v2 ->
  v1 + v2
end)
end)
end
end

Lessons learned!

I may not win the prize, as others may have solved these challenges in simpler, more efficient ways. But I'm happy I took the time to figure out some Elixir. This is the second programming language I've learned (after Javascript), and it gives me a great deal of confidence moving forward.

The life of a bounty hunter is feast or famine, after all. Us Mandalorians must accept our fates, acquire our targets, and go where the bounties lead. Win or lose, we can accept that every hunt that does not kill us makes us stronger.

If you want to check out more of my adventures, swing by my website. Til next time!

Creating a DIY Digital Garden with Obsidian and Gatsby

Joe Holmes — Thu, 15 Oct 2020 16:45:26 +0000

Are you intrigued by networked note-taking apps?

Do you want to build "blazing" fast sites on the JAMStack?

Have you heard about the digital garden craze sweeping the nation and want to make one of your own?

Maybe Obsidian + Gatsby will be as good to you as they have been to me.

In addition to being a great note-taking tool, Obsidian functions as an excellent content manager. When combined with a git-based deployment solution like Netlify (and a few plugins), it compares favorably to other git-based CMS's such as Forestry and Netlify CMS, with the added benefit of backlinks, graph views, and a bunch of bells and whistles.

Open a new Obsidian vault in your content folder, .gitignore the config folder and its CSS, add frontmatter templates to your Obsidian settings, and voila! You've got a slick new CMS.

Add MDX to Gatsby and you can pass shortcodes through your Obsidian files too. This allows you to display interactive, custom React components in your notes. If this sounds fun to you, I've cataloged the steps I took to set it up below.

I have made the repo for this demo public. You can find it here.

Getting Started

Note: I am a beginner developer and this is a beginner-friendly tutorial. If y'all spot anything that could be improved upon, please let me know!

We will start with the totally empty gatsby-hello-world starter to keep things simple.

Navigate to the folder where you plan to house your site and enter

gatsby new obsidian-cms-demo https://github.com/gatsbyjs/gatsby-starter-hello-world.git`

Once the site is booted up let's install some dependencies. This set up will depend on gatsby-source-filesystem and gatsby-plugin-mdx.
Navigate to your new project directory, then install them at the command line:
npm i gatsby-source-filesystem
npm i gatsby-plugin-mdx

Configure plugins

Add both plugins to gatsby-config.js. Make sure the MDX plugin reads markdown files as .mdx as well, since .md files are what Obsidian creates. Tell gatsby-source-filesystem to read a folder notes that is inside a folder named content:

//gatsby-config.js
...
    {
      resolve: `gatsby-plugin-mdx`,
      options: {
        extensions: [`.mdx`, `.md`],
      },
    },
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `notes`,
        path: `${__dirname}/content/notes/`,
      },
    },

Setting up Obsidian

Make a folder in the root of your project named content.
mkdir ./content

Then, after downloading and installing Obsidian, open that folder as a pre-existing vault.

Navigate to your project folder and open content.
You may wish to configure this differently, but I created a second folder within content called notes. This way, all content you want published to your website is automatically kept apart from the Obsidian config files.

Create your first note! We will test out our page creation file with it soon.
You'll notice a new folder, .obsidian, was added to the content folder. We can tell git to ignore it. We will also be adding a frontmatter template to Obsidian soon, so I've created a _private folder at content/_private to house it (as well as any drafts and journals you may want in the future). If you install any custom CSS to your vault, you can add those to the .gitignore too.
In the .gitignore file:

# Obsidian Files
/content/.obsidian
/content/_private
# Optional custom CSS
obsidian.css

Now, gatsby-source-filesystem will only read content in the notes folder, and will not push any other files to your repo. You can write and setup Obsidian in peace.

Creating pages using gatsby-node.js

We now need a way to programmatically create pages from the files Obsidian creates.

There's a lot of ground to cover here and I may gloss over it, but I'm getting the code from the official docs at: Creating Pages from Data Programmatically | Gatsby. It is a great walkthrough and there are plenty like it on the Gatsby docs.

First, in gatsby-node.js:

exports.createPages = async ({ actions, graphql, reporter }) => {
  const { createPage } = actions

  const notesTemplate = require.resolve(`./src/templates/noteTemplate.js`)

  const result = await graphql(`
    {
      allFile {
        edges {
          node {
            childMdx {
              slug
            }
          }
        }
      }
    }
  `)

  if (result.errors) {
    reporter.panicOnBuild(`Error while running GraphQL query.`)
    return
  }

  result.data.allFile.edges.forEach(({ node }) => {
    createPage({
      path: `notes/${node.childMdx.slug}`,
      component: notesTemplate,
      context: {
        // additional data can be passed via context
        slug: node.childMdx.slug,
      },
    })
  })
}

Note that the GraphQL query is set to query allFiles instead of allMdx. There may be a better solution than this, but I set it up in case I wanted to create pages via other sources in the future (GraphQL has a handy piece of data called sourceInstanceName that can help you sort different sources, available from allFile.)

We've also specified a noteTemplate file for all incoming notes, so let's make that now.

Create a new folder in src called templates, then add noteTemplate.js to the templates folder.

This is a very barebones template and you will probably want to add a layout component and styles to it as well. I've added a link to go Back Home for easier navigation.

//in src/templates/noteTemplate.js

import React from "react"
import { graphql, Link } from "gatsby"
import { MDXRenderer } from "gatsby-plugin-mdx"

export default function noteTemplate({ data }) {
  const { mdx } = data

  return (
    <article>
      <MDXRenderer>{mdx.body}</MDXRenderer>
      <Link to="/">Back Home</Link>
    </article>
  )
}

export const query = graphql`
  query($slug: String!) {
    mdx(slug: { eq: $slug }) {
      body
    }
  }
`

We import the MDXRenderer component to display the body of our note as an MDX file. In the GraphQL query, we're taking in the variable that we passed at the bottom of gatsby-node.js in the context: section.

Setting up a Basic Frontmatter Template in Obsidian

We will probably want an easy way to grab a note's title and date of creation. Thankfully, this is simple and relatively frictionless using Obsidian's template plugin. In your vault, navigate to Settings/Plugin and turn on the Template plugin. Specify a _templates folder inside _private, then create a file called "Frontmatter" in it with the following:

---
title: "{{title}}"
date: {{date}}T{{time}}
---

Obsidian will automatically fill in the title and date values every time you call the template. Hook the template up to a hotkey and it can be very fluid to create new notes with formatted frontmatter.

Note: I struggled to find the best way to format the date. I found that creating it in the style above allows notes to be sorted by date, and formatting the date to be more readable (either via the template or GraphQL) gave me troubles when I tried to sort. So instead, I imported Day.js in pages where the date was displayed and it worked without trouble.

After inserting the template at the top of the page, our Hello, World note now looks like this:

---
title: Hello world
date: 2020-10-14T13:22
---

This is the first note in my Gatsby digital garden.

Showing All Notes on the Index Page

Just to illustrate the idea, we'll make a list of all note pages on our home page.
We can easily accomplish this with a GraphQL query for each note's title, date, and slug.

In pages/index.js:

import React from "react"
import { Link, graphql } from "gatsby"

const Home = ({
  data: {
    allMdx: { edges },
  },
}) => {
  const Notes = edges.map(edge => (
    <article>
      <Link to={`/notes/${edge.node.slug}`}>
        <h1>{edge.node.frontmatter.title}</h1>
      </Link>
      <p>{edge.node.frontmatter.date}</p>
    </article>
  ))
  return <section>{Notes}</section>
}

export default Home
export const pageQuery = graphql`
  query MyQuery {
    allMdx {
      edges {
        node {
          slug
          frontmatter {
            title
            date
          }
        }
      }
    }
  }
`

To walk through what we've done here:

We pass the edges of the query into our page's function, which allows us to retrieve the data in our markup
We take the array of edges and use the .map() array method to create a new array of markup with link to each note page, displaying its title and (ugly-formatted) date (I recommend fixing this with Day.js)
We pass this new array Notes to the JSX that's returned by the function
We export the Home page function
We export the GraphQL query that retrieves our data

Now fire up the dev server using gatsby develop, and you should see your first note displayed there!

Add Wikilink and Reference Plugins

Wikilinks and Gatsby-plugin-catch-links

Right now our site is pretty unimpressive and its functionality is more or less the same as any old markdown blog with posts written in the IDE. We will fix that!

Two fundamental features of networked note software are

support for [[wikilink]] syntax
support for linked references

and we have some simple plugins that can accomplish both of these things!

Thanks to the excellent work of developer Mathieu Dutour on his gatsby-digital-garden, we can easily get both of these features running.

We will use two items in his packages directory: gatsby-remark-double-brackets-link and gatsby-transformer-markdown-references.

First, let's install them in our project:

npm i gatsby-remark-double-brackets-link
npm i gatsby-transformer-markdown-references

We can now configure the plugins in our gatsby-config.js. First let's set up the double brackets link. It will be configured inside the MDX plugin:

//in gatsby-config.js
plugins: [
...
{
      resolve: `gatsby-plugin-mdx`,
      options: {
        extensions: [`.mdx`, `.md`],
        gatsbyRemarkPlugins: [
          {
            resolve: "gatsby-remark-double-brackets-link",
            options: {
              titleToURLPath: `${__dirname}/resolve-url.js`,
              stripBrackets: true,
            },
          },
        ]
      },
    },
]

Both of these options are... optional. Because our note pages are created at index/notes/note-slug, we need a way to tell the autogenerated wikilinks to follow that same convention.

(I did this because I set up a blog in addition to a garden on my personal site, and I think it's good practice to separate the note files from any additional main-level pages as well.)

At the root of the project, create a file named resolve-url.js. The code here is quite simple:

const slugify = require("slugify")
module.exports = title => `/notes/${slugify(title)}`

There you have it! Now any [[double bracket link]] in our MDX notes will automatically turn into a link to another note page.

If you care to try it out, create a new wikilink in your first note. Open the link in Obsidian by Ctrl + clicking it and add its frontmatter via the template.

Make sure your new note is inside the notes folder. You can configure where new files are stored in the Vault settings.

Boot up the dev server again and you should see the link in the Hello World note. Click it and you'll be routed to your new note.

You may notice something wrong, though-- the link takes a long time to load. Isn't lightning-fast linking one of the core features of Gatsby? Yes it is and there's a very easy plugin solution in gatsby-plugin-catch-links.

Install with npm i gatsby-plugin-catch-links, and throw it in your gatsby-config.js file:

//gatsby-config.js
plugins: [
...
`gatsby-plugin-catch-links`,
...

Now the next time you click the link it should be "blazing".

Markdown References

After installing gatsby-transformer-markdown-references, add it to the root level of your gatsby-config (ie, not inside of the gatsby-mdx plugin).

//gatsby-config.js

plugins: [
...
    {
      resolve: `gatsby-transformer-markdown-references`,
      options: {
        types: ["Mdx"], // or ['RemarkMarkdown'] (or both)
      },
    },
]

Now if you check out GraphiQL, the super-handy GraphQL tool at http://localhost:8000/___graphql, you should see nodes for inbound and outbound references in each of your mdx files!

Turns into...

Since the Hello, World note contained a link to "Second Note," Second Note is aware of it and lists it in the inboundReferences array. Pretty sweet!
We can use this to list linked references to each note file we have, a la Roam Research, Obsidian, and all the other fancy note apps.

Let's set it up in our noteTemplate.js file.

First, let's add it to our GraphQL query:

//noteTemplate.js

export const query = graphql`
  query($slug: String!) {
    mdx(slug: { eq: $slug }) {
    body
    inboundReferences {
      ... on Mdx {
        frontmatter {
          title
        }
        slug
      }
    }
    }
  }
`

Then we can map a simple new array that lists the references in a <ul> tag. I added a ternary operator for the "Referenced in:" line, so it wouldn't display if there were no references.

//inside noteTemplate.js
return (
    <article>
      <MDXRenderer>{mdx.body}</MDXRenderer>
      {mdx.inboundReferences.length > 0 ? <p>Referenced in:</p> : ""}
      <ul>
        {mdx.inboundReferences.map(ref => (
          <li>
            <Link to={`/notes/${ref.slug}`}>{ref.frontmatter.title}</Link>
          </li>
        ))}
      </ul>
      <Link to="/">Back Home</Link>
    </article>
  )

Not so hard, right? Our Second Note page should now look like this:

The Cherry on Top: MDX Shortcodes

None of the early MDX config would be worth it if we didn't try out some custom components. The Gatsby MDX Docs are pretty thorough, so I'll be brief. I'm using the lovely MDX Embed for Gatsby, a freshly updated plugin worked on by some really nice people. It requires no imports.

Simply:

npm install mdx-embed gatsby-plugin-mdx-embed --save

then


// gatsby-config.js
module.exports = {
  ...
  plugins: [
  ...
  `gatsby-plugin-mdx-embed`
  ]
  ...
}

It's a great plugin with lots of different embeddable content. Try it out!

Obsidian autosaves, so it'll crash the dev server if you type in it for too long. But just for fun:

If you want to see what this system looks like with some styling (and a newly added Algolia search), I set it up on the Notes section of my personal website. Thanks for reading, and if you make something cool with this or want to chat, feel free to send me an email or say hi on Twitter.

Forem: Joe Holmes

GPT-4 Spaced Repetition with "The Making of Modern Ukraine"

tl;dr

Background history

How it was built

Flashcard generation

Results

Take-aways and next steps

Templater Treats for Obsidian October

Daily Zen Quote in Your Journal

Fetch YouTube Transcripts in Your Notes

Try them out!

How to Create a Todo List with React and Sanity

Requirements

Clone starter

Register for Magic.link

Deploy a todo schema from Sanity

Get Sanity permissions

Creating the input form

Creating states to store form values

Create handleSubmit and handleChange functions

Creating the input form

Make our API route

Display our todos

Creating todo components

Updating todos.js

Add the new routes to our API

The most user-friendly database in the world

Next Steps

How to Create a Telegram Bot with Node.js and Sanity

Requirements

Initializing the Telegram bot with Botfather

Starting a new Replit with NodeJS

Setting up our environment variables

Booting up our bot

Giving our bot some basic functionality

Initializing Sanity

Secure the write token

Testing the Sanity client

Send memes from groupchat to Sanity

Create helper function and temp folder

Testing photo functionality

Importing more modules

Use Node to download our images from the Telegram server to Replit

Sending image from temp folder to Sanity studio

Robot complete!

Build an e-commerce site with the Stripe API

Requirements

Get your Stripe API keys

Clone our starter repo

Integrate Stripe with Next.js

Making the cart component for the merch page

Making the product display component

Initialize your Sanity Studio

Create products schema in Sanity

Create a GROQ query for Sanity

Create merch page

Preparing to write our API

Creating the checkout page component

Building our serverless API

Displaying order confirmation from Stripe

Confirming the order on the Stripe dashboard

Next steps

Bounty Hunting with Elixir

Parallel Cores, Engage!

Installing Elixir and Exercism

Gigasecond Challenge

Diffie-Hellman Key Exchange

Parallel Letter Frequency

Lessons learned!

Creating a DIY Digital Garden with Obsidian and Gatsby

Getting Started

Configure plugins

Setting up Obsidian

Creating pages using gatsby-node.js

Setting up a Basic Frontmatter Template in Obsidian

Showing All Notes on the Index Page

Add Wikilink and Reference Plugins

Wikilinks and Gatsby-plugin-catch-links

Markdown References

Create `handleSubmit` and `handleChange` functions

Updating `todos.js`

Sending image from `temp` folder to Sanity studio