Forem: ⧉ infominer

Creating the back end for a weekly newsletter on decentralized identity

⧉ infominer — Sun, 15 Nov 2020 13:52:25 +0000

Interested in the development of decentralized-identity technology, I began creating an Awesome List which quickly grew into it's own website Decentralized-ID.com a few years ago. (read my IndieHackers intro for some background)

Developing the backend for a sustainable weekly newsletter

I'd been chatting with Kaliya Identity Woman for around a year, after contacting her about the potential for our collaborating on decentralized identity. At some point, she proposed the idea of writing a newsletter together, under the Identosphere.net domain.

Instead of jumping in head first, like I usually do, we've spent a lot of time figuring out how to run a newsletter, sustainably, with as few third party services as possible, while I'm learning my way around various web-tools.

We're tackling a field that touches every domain, has a deep history, and is currently growing faster than anyone can keep up with. But this problem of fast-moving information streams isn't unique to digital identity, and I'd like to share this process for others to benefit from.

GitHub Pages

Once my 'Awesome List' outgrew the Awesome format, I began learning to create static web-sites with GitHub Pages and Jekyll.

GitHub Pages Starter Pack (a resource I've created along that journey)

Static Websites are great for security and easy to set up, but if you're an indie hacker, you're gonna want some forms so you can begin collecting subscribers! However, forms are not supported natively through Jekyll or GitHub Pages.

Enter Staticman

Staticman is a comments engine for static websites, but can be used for any kind of form, with the proper precautions.

It can be deployed to Heroku with a click of a button, made into a GitHub App, or run on your own server. Once set up, it will submit a pull-request to your repository with the form details (and an optional mailgun integration).

I set it up on my own server and created a bot account on GitHub with permissions to a private repository for the Staticman app to update with subscriptions e-mails to.

Made the form, and a staticman.yml config file in the root of the private repository where I'm collecting e-mail addresses.

The Subscription Form

<center>
<h3>Subscribe for Updates</h3>
<form class="staticman" method="POST" action="https://identosphere.net/staticman/v2/entry/infominer33/subscribe/master/subscribe">
    <input name="options[redirect]" type="hidden" value="https://infominer.xyz/subscribed">
    <input name="options[slug]" type="hidden" value="infohub">
    <input name="fields[name]" type="text" placeholder="Name (optional)"><br>
    <input name="fields[email]" type="email" placeholder="Email"><br>
    <input name="fields[message]" type="text" placeholder="Areas of Interest (optional)"><br>
    <input name="links" type="hidden" placeholder="links">
    <button type="submit">Subscribe</button>
</form>
</center>

The staticman.yml config in the root of my private subscribe repo

subscribe:
  allowedFields: ["name", "email", "message"]
  allowedOrigins: ["infominer.xyz","identosphere.net"]
  branch: "master"
  commitMessage: "New subscriber: {fields.name}"
  filename: "subscribe-{@timestamp}"
  format: "yaml"
  generatedFields:
    date:
      type: "date"
      options:
        format: "iso8601" 
  moderation: false
  name: "infominer.xyz"
  path: "{options.slug}" 
  requiredFields: ["email"]

It seems to be struggling with GitHub's recent move to change the name of your default branch from master to main (for new repositories). So, unfortunately, I had to re-create a master branch to get it running.

Planet Pluto Feed Reader

One of the most promising projects I found, in pursuit of keeping up with all the info, is Planet Pluto Feed Reader, by Gerald Bauer.

In online media a planet is a feed aggregator application designed to collect posts from the weblogs of members of an internet community and display them on a single page. - Planet (Software)

For the uninitiated, I should add that websites generate RSS feeds that can be read by a newsreader, allowing users to keep up with posts from multiple locations without needing to visit each site individually. You very likely use RSS all the time without knowing, for example, your podcast player depends on RSS feeds to bring episodes directly to your phone.

What Pluto Feed reader does is just like your podcast app, except, instead of an application on your phone that only you can browse, it builds a simple webpage from the feeds you add to it, that can be published on GitHub, your favorite static web-hosting service, or on your own server in the cloud.

Pluto is built with Ruby, using the ERB templating language for web-page design.

One of the cool things about ERB is it lets you use any ruby function in your web-page template, supporting any capability you might want to enable while rendering your feed. This project has greatly helped me to learn the basics of Ruby while customizing its templates to suit my needs.

Feed Search

I use the RSSHub Radar browser extension to find feeds for sites while I'm browsing. However, this would be a lot of work when I want to get feeds for a number of sites at once.

I found a few simple python apps that find feeds for me. They aren't perfect, but they do allow me to find feeds for multiple sites at the same time, all I have to do is format the query and hit enter.

As you can see below, these are not fully formed applications, just a few lines of code. To run them, it's necessary to install Python, install the package with pip (pip install feedsearch-crawler), and type python at the command prompt, which takes you to a Python terminal that will recognize these commands.

From there you can type\paste python commands for demonstration, practice, or for simple scripts like this. I could also put the following scripts into their own feedsearch.py file and type python feedsearch.py, but I haven't gotten around to doing anything like that.

Depending on the site, and the features you're interested in, either of these feed seekers has their merits.

Feedsearch Crawler

DBeath/feedsearch-crawler

from feedsearch_crawler import search
import logging
logging.basicConfig(filename='example.log',level=logging.DEBUG)
import output_opml
list = ["http://bigfintechmedia.com/Blog/","http://blockchainespana.com/","http://blog.deanland.com/"]
for items in list:
    feeds = search(items)
    output_opml(feeds).decode()
    logger = logging.getLogger("feedsearch_crawler")

Feed seeker

mitmedialab/feed_seeker

from feed_seeker import generate_feed_urls
list = ["http://bigfintechmedia.com/Blog/","http://blockchainespana.com/","http://blog.deanland.com/"]
for items in list:
    for url in generate_feed_urls(items):
        print(url)

GitHub Actions

Pluto Feed Reader is great, but I needed to find a way for it to run a regular schedule, so I wouldn't have to run the command every time I wanted to check for new feeds. For this, I've used GitHub actions.

This is an incredible feature of GitHub that allows you to spin up a virtual machine, install an operating system, dependencies supporting your application, and whatever commands you'd like to run, on a schedule.

name: Build BlogCatcher
on:
  schedule:
    # This action runs 4x a day.
    - cron:  '0/60 */4 * * *'
  push:
    paths:
    # It also runs whenever I add a new feed to Pluto's config file.
    - 'planetid.ini'
jobs:
  updatefeeds:
    # Install Ubuntu
    runs-on: ubuntu-latest
    steps:
    # Access my project repo to apply updates after pluto runs
    - uses: actions/checkout@v2
    - name: Set up Ruby
      uses: ruby/setup-ruby@v1
      with:
        ruby-version: 2.6
    - name: Install dependencies
      # Download and install SQLite (needed for Pluto), then delete downloaded installer
      run: | 
        wget http://security.ubuntu.com/ubuntu/pool/main/s/sqlite3/libsqlite3-dev_3.22.0-1ubuntu0.4_amd64.deb
        sudo dpkg -i libsqlite3-dev_3.22.0-1ubuntu0.4_amd64.deb
        rm libsqlite3-dev_3.22.0-1ubuntu0.4_amd64.deb
        gem install pluto && gem install nokogiri && gem install sanitize
    - name: build blogcatcher # This is the command I use to build my pluto project
      run: pluto b planetid.ini -t planetid -o docs
    - name: Deploy Files # This one adds the updates to my project
      run: |
        git remote add gh-token "https://github.com/identosphere/identity-blogcatcher.git"
        git config user.name "github-actions[bot]" 
        git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
        git add .
        git commit -a -m "update blogcatcher"
        git pull
        git push gh-token master

Identosphere Blogcatcher

Identosphere Blogcatcher (source) is a feed aggregator for personal blogs of people who've been working on digital identity through the years, inspired by the original Planet Identity.

We also have a page for companies, and another for organizations working in the field.

Identosphere Weekly Highlights

Last month, Kaliya suggested that since we have these pages up and running smoothly, we were ready to start our newsletter. This is just a small piece of the backend information portal we're working towards, and not enough to make this project as painless and comprehensive as possible, but we had enough to get started.

Every weekend we get together, browse the BlogCatcher, and share essential content others in our field will appreciate.

We'll be publishing our 6th edition, at the start of next week, and our numbers are doing well!

This newsletter is free, and a great opportunity for us to work together on something consistent while developing a few other ideas.

identosphere.substack.com

Setting up a newsletter without third-party intermediaries is more of a challenge than I'm currently up for, so we've settled on Substack for now, which seems to be a trending platform for tech newsletters.

It has a variety of options for both paid and free content, and you can read our content before subscribing.

Support us on Patreon

While keeping the newsletter free, we are accepting contributions via Patreon. (yes another intermediary, but we can draw upon a large existing userbase, and it's definitely easier than setting up a self-hosted alternative.)

So far, we have enough to cover a bit more than server costs, and this will ideally grow to support our efforts, and enable us to sustainably continue developing these open informational projects.

Python, Twitter Api, and GitHub Actions

Since we're publishing this newsletter, and I've gotten a better handle on my inner state, I decided it was time to come back to twitter. However, I knew I couldn't do it the old way, where I manually re-tweeted everything of interest, spending hours a day scrolling multiple accounts trying to stay abreast of important developments.

Instead, I dove into the twitter api. The benefits of using twitter programmatically can't be understated. For my first project, I decided to try an auto-poster, which could enable me to keep an active twitter account, without having to regularly pay attention to twitter.

I found a simple guide How To Write a Twitter Bot with Python and tweepy composed of a dozen lines of python. That simple script posts a tweet to your account, but I wanted to post from a pre-made list, and so figured out how to read from a yaml file, and then used GitHub actions to run the script on a regular schedule.

While that didn't result in anything I'm ready to share here, quite yet, somewhere during that process I realized that I could write python. After playing around with Ruby, in ERB, to build the BlogCatcher, and running various python scripts that other people wrote, tinkering where necessary, eventually I had pieced together enough knowledge I could actually write my own code!

Decentralized ID Weekly Twitter Collections

With that experience as a foundation I knew I was ready to come back to Twitter, begin trying to make more efficient use of its wealth of knowledge, and see about keeping up my accounts without losing too much hair.

I made a script that searches twitter for a variety of keywords related to decentralized identity, and write the tweet text and some other attributes to a csv file. From there, I can sort through those tweets, and save only the most relevant, and publish a few hundred tweets about decentralized identity to a weekly twitter collection, that make our job a lot easier than going to 100's of websites to find out what's happening. :D

Soon, these will be regularly published to decentralized-id.com, which I found out is an approved method of re-publishing tweets, unlike the ad hoc method I was using before, sharing them to discord channels (which grabs metadata and displays the preview image \ text), exporting their contents and re-publishing that.

I do intend to share my source for all that after I've gotten the kinks worked out, and set it running on an action.

Twitter collections I've made so far

Thanks for Reading!

Look forward to any feedback, suggestions or any ideas you have!

Jekyll HPSTR to Hugo HPSTR Theme Migration

⧉ infominer — Wed, 05 Jun 2019 23:58:56 +0000

I've tried to learn Hugo a few different ways, and finally found one that works for me, and I hope will help anyone who's trying to make the leap.

This guide and the accompanying repository should assist anyone switching from either Jekyll to Hugo, or Hugo to Jekyll.

If you are familiar with other Jekyll themes by Michael Rose, such as Minimal Mistakes, then HPSTR shouldn't come with too many surprises.

Jekyll HPSTR has its quirks like any theme, but I'm a novice and didn't have trouble forking it and building locally without any changes to the code.

When I found out there was a Hugo version of the HPSTR theme, I knew this was my opportunity to figure out Hugo, once and for all!

Now, I've got two branches of the same exact website!

One powered by Jekyll and one the other by Hugo!!

That's pretty neat, if you ask me!

Why

mmistakes/hpstr-jekyll-theme is archived, and you can no longer submit issues or pull-requests.
Hugo has a lot of support from the development community, and is growing quickly in features and popularity.
There are a lot of awesome Hugo themes I've been itching to try, particularly those supporting indieweb.
/themes/ directory allows for easy testing and switching between new themes.

HPSTR Jekyll vs Hugo

Since HPSTR is an older Hugo theme, I need to find an older version of Hugo to run it. I started near the release and worked my way forward through the releases.

Releases

It turns out that each version was released at the same time, and that they were built together from the very beginning!

The nice thing about hpstr-hugo-theme is that its not archived, and if you want to open any issues or pull-requests, that can be done.

Root Directories

Which Version of Hugo Should I Run?

Scrolling back through the hugo releases when the HPSTR theme was in active development, I come across:

0.16.0 June 6th 2016

Hugo 0.16 is our best and biggest release ever. The Hugo community has outdone itself with continued performance improvements, beautiful themes for all types of sites from project sites to documentation to blogs to portfolios, and increased stability.

.deb packages for Debian, Ubuntu, etc.

Hugo has become part of the official Debian and Ubuntu repositories since January 2016!

That's a nice note to find, since I'm an Ubuntu user.

v0.17 - October 2016

There are numerous options for different platforms, and this version of Hugo was released in October, one month after the final release of HPSTR.

Hugo is going global with our 0.17 release. [...] Adding additional languages to your website is simple and straightforward.

Hugo continues its trend of each release being faster than the last.

Apparently it's fast enough that people began using hugo's webserver in production, around this time.

New in 0.17: Available as Snap package

Thanks to the contribution #2443 and guidance from @dholbach, Hugo is now available as a Snap package! (Snaps are a new kind of universal Linux packages.) Check it out at https://uappexplorer.com/app/hugo.hugo-authors

Download links for Hugo v0.17

Now that I've got Hugo working, lets see how far I can go..

v0.30.1 - Oct 16, 2017

Hugo 0.30 is the Race Car Edition. Hugo is already very very fast, but much wants more. So we added Fast Render Mode. It is hard to explain, so start the Hugo development server with hugo server and start editing. Live reloads just got so much faster! The "how and what" is discussed at length in other places, but the short version is that we now re-render only the parts of the site that you are working on.

This is a big release, so I'm going to look ahead for the closest bug-fixes, but avoid any feature releases.

v0.31.1 - Nov 27, 2017

So.. there's another big release after this, at the start of 2018, but I think I'll stick with this one. It was published one year after HPSTR completed its development.

Test Install

Once it's installed, type I hugo version and read:

perfect!!

Clone the Hugo Theme.

With some pre-requisites out of the way, lets jump in at the first step in the theme setup:

Theme Setup - HPSTR Hugo

Even Migrating from one to the other, it's easier to start fresh in a new directory and pull in content from the original as necessary.

$ mkdir newProject
$ cd newProject
$ mkdir themes
$ cd themes
$ git clone https://github.com/dldx/hpstr-hugo-theme.git hpstr

and you will see hpster located in: /web-work/themes/hpstr.

Example Site

Once you have the newProject/themes/hpstr you'll find the folder exampleSite in the hpstr directory.

Just Copy the contents of exampleSite to the root of newProject, and test to see if it will run.

$ hugo

It should print something like this:

Started building sites ...
Built site for language en:
0 draft content
0 future content
0 expired content
10 pages created
0 non-page files copied
15 paginator pages created
9 tags created
0 categories created
total in 100 ms

Unless you get that print-out, don't bother changing your whole sites configuration just yet.

Make sure you placed the contents of exampleSite into the root of your project directory, and that your directories are structured properly.

Just to be sure! I'll test the server, also, and see that I get a website.

$ hugo server

Started building sites ...
Built site for language en:
0 draft content
0 future content
0 expired content
10 pages created
0 non-page files copied
15 paginator pages created
9 tags created
0 categories created
total in 61 ms
Watching for changes in /newProject/ {data,content,static,themes}

Yay!!!

Directory Structure HPSTR Jekyll vs Hugo

mmistakes.github.io/hpstr-jekyll-theme/theme-setup/

hpstr-hugo-theme/theme-setup

In with the new.

Now, it's simply moving over the content, and swapping out some frontmatter, and configuration formating.

Content

Posts are stored in the content directory. By default, only content in the content/posts will show up in the All Posts section, however, you can link to other sections manually. For example, if you create a post at gallery/photo1.md, your post will appear both under the home page and under /gallery.

You'll notice that your root directory mirrors the themes directory structure, because the theme always keeps a backup file of everything necessary to function.

archetypes/

I entered some tags and categories:

web-work/themes/hpstr/archetypes/default.md

+++
Description = ""
Tags = ["resources", "web-work"]
Categories = ["howto", "tools"]
menu = "main"
+++

data/

From YAML to TOML

In the theme directory is a navigation.yml file, and even awhole exampleSite that we can copy over to our root data directory and customize.

Be sure to change title: to title = and so forth.

[[links]]
title = "Webwork.tools"
url = "/webwork.tools/"

[[links]]
title = "Services"
url = "/services/"

[[links]]
title = "Mostly Free SEO Tools"
url = "/seo-tools/"

[[links]]
title = "GitHub Pages Starter Pack"
url = "/github-pages-starter-pack/"

[[links]]
title = "Practical Public Key Crypto"
url = "/practical-public-key-crypto/"

[[links]]
title = "InfoMine"
url = "http://infominer.id/"

Frontmatter

The main differences:

using type rather than layout
using the frontmatter variable date to signify publication date, rather than hardcoding it into the title.

HPSTR-Jekyll

---
layout: post
title: "Sample Code Post"
description: "Examples and code for various HPSTR functions."
tags: [samples, code, snippets]
comments: true
image:
  thumb: /images/pgp-og.png
  feature: pgp-banner.png
  background: triangular.png
modified: 2019-05-30T13:15:59-23:00
permalink: /sample-code/
---

HPSTR-Hugo

---
type: post
title: Sample Post
description: "Just about everything you'll need to style in the theme: headings, paragraphs, blockquotes, tables, code blocks, and more."
date: 2011-03-10

tags: [sample post]
image:
  feature: /images/abstract-3.jpg
  credit: dargadgetz
  creditlink: http://www.dargadgetz.com/ios-7-abstract-wallpaper-pack-for-iphone-5-and-ipod-touch-retina/
---

config.toml

Now we are getting places! Next step is to copy the config.toml from the root of our example site into the root of our repository.

This is the final stretch, and we should be good to go

baseurl = "https://infominer.id/web-work"
languageCode = "en-gb"
title = "Webwork.tools: Independent Web-Work and Skyscraper Publishing."
theme = "hpstr"
pluralizelisttitles = false
PygmentsCodeFences = true
Paginate = 5
disqusShortname = "hpstrhugo"
publishdir = "docs"
enableEmoji = true
[params]
    subtitle = "Digital Tools for a Digital Transformation."
    [params.author]
        name = "⧉ Infominer"
        avatar = "/images/info-id.png"
        bio = "Full-Time Crypto-Curation and Histories ⧉ #Bitcoin #Blockchain #DecentralizedID ⧉ Research, Publishing, #WebWork #Indieweb ⧉"
        github = "infominer33/web-work"
    [params.image]
      feature = "/images/web-work-tools.png"

Configuring Hugo

publishdir = "docs"

By setting this in our config.toml, we will be able to ask github pages to publish from the docs folder.

Redirects

One thing that's important to note, if you are switching from Jekyll HPSTR to HPSTR Hugo with me. HPSTR Hugo makes all your posts live in the /posts/ directory.

If you used to let your blog live at the root of the site, like me, then add aliases, which are how Hugo does redirects.

Content Manegement - Aliases

aliases:
  - /title-goes-here/
  - /other-title-too/

Build and Serve

You must type hugo to actually build and publish to the docs directory.

Even though hugo server builds new updates, it doesn't seem to update the files going back to our hosting provider. I don't really understand, but that seems to be the case.

Use a line-break before each list!

Twitter Cards

This part, I didn't figure out, and am too happy playing with various Hugo themes to worry about it, for the moment. I'll update here when I have a definitive answer.

I don't understand why .Title is not .title.

I'm pretty sure I need to add

<meta name="twitter:site" content="{{ .site }}">

but I didn't have success with that yet.

<!-- Twitter Cards -->
<meta name="twitter:title" content="{{ .Title }}">
<meta name="twitter:description" content="{{ with .Description }}{{ . }}{{ else }}{{ $.Site.Params.subtitle }}{{ end }}">
{{ with .Site.Params.owner.twitter }}<meta name="twitter:creator" content="{{ . }}">{{ end }}
{{ if isset ($.Scratch.Get "Params") "image" }}
    {{ $imageparams := index ($.Scratch.Get "Params") "image" }}
    {{ with $imageparams.thumb }}
        <meta name="twitter:card" content="summary">
        <meta name="twitter:image" content="{{ . | absURL }}">
    {{ else }}
        <meta name="twitter:card" content="summary_large_image">
        <meta name="twitter:image" content="{{ $imageparams.feature | absURL }}">
    {{ end }}
{{ end }}

I also added these lines to my config.toml


[twitter]
  card = "summary_image_large"
  site = "@infominer33"
  title =  "Webwork.tools: Independent Web-Work and Skyscraper Publishing."
  description = "Resources for Independent Webworkers: Web-Publishing, SEO, Static Site Generators, Ubuntu, Research Driven Content."
  image = "/images/web-work-tools.png"

Test Branches

I'm already trying a few Hugo Themes for different sites, in particularly one designed to support indieweb priniciples. I have a few demo images of HPSTR-Hugo in the Master branch. I didn't think ahead and save images of the Jekyll version.

However, I'll leave those branches to make sure anyone has a little easier time migrating from Jekyll to Hugo.

Resources

Other Hugo Themes

https://github.com/EmielH/tale-hugo

Over the Rainbow

Blogging with Org-Mode and Hugo

Reach out in the Comments

I'd love to hear what your favorite themes are, Jekyll or Hugo.

Feel free to ask any questions, I'm no expert, but am darn good with a search engine, and have plenty of opinions. :)