Forem: Fabio Manganiello

Correlation without tears

Fabio Manganiello — Fri, 11 Feb 2022 12:09:52 +0000

Assessing the degree of correlation between two numeric series is a notoriously challenging task in every scientific discipline, as well as a crucial aspect of every scientific research (ever wondered if there's a correlation between the usage of Internet Explorer and the murder rate?).

Different coefficients have been proposed over the years for different series with different properties (most notably, Pearson's, Spearman's and Kendall's coefficients), and the quest for a "universal" correlation coefficient has often been unproductive.

It took me a while to digest the math behind the new paper from Sourav Chatterjee (https://arxiv.org/abs/1909.10140), but once I modelled the proposed coefficient into Python code I was surprised by how well it performed on arbitrary numeric series (not necessarily monotonic) compared to most of the coefficients out there. And it's also very easy to calculate compared to other coefficients.

So I've put together a Gist notebook that shows how the new coefficient works on some simple data with increasing levels of noise.

Feel free to reuse the code if you need it!

[Trivia] What physical address is mentioned the most in all the software ever written?

Fabio Manganiello — Fri, 28 Jan 2022 21:27:26 +0000

While contributing to some open-source repo the other day, I diligently copied that copyright license boilerplate that every good developer adds to their code without ever reading. For the first time in two decades, I decided to actually read it, and the last lines particularly triggered my attention:

* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
* 02111-1307, USA.

The first obvious question is: does the FSF today still process paper letters coming from people all over the world who want a paper copy of a license that can be found on the web within 2 ms? I've been honestly tempted by the idea of sending a physical letter to the FSF and request a hard copy of the GPL 3 license, just to frame it in my room (fax could also be an interesting option, who know if they provide the feature).

The second obvious question is: just how common is this address in today's code? So I asked Github, and got back an astonishing 131M results.

Do you know any other address that may be referenced more often in all the software ever written?

Web 3.0 and the undeliverable promise of decentralization

Fabio Manganiello — Mon, 17 Jan 2022 09:14:11 +0000

What was the Web 3.0 supposed to mean again?

The Semantic Web

The definition of "Web 3.0" has assumed different incarnations over the past few years, depending on whom you asked and when. We initially thought it would have been some kind of semantic web, with shared ontologies and taxonomies that described all the domains relevant to the published content. The Internet, according to this vision, will be a collective semantic engine driven by a shared language to describe entities and the way they connect and interact with one another. It will be easy for a machine to extract the "meaning" and the main entities of another page, so the Internet will be a network of connected concepts rather than a network of connected pages.

Looking at things in 2022, we can arguably say that this vision hasn't (yet?) been realized. Despite being envisioned for at least the past 20 years, today approximately 1-2% of the Internet domains contain or support the semantic markup extensions proposed by RDF / OWL. I still believe that there's plenty of untapped potential in that area but, since there hasn't been a compelling commercial use-case to invest tons of resources in building a shared Internet of meaning instead of an Internet of links, the discussion has gotten stranded between academia and W3C standards (see also the curse of protocols and shared standards).

The Web-of-Things

More recently, we envisioned the Web 3.0 as the Internet of Things. According to this vision, IPv6 should have taken over within a few years, every device would have been directly connected to the Internet with no need of workarounds such as NAT (IPv6 provides enough addresses for each of the atoms on the planet), all the devices will ideally be talking directly to one another over some extension of HTTP.

Looking at things in 2022, we can say that this vision hasn't (yet) become a reality either. Many years down the line, only one third of the traffic on the Internet is IPv6. In countries like Italy and Spain, the adoption rate is less than 4-5%. So NAT-ting is still a thing, with the difference that it has now been largely delegated to a few centralized networks (mainly Google, Apple and Amazon and their ecosystems): so much for the dream of a decentralized Internet of things. And the situation on the protocols side isn't that rosy either. The problem of fragmentation in the IoT ecosystem is well documented. The main actors in this space have built their own ecosystems, walled gardens and protocols. The Matter standard should solve some of these problems, but after a lot of hype there isn't much usable code on it yet, nor usable specifications (2022 should be the year of Matter though, that's what everybody says). Regardless of how successful the Matter project will be in solving the fragmentation that prevents the Internet of things from being an actual thing, it's sobering to look back at how far we've come from the original vision. We initially envisioned a decentralized web of devices, each with its own address, all talking the same protocols, all adhering to the same standards. We have instead ended up with an IoT run over a few very centralized platforms, with limited capabilities when it comes to talking to one another, and the most likely to solution to this problem will be a new standard which hasn't been established by academic experts, ISO or IEEE, but by the same large companies that run the current oligopoly. Again, so much for the dream of a decentralized Internet of things.

Crypto to the rescue?

Coming to the definition of Web 3.0 most popular in the last couple of years, we've got the Internet-over-Blockchain, a.k.a. crypto-web, a.k.a. dApps. The idea is pretty straightforward: if Blockchain can store financial transactions in a decentralized way, and it can also run some code that transparently manipulates its data in the form of smart contracts, then we've got the foundations to build both data and code in a decentralized way.

To understand how we've got here, let's take a step back and let's ask ourselves a simple question.

What problems is the Web 3.0 actually trying to solve?

What is wrong with the Web 2.0 that actually wants us to move towards a Web 3.0, with all the costs involved in changing the way we organize software, networks, companies and whole economies around them?

Even if they envision different futures, the Semantic Web, the Web-of-Things and the Crypto-Web have several elements in common. And the common trait to all those elements can be summarized by a single word: decentralization.

So why is the Web 3.0, in all of its incarnations, so much into decentralization?

An obvious part to the answer is, of course, that the Web 2.0 has become too much about centralization (of data, code, services, and infrastructure) in the hands of a few corporations.

The other part of the answer to the question is the fact that software engineering, in all of its forms, is a science that studies a pendulum that swings back and forth between centralization and decentralization (the most cynical would argue that this is the reason why information engineers will never be jobless). This common metaphor, however, provides only a low-resolution picture of reality. I believe that the information technology has a strong center of mass that pulls towards centralization. When the system has enough energy (for various reason that we'll investigate shortly) it flips towards decentralization, just to flip back to a centralized approach when the cost of running a system with an increasingly high number of moving parts becomes too high. At that point, most of the people are happy to step back and accept somebody providing them with infrastructure/platform/software-as-a-service®, so they can just focus on whatever their core business is. At that point, the costs and risks of running the technology themselves are too high compared to simply using it as a service provided by someone who has invested enough resources in commoditising that technology leveraging an economy of scale.

The few companies that happen to be in that sweet spot in that precise moment during the evolution of the industry (an economist would call it the moment where the center of mass between demand and supply naturally shifts) are blessed with the privilege of monopoly or oligopoly in the industry for at least the next decade to come - which, when it comes to the IT industry, is the equivalent of a geologic eon. How long it will take until the next brief "decentralized revolution" (whose only possible outcome is the rising of a new centralized ruling oligarchy) depends on how much energy the system accumulates during their reign.

There is a simple reason why technology inertially moves towards centralization unless a sufficient external force is applied, and I think that this reason goes often under-mentioned:

Most of the folks don't want to run and maintain their own servers.

As an engineer and a hobbyist maker that has been running his own servers for almost two decades (starting with a repurposed Pentium computer in my cabinet a long time ago), as well as a long-time advocate of decentralized solutions, this is a truth that hurts. But the earlier we accept this truth, the better it is for our sanity - and the sanity of the entire industry.

The average Joe doesn't want to learn how to install, run and maintain a web server and a database to manage their home automation, their communication with family and friends or their news feed unless they really have to.

Just like most of the folks out there don't want to run and maintain their own power generators, their own water supply and sewage systems, their own telephone cables or their own gasoline refineries. When a product or a technology becomes so pervasive that most of the people either want it or expect it, then they'll expect somebody else to provide it to them as a service.

However, people may still run their own decentralized solutions if it's required. Usually this happens when at least one of the following conditions apply:

A shortage occurs: think of an authoritarian government cutting its population out of the most popular platforms for messaging or social network, prompting people to flock to smaller, often decentralized or federated, solutions.
The technology for a viable centralized solution isn't out there yet: think of the boom of self-administered forums and online communities that preceded the advent of social networks and the Web 2.0.
Some new technology provides many more benefits compared to those provided by the centralized solution: think of the advantage of using a personal computer compared to using an ad-hoc, time-slotted connection to a mainframe.
People don't trust the centralized solutions out there: this has definitely been the case after the Cambridge Analytica scandal brought in the spotlight the fact that free online platform aren't really free, and that giving lots of your data to a small set of companies can have huge unintended consequences.

When one of these conditions apply, the industry behaves like a physical system that has collected enough energy, and it briefly jumps to its next excited state. Enough momentum may build up that leads to the next brief decentralized revolution that may or may not topple the current ruling class. However, once the initial momentum settles, new players and standards emerge from the initial excitement, and most of the people are just happy to use the new solutions as on-demand services.

Unlike an electron temporarily excited by a photon, however, the system rarely goes back to its exact previous state. The new technologies are usually there to stay, but they are either swallowed by new large centralized entities, packaged as commodified services, or they are applied in much more limited scopes than those initially envisioned.

How is the Web 3.0 going to achieve decentralization?

The different incarnations of the Web 3.0 propose different solutions to implement the vision of a more decentralized Web.

The Semantic Web proposes decentralization through presentation protocols. If we have defined a shared ontology that describes what an apple is, what it looks, smells and tastes like, how it can be described in words, what are its attributes, and we have also agreed to a universal identifier for the concept of "apple", then we can build a network of content where we may tag this concept as an ID in a shared ontology in our web pages (that may reference it as an image, as a word, as a description in the form of text or audio), and any piece of code out there should be able to understand that some parts of that web page page talk about apples. In this ideal world the role of a centralized search engine becomes more marginal: the network itself is designed to be a network of servers that can understand content and context, and they can render both requests and responses from/to human/machine-friendly formats with no middlemen required. Web crawling is not only tolerated and advised: it's the way the Web itself is built. Machines can talk to one another and exchange actual content and context, instead of HTML pages that are only rendered for human eyes.
The Web-of-Things proposes decentralization through network protocols, as well as scaling up the number of devices. If instead of having a few big computers that do a lot of things we break them up into smaller devices with lower power requirements, each doing a specific task, all part of the same shared global network, all with their own unique address, all communicating with one another through shared protocols, then the issues of fragmentation and lock-in effects of the economies of platforms can be solved by the natural benefits of the economies of scale. We wouldn't even need a traditional client/server paradigm in such a solution: each device can be both client and server depending on the context.
The Crypto-Web proposes decentralization through distributed data and code storage. If Blockchains can store something as sensitive as an individual's financial transactions on a publicly accessible decentralized network, and in a way that is secure, transparent and anonymous at the same time, and they can even run smart contracts to manipulate these transactions, then they are surely the tool we need to build a decentralized Web.

Why decentralization hasn't happened before

We have briefly touched in the previous paragraphs why decentralization usually doesn't work on the long run. To better formalize them.

People don't like to run and maintain their own servers

Let's repeat this concept, even if it hurts many of us. Decentralization often involves having many people running their own services, and people usually don't like to run and maintain their own services for too long, unless they really have to. Especially if there are centralized solutions out there that can easily take off that burden.

Decentralized protocols move much slower than centralized platforms

Decentralization involves coming up with shared protocols that everybody agrees on. This is a slow process that usually requires reaching consensus between competing businesses, academia, standards organizations, domain experts and other relevant stakeholders.

Once those protocols are set in stone, it's very hard to change them or extend them, because a lot of hardware and software has been built on those standards. You can come up with extension, plugins and patch versions, or introduce a breaking major version at some point, but all these solutions are usually painful and bring fragmentation issues. The greater the success of your distributed protocol, the harder it is to change, because many nodes will be required to keep up with the changes.

The HTTP 1.1 protocol (a.k.a. RFC 2068 standard) that still largely runs the Web today was released in 1997, long before many of today's TikTok celebrities were even born. On the other hand, Google developed and released gRPC within a couple of years. The IMAP and SMTP protocols that have been delivering emails for decades will probably never manage to implement official support for encryption, while WhatsApp managed to roll out support for end-to-end encryption to all of its users with a single app update. IRC is unlikely to ever find a standard way to support video chat, and its logic for sending files still relies on non-standard extensions built over DCC/XDCC, while a platform like Zoom managed to go from zero to becoming new standard de facto for the videochat within a couple of years.

The same bitter truth applies to the protocols proposed both by the semantic web and the IoT web. The push towards IPv6 has moved with the pace of a snail over the past 25 years, both for political reasons (the push, at least in the past, often came from the developing world, eager to get more IP addresses for their new networks, while the US and Europe, which hoarded most of the IPv4 subnets, often preferred to kick the can down the road), but, most importantly, for practical reasons. Millions of lines of code have been written over decades assuming that an IP address has a certain format, and that it can be operated by a certain set of functions. All that code out there needs to be updated and tested, and all the network devices that run that code, from your humble smoke sensor all the way up to the BGP routers that run the backbone of the Internet, need to be updated. Even though most of the modern operating systems have been supporting IPv6 for a long time by now, there is a very long tail of user-space software that needs to be updated.

While the Internet slowly embraces a new version that allows it to allocate more addresses, companies like Google and Amazon have come in to fill that gap. We still need many devices to be able to transfer messages, but instead of communicating directly to one another, they now communicate with Google's or Amazon's clouds, which basically manage them remotely, and allow communication between two different devices only if both of them support their protocols. So yes, an Internet-of-Things revolution is probably happening already, but it's unlikely to be decentralized, and it smells more like some cloud-based form of NAT.

A shared standard may also be on the way (after many years) in the form of Matter, but it's a different standard than the open standards we are used to. It's a standard agreed mainly by the largest platforms on the market, with little to no involvement from the other traditional actors (academia, standards organizations, other developers).

Money runs the world, not protocols and technologies

Things at these scales often move only when loads of money and human resources are poured into a field. No matter how good the new way of doing things is, or how bad the current way of doing thing is. The semantic web and its open protocols haven't really taken off because they were pursued mostly by academia, standards organizations and Tim Berners-Lee. It didn't really pick up momentum in the rest of the industry because at some point solutions like Google came up. Why would you radically change the way you build, present and distribute your web pages, when Google already provides some black magic to infer content and context and search for anything you want on the Web? When Google already fills that gap somehow, it doesn't matter anymore whether the solution is centralized or not: most folks only care that it works well enough for their daily use. Not only: the vision of a semantic web where content is exposed in a form that is both human-readable and machine-readable, where web crawling becomes a structural part of the new experience, would jeopardise the very reason why the world needs services like the Google search engine. The idea of a semantic web, at least as it was proposed by the W3C, is amazing on paper, but it's hard to monetize. Why would I make my website easier for another machine to scrape and extract content from? We can arguably say that the Web has gone in the opposite direction in the meantime. Many sites have paywalls to protect their precious content from unpaid access. Many have built barriers against robots and scrapers. Many even sue companies like Google for ripping them (through Google News) of the precious profits that come from people dropping their eyeballs on the ads running on their websites. Why would a for-profit website want to invest time to rewrite its code in a way that makes its content easier to access, embed and reuse on other websites, when the whole Internet economy basically runs on ads that people can see only if they come to your website? No matter how good the technological alternatives are, they will never work if the current economic model rewards walled gardens or those that run ads that are seen by the highest number of people, especially if the system is already structurally biased towards centralization.

Will the crypto-web succeed where others have failed?

The crypto-web tries to solve some of the hyper-centralization issues of the Web 2.0 by leveraging the features of Blockchain. Here are some of the proposed solutions:

Monetization problem: the issue is addressed by simply making money a structural part of the architecture. Your digital wallet is also the holder of your identity. You can pay and get paid for content on the platform by using one of the virtual currencies it supports. The currency, the ledger of transactions and the platform are, basically, one single entity. So we have a digital currency at the core, and enough people who agree that it has some form of value, but very few ways to make use of that value - unless you live in El Salvador, you probably don't use Bitcoins to pay for your groceries. For that reason, most of the cryptocurrencies have been, over the past years, mainly instruments for investment and speculation, rather than actual ways to purchase goods and services. The new vision of the Web 3.0 tries to close this loop, by creating a whole IT ecosystem and a whole digital economy around the cryptocurrencies (the most cynical would argue that it's been designed so those who hoarded cryptocurrencies over the past years can have a way of cashing their assets, by artificially creating demand around a scarcely fungible asset). If cryptocurrencies were a solution looking for a problem, then we've now got a problem they can finally solve.
Identity: a traditional problem in any network of computers is how to authenticate remote users. The Web 2.0 showed the power of centralization when it comes to identity management: if a 3rd-party like Google, Facebook or Apple can guarantee (through whatever identification mechanism they have implemented) that somebody is really a certain person, then we don't have to reinvent the authentication/authorization wheel from scratch on every different system. We delegate these functionalities to a centralized 3rd-party that owns your digital identity through mechanisms like OAuth2. By now, we also all know the obvious downsides of this approach - loss of privacy, single-point of failure (if your Google/Facebook account gets blocked/deleted/compromised then you won't be able to access most of your online services), your digital identity is at the mercy of a commercial business that doesn't really care a lot about you, Big Button Brother effect (no matter what you visit on the Internet, you'll always have some Google/Facebook button/tracker/hidden pixel somewhere that sees what you're doing). The crypto-web proposes a solution that maintains the nice features of a federated authentication, while being as anonymous as you can get. Your digital wallet becomes a proxy to your identity. If you can prove that you are the owner of your wallet (usually by proving that you can sign data with its private key), then that counts as authentication. However, since your wallet is just a pair of keys registered on the Blockchain with no metadata linked to you as a physical person, this can be a way to authenticate you without identifying you as a physical person. An interesting solution is also provided for the problem of "what if a user loses their password/private key/the account gets compromised?" in the form of social recovery. In the world of Web 2.0, identity recovery isn't usually a big deal. Your digital identity is owned by a private company, which also happens to know your email address or phone number, or it also happens to run the OS on your phone, and they have other ways to reach out to you and confirm your identity. In the world of crypto, where an account is just a pair of cryptographic keys with no connection to a person, this gets tricky. However, you can select a small circle of family members or friends that have the power to confirm your identity and migrate your data to a new keypair.
Protocols and standards: they are replaced by the Blockchains. Blockchains are ledgers of transactions, and the largest ones (such as Bitcoin and Ethereum) are unlikely to be going anywhere because a lot of people have invested a lot of money into them. Since Blockchains can be used a general-purpose storage for data and code, then we've got the foundations for a new technology that doesn't require us to enforce new protocols that may or may not be adopted.
Scalability: decentralized systems are notoriously difficult to scale, but that's simpler for Blockchains. Everybody can run a node, so you just add more nodes to the network. If people don't want to run their nodes, then they can use a paid service (like Infura, Alchemy) to interact with the Blockchain.

So far so good, right? Well, let's explore why these promises may be based on a quite naive interpretation of how things work in reality.

Blockchains are scalability nightmares

Let's first get the elephant out of the room. From an engineering perspective, a Blockchain is a scalability nightmare. And, if implemented in its most decentralized way (with blocks added by nodes through proof-of-work), they are environmental and geopolitical disasters waiting to happen. This is because the whole assumption that anybody can run their own node on a large Blockchain is nowadays false.

Running a Blockchain is a dirty business

In the early days of the crypto-mania, anybody could indeed run a small clusters of spare Raspberry Pis, mine Bitcoins and make some money on the side. Nowadays, big mining companies have installed server farms in the places where energy is the cheapest, such as near Inner Mongolia's hydroelectric dams, near Xinjiang's coal mines or leveraging Kazakhstan's cheap natural gas.

The author of the post on Tom's Hardware mentions that in February 2021 he made $0.00015569 after running mining software on his Raspberry Pi 4 for 8 hours. His hash rate (the number of block hashes added to the Blockchain per unit of time) varied from 1.6 H/s to 33.3 H/s. For comparison, the average hash rate of the pool was 10.27 MH/s. That's 3-4 million times more than the average hash rate of a Raspberry Pi.

The examples I mentioned (Inner Mongolia, Xinjiang and Kazakhstan) are not casual. At its peak (around February 2020) Chinese miners contributed more than 72% of the Bitcoin hashes. The share hovered between 55% and 67% until July 2021, when the government imposed a nationwide crackdown on private crypto-mining. The motivations, as usual, are complex, and they obviously include the government's efforts of creating a digital yuan. However, the fact that private miners scoop up a lot of cheap energy before it can get downstream to the rest of the grid, especially during times of energy supply shortages and inflationary pushes on the economy, has surely played a major role in Beijing's decisions.

The fall of Beijing in the crypto market left a void that other players have rushed to fill in. The share of Bitcoin hashes contributed by nodes in Kazakhstan has gone from 1.42% in September 2019 up to 8.8% in June 2021, when China announced its crackdown on crypto. Since then, it climbed all the way up to 18% in August 2021 - and that's the latest available data point on the Cambridge Bitcoin Electricity Consumption Index. From more recent news, higher demand for Kazakh gas, mixed with the high inflation, pushed energy prices up, triggering mass protests and the resignation of the government.

The curse of the proof-of-work

Why do decentralized Blockchains consume so much energy that they can impact the energy demand and production of whole countries, as well as destabilise whole geopolitical landscapes?

The Bitcoin Blockchain processes less than 300 thousands transactions per day. For a comparison, as of 2019 the credit card circuits processed about 108.6 millions transactions per day in the US alone.

We therefore have a proposed new approach whose underlying infrastructure currently processes less than 0.28% of the transactions processed in the US alone. Interestingly, the volumes of transactions processed by Visa and Mastercard have never caused geopolitical instabilities nor drained the energy supply of whole countries. How come?

To answer this question, we have to understand how Blockchains (or, at least, the most popular decentralized flavour initially proposed by Natoshi Sakamoto's paper) work under the hood.

We have often heard that a Blockchain is secure by design, and it allows the distributed creation of a ledger of transactions among peers that don't necessarily trust one another. This is possible through what is commonly known as proof-of-work. Basically, any node can add new blocks to the distributed chain, provided that the operation satisfies two requirements:

The new blocks are cryptographically linked to one another and consistent with the hashes of the previous latest block. A Blockchain is, as the name suggest, a chain of blocks. Each of these blocks contains a transaction (e.g. the user associated to the wallet ABC has paid 0.0001 Bitcoins to the user associated to the user associated to the wallet XYZ on time t), and a pointer with the hash of the latest block in the chain at the time t. Any new added blocks must satisfy this contraint: basically, blocks are always added in consecutive order to the chain, they cannot be removed (that would require all the hashes of the following blocks to be updated), and this is usually a good way of ensuring temporal consistency of events on a distributed network.
In order to add new blocks, the miner is supposed to solve a numeric puzzle to prove that they have spent a certain amount of computational resources before requesting to add new blocks. This puzzle is defined in such a way that:

2.1. It's hard to find a solution to it (or, to be more precise, a solution can only be found through CPU-intensive brute force);

2.2. It's easy to verify if a solution is valid (it usually involves calculating a simple hash of a string through an algorithm like SHA-1).

The most popular approach (initially proposed in 1997 by Hashcash to prevent spam/denial-of-service, and later adopted by Bitcoin) uses partial hash inversions to verify that enough work has been done by a node before accepting new data from it. The network periodically calibrates the difficulty of the task by establishing the number of hash combinations in the cryptographic proofs provided by the miners (on Bitcoin, this usually means that the target hash should start with a certain number of zeros). This is tuned in such a way that the block time (i.e. the average time required to add a new block to the network) remains constant. If the only way to find a solution to the riddle is by brute-forcing all the possible combinations, then you can tune the time required to solve the riddle by changing the total number of combinations that a node should explore. If a lot more powerful mining machines are added to the pool, then it'll likely push down the block hashing time, and therefore at some point the network may increase the difficulty of the task by increasing the number of combinations a CPU is supposed to go through.

For example, let's suppose that at some point the network requires the target hashes to start with 52 binary zeros (i.e. 13 hexadecimal zeros). On the original Hashcash network, an email could be sent on January 19, 2038 to calvin@comics.net if it had a header structured like this:

X-Hashcash: 1:52:380119:calvin@comics.net:::9B760005E92F0DAE

Where:

X-Hashcash is the header of the string to be used as a cryptographic challenge;
52 is the number of leading binary zeros that the target hash is supposed to have;
380119 represents the timestamp of the transaction;
9B760005E92F0DAE is the hexadecimal digest calculated by the miner that, appended to the metadata of the block, satisfies the hashing requirements.

If we take the string above (except for the X-Hashcash: header title and any leading whitespaces) and we calculate its SHA-1 hash we get:

0000000000000756af69e2ffbdb930261873cd71

This hash has 13 leading hexadecimal zeros (i.e. 52 binary zeroes), and it therefore satisfies the network's hashing requirements: the node is allowed to process the requested information.

Here you go: you have an algorithm that makes it computationally expensive to add new data (a miner has to iterate through zillions of combinations to find a string whose hash satisfies a certain condition), but computationally easy to verify that the hash of a block is correct (you just have to calculate a SHA hash and verify that it meets the condition).

Miners are usually rewarded for their work by being paid a certain fee of the amount of the processed transaction. It's easy to see how early miners who invested the unused power of a laptop or a Raspberry Pi to mine Bitcoins managed to make quite some money on the side, while making a profit out of mining nowadays, given how high the bar has moved over the past years, is nearly impossible unless you invest on some specialised boards that suck up a lot of power.

The proof-of-work provides a secure mechanism to process information among untrusted parties. A malicious actor who wants to add invalid data to the chain (let's say, move one Bitcoin to their wallet from another wallet that hasn't really authorised the transfer) will first have to crack the private key of the other wallet in order to sign a valid transaction, then invest enough computational power to add the block - and this usually means being faster in average than all the nodes on the Blockchain, which already sets a quite high bar on large Blockchains. The new block needs to be accepted through shared consensus mechanisms from a qualified majority of the nodes on network, and usually long chains are more likely to be added than short ones - meaning that you have to either control a large share of the nodes on the network (which is usually unlikely unless you have invested the whole GDP of a country on it), or build a very long chain, therefore spending more power in average than all the other nodes on the network, to increase the chances that your rogue block gets included. Not only, but the new rogue block needs to be cryptographically linked to all the previous and next transactions in the ledger. So the rogue actor will basically have to forever run a system that is faster than all the other nodes in order to "keep up" with the lie. This is, in practice, impossible. In some very rare cases, rogue blocks may eventually be added to the Blockchain. But, when that happens, the network can usually spot the inconsistency within a short time and fork the chain (i.e. cut a new chain that excludes the rogue block and any successive blocks).

This also means that rogue nodes are implicitly punished by the network itself. An actor who tries to insert invalid nodes will eventually be unable to keep up their lie, but they will have wasted the energy required to calculate the hashes necessary to add those blocks. This means that they'll have wasted a lot of energy without being rewarded in any way.

A problem that wasn't originally addressed by the Hashcash proposal (and wasn't addressed by earlier cryptocurrencies either) was that of double-spending. This has been addressed by Bitcoin by basically implementing a P2P validation where a distributed set of nodes needs to reach consensus before a new block is added to the chain, and temporal consistency of transactions is enforced by design with a double-linked chain of nodes.

The proof-of-work is a proof-of-concept that was never supposed to scale

Now that we know how a Blockchain manages to store a distributed ledger of transactions among untrusted parties, it's easy to see why it's such a scalability nightmare. Basically, the whole system is designed around the idea that its energy consumption increases with the number of transactions processed by the network. Not only: since the whole algorithm for adding new blocks relies on a massively CPU-intensive brute-force logic to calculate a hash that start with a certain number of leading zeros, the amount of CPU work (and, therefore, electricity usage) is basically the underlying currency of the network. If the average block hashing time decreases too much, then the likelihook of success for rogue actors who want to add invalid transactions and invest enough in hardware resources increases. We therefore need to keep the cost of mining high so that it's technically unfeasible for malicious actors to spoof transactions.

As engineers, we should be horrified by a technical solution whose energy requirements increase linearly (or on even steeper curves) with the amount of data it processes per unit of time. Managing the scale of a system so that the requirements to run it don't increase at the same pace as the volumes of data that it processes is corner-stone of engineering, and Blockchains basically throws this principle out of window while trying to convince us that their solution is better. Had the "Blockchain revolution" happened under other circumstances, we'd be quick to dismiss its proposals as hype in the best case, and outright scam in the worst case. In an economy that has to increasingly deal with limited supply of resources and increasingly tighter energy constraints, a solution that basically requires wasting CPU power just to add a new block (not only: a system where the amount of electric energy spent is the base currency used to reward nodes) is not only unscalable: it's immoral.

Not only: the requirement to keep a block mining time constant means that, by design, the network can only handle a limited amount of transactions. In other words, we are capping the throughput of the system by design in order to maintain the system secure and consistent. I'm not sure how anybody with any foundation of logical reasoning can believe that such a system can scale to the point that it replaces the way we do things today.

Proof-of-stake to the rescue?

The scalability issues of the proof-of-work mechanism are well-known to anybody who has any level of technical clue. Those who deny it are simply lying.

In fact, Ethereum is supposed to migrate during 2022 away from proof-of-work and towards a proof-or-stake mechanism.

Under such a mechanism for distributed consensus, miners don't compete with one another to add new blocks, and they aren't rewarded on the basis of the electric energy consumed. Instead, they have a certain amount of tokens that represent votes that they can use to validate transactions added to the network. Under the newly proposed approach, for example, a node would have to spend 32 ETH in order to become a validator on the network (invested stake). Whenever a node tries to add a new transaction to the network, a pool of validating nodes is chosen a random. The behaviour of a validator determines the value of its stake - in other words, rogue nodes would be punished by karma:

Nodes are rewarded (their stake is increased) when they add/validate new valid transactions.
Nodes can be punished (their stake is decreased), for example, when they are offline (as it is seen as a failure to validate blocks), or when they add/validate invalid transactions. Under the worst scenario (collusion), a node could lose its entire stake.

On paper, this approach should guarantee the same goal as proof-of-work (reaching a shared consensus among a distributed network of untrusted parties), without burning up the whole energy budget of the planet in the process. Why don't we just go all-in on the proof-of-stake then?

Well, one reason is that it's a system that is definitely less tested on wide networks compared to proof-of-work. PoW works because it leverage physical constraints (like the amount of consumed energy) to add new blocks. PoS removes those constraints, it proposes instead a stake-based system that leverages some well-documented mechanisms of reward and punishment from game theory, but once the physical constraint is removed there are simply too many things to take into account, and different ways a malicious actor may try and trick the system.

A common problem of PoS systems is that of a 51% attack. An attacker who owns 50%+1 of the share of stakes of a currency can easily push all the transactions that they want - consensus through qualified majority is basically guaranteed. The risk for this kind of attack is indeed very low in the case of Ethereum - one Ethereum coin is expensive, buying such a large share of them is even more expensive, and anyone who invests such a large amount has no interest in fooling the Blockchain, cause a loss of trust and therefore erode the value of their own shares. However, this risk is still very present in smaller Blockchains. PoS is a mechanism that probably only works in the case of large Blockchains where it's expensive to buy such a large share of tokens. On smaller Blockchains with a lower barrier to buy a majority stake it can still be a real issue.

There is another issue, commonly known as nothing-at-stake. This kind of issues can occur when a Blockchain forks - either because of malicious action or because of two nodes actually propose two blocks at the exact same time, therefore making temporal causality tricky.

When this happens, it's in the interest of the miners to keep mining both the chains. Suppose that they keep mining only one chain, and the other one is eventually the one that picks up steam: the miner's revenue will sink down. In other words, mining all the forks of a chain maximises the revenue of a miner.

Suppose now that an attacker wants to attempt a double-spending attack. They may do so by attempting to create a fork in the Blockchain (for example, by submitting two blocks simultaneously from two nodes that they own) just before spending some coins. If the attacker only keeps mining their fork, while all the other miners act in their self-interest and keep mining both the forks, then the attacker's fork eventually becomes the longest chain, even if the attacker only had a small initial stake.

A simpler version of this attack is the one where a validator cashes out their whole stake but the network does not revoke their keys. In this case, they may still be able to sign whatever transactions that they wish, but the system may not be able to punish them - the node has nothing at stake, therefore it can't be punished by slashing their stake.

Ethereum proposes to mitigate these issues by linking the stake onto the network to an actual financial deposit. If an attacker tries to trick the system, they may lose real money that they have invested.

There is, however, a problem with both the mitigation policies proposed by Ethereum. Their idea is basically to disincentivise malicious actors by increasing the initial stake required to become a validator. The idea is that, if the stake you lose is higher than the money you can make by tricking the system, you'll think twice before doing it. What this means is that, in order for the system to work, you need to increase the initial barriers as the value of your currency (and the number of transaction it processes) increases. So, if it works, it'll basically turn the Blockchain into a more centralized system where a limited amount of nodes can afford to pay the high cost required to validate new blocks (so much for the decentralization dream). As of now, the entry fee to become a validating node is already as high as 32 ETH - more than $100k. As the system needs to ensure that the stake a node loses for rogue behaviour is higher than the average gain they can make from tricking the system, this cost is only likely to go up, making the business of becoming a validating node on the Blockchain very expensive. If it doesn't, if the value of the cryptocurrency plunges and the perceived cost of tricking the system gets lower, then there's nothing protecting the system from malicious actors.

In other words, proof-of-stake is a system that works well under some very well defined assumptions - and, when it works, it's probably doomed to swing towards a centralized oligarchy of people loaded with money anyways.

How about data storage scalability?

Another scalability pain point of Blockchains is on the data storage side. Remember that any mining/validating node on the network needs to be able to validate any transaction submitted to the network. In order to perform this task, they need a complete synchronized snapshot of the whole Blockchain, starting with its first block all the way up to the latest one.

If 300k blocks per day are submitted, and supposing that the average size of a block is 1 KB, this translates into a data storage requirement of about 100 GB per year just to store the ledger of transactions.

As an engineer, suppose that you have a technological system that requires thousands of nodes on the network to process about 0.2% of the volumes of transactions currently processed by Visa/Mastercard/Amex in the US alone. The amount of nodes required on the network increases proportionally with the amount of transactions that the network is able to process, because we want the process of adding new blocks to be artificially CPU-bound. Moreover, each of these nodes has to keep an up-to-date snapshot of the whole history of the transactions, and it requires about 100 GB for each node to store a year of transactions. Since every node has to keep an up-to-date snapshot of the whole ledger, you can't rely on popular scalability solutions such as data sharding or streaming pipelines. Not only, but adding a new block to this "decentralized filesystem" requires that you run some code that wastes CPU cycles in order to solve a stupid numeric puzzle with SHA hashes. How would you scale up such a system to manage not only the amount of financial transactions currently handled by the existing solutions, but run a whole web of decentralized apps on it? I think that most of the experienced engineers out there would feel either uncomfortable or powerless if faced with such a scalability challenge.

Castles built on sand

The Blockchain has promised us a system where any nodes could join the network, add and validate transactions (distributed promise) in a scalable and secure way, and that system could soon disrupt not only the way we manage financial transactions, but the way we run the entire Web. This whole promise was based around mechanisms to reach shared consensus on distributed networks.

It turns out that both those mechanism (proof-of-work and proof-of-stake) aren't ready to fullfil those promises.

Proof-of-work is an environmental and scalability disaster by design, with capped throughput, ability to process only a tiny fraction of the transactions we process nowadays on traditional circuits, and the amount of energy wasted to add new blocks being the fundamental currency of the network.

Proof-of-stake is more promising, but it hasn't yet fully addressed its underlying security issues - it has proposed mitigation measures, not real structural solutions - and it's not been tested on large scales yet.

Moreover, both the solutions are doomed, by design, to break their promises about decentralizations.

As the example of Bitcoin clearly shows, when the amount of resources invested by miners increases (because the value of the currency itself increases, therefore mining blocks becomes more profitable), the entry barriers for mining also increase. Unless you have a dedicated server farm located in some of the places on the planet where energy is cheapest, you probably won't make much money out of it. So mining basically becomes a game played by a limited amount of players, with a long tail left to eat breadcrumbs.

When it comes to proof-of-stake, instead, the security of the network, as the number of nodes or the value of the currency increases, can only be guaranteed by increasing the initial deposit required by nodes who want to join as validators. As the margin of revenue increases, you can bet that more affluent players will join the club, eventually pushing up the initial stake and therefore increasing the entry barriers - it's basically a network where the actors with more power are the ones who can afford to pay more money. So much for the idea of a decentralized and more equitable network.

In other words, it can be proven that a Blockchain can't scale to the demands of throughput required by modern networks without either giving up on one of its promises (either of security of true decentralization), or causing unacceptable side effects. So why are we even talking about it as if it actually was a serious option to consider?

We are not in the initial days of the technology either - some rough corners would be a bit more tolerable if that was the case. Sakamoto's paper was published 12 years ago - an eternity when it comes to technology. In the meantime, Bitcoin's unsustainability has been proved again and again, to the point where it currently takes the whole yearly energy output of a country like Argentina to mine only 0.2% of the credit card transactions currently processed in the US alone. Its high energy cost has already had consequences on countries like China and Kazakhstan. 12 years down the line, we don't have yet a proof-of-work 2.0 that could address or at least mitigate its own unsustainability issues. And we have another proposed solution (proof-of-stake) that is either doomed to make Blockchains more centralized than the traditional circuits, or it is doomed to dilute its security constraints with "mitigation actions" for commonly known issues. Would we accept Visa or Mastercard to build their own security around "structural assumptions" and "mitigation policies"? Would we feel confident such an infrastructure to be in charge of our money or personal data? Then, again, we should stop talking of Blockchains as if they were a serious solution.

The Web 3.0 is already much more centralized than we think

This is an obvious corollary of the conclusions we have reached in the previous paragraph.

In order to build any kind of web, you need to be able to build software on it. In the crypto world, software takes the shape of dApps (distributed apps).

dApps basically leverage one of the core features of Ethereum (smart contracts) to run software that can read or manipulate the Blockchain.

Some example standard use-cases for smart contracts are:

You want to create a logic that moves 10% of a certain sum from your wallet to another wallet every month until the original sum is exhausted (payments in installments).
You want to create a logic that moves a certain sum to another wallet if the value of an underlying asset (e.g. a stock, a bond or another financial instrument) goes below/above a certain value (a derivative).

The standard use-case for a smart contract, in other words, is to model a financial instrument that manipulates the ledger and moves money across wallets if/when a certain set of conditions is met.

The whole idea behind the Web 3.0 is to leverage these technological instruments to build all kind of software. In theory, this solution solves the problem of authentication by design (the identity of users is mapped to their digital wallets, which is something recognized by everyone on the Blockchain, yet they mask the user's real identity unless they want to reveal it), while providing a stable secure and immutable storage by design under the form of a Blockchain. Not only, but financial transactions between users can happen on the fly, without having to manage complex integrations for payments, without having to give up the control of the money flow to a handful of major players. This would ideally open a whole world of decentralized micro-payments that don't rely on intermediary banks with their fees.

Not only: the Blockchain allows you to mint Non-Fungible Tokens (NFT_s). Unlike any other tokens, which are equally _fungible (i.e. you can swap any token A for a token B, and everybody agrees that they have the same value, just like a $1 bill has the same value of another $1 bill), NFTs are minted only once, and they can be univocally linked to a digital asset (such as a media file or game mod). A user who buys that token basically buys the underlying asset, and since the transaction can be easily verified by anyone on the Blockchain, everybody can agree that the user owns that asset. This would open a whole new set of revenue streams for artists and digital creators.

So far so good, right? Well, as you may have guessed, there are a few problems with this vision.

Real dApps come with real gatekeepers

We have extensively analysed why Blockchains are no longer that small niche world where everyone can run their server on a Raspberry Pi and seriously be a part of a distributed network. The entry barriers to run a node that is seriously able to push blocks to the network are very high - either because of high costs in terms of energy/hardware requirements (PoW), or high financial costs (PoS, realistic forecast).

So, if the cost of running a node on a Blockchain is high, then how do we interact with it in our software? How do we actually run these marvelous smart contracts? Well, by delegating access to the Blockchain to a few major players that provide API access to interact with it. Right now, the market is rather consolidating around Infura and Alchemy. Even when it comes to authentication-over-wallet, most of the distributed applications nowadays delegate the task to MetaMask, which already provides integrations with most of the popular wallets out there. MetaMask itself, under the hood, simply routes API calls to Infura.

But wait a minute - the whole premise of this crypto-web was that we wanted more decentralization. We were tired of delegating tasks such as user authentication, software execution and financial transactions to a handful of major Web 2.0 technological providers. So we decided to invest tons of resources and efforts to design a decentralized system with shared trust management. How come then have we decided to replace the Google/Facebook login button with a MetaMask login button, and running software on Google/Amazon/Microsoft's clouds with running it over the APIs provided by Infura/Alchemy?

If you have come so far in the article, you probably already know the answer to this question: it was inevitable. Technological systems already have a built-in bias towards centralization. If you model a system in such a way that its entry barriers also scale up with its size, then such a bias becomes inevitability. The reasons, again, are a corollary of the previous paragraphs:

On average, people are lazy. They don't want to run servers unless they really have to. So, if somebody provides them with easy access to some services, and the subscription/on-demand cost of these services is lower than the cost of running their own hardware and software, people will inevitably tend to favour that option. Especially when the system is designed in such a way that being an element of the distributed network requires you to run a small data center and add at least 100 GB of physical storage every year.
On average, developers are lazy (corollary of the previous point). Sure, they may initially want to experiment with new technologies and frameworks. But once the cost to competitively run production-ready hardware and software increases because the complexity of the system itself increases (because there are many cryptocurrencies and types of wallets to support, or because there are many protocols and APIs to support, or because the cost of running a node or investing an initial stake is too high), then they may just opt for someone who offers themselves as an intermediary. If somebody can solve the problems of user authentication, data validation and data integration for us, then we're likely to use their services, APIs or SDKs, instead of reinventing the wheel.

Not only this flavour of Web 3.0 was doomed to swing back towards centralization, but its proposed approached is even much more centralized than today's Web.

Nowadays, nobody prevents you from buying your own domain, running your own server, installing Apache/nginx (or just use an off-the-shelft solution like Wordpress), maybe an integration with PayPal/Stripe/Adyen, and having your own website ready to accept users and payments. You can even store your own database of users, so you don't need to rely on Google/Facebook login buttons and trackers. No fancy clouds involved, no centralization. Sure, the average Joe that isn't that fluent with HTML, JavaScript or DNS management may still prefer an off-the-shelf centralized solution, but nothing prevents anybody with the right tools and skills from running a website without relying on the Web 2.0 mafia.

In the case of the Web 3.0, instead, dApps can run their logic only if they run on a node that is part of the network. Running a node on the network while being profitable, as we have extensively explored, is practically an out-of-reach task even for the most experienced and affluent engineer. So basically, by design, you have no way of running software on the Web 3.0 other than delegating the task to a handful of gatekeepers. Not only: since these gatekeepers manage all the transactions between you and the Blockchain, you have no choice but trusting what they say - their API responses are basically the only source of truth. And you have no options to test your software locally either - either you run your own local Blockchain (which is highly unpractical), or you run your software in their sandboxes. Imagine a world where all the interactions with any website out there require all the HTTP requests and responses to be proxied through Google's servers, and you have no options outside of Google's cloud even to run/test your software. Those who dream of the Web 3.0 as a place to escape the current technological oligopoly should probably ponder the fact that they are just shifting power to another oligopoly with even higher entry barriers and concentration of power.

Transaction fees are actually higher than most of the credit card circuits

How about micro-payments and escaping the jail of credit card circuits with their transaction fees?

Well, one should keep in mind that Blockchains are systems made of miners that consume energy in order to add transactions. In order to incentivise miners to run nodes on the Blockchain, we need to provide them with a reward. This rewards takes the shape of a fee, which can be either fixed or proportional either to the transacted amount or the amount of consecutive blocks that are mined.

Not only: remember that the network has to ensure that it's always profitable enough for miners to run their nodes. If that's no longer the case, the Blockchain may die off - nobody likes to waste electric power with no prospect of making a profit. This means that the fees on a Blockchain are constrained to the demand/supply mechanisms that rule any other markets. A shortage of miners compared to the demand for transactions (either because mining a block isn't profitable when weighed against its cost, or because the network has to handle a spike in the number of transactions, or because a whole country decides to crack down on cryptocurrencies and take supply out of the pool) means that the average fee needs to increase as well - you need to pay miners better if you want to attract more of them. This cost is simply offloaded to the party that initiated the transaction.

Nowadays, the average transaction fee for Bitcoin is around $1.7 (remember, to process only a tiny fraction of the transactions processed by today's circuits). But this can vary widely depending on the supply of miners on the network. Shortage of miners equals higher fees: the few miners on the network will process the transactions that offer higher gains. In April 2021, for example, the average transaction fee to process Bitcoins spiked at nearly $63. How can you build any solution to handle micro-payments around a system whose transaction fees can go from a few cents to $60 depending on the supply of miners on the network? How is it even financially feasible to move $0.05 from my wallet to a band's digital wallet after streaming an mp3 on their website, if the fee to process such a transaction can go all the way from 20 to 1000 times the value of the transacted amount?

Yes, micro-payments are a real technological problem that deserves to be solved. Yes, today's solutions to handle online payments aren't perfect. But Blockchains only seem to make the problem worse. What we may need a shared protocol to handle payments - some parts of it have already been envisioned, like the HTTP 402 code, or the proposed W3C standard for handling payments. Yes, the Web 3.0 proposes similar solutions through shared protocols, but I'd rather extend that we already have (HTTP and HTML) instead of relying on something completely new, especially if that new solution comes with so many drawbacks and shortcomings.

Let's come to another promise that the Web 3.0 can't fullfil: data storage.

A Blockchain does not really store data

Anybody who tells you that your data will be safely stored on a Blockchain is a scammer.

A Blockchain was never supposed to store general-purpose data. Hell, with a throughput of 300k blocks per day, a cost per block that equals the daily energy demand of a small apartment, the requirement for all the nodes to hold an up-to-date snapshot of the whole database and no margins for data sharding, it'd be the worst and most unscalable data storage that I can think of. The only thing that could outcompete it would be storing data through butterflies.

Blockchains were designed to store financial transactions on a network of untrusted nodes. Such transactions are usually small blocks of text that say "wallet A moved X Bitcoins to wallet B at time T, and the latest transaction in the chain at time T was Z". Forget storing your GIFs on such an architecture, let alone your whole website or your self-made movie - it'd be like storing a Blueray DVD on floppy disks. Remember that it takes the average node on the network about 100 GB of storage investment to store one year of transactions, given the current volumes of Bitcoin transactions and assuming an average block size of 1 KB. Storing all kind of media data from every user on top of this would simply be unacceptable.

So, if the data isn't stored on the Blockchain directly (outside of a wallet's metadata and its transactions), where is it stored?

Well, once you scrape the crypto-facade the dear ol' web appears. Take the example of an NFT. The minted token contains a metadata section that references a URL - the linked digital asset. By buying the token, in theory you buy the ownership of the underlying asset. And, since this transaction occurs on the Blockchain, everybody can confirm that you are the actual owner of the asset.

But wait - the digital asset is actually referenced by URL, it's not actually stored on the Blockchain. What this means, in reality, is that most of the NFTs currently on sale point to web servers that run an Apache instance and expose static files.

What happens if the owner of the domain stops paying for the domain name? Or for the web hosting? Or for the SSL certificate? What if the machine goes down? What if the web server is compromised and all the static assets are either scrambled by ransomware or replaced with shit emojis, or the file is simply removed from the server? If instead it's stored in a Dropbox/Google Drive folder, what happens if that account gets deleted/blocked? If the URL is publicly accessible, what prevents me from simply copying the file to my own web server, put it on sale for $0.01, buy it and say that I'm the actual owner? If the underlying physical file has the same content, and URLs are used as identifiers of digital assets, who is supposed to establish what URL is valid if two of them point to files with the same content?

Another guy had similar concerns recently, so he created a piece of digital art and he put it on sale via NFT.

Remember: an NFT simply points to a URL. A URL simply points to a web server. It's quite easy to embed some pieces of code into your images (for example, using GD in PHP) so that the image is rendered differently depending on the context of the HTTP request. For example, the underlying image would render in different ways on different NFT marketplaces, and, if visualized within the context of a wallet (i.e. a user that purchased the underlying asset), it would simply render a "shit" emoji. His point was quite simple: if an NFT simply points to a URL that points to a web server that any dude can run, what prevents all the digital assets purchased via NFTs from turning into a 404 error page (or "shit" emojis) at the moment of purchase or any later moment?

Again, the Web 3.0 tries to address a real problem here: how to compensate digital creators for their work, while cutting away all the middlemen that may eat large shares of the profits. I've been very sensitive myself on this topic myself for a long time. But, again, it proposes the wrong solution. Middlemen aren't really cut out (again, a Blockchain isn't really decentralized), fees aren't really slashed (again, it depends on the demand/supply mechanisms that regulate the mining industry), and having digital content paid thousands of dollars living on some dude's Apache server or Wordpress blog wasn't exactly the solution to the problem of "digital ownership" that I had in mind. I mean, come on, even a naive solution like referencing the SHA-1 hash of the underlying file would have been better than this grotesque sham.

Nobody can touch your digital wallets, transactions and assets - unless they can?

Another argument often proposed by the crypto-web enthusiasts is that, once something is on the Blockchain, it's forever. There's no censorship, no risk for later data manipulation, nobody can "cancel" you.

Remember the "shit emoji" dude from the previous paragraph? Well, his digital asset was later removed from OpenSea (one of the main NFT marketplaces). It is quite likely that somebody complained about the "polymorphic" nature of the image:

Not only: it also disappeared from the wallets of those who had legitimately purchased it.

How is this possible? Well, it's quite easy to understand once we keep the previous observations in mind:

An NFT is simply a token that uniquely points to a URL;
Access to the Ethereum Blockchain is actually already quite centralized.

And, since wallets and transactions on the Blockchain are actually linked to users and accounts hosted on a normal website, nothing prevents the owner of the website from doing what the owner of any website can do - included blocking/banning users, modifying database records, and so on.

Not only: if Blockchains are really so secure, how come do we read so much about thefts of tokens or whole wallets? Well, while the underlying Blockchains are often quite secure (rogue nodes may occasionally still be able to push illegitimate transactions, but such events are very rare), remember that digital wallets are still associated to a user on a normal web site that logs in like any other user would log into a normal web site - with username and password. Remember that the strenght of a chain always equals the strength of its weakest link. Having a standard web layer wrapped around a Blockchain doesn't really improve security.

Not only: another promise of Blockchains is that of anonymity. You are identified on the Blockchain by your wallet, and your wallet is just a pair of randomly generated cryptographic keys. But if access to the Blockchain is centralized, then you'll probably end up managing multiple wallets with a small set of service providers. These providers can easily link multiple wallets that you manage on their website to your account, therefore enabling all kind of user analytics already performed by today's platforms.

A Blockchain is a distributed ledger, but there's no point in investing the effort in building a distributed ledger if it's wrapped by a centralized layer that doesn't offer significant gains in terms of data control, security and anonymity.

Conclusions

The vision of a crypto-based Web 3.0 revolves around promises of greater decentralization, scalability, privacy, security, data control, transparency, and payments that cut the middlemen. It turns out that, at the current state, it cannot deliver any of these promises:

Decentralization: if the cost to join the network is a node is artificially high, either in terms of energy/hardware requirements (PoW) or initial financial investment (PoS), then we are in a situation where the entry barriers are higher, not lower, than running your own web site on your own web server today.
Scalability: if the entry barriers are high in the first place, and they are designed to structurally increase as the network grows, then the network can hardly scale up. If we talk of a system that consumes the whole energy demand of a country to handle 300k transactions per day, and where all the nodes are supposed to keep a synchronized copy of the whole dataset, then we are talking of a system that performs much worse, not better, than the systems that we currently have.
Security: security is always as strong as the weakest link in the chain. If you have an underlying data structure that is hard to tamper with, but access to that data structure is managed through a username/password login on a normal web site or app, then you can't overall expect the security level to go up.
Privacy, transparency and data control: all the interactions with the major Blockchain-based web solutions are processed by a small number of players in the industry. Consolidation in this space is already strong. Privacy, transparency and data control are as good as when you delegate complete control of your data to a third-party on the Web. You are trusting third-parties to authenticate you, to process your online payments or to keep track of the assets that you own. What they return on their API responses is also the ultimate source of truth. The Web 3.0 basically is attempting to shift power from a tech oligarchy to another - presumably composed of those who have invested a lot into cryptocurrencies and are just searching for problems that Blockchains can solve while making them rich.
Transaction fees: fees are actually linked by design to the demand/supply balance of mining resources. This makes them much more volatile than the fees you would pay for a normal credit card transaction. Not only: they are also higher on average, since the cost of adding a new block to a distributed ledger through shared consensus is higher than the cost of handling the same transaction on a centralized ledger. This makes the solutions proposed by the Web 3.0 unfeasible to handle problems such as micro-payments or royalties.

I still believe that the Blockchain can be a promising technology, but, after this long hype phase, I believe that expectations about what it can actually deliver will be downsized. I don't believe that it can be used as a system for shared consensus in the first place. The original approach, the proof-of-work, is a scalability disaster by design - it was a good proof-of-concept that recycled an idea from some late-90s spam mitigation protocol, nothing more. The newly proposed approach, the proof-of-stake, is more promising, but it hasn't yet been tested enough on large scale and it hasn't yet addressed some security and consistency issues. And, even if it's successful, it's probably doomed to converge towards centralization, since the initial stakes for being a part of the network are likely to increase either with the size of the network or with the value of the underlying cryptocurrency. In short, the Blockchain proposed itself a solution to the problem of distributed consensus, but, 12 years down the line, it hasn't yet figured out how to achieve consensus in a way that is scalable, secure and distributed at the same time.

I still believe that Blockchains will have a place in cutting the middlemen when it comes to B2B financial transactions, or creating financial instruments that currently require fat contracts signed by fat notaries with fat fees. There is quite an ironic twist here: Bitcoin was initially proposed in the aftermath of the 2008-2009 financial crisis. It was supposed to punish the financial system that had caused the crisis (and, in particular, the underlying level of unregulated financial speculation) by creating a new distributed financial system. Instead, smart contracts may end up powering a revolution in the very financial instruments that cryptocurrencies were supposed to destroy.

Blockchains may also revolutionarise logistics: shipping an item from one side of the world to the other currently requires integrating data from many different sources, spanning over different countries and subject to different regulations. If everybody agrees to write both their transactions and their business logic to the same distributed ledger, it would solve a lot of issues in a structural way.

But these use-cases probably depict very different scenarios from the ones originally envisioned. This will be a distributed ledger managed through some proof-of-stake-like mechanism between businesses or financial institutions that already have some ways of identifying their identities - once you relax the constraints of trust enforcement and decentralization, Blockchains can actually be interesting technological tools.

They might also be used to run whole national currencies. China is already working of it. This is another use-case that makes totally sense: if you own a digital wallet directly with your own central bank, then the central bank has much more direct visibility and control over its monetary policies, and it doesn't have to rely on the middlemen (e.g. retail banks) to manage the money of citizens. However, this is, again, a far cry from the Blockchain's original idea. It was initially conceived as a system to let citizens take back control of their money. It may probably end up cutting retail banks out, but only to further centralize control in the hands of the government or the central bank.

However, I also believe that the crypto-based Web 3.0 addresses real problems that are still searching for solutions. Centralization of identity management and data management, and in general an increasing consolidation of power in the hands of a half a dozen of companies, are still real problems. Affordable, easy-to-setup cross-border micro-payments are still looking for a solution. The problem of handling and enforcing ownership of digital assets, and rewarding digital creators for their work, is still looking for a solution. I just believe that the Blockchain is the wrong solution for this problem, because - as many already said - it's a solution still looking for a problem to solve.

In order to address the issues that caused the consolidation of technological power that plagues the Web 2.0, we must first analyse what kind of phenomena lead to technological consolidation in the first place. Once we acknowledge that the level of consolidation is proportional to the entry barriers required to compete in the market, then we may want to design systems that lower those barriers, not systems that increase them.

I also believe that there is a lot of untapped potential in the previous incarnations of the Web 3.0 - both the semantic web and the web-of-things. Sure, they had to face a reality where it's easier to build a platform with your own set of rules rather than getting everybody to agree on the same shared set of protocols and standards, and with an industry that has a strong bias towards centralization and de-facto standards anyways. But at least they proposed incremental solutions built on top of existing standards that address real issues in building a distributed Web - data and protocols fragmentation in the first place.

Build your self-hosted Evernote

Fabio Manganiello — Thu, 06 Jan 2022 19:58:18 +0000

The need for an online second brain

When Evernote launched the idea of an online notebook as a sort of "second brain" more than a decade ago, it resonated so much with what I had been trying to achieve for a while. By then I already had tons of bookmarks, text files with read-it-later links, notes I had taken across multiple devices, sketches I had taken on physical paper and drafts of articles or papers I was working on. All of this content used to be sparse across many devices, it was painful to sync, and then Evernote came like water in a desert.

I have been a happy Evernote user until ~5-6 years ago, when I realized that the company had run out of ideas, and I could no longer compromise with its decisions. If Evernote was supposed to be my second brain then it should have been very simple to synchronize it with my filesystem and across multiple devices, but that wasn't as simple as it sounds. Evernote had a primitive API, a primitive web clipper, no Linux client, and, as it tried harder and harder to monetize its product, it put more and more features behind expensive tiers. Moreover, Evernote experienced data losses, security breaches and privacy controversies that in my eyes made it unfit to handle something as precious as the notes from my life and my work. I could not compromise with a product that would charge me $5 more a month just to have it running on an additional device, especially when the product itself didn't look that solid to me. If Evernote was supposed to be my second brain then I should have been able to take it with me wherever I wanted, without having to worry on how many devices I was using it already, without having to fear future changes or more aggressive monetization policies that could have limited my ability to use the product.

So I started my journey as a wanderer of note-taking and link-saving services. Yes, ideally I want something that can do both: your digital brain consists both of the notes you've taken and the links you've saved.

I've tried many of them over the following years (Instapaper, Pocket, Readability, Mercury Reader, SpringPad, Google Keep, OneNote, Dropbox Paper...), but eventually got dissatisfied by most of them:

In most of the cases those products fall into the note-taking category or web scraper/saver category, rarely both.
In most of the cases you have to pay a monthly/yearly fee for something as simple as storing and syncing text.
Many of the products above either lack an API to programmatically import/export/read data, or they put their APIs behind some premium tiers. This is a no-go for me: if the company that builds the product goes down, the last thing I want is my personal notes, links and bookmarks to go down with it with no easy way to get them out.
Most of those products don't have local filesystem sync features: everything only works in their app.

My dissatisfaction with the products on the market was a bit relieved when I discovered Obsidian. A Markdown-based, modern-looking, multi-device product that transparently stores your notes on your own local storage, and it even provides plenty of community plugins? That covers all I want, it's almost too good to be true! And, indeed, it is too good to be true. Obsidian charges $8 a month just for syncing content across devices (copying content to their own cloud), and $16 a month if you want to publish/share your content. Those are unacceptably high prices for something as simple as synchronizing and sharing text files! This was the trigger that motivated me to take the matter into my own hands, so I came up with the wishlist for my ideal "second brain" app:

It needs to be self-hosted. No cloud services involved: it's easy to put stuff on somebody else's cloud, it's usually much harder to take it out, and cloud services are unreliable by definition - they may decide from a moment to another that they aren't making enough money, charge more for some features you are using, while keeping your own most precious data as hostage. Or, worse, they could go down and take all of your data with them.
Each device should have a local copy of my notebook, and it should be simple to synchronize changes across these copies.
It ought to be Markdown-based. Markdown is portable, clean, easy to index and search, it can easily be converted to HTML if required, but it's much less cumbersome to read and write, and it's easy to import/export. To give an idea of the underestimated power and flexibility of Markdown, keep in mind that all the articles on the Platypush blog are static Markdown files on a local server that are converted on the fly to HTML before being served to your browser.
It needs to be able to handle my own notes, as well as parse and convert to Markdown web pages that I'd like to save or read later.
It must be easy to add and modify content. Whether I want to add a new link from my browser session on my laptop, phone or tablet, or type some text on the fly from my phone, or resume working on a draft from another device, I should be able to do so with no friction, as if I were working always on the same device.
It needs to work offline. I want to be able to work on a blog article while I'm on a flight with no Internet connection, and I expect the content to be automatically synced as soon as my device gets a connection.
It needs to be file-based. I'm sick of custom formats, arcane APIs and other barriers and pointless abstractions between me and my text. The KISS rule applies here: if it's a text file, and it appears on my machine inside a normal directory, then expose it as a text file, and you'll get primitives such as read/create/modify/copy/move/delete for free.
It needs to encapsulate some good web scraping/parsing logic, so every web page can be distilled into a readable and easily exportable Markdown format.
It needs to allow automated routines - for instance, automatically fetch new content from an RSS feed and download it in readable format on the shared repository.

It looks like a long shopping list, but it actually doesn't take that much to implement it. It's time to get to the whiteboard and design its architecture.

High-level architecture

From a high-level perspective, the architecture we are trying to build resembles something like this:

The git repository

We basically use a git server as the repository for our notes and links. It could be a private repo on GitHub or Gitlab, or even a static folder initialized as a git repo on a server accessible over SSH. There are many advantages in choosing a versioning system like git as the source of truth for your notebook content:

History tracking comes for free: it's easy to keep track of changes commit by different devices, as well as rollback to previous versions - nothing is ever really lost.
Easy synchronization: pushing new content to your notes can be mapped to a git push, synchronizing new content on other devices can be mapped to a git pull.
Native Markdown-friendly interfaces: both GitHub and Gitlab provide native good interfaces to visualize Markdown content. Browsing and managing your notebook is as easy as browsing a git repo.
Easy to import and export: exporting your notebook to another device is as simple as running a git clone.
Storage flexibility: you can create the repo on a cloud instance, on a self-hosted instance, or on any machine with an SSH interface. The repo can live anywhere, as long as it is accessible to the devices that you want to use.

So the first requirement for this project is to set up a git repository on whatever source you want to use a central storage for your notebook. We have mainly three options for this:

Create a new repo on a GitHub/Gitlab cloud instance.

Pros: you don't have to maintain a git server, you just have to create a new project, and you have all the fancy interfaces for managing files and viewing Markdown content.
Cons: it's not really 100% self-hosted, isn't it? :)

Host a Gitlab instance yourself.

Pros: plenty of flexibility when it comes to hosting. You can even run the server on a machine only accessible from the outside over a VPN, which brings some nice security features and content encapsulation. Plus, you have a modern interface like Gitlab to handle your files, and you can also easily set up repository automation through web hooks.
Cons: installing and running a Gitlab instance is a process with its own learning curve. Plus, a Gitlab instance is usually quite resource-hungry - don't run it on a Raspberry Pi if you want the user experience to be smooth.

Initialize an empty repository on any publicly accessible server (or accessible over VPN) with an SSH interface.

An often forgotten feature of git is that it's basically a wrapper on top of SSH, therefore you can create a repo on the fly on any machine that runs an SSH server - no need for a full-blown web framework on top of it. It's as simple as:

# Server machine
$ mkdir -p /home/user/notebook.git
$ cd /home/user/notebook.git
$ git init --bare

# Client machine
$ git clone user@remote-machine:/home/user/notebook.git

Pros: the most flexible option: you can run your notebook storage on literally anything that has a CPU, an SSH interface and git.
Cons: you won't have a fancy native interface to manage your files, nor repository automation features such as actions or web hooks (available with GitHub and Gitlab respectively).

The Markdown web server

It may be handy to have a web server to access your notes and links from any browser, especially if your repository doesn't live on GitHub/Gitlab, and therefore it doesn't have a native way to expose the files over the web.

Clone the notebook repo on the machine where you want to expose the Markdown web server and then install Madness and its dependencies:

$ sudo apt install ruby-full
$ gem install madness

Take note of where the madness executable was installed and create a new user systemd service file under ~/.config/systemd/user/madness.service to manage the server on your repo folder:

[Unit]
Description=Serve Markdown content over HTML
After=network.target

[Service]
ExecStart=/home/user/.gem/ruby/version/bin/madness /path/to/the/notebook --port 9999
Restart=always
RestartSec=10

[Install]
WantedBy=default.target

Reload the systemd daemon and start/enable the server:

$ systemctl --user daemon-reload
$ systemctl --user start madness
$ systemctl --user enable madness

If everything went well you can head your browser to http://host:9999 and you should see the Madness interface with your Markdown files.

You can easily configure a nginx reverse proxy or an SSH tunnel to expose the server outside of the local network.

The MQTT broker

An MQTT broker is another crucial ingredient in this set up. It is used to asynchronously transmit events such as a request to add a new URL or update the local repository copies.

Any of the open-source MQTT brokers out there should do the job. I personally use Mosquitto for most of my projects, but RabbitMQ, Aedes or any other broker should all just work.

Just like the git server, you should also install the MQTT on a machine that is either publicly accessible, or it is accessible over VPN by all the devices you want to use your notebook on. If you opt for a machine with a publicly accessible IP address then it's advised to enable both SSL and username/password authentication on your broker, so unauthorized parties won't be able to connect to it.

Taking the case of Mosquitto, the installation and configuration is pretty straightforward. Install the mosquitto package from your favourite package manager, the installation process should also create a configuration file under /etc/mosquitto/mosquitto.conf. In the case of an SSL configuration with username and password, you would usually configure the following options:

# Usually 1883 for non-SSL connections, 8883 for SSL connections
port 8883

# SSL/TLS version
tls_version tlsv1.2

# Path to the certificate chain
cafile /etc/mosquitto/certs/chain.crt

# Path to the server certificate
certfile /etc/mosquitto/certs/server.crt

# Path to the server private key
keyfile /etc/mosquitto/certs/server.key

# Set to false to disable access without username and password
allow_anonymous false

# Password file, which contains username:password pairs
# You can create and manage a password file by following the
# instructions reported here:
# https://mosquitto.org/documentation/authentication-methods/
password_file /etc/mosquitto/passwords.txt

If you don't need SSL encryption and authentication on your broker (which is ok if you are running the broker on a private network and accessing it from the outside over VPN) then you'll only need to set the port option.

After you have configured the MQTT broker, you can start it and enable it via systemd:

$ sudo systemctl start mosquitto
$ sudo systemctl enable mosquitto

You can then use an MQTT client like MQTT Explorer to connect to the broker and verify that everything is working.

The Platypush automation

Once the git repo and the MQTT broker are in place, it's time to set up Platypush on one of the machines where you want to keep your notebook synchronized - e.g. your laptop.

In this context, Platypush is used to glue together the pieces of the sync automation by defining the following chains of events:

When a file system change is detected in the folder where the notebook is cloned (for example because a note was added, removed or edited), start a timer than within e.g. 30 seconds synchronizes the changes to the git repository (the timer is used to throttle the frequency of update events). Then send a message to the MQTT notebook/sync topic to tell the other clients that they should synchronize their copies of the repository.
When a client receives a message on notebook/sync, and the originator is different from the client itself (this is necessary in order to prevent "sync loops"), pull the latest changes from the remote repository.
When a specific client (which will be in charge of scraping URLs and adding new remote content) receives a message on the MQTT notebook/save topic with a URL attached, the content of the associated web page will be parsed and saved to the notebook ("Save URL" feature).

The same automation logic can be set up on as many clients as you like.

The first step is to install the Redis server and Platypush on your client machine. For example, on a Debian-based system:

# Install Redis
$ sudo apt install redis-server
# Start and enable the Redis server
$ sudo systemctl start redis-server
$ sudo systemctl enable redis-server
# Install Platypush
$ sudo pip install platypush

You'll then have to create a configuration file to tell Platypush which services you want to use. Our use-case will require the following integrations:

mqtt (backend and plugin), used to subscribe to sync/save topics and dispatch messages to the broker.
file.monitor backend, used to monitor changes to local folders.
[Optional] pushbullet, or an alternative way to deliver notifications to other devices (such as telegram, twilio, gotify, mailgun). We'll use this to notify other clients when new content has been added.
[Optional] the http.webpage integration, used to scrape a web page's content to Markdown or PDF.

Start by creating a config.yaml file with your integrations:

# The name of your client
device_id: my-client

mqtt:
  host: your-mqtt-server
  port: 1883
  # Uncomment the lines below for SSL/user+password authentication
  # port: 8883
  # username: user
  # password: pass
  # tls_cafile: ~/path/to/ssl.crt
  # tls_version: tlsv1.2

# Specify the topics you want to subscribe here
backend.mqtt:
  listeners:
    - topics:
      - notebook/sync

# The configuration for the file monitor follows.
# This logic triggers FileSystemEvents whenever a change
# happens on the specified folder. We can use these events
# to build our sync logic
backend.file.monitor:
  paths:
    # Path to the folder where you have cloned the notebook
    # git repo on your client
    - path: /path/to/the/notebook
      recursive: true
      # Ignore changes on non-content sub-folders, such as .git or
      # other configuration/cache folders
      ignore_directories:
        - .git
        - .obsidian

Then generate a new Platypush virtual environment from the configuration file:

$ platyvenv build -c config.yaml

Once the command has run, it should report a line like the following:

Platypush virtual environment prepared under /home/user/.local/share/platypush/venv/my-client

Let's call this path $PREFIX. Create a structure to store your scripts under $PREFIX/etc/platypush (a copy of the
config.yaml file should already be there at this point). The structure will look like this:

$PREFIX
 -> etc
   -> platypush
    -> config.yaml      # Configuration file
    -> scripts          # Scripts folder
      -> __init__.py    # Empty file
      -> notebook.py    # Logic for notebook synchronization

Let's proceed with defining the core logic in notebook.py:

import logging
import os
import re
from threading import RLock, Timer

from platypush.config import Config
from platypush.event.hook import hook
from platypush.message.event.file import FileSystemEvent
from platypush.message.event.mqtt import MQTTMessageEvent
from platypush.procedure import procedure
from platypush.utils import run

logger = logging.getLogger('notebook')
repo_path = '/path/to/your/git/repo'

sync_timer = None
sync_timer_lock = RLock()


def should_sync_notebook(event: MQTTMessageEvent) -> bool:
    """
    Only synchronize the notebook if a sync request came from
    a source other than ourselves - this is required to prevent
    "sync loops", where a client receives its own sync message
    and broadcasts sync requests again and again.
    """
    return Config.get('device_id') != event.msg.get('origin')


def cancel_sync_timer():
    """
    Utility function to cancel a pending synchronization timer.
    """
    global sync_timer
    with sync_timer_lock:
        if sync_timer:
            sync_timer.cancel()
            sync_timer = None


def reset_sync_timer(path: str, seconds=15):
    """
    Utility function to start a synchronization timer.
    """
    global sync_timer
    with sync_timer_lock:
        cancel_sync_timer()
        sync_timer = Timer(seconds, sync_notebook, (path,))
        sync_timer.start()


@hook(MQTTMessageEvent, topic='notebook/sync')
def on_notebook_remote_update(event, **_):
    """
    This hook is triggered when a message is received on the
    notebook/sync MQTT topic. It triggers a sync between the
    local and remote copies of the repository.
    """
    if not should_sync_notebook(event):
        return

    sync_notebook(repo_path)


@hook(FileSystemEvent)
def on_notebook_local_update(event, **_):
    """
    This hook is triggered when a change (i.e. file/directory
    create/update/delete) is performed on the folder where the
    repository is cloned. It starts a timer to synchronize the
    local and remote repository copies.
    """
    if not event.path.startswith(repo_path):
        return

    logger.info(f'Synchronizing repo path {repo_path}')
    reset_sync_timer(repo_path)


@procedure
def sync_notebook(path: str, **_):
    """
    This function holds the main synchronization logic.
    It is declared through the @procedure decorator, so you can also
    programmatically call it from your requests through e.g.
    `procedure.notebook.sync_notebook`.
    """
    # The timer lock ensures that only one thread at the time can
    # synchronize the notebook
    with sync_timer_lock:
        # Cancel any previously awaiting timer
        cancel_sync_timer()
        logger.info(f'Synchronizing notebook - path: {path}')
        cwd = os.getcwd()
        os.chdir(path)
        has_stashed_changes = False

        try:
            # Check if the local copy of the repo has changes
            git_status = run('shell.exec', 'git status --porcelain').strip()

            if git_status:
                logger.info('The local copy has changes: synchronizing them to the repo')

                # If we have modified/deleted files then we stash the local changes
                # before pulling the remote changes to prevent conflicts
                has_modifications = any(re.match(r'^\s*[MD]\s+', line) for line in git_status.split('\n'))
                if has_modifications:
                    logger.info(run('shell.exec', 'git stash', ignore_errors=True))
                    has_stashed_changes = True

                # Pull the latest changes from the repo
                logger.info(run('shell.exec', 'git pull --rebase'))
                if has_modifications:
                    # Un-stash the local changes
                    logger.info(run('shell.exec', 'git stash pop'))

                # Add, commit and push the local changes
                has_stashed_changes = False
                device_id = Config.get('device_id')
                logger.info(run('shell.exec', 'git add .'))
                logger.info(run('shell.exec', f'git commit -a -m "Automatic sync triggered by {device_id}"'))
                logger.info(run('shell.exec', 'git push origin main'))

                # Notify other clients by pushing a message to the notebook/sync topic
                # having this client ID as the origin. As an alternative, if you are using
                # Gitlab to host your repo, you can also configure a webhook that is called
                # upon push events and sends the same message to notebook/sync.
                run('mqtt.publish', topic='notebook/sync', msg={'origin': Config.get('device_id')})
            else:
                # If we have no local changes, just pull the remote changes
                logger.info(run('shell.exec', 'git pull'))
        except Exception as e:
            if has_stashed_changes:
                logger.info(run('shell.exec', 'git stash pop'))

            # In case of errors, retry in 5 minutes
            reset_sync_timer(path, seconds=300)
            raise e
        finally:
            os.chdir(cwd)

        logger.info('Notebook synchronized')

Now you can start the newly configured environment:

$ platyvenv start my-client

Or create a systemd user service for it under ~/.config/systemd/user/platypush-notebook.service:

$ cat <<EOF > ~/.config/systemd/user/platypush-notebook.service
[Unit]
Description=Platypush notebook automation
After=network.target

[Service]
ExecStart=/path/to/platyvenv start my-client
ExecStop=/path/to/platyvenv stop my-client
Restart=always
RestartSec=10

[Install]
WantedBy=default.target
EOF

$ systemctl --user daemon-reload
$ systemctl --user start platypush-notebook
$ systemctl --user enable platypush-notebook

While the service is running, try and create a new Markdown file under the monitored repository local copy. Within a few seconds the automation should be triggered and the new file should be automatically pushed to the repo. If you are running the code on multiple hosts, then those should also fetch the updates within seconds. You can also run an instance on the same server that runs Madness to synchronize its copy of the repo, and your web instance will remain in sync with any updates. Congratulations, you have set up a distributed network to synchronize your notes!

Android setup

You may probably want a way to access your notebook also on your phone and tablet, and keep the copy on your mobile devices automatically in sync with the server.

Luckily, it is possible to install and run Platypush on Android through Termux, and the logic you have set up on your laptops and servers should also work flawlessly on Android. Termux allows you to run a Linux environment in user mode with no need for rooting your device.

First, install the Termux app on your Android device. Optionally, you may also want to install the following apps:

Termux:API: to programmatically access Android features (e.g. SMS texts, camera, GPS, battery level etc.) from your scripts.
Termux:Boot: to start services such as Redis and Platypush at boot time without having to open the Termux app first (advised).
Termux:Widget: to add scripts (for example to manually start Platypush or synchronize the notebook) on the home screen.
Termux:GUI: to add support for visual elements (such as dialogs and widgets for sharing content) to your scripts.

After installing Termux, open a new session, update the packages, install termux-services (for services support) and enable SSH access (it's usually more handy to type commands on a physical keyboard than a phone screen):

$ pkg update
$ pkg install termux-services openssh
# Start and enable the SSH service
$ sv up sshd
$ sv-enable sshd
# Set a user password
$ passwd

A service that is enabled through sv-enable will be started when a Termux session is first opened, but not at boot time unless Termux is started. If you want a service to be started a boot time, you need to install the Termux:Boot app and then place the scripts you want to run at boot time inside the ~/.termux/boot folder.

After starting sshd and setting a password, you should be able to log in to your Android device over SSH:

$ ssh -p 8022 anyuser@android-device

The next step is to enable access for Termux to the internal storage (by default it can only access the app's own data folder). This can easily be done by running termux-setup-storage and allowing storage access on the prompt. We may also want to disable battery optimization for Termux, so the services won't be killed in case of inactivity.

Then install git, Redis, Platypush and its Python dependencies, and start/enable the Redis server:

$ pkg install git redis python3
$ pip install platypush

If running the redis-server command results in an error, then you may need to explicitly disable a warning for a COW bug for ARM64 architectures in the Redis configuration file. Simply add or uncomment the following line in /data/data/com.termux/files/usr/etc/redis.conf:

ignore-warnings ARM64-COW-BUG

We then need to create a service for Redis, since it's not available by default. Termux doesn't use systemd to manage services, since that would require access to the PID 1, which is only available to the root user. Instead, it uses it own system of scripts that goes under the name of Termux services.

Services are installed under /data/data/com.termux/files/usr/var/service. Just cd to that directory and copy the available sshd service to redis:

$ cd /data/data/com.termux/files/usr/var/service
$ cp -r sshd redis

Then replace the content of the run file in the service directory with this:

#!/data/data/com.termux/files/usr/bin/sh
exec redis-server 2>&1

Then restart Termux so that it refreshes its list of services, and start/enable the Redis service (or create a boot script for it):

$ sv up redis
$ sv-enable redis

Verify that you can access the /sdcard folder (shared storage) after restarting Termux. If that's the case, we can now clone the notebook repo under /sdcard/notebook:

$ git clone git-url /sdcard/notebook

The steps for installing and configuring the Platypush automation are the same shown in the previous section, with the following exceptions:

repo_path in the notebook.py script needs to point to /sdcard/notebook - if the notebook is cloned on the user's home directory then other apps won't be able to access it.
If you want to run it in a service, you'll have to follow the same steps illustrated for Redis instead of creating a systemd service.

You may also want to redirect the Platypush stdout/stderr to a log file, since Termux messages don't have the same sophisticated level of logging provided by systemd. The startup command should therefore look like:

platyvenv start my-client > /path/to/logs/platypush.log 2>&1

Once everything is configured and you restart Termux, Platypush should automatically start in the background - you can check the status by running a tail on the log file or through the ps command. If you change a file in your notebook on either your Android device or your laptop, everything should now get up to date within a minute.

Finally, we can also leverage Termux:Shortcuts to add a widget to the home screen to manually trigger the sync process - maybe because an update was received while the phone was off or the Platypush service was not running. Create a ~/.shortcuts folder with a script inside named e.g. sync_notebook.sh:

#!/data/data/com.termux/files/usr/bin/bash

cat <<EOF | python
from platypush.utils import run

run('mqtt.publish', topic='notebook/sync', msg={'origin': None})
EOF

This script leverages the platypush.utils.run method to send a message to the notebook/sync MQTT topic with no origin to force all the subscribed clients to pull the latest updates from the remote server.

You can now browse to the widgets' menu of your Android device (usually it's done by long-pressing an empty area on the launcher), select Termux shortcut and then select your newly created script. By clicking on the icon you will force a sync across all the connected devices.

Once Termux is properly configured, you don't need to repeat the whole procedure on other Android devices. Simply use the Termux backup scripts to back up your whole configuration and copy it/restore it on another device, and you'll have the whole synchronization logic up and running.

The Obsidian app

Now that the backend synchronization logic is in place, it's time to move to the frontend side. As mentioned earlier, Obsidian is an option I really like - it has a modern interface, it's cross-platform, it's electronjs-based, it has many plugins, it relies on simple Markdown, and it just needs a local folder to work. As mentioned earlier, you would normally need to subscribe to Obsidian Sync in order to synchronize notes across devices, but now you've got a self-synchronizing git repo copy on any device you like. So just install Obsidian on your computer or mobile, point it to the local copy of the git notebook, and you're set to go!

The NextCloud option

Another nice option is to synchronize your notebook across multiple devices is to use a NextCloud instance. NextCloud provides a Notes app that already supports notes in Markdown format, and it also comes with an Android app.

If that's the way you want to go, you can still have notes<->git synchronization by simply setting up the Platypush notebook automation on the server where NextCloud is running. Just clone the repository to your NextCloud Notes folder:

$ git clone git-url /path/to/nextcloud/data/user/files/Notes

And then set the repo_path in notebook.py to this directory.

Keep in mind however that local changes in the Notes folder will not be synchronized to the NextCloud app until the next cron is executed. If you want the changes to be propagated as soon as they are pushed to the git repo, then you'll have to add an extra piece of logic to the script that synchronizes the notebook, in order to rescan the Notes folder for changes. Also, Platypush will have to run with the same user that runs the NextCloud web server, because of the requirements for executing the occ script:

import logging
from platypush.utils import run

...

logger = logging.getLogger('notebook')
# Path to the NextCloud occ script
occ_path = '/srv/http/nextcloud/occ'

...

def sync_notebook(path: str, **_):
    ...
    refresh_nextcloud()

def refresh_nextcloud():
    logger.info(run('shell.exec', f'php {occ_path} files:scan --path=/nextcloud-user/files/Notes'))
    logger.info(run('shell.exec', f'php {occ_path} files:cleanup'))

Your notebook is now synchronized with NextCloud, and it can be accessed from any NextCloud client!

Automation to parse and save web pages

Now that we have a way to keep our notes synchronized across multiple devices and interfaces, let's explore how we can parse web pages and save them in our notebook in Markdown format - we may want to read them later on another device, read the content without all the clutter, or just keep a persistent track of the articles that we have read.

Elect a notebook client to be in charge of scraping and saving URLs. This client will have a configuration like this:

# The name of your client
device_id: my-client

mqtt:
  host: your-mqtt-server
  port: 1883
  # Uncomment the lines below for SSL/user+password authentication
  # port: 8883
  # username: user
  # password: pass
  # tls_cafile: ~/path/to/ssl.crt
  # tls_version: tlsv1.2

# Specify the topics you want to subscribe here
backend.mqtt:
  listeners:
    - topics:
       - notebook/sync
       # notebook/save will be used to send parsing requests
       - notebook/save

# Monitor the local repository copy for changes
backend.file.monitor:
  paths:
    # Path to the folder where you have cloned the notebook
    # git repo on your client
    - path: /path/to/the/notebook
      recursive: true
      # Ignore changes on non-content sub-folders, such as .git or
      # other configuration/cache folders
      ignore_directories:
        - .git
        - .obsidian

# Enable the http.webpage integration for parsing web pages
http.webpage:
  enabled: true

# We will use Pushbullet to send a link to all the connected devices
# with the URL of the newly saved link, but you can use any other
# services for delivering notifications and/or messages - such as
# Gotify, Twilio, Telegram or any email integration
backend.pushbullet:
  token: my-token
  device: my-client

pushbullet:
  enabled: true

Build an environment from this configuration file:

$ platyvenv build -c config.yaml

Make sure that at the end of the process you have the node and npm executables installed - the http.webpage integration uses the Mercury Parser API to convert web pages to Markdown.

Then copy the previously created scripts folder under <environment-base-dir>/etc/platypush/scripts. We now want to add a new script (let's name it e.g. webpage.py) that is in charge of subscribing to new messages on notebook/save and use the http.webpage integration to save its content in Markdown format in the repository folder. Once the parsed file is in the right directory, the previously created automation will take care of synchronizing it to the git repo.

import logging
import os
import re
import shutil
import tempfile
from datetime import datetime
from typing import Optional
from urllib.parse import quote

from platypush.event.hook import hook
from platypush.message.event.mqtt import MQTTMessageEvent
from platypush.procedure import procedure
from platypush.utils import run

logger = logging.getLogger('notebook')
repo_path = '/path/to/your/notebook/repo'
# Base URL for your Madness Markdown instance
markdown_base_url = 'https://my-host/'


@hook(MQTTMessageEvent, topic='notebook/save')
def on_notebook_url_save_request(event, **_):
    """
    Subscribe to new messages on the notebook/save topic.
    Such messages can contain either a URL to parse, or a
    note to create - with specified content and title.
    """
    url = event.msg.get('url')
    content = event.msg.get('content')
    title = event.msg.get('title')
    save_link(url=url, content=content, title=title)


@procedure
def save_link(url: Optional[str] = None, title: Optional[str] = None, content: Optional[str] = None, **_):
    assert url or content, 'Please specify either a URL or some Markdown content'

    # Create a temporary file for the Markdown content
    f = tempfile.NamedTemporaryFile(suffix='.md', delete=False)

    if url:
        logger.info(f'Parsing URL {url}')

        # Parse the webpage to Markdown to the temporary file
        response = run('http.webpage.simplify', url=url, outfile=f.name)
        title = title or response.get('title')

    # Sanitize title and filename
    if not title:
        title = f'Note created at {datetime.now()}'

    title = title.replace('/', '-')
    if content:
        with open(f.name, 'w') as f:
            f.write(content)

    # Download the Markdown file to the repo
    filename = re.sub(r'[^a-zA-Z0-9 \-_+,.]', '_', title) + '.md'
    outfile = os.path.join(repo_path, filename)
    shutil.move(f.name, outfile)
    os.chmod(outfile, 0o660)
    logger.info(f'URL {url} successfully downloaded to {outfile}')

    # Send the URL
    link_url = f'{markdown_base_url}/{quote(title)}'
    run('pushbullet.send_note', title=title, url=link_url)

We now have a service that can listen for messages delivered on notebook/save. If the message contains some Markdown content, it will directly save it to the notebook. If it contains a URL, it will use the http.webpage integration to parse the web page and save it to the notebook. What we need now is a way to easily send messages to this channel while we are browsing the web. A common use-case is the one where you are reading an article on your browser (either on a computer or a mobile device) and you want to save it to your notebook to read it later through a mechanism similar to the familiar Share button. Let's break down this use-case in two:

The desktop (or laptop) case
The mobile case

Sharing links from the desktop

If you are reading an article on your personal computer and you want to save it to your notebook (for example to read it later on your mobile) then you can use the Platypush browser extension to create a simple action that sends your current tab to the notebook/save MQTT channel.

Download the extension on your browser (Firefox version, Chrome version) - more information about the Platypush browser extension is available in a previous article. Then, click on the extension icon in the browser and add a new connection to a Platypush host - it could either be your own machine or any of the notebook clients you have configured.

Side note: the extension only works if the target Platypush machine has backend.http (i.e. the web server) enabled, as it is used to dispatch messages over the Platypush API. This wasn't required by the previous set up, but you can now select one of the devices to expose a web server by simply adding a backend.http section to the configuration file and setting enabled: True (by default the web server will listen on the port 8008).

Then from the extension configuration panel select your host -> Run Action. Wait for the autocomplete bar to populate (it may take a while the first time, since it has to inspect all the methods in all the enabled packages) and then create a new mqtt.publish action that sends a message with the current URL over the notebook/save channel:

Click on the Save Action button at the bottom of the page, give your action a name and, optionally, an icon, a color and a set of tags. You can also select a keybinding between Ctrl+Alt+0 and Ctrl+Alt+9 to automatically run your action without having to grab the mouse.

Now browse to any web page that you want to save, run the action (either by clicking on the extension icon and selecting it or through the keyboard shortcut) and wait a couple of seconds. You should soon receive a Pushbullet notification with a link to the parsed content and the repo should get updated as well on all of your devices.

Sharing links from mobile devices

An easy way to share links to your notebook through an Android device is to leverage Tasker with the AutoShare plugin, and choose an app like MQTT Client that comes with a Tasker integration. You may then create a new AutoShare intent named e.g. Save URL, create a Tasker task associated to it that uses the MQTT Client integration to send the message with the URL to the right MQTT topic. When you are browsing a web page that you'd like to save then you simply click on the Share button and select AutoShare Command in the popup window, then select the action you have created.

However, even though I really appreciate the features provided by Tasker, its ecosystem and the developer behind it (I have been using it for more than 10 years), I am on a path of moving more and more of my automation away from it. Firstly, because it's a paid app with paid services, and the whole point of setting up this whole automation is to have the same quality of a paid service without having to pay for - we host it, we own it. Secondly, it's not an open-source app, and it's notably tricky to migrate configurations across devices.

Termux also provides a mechanism for intents and hooks, and we can easily create a sharing intent for the notebook by creating a script under ~/bin/termux-url-opener. Make sure that the binary file is executable and that you have Termux:GUI installed for support for visual widgets:

#!/data/data/com.termux/files/usr/bin/bash

arg="$1"

# termux-dialog-radio show a list of mutually exclusive options and returns
# the selection in JSON format. The options need to be provided over the -v
# argument and they are comma-separated
action=$(termux-dialog radio -t 'Select an option' -v 'Save URL,some,other,options' | jq -r '.text')

case "$action" in
    'Save URL')
      cat <<EOF | python
from platypush.utils import run

run('mqtt.publish', topic='notebook/save', msg={'url': '$arg'})
EOF
      ;;

    # You can add some other actions here
esac

Now browse to a page that you want to save from your mobile device, tap the Share button, select Termux and select the Save URL option. Everything should work out of the box.

Delivering RSS digests to your notebook

As a last step in our automation set up, let's consider the use-case where you want a digest of the new content from your favourite source (your favourite newspaper, magazine, blog etc.) to be automatically delivered on a periodic basis to your notebook in readable format.

It's relatively easy to set up such automation with the building blocks we have put in place and the Platypush rss integration. Add an rss section to the configuration file of any of your clients with the http.webpage integration. It will contain the RSS sources you want to subscribe to:

rss:
  subscriptions:
    - https://source1.com/feed/rss
    - https://source2.com/feed/rss
    - https://source3.com/feed/rss

Then either rebuild the virtual environment (platyvenv build -c config.yaml) or manually install the required dependency in the existing environment (pip install feedparser).

The RSS integration will trigger a NewFeedEntryEvent whenever an entry is added to an RSS feed you are subscribed to. We now want to create a logic that reacts to such events and does the following:

Whenever a new entry is created on a subscribed feed, add the corresponding URL to a queue of links to process
A cronjob that runs on a specified basis will collect all the links in the queue, parse the content of the webpages and save them in a digests folder on the notebook.

Create a new script under $PREFIX/etc/platypush/scripts named e.g. digests.py:

import logging
import pathlib
import os
import tempfile
from datetime import datetime
from multiprocessing import RLock

from platypush.cron import cron
from platypush.event.hook import hook
from platypush.message.event.rss import NewFeedEntryEvent
from platypush.utils import run

from .notebook import repo_path

logger = logging.getLogger('digest-generator')
# Path to a text file where you'll store the processing queue
# for the feed entries - one URL per line
queue_path = '/path/to/feeds/processing/queue'
# Lock to ensure consistency when writing to the queue
queue_path_lock = RLock()
# The digests path will be a subfolder of the repo_path
digests_path = f'{repo_path}/digests'


@hook(NewFeedEntryEvent)
def on_new_feed_entry(event, **_):
    """
    Subscribe to new RSS feed entry events and add the
    corresponding URLs to a processing queue.
    """
    with queue_path_lock:
        with open(queue_path, 'a') as f:
            f.write(event.url + '\n')


@cron('0 4 * * *')
def digest_generation_cron(**_):
    """
    This cronjob runs every day at 4AM local time.
    It processes all the URLs in the queue, it generates a digest
    with the parsed content and it saves it in the notebook folder.
    """
    logger.info('Running digest generation cronjob')

    with queue_path_lock:
        try:
            with open(queue_path, 'r') as f:
                md_files = []
                for url in f:
                    # Create a temporary file for the Markdown content
                    tmp = tempfile.NamedTemporaryFile(suffix='.md', delete=False)
                    logger.info(f'Parsing URL {url}')

                    # Parse the webpage to Markdown to the temporary file
                    response = run('http.webpage.simplify', url=url, outfile=tmp.name)
                    title = response.get('title', url)
                    md_files.append(tmp.name)
        except FileNotFoundError:
            pass

        if not md_files:
            logger.info('No URLs to process')
            return

        try:
            pathlib.Path(digests_path).mkdir(parents=True, exist_ok=True)
            digest_file = os.path.join(digests_path, f'{datetime.now()}_digest')
            digest_content = f'# Digest generated on {datetime.now()}\n\n'

            for md_file in md_files:
                with open(md_file, 'r') as f:
                    digest_content += f.read() + '\n\n'

            with open(digest_file, 'w') as f:
                f.write(digest_content)

            # Clean up the queue
            os.unlink(queue_path)
        finally:
            for md_file in md_files:
                os.unlink(md_file)

Now restart the Platypush service. On the first start after configuring the rss integration it should trigger a bunch of NewFeedEntryEvent with all the newly seen content from the subscribed feed. Once the cronjob runs, it will process all these pending requests and it will generate a new digest in your notebook folder. Since we previously set up an automation to monitor changes in this folder, the newly created file will trigger a git sync as well as broadcast sync request on MQTT. At there you go - your daily or weekly subscriptions, directly delivered to your custom notebook!

Conclusions

In this article we have learned:

How to design a distributed architecture to synchronize content across multiple devices using Platypush scripts as the glue between a git repository and an MQTT broker.
How to manage a notebook based on Markdown and which popular options are available for the visualization - Github/Gitlab, Obsidian, NextCloud Notes, Madness.
How to install a Platypush virtual environment on the fly from a configuration file through platyvenv command (in the previous articles I mainly targeted manual installations). Just for you to know, a platydock command is also available to create Docker containers on the fly from a configuration file, but given the hardware requirements or specific dependency chains that some integrations may require the mileage of platydock may vary.
How to install and run Platypush directly on Android through Termux. This is actually quite huge: in this specific article we targeted a use-case for folder synchronization between mobile and desktop, but given the high number of integrations provided by Platypush, as well as the powerful scripts provided by Termux:API, it's relatively easy to use Platypush to set up automations that replace the need of paid (and closed-source) services like Tasker.
How to use the http.webpage integration to distill web pages into readable Markdown.
How to push links to our automation chain through a desktop browser (using the Platypush browser extension) or mobile (using the termux-url-opener mechanism).
How to use the rss integration to subscribe to feeds, and how to hook it to http.webpage and cronjobs to generate periodic digests delivered to our notebook.

You should now have some solid tools to build your own automated notebook. A few ideas on possible follow-ups:

Use your notebook to manage databases (a feature provided by Notion) in CSV format.
Set up a similar distributed sync mechanism to synchronize photos across devices.
Host your own Markdown-based wiki or website built on top of such an automation pipeline, so on each update the website is automatically refreshed with the new content.

Happy hacking!

Set up self-hosted CI/CD git pipelines with Platypush

Fabio Manganiello — Sun, 07 Mar 2021 21:11:52 +0000

Git automation, either in the form of Gitlab pipelines or Github actions, is amazing. It enables you to automate a lot of software maintenance tasks (testing, monitoring, mirroring repositories, generating documentation, building and distributing packages etc.) that until a couple of years ago used to take a lot of development time. These forms of automation have democratized CI/CD, bringing to the open-source world benefits that until recently either belonged mostly to the enterprise world (such as TeamCity) or had a steep curve in terms of configuration (such as Jenkins).

I have been using Github actions myself for a long time on the Platypush codebase, with a Travis-CI integration to run integration tests online and a ReadTheDocs integration to automatically generate online documentation.

You and whose code?

However, a few things have changed lately, and I don't feel like I should rely much on the tools mentioned above for my
CI/CD pipelines.

Github has too often taken the wrong side in DMCA disputes since it's been acquired by Microsoft. The CEO of Github has in the meantime tried to redeem himself, but the damage in the eyes of many developers, myself included, was done, all in spite of the friendly olive branch to the community handed over IRC. Most of all, that doesn't change the fact that Github has taken down more than 2000 other repos in 2020 alone, often without any appeal or legal support - the CEO bowed down in the case of youtube-dl only because of the massive publicity that the takedown attracted. Moreover, Github has yet to overcome its biggest contradiction: it advertises itself like the home for open-source software, but its own source code is not open-source, so you can't spin up your own instance on your own server. There's also increasing evidence in support of my initial suspicion that the Github acquisition was nothing but another old-school Microsoft triple-E operation. When you want to clone a Github repo you won't be prompted anymore with the HTTPS/SSH link by default, you'll be prompted with the Github CLI command, which extends the standard git command, but it introduces a couple of naming inconsistencies here and there. They could have contributed to improve the git tool for everyone's benefit instead of providing their new tool as the new default, but they have opted not to do so. I'm old enough to have seen quite a few of these examples in the past, and it never ended well for the extended party. As a consequence of these actions, I have moved the Platypush repos to a self-hosted Gitlab instance - which comes with much more freedom, but also no more Github actions.

And, after the announcement of the planned migration from travis-ci.org to travis-ci.com, with greater focus on enterprise, a limited credit system for open-source projects and a migration process that is largely manual, I have also realized that Travis-CI is another service that can't be relied upon anymore when it comes to open-source software. And, again, Travis-CI is plagued by the same contradiction as Github - it claims to be open-source friendly, but it's not open-source itself, and you can't install it on your own machine.

ReadTheDocs, luckily, seems to be still coherent with its mission of supporting open-source developers, but I'm also keeping an eye on them just in case :)

Building a self-hosted CI/CD pipeline

Even though abandoning closed-source and unreliable cloud development tools is probably the right thing to do, that leaves a hole behind: how do we bring the simplicity of the automation provided by those tools to our new home - and, preferably, in such a format that it can be hosted and moved anywhere?

Github and Travis-CI provide a very easy way of setting up CI/CD pipelines. You read the documentation, upload a YAML file to your repo, and all the magic happens. I wanted to build something that was that easy to configure, but that could run anywhere, not only in someone else's cloud.

Building a self-hosted pipeline, however, also brings its advantages. Besides liberating yourself of the concern of handing your hard-worked code to someone else who can either change their mind about their mission, or take it down overnight, you have the freedom of setting up the environment for build and test however you please and customize it however you please. And you can easily set up integrations such as automated notifications over whichever channel you like, without the headache of installing and configuring all the dependencies to run on someone else's cloud.

In this article we'll how to use Platypush to set up a pipeline that:

Reacts to push and tag events on a Gitlab or Github repository and runs custom Platypush actions in YAML format or Python code.
Automatically mirrors the new commits and tags to another repo - in my case, from Gitlab to Github.
Runs a suite of tests.
If the tests succeed, then it proceeds with packaging the new version of the codebase - in my case, I run the automation to automatically create the new platypush-git package for Arch Linux on new pushes, and the new platypush Arch package and the pip package on new tags.
If the tests fail, it sends a notification (over email, Telegram, Pushbullet or whichever plugin supported by Platypush). Also send a notification if the latest run of tests has succeeded and the previous one was failing.

Note: since I have moved my projects to a self-hosted Gitlab server, I could have also relied on the native Gitlab CI/CD pipelines, but I have eventually opted not to do so for two reasons:

Setting up the whole Docker+Kubernetes automation required for the CI/CD pipeline proved to be a quite cumbersome process. Additionally, it may require some properly beefed machine in order to run smoothly, while ideally I wanted something that could run even on a RaspberryPi, provided that the building and testing processes aren't too resource-heavy themselves.
The alternative provided by Gitlab to setting up your Kubernetes instance and configuring the Gitlab integration is to get a bucket on the cloud to spin a container that runs all you have to run. But if I have gone so far to set up my own self-hosted infrastructure for hosting my code, I certainly don't want to give up on the last mile in exchange of a small discount on the Google Cloud services :)

However, if either you have enough hardware resources and time to set up your own Kubernetes infrastructure to integrate with Gitlab, or you don't mind running your CI/CD logic on the Google cloud, Gitlab CI/CD pipelines are something that you may consider - if you don't have the constraints above then they are very powerful, flexible and easy to set up.

Installing Platypush

Let's start by installing Platypush with the required integrations.

If you want to set up an automation that reacts on Gitlab events then you'll only need the http integration, since we'll use Gitlab webhooks to trigger the automation:

$ [sudo] pip install 'platypush[http]'

If you want to set up the automation on a Github repo you'll only have one or two additional dependencies, installed through the github integration:

$ [sudo] pip install 'platypush[http,github]'

If you want to be notified of the status of your builds then you may want to install the integration required by the communication mean that you want to use. We'll use Pushbullet in this example because it's easy to set up and it natively supports notifications both on mobile and desktop:

$ [sudo] pip install 'platypush[pushbullet]'

Feel free however to pick anything else - for instance, you can refer to this article for a Telegram set up or this article for a mail set up, or take a look at the Twilio integration if you want automated notifications over SMS or Whatsapp.

Once installed, create a ~/.config/platypush/config.yaml file that contains the service configuration - for now we'll just enable the web server:

# The backend listens on port 8008 by default
backend.http:
  enabled: True

Setting up a Gitlab hook

Gitlab webhooks are a very simple and powerful way of triggering things when something happens on a Gitlab repo. All you have to do is setting up a URL that should be called upon a repository event (push, tag, new issue, merge request etc.), and set up an automation on the endpoint that reacts to the event.

The only requirement for this mechanism to work is that the endpoint must be reachable from the Gitlab host - it means that the host running the Platypush web service must either be publicly accessible, on the same network or VPN as the Gitlab host, or the Platypush web port must be tunneled/proxied to the Gitlab host.

Platypush offers a very easy way to expose custom endpoints through the WebhookEvent. All you have to do is set up an event hook that reacts to a WebhookEvent at a specific endpoint. For example, create a new event hook under ~/.config/platypush/scripts/gitlab.py:

from platypush.event.hook import hook
from platypush.message.event.http.hook import WebhookEvent

# Token to be used to authenticate the calls from Gitlab
gitlab_token = 'YOUR_TOKEN_HERE'


@hook(WebhookEvent, hook='repo-push')
def on_repo_push(event, **context):
    # Check that the token provided over the
    # X-Gitlab-Token header is valid
    assert event.headers.get('X-Gitlab-Token') == gitlab_token, \
      'Invalid Gitlab token'

    print('Add your logic here')

This hook will react when an HTTP request is received on http://your-host:8008/hook/repo-push. Note that, unlike most of the other Platypush endpoints, custom hooks are not authenticated - that's because they may be called from any context, and you don't necessarily want to share your Platypush instance credentials or token with 3rd-parties. Instead, it's up to you to implement whichever authentication policy you like over the requests.

After adding your endpoint, start Platypush:

$ platypush

Now, in order to set up a new webhook, navigate to your Gitlab project -> Settings -> Webhooks.

Enter the URL to your webhook and the secret token and select the events you want to react to - in this example, we'll select new push events.

You can now test the endpoint through the Gitlab interface itself. If it all went well, you should see a Received event line with a content like this on the standard output or log file of Platypush:

{
  "type": "event",
  "target": "platypush-host",
  "origin": "gitlab-host",
  "args": {
    "type": "platypush.message.event.http.hook.WebhookEvent",
    "hook": "repo-push",
    "method": "POST",
    "data": {
      "object_kind": "push",
      "event_name": "push",
      "before": "previous-commit-id",
      "after": "current-commit-id",
      "ref": "refs/heads/master",
      "checkout_sha": "current-commit-id",
      "message": null,
      "user_id": 1,
      "user_name": "Your User",
      "user_username": "youruser",
      "user_email": "you@email.com",
      "user_avatar": "path to your avatar",
      "project_id": 1,
      "project": {
        "id": 1,
        "name": "My project",
        "description": "Project description",
        "web_url": "https://git.platypush.tech/platypush/platypush",
        "avatar_url": "https://git.platypush.tech/uploads/-/system/project/avatar/3/icon-256.png",
        "git_ssh_url": "git@git.platypush.tech:platypush/platypush.git",
        "git_http_url": "https://git.platypush.tech/platypush/platypush.git",
        "namespace": "My project",
        "visibility_level": 20,
        "path_with_namespace": "platypush/platypush",
        "default_branch": "master",
        "ci_config_path": null,
        "homepage": "https://git.platypush.tech/platypush/platypush",
        "url": "git@git.platypush.tech:platypush/platypush.git",
        "ssh_url": "git@git.platypush.tech:platypush/platypush.git",
        "http_url": "https://git.platypush.tech/platypush/platypush.git"
      },
      "commits": [
        {
          "id": "current-commit-id",
          "message": "This is a commit",
          "title": "This is a commit",
          "timestamp": "2021-03-06T20:02:25+01:00",
          "url": "https://git.platypush.tech/platypush/platypush/-/commit/current-commit-id",
          "author": {
            "name": "Your Name",
            "email": "you@email.com"
          },
          "added": [],
          "modified": [
            "tests/my_test.py"
          ],
          "removed": []
        }
      ],
      "total_commits_count": 1,
      "push_options": {},
      "repository": {
        "name": "My project",
        "url": "git@git.platypush.tech:platypush/platypush.git",
        "description": "Project description",
        "homepage": "https://git.platypush.tech/platypush/platypush",
        "git_http_url": "https://git.platypush.tech/platypush/platypush.git",
        "git_ssh_url": "git@git.platypush.tech:platypush/platypush.git",
        "visibility_level": 20
      }
    },
    "args": {},
    "headers": {
      "Content-Type": "application/json",
      "User-Agent": "GitLab/version",
      "X-Gitlab-Event": "Push Hook",
      "X-Gitlab-Token": "YOUR GITLAB TOKEN",
      "Connection": "close",
      "Host": "platypush-host:8008",
      "Content-Length": "lenght"
    }
  }
}

These are all fields provided on the event object that you can use in your hook to build your custom logic.

Setting up a Github integration

If you want to keep using Github but run the CI/CD pipelines on another host with no dependencies on the Github actions, you can leverage the Github backend to monitor your repos and fire Github events that you can build your hooks on when something happens.

First, head to your Github profile to create a new API access token. Then add the configuration to ~/.config/platypush/config.yaml under the backend.github section:

backend.github:
  user: your_user
  user_token: your_token
  # Optional list of repos to monitor (default: all user repos)
  repos:
    - https://github.com/you/myrepo1.git
    - https://github.com/you/myrepo2.git

  # How often the backend should poll for updates (default: 60 seconds)
  poll_seconds: 60

  # Maximum events that will be triggered if a high number of events has
  # been triggered since the last poll (default: 10)
  max_events_per_scan: 10

Start the service, and on e.g. the first repository push event you should see a Received event log line like this:

{
  "type": "event",
  "target": "your-host",
  "origin": "your-host",
  "args": {
    "type": "platypush.message.event.github.GithubPushEvent",
    "actor": {
      "id": 1234,
      "login": "you",
      "display_login": "You",
      "url": "https://api.github.com/users/you",
      "avatar_url": "https://avatars.githubusercontent.com/u/1234?"
    },
    "event_type": "PushEvent",
    "repo": {
      "id": 12345,
      "name": "you/myrepo1",
      "url": "https://api.github.com/repos/you/myrepo1"
    },
    "created_at": "2021-03-03T18:20:27+00:00",
    "payload": {
      "push_id": 123456,
      "size": 1,
      "distinct_size": 1,
      "ref": "refs/heads/master",
      "head": "current-commit-id",
      "before": "previous-commit-id",
      "commits": [
        {
          "sha": "current-commit-id",
          "author": {
            "email": "you@email.com",
            "name": "You"
          },
          "message": "This is a commit",
          "distinct": true,
          "url": "https://api.github.com/repos/you/myrepo1/commits/current-commit-id"
        }
      ]
    }
  }
}

You can easily create an event hook that reacts to such events to run your automation - e.g. under ~/.config/platypush/scripts/github.py:

from platypush.event.hook import hook
from platypush.message.event.github import GithubPushEvent


@hook(GithubPushEvent)
def on_repo_push(event, **context):
  # Run this action only for a specific repo
  if event.repo['name'] != 'you/myrepo1':
    return

  print('Add your logic here')

And here you go - you should now be ready to create your automation routines on Github events.

Automated repository mirroring

Even though I have moved the Platypush repos to a self-hosted domain, I still keep a mirror of them on Github. That's because lots of people have already cloned the repos over the years and may lose updates if they haven't seen the announcement about the transfer. Also, registering to a new domain is often a barrier for users who want to create issues. So, even though me and Github are no longer friends, I still need a way to easily mirror each new commit on my domain to Github - but you might as well have another compelling case for backing up/mirroring your repos. The way I'm currently achieving this is by cloning the main instance of the repo on the machine that runs the Platypush service:

$ git clone git@git.you.com:you/myrepo.git /opt/repo

Then add a new remote that points to your mirror repo:

$ cd /opt/repo
$ git remote add mirror git@github.com:/you/myrepo.git
$ git fetch

Then try a first git push --mirror to make sure that the repos are aligned and all conflicts are solved:

$ git push --mirror -v mirror

Then add a new sync_to_mirror function in your Platypush script file that looks like this:

import logging
import os
import subprocess

repo_path = '/opt/repo'

# ...

def sync_to_mirror():
    logging.info('Synchronizing commits to mirror')
    os.chdir(repo_path)
    # Pull the updates from the main repo
    subprocess.run(['git', 'pull', '--rebase', 'origin', 'master'])
    # Sync the updates to the repo
    subprocess.run(['git', 'push', '--mirror', '-v', 'mirror'])
    logging.info('Synchronizing commits to mirror: DONE')

And just call it from the previously defined on_repo_push hook, either the Gitlab or Github variant:

# ...
def on_repo_push(event, **_):
    # ...
    sync_to_mirror()
    # ...

Now on each push the repository clone stored under /opt/repo will be updated and any new commits and tags will be mirrored to the mirror repository.

Running tests

If our project is properly set up, then it probably has a suite of unit/integration tests that is supposed to be run on each change to verify that nothing is broken. It's quite easy to configure the previously created hook so that it runs the tests on each push. For instance, if your tests are stored under the tests folder of your project and you use pytest:

import datetime
import os
import pathlib
import shutil
import subprocess

from platypush.event.hook import hook
from platypush.message.event.http.hook import WebhookEvent

# Path where the latest version of the repo will be cloned
tmp_path = '/tmp/repo'
# Path where the results of the tests will be stored
logs_path = '/var/log/tests'

# ...

def run_tests():
  # Clone the repo in /tmp
  shutil.rmtree(tmp_path, ignore_errors=True)
  subprocess.run(['git', 'clone', 'git@git.you.com:you/myrepo.git', tmp_path])
  os.chdir(os.path.join(tmp_path, 'tests'))
  passed = False

  try:
    # Run the tests
    tests = subprocess.Popen(['pytest'],
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)

    stdout = tests.communicate()[0].decode()
    passed = tests.returncode == 0

    # Write the stdout to a logfile
    pathlib.Path(logs_path).mkdir(parents=True, exist_ok=True)
    logfile = os.path.join(logs_path,
                           f'{datetime.datetime.now().isoformat()}_'
                           f'{"PASSED" if passed else "FAILED"}.log')

    with open(logfile, 'w') as f:
      f.write(stdout)
  finally:
    shutil.rmtree(tmp_path, ignore_errors=True)

  # Return True if the tests passed, False otherwise
  return passed

# ...

@hook(WebhookEvent, hook='repo-push')
# or
# @hook(GithubPushEvent)
def on_repo_push(event, **_):
  # ...
  passed = run_tests()
  # ...

Upon push event, the latest version of the repo will be cloned under /tmp/repo and the suite of tests will be run. The output of each session will be stored under /var/log/tests in a file formatted like <ISO timestamp>_<PASSED|FAILED>.log. To make things even more robust, you can create a new virtual environment under the temporary directory, install your repo with all of its dependency in the new virtual environment and run the tests from there, or spin a Docker instance with the required configuration, to make sure that the tests would also pass on a fresh installation and prevent the "but if works on my box" issue.

Serve the test results over HTTP

Now you can simply serve /var/log/tests over an HTTP server and the logs can be accessed from your browser. Simple case:

$ cd /var/log/tests
$ python -m http.server 8000

The logs will be served on http://host:8000. You can also serve the directory through a proper web server like nginx or Apache.

It doesn't come with all the bells and whistles of the Jenkins or Travis-CI UI, but it's simple and good enough for its job - and it's not hard to extend it with a fancier UI if you like.

Another nice addition is to download some of those nice passed/failed badge images that you find on many Github repositories to your Platypush box. When a test run completes, just edit your hook to copy the associated banner image (e.g. passed.svg or failed.svg) to e.g. /var/log/tests/status.svg:

import os
import shutil

# ...

def run_tests():
    # ...
    passed = tests.returncode == 0
    badge_path = '/path/to/passed.svg' if passed else '/path/to/failed.svg'
    shutil.copy(badge_path, os.path.join(logs_path, 'status.svg'))
    # ...

Then embed the status in your README.md:

[![Tests Status](http://your-host:8000/status.svg)](http://your-host:8000)

And there you go - you can now show off a dynamically generated and self-hosted status badge on your README without relying on any cloud runner.

Automatic build and test notifications

Another useful feature of the popular cloud services is the ability to send notification when a build status changes. This is quite easy to set up with Platypush, as the application provides several plugins for messaging. Let's look at an example where a change in the status of our tests triggers a notification to our Pushbullet account, which can be delivered both to our desktop and mobile devices. Download the Pushbullet app if you want the notifications to be delivered to your mobile, get an API token and then install the dependencies for the Pushbullet integration for Platypush:

$ [sudo] pip install 'platypush[pushbullet]'

Then configure the Pushbullet plugin and backend in ~/.config/platypush/config.yaml:

backend.pushbullet:
    token: YOUR_PUSHBULLET_TOKEN
    device: platypush

pushbullet:
    enabled: True

Now simply modify your push hook to send a notification when the status of build changes. We will also use the variable plugin to retrieve and store the latest status, so that notifications are triggered only when the status changes:

from platypush.context import get_plugin
from platypush.event.hook import hook
from platypush.message.event.http.hook import WebhookEvent

# Name of the variable that holds the latest run status
last_tests_passed_var = 'LAST_TESTS_PASSED'

# ...

def run_tests():
  # ...
  passed = tests.returncode == 0
  # ...
  return passed

# ...

@hook(WebhookEvent, hook='repo-push')
# or
# @hook(GithubPushEvent)
def on_repo_push(event, **_):
  variable = get_plugin('variable')
  pushbullet = get_plugin('pushbullet')

  # Get the status of the last run
  response = variable.get(last_tests_passed_var).output
  last_tests_passed = int(response.get(last_tests_passed_var, 0))

  # ...

  passed = run_tests()
  if passed and not last_tests_passed:
    pushbullet.send_note(body='The tests are now PASSING',
                         # If device is not set then the notification will
                         # be sent to all the devices connected to the account
                         device='my-mobile-name')
  elif not passed and last_tests_passed:
    pushbullet.send_note(body='The tests are now FAILING',
                         device='my-mobile-name')

  # Update the last_test_passed variable
  variable.set(**{last_tests_passed_var: int(passed)})

  # ...

The nice addition of this approach is that any other Platypush device with the Pushbullet backend enabled and connected to the same account will receive a PushbulletEvent when a Pushbullet note is sent, and you can easily leverage this to build some downstream logic with hooks that react to these events.

Continuous delivery

Once we have a logic in place that automatically mirrors and tests our code and notifies us about status changes, we can take things a step further and set up our pipeline to also build a package for our applications if the tests are successful.

Let's consider in this article the example of a Python application whose new releases are tagged through git tags, and each time a new version is released we want to create a pip package and upload it to the online PyPI registry. However, you can easily adapt this example to work with any build and release process.

Twine is a quite popular option when it comes to uploading packages to the PyPI registry. Let's install it:

$ [sudo] pip install twine

Then create a Gitlab webhook that reacts to tag events, or react to a GithubCreateEvent if you are using Github, and create a Platypush hook that reacts to tag events by running the logic of on_repo_push, and additionally make a package build and upload it with Twine if the tests are successful:

import importlib
import os
import subprocess

from platypush.event.hook import hook
from platypush.message.event.http.hook import WebhookEvent

# Path where the latest version of the repo has been cloned
tmp_path = '/tmp/repo'

# Initialize these variables with your PyPI credentials
os.environ['TWINE_USERNAME'] = 'your-pypi-user'
os.environ['TWINE_PASSWORD'] = 'your-pypi-pass'

# ...

def upload_pip_package():
  os.chdir(tmp_path)
  # Build the package
  subprocess.run(['python', 'setup.py', 'sdist', 'bdist_wheel'])

  # Check the version of your app - for example from the
  # yourapp/__init__.py __version__ field
  app = importlib.import_module('yourapp')
  version = app.__version__

  # Check that the archive file has been created
  archive_file = os.path.join('.', 'dist', f'yourapp-{version}.tar.gz')
  assert os.path.isfile(archive_file), \
    f'The target file {archive_file} was not created'

  # Upload the archive file to PyPI
  subprocess.run(['twine', 'upload', archive_file])

@hook(WebhookEvent, hook='repo-tag')
# or
# @hook(GithubCreateEvent)
def on_repo_tag(event, **_):
  # ...

  passed = run_tests()
  if passed:
    upload_pip_package()

  # ...

And here you go - you now have an automated way of building and releasing your application!

Continuous delivery of web applications

We have seen in this article some examples of CI/CD for stand-alone applications with a complete test+build+release pipeline. The same concept also applies to web services and applications. If your repository stores the source code of a website, then you can easily create automations that react to push events and pull the changes on the web server and restart the web service if required. This is in fact the way I'm currently managing updates on the Platypush blog and homepage. Let's see a small example where we have a Platypush instance running on the same machine as the web server, and suppose that our website is served under /srv/http/myapp (and, of course, that the user that runs the Platypush service has write permissions on this location). It's quite easy to tweak the previous hook example so that it reacts to push events on this repo by pulling the latest changes, runs e.g. npm run build to build the new dist files and then copies the dist folder to our web server directory:

import os
import shutil
import subprocess

from platypush.event.hook import hook
from platypush.message.event.http.hook import WebhookEvent

# Path where the latest version of the repo has been cloned
tmp_path = '/tmp/repo'

# Path of the web application
webapp_path = '/srv/http/myapp'

# Backup path of the web application
backup_webapp_path = '/srv/http/myapp-backup'

# ...

def update_webapp():
  os.chdir(tmp_path)
  # Build the app
  subprocess.run(['npm', 'install'])
  subprocess.run(['npm', 'run', 'build'])

  # Verify that the dist folder has been created
  dist_path = os.path.join('.', 'dist')
  assert os.path.isdir(dist_path), 'dist path not created'

  # Remove the previous app backup folder if present
  shutil.rmtree(backup_webapp_path, ignore_errors=True)
  # Backup the old web app folder
  shutil.move(webapp_path, backup_webapp_path)
  # Move the dist folder to the web app folder
  shutil.move(dist_path, webapp_path)

@hook(WebhookEvent, hook='repo-push')
# or
# @hook(GithubPushEvent)
def on_repo_tag(event, **_):
  # ...

  passed = run_tests()
  if passed:
    update_webapp()

  # ...

Deliver customized newsletters from RSS feeds with Platypush

Fabio Manganiello — Sun, 06 Sep 2020 21:57:15 +0000

I've always been a fan of well-curated newsletters. They give me an opportunity to get a good overview of what happened in the fields I follow within a span of a day, a week or a month. However, not all the newsletters fit this category. Some don't think three times before selling email addresses to 3rd-parties, and within the blink of an eye, your mailbox can easily get flooded with messages that you didn't request. Others may sign up your address for other services or newsletters as well, and often they don't often much granularity to configure which communications you want to receive. Even in the best-case scenario, the most privacy-savvy user may still think twice before signing up for a newsletter - you're giving your personal email address to someone else you don't necessarily trust, implying "yes, this is my address and I'm interested in this subject". Additionally, most of the newsletters spice up their URLs with tracking parameters, so they can easily measure user engagement - something you may not necessarily be happy with. Moreover, the customization junkie may also have a use case for a more finely tuned selection of content in his newsletter - you may want to group some sources together into the same daily/weekly email, or you may be interested only in some particular subset of the subjects covered by a newsletter, filtering out those that aren't relevant, or customize the style of the digest that gets delivered. Finally, a fully automated way to deliver newsletters through 5 lines of code and the tuning of a couple of parameters is the nirvana for many companies of every size out there.

Feed up the newsletter

Those who read my articles in the past may know that I'm an avid consumer of RSS feeds. Despite being a 21-year-old technology, they do their job very well when it comes to deliver the information that matters without all the noise and trackers, and they provide a very high level of integration being simple XML documents. However, in spite of all the effort I put to be up-to-date with all my sources, a lot of potentially interesting content inevitably slips through - and that's where newsletters step in, as they filter and group together all the content that was generated in a given time frame and periodically deliver it to your inbox.

My ideal solution would be something that combines the best aspects of both the worlds: the flexibility of an RSS subscription, combined with a flexible way of filtering and aggregating content and sources, and get the full package delivered at my door in whichever format I like (HTML, PDF, MOBI…). In this article I'm going to show how to achieve this goal with a few tools:

One or more sources that you want to track and that support RSS feeds (in this example I'll use the MIT Technology Review RSS feed, but the procedure works for any RSS feed).
An email address.
Platypush to do the heavy-lifting job - monitor the RSS sources at custom intervals, trigger events when a source has some new content, create a digest out of the new content, and deliver the full package to a list of email addresses.

Let's cover these points step by step.

Installing and configuring Platypush

Those who have already read my previous articles may have heard of Platypush - the automation platform I've been building in the past few years. For those who aren't familiar, an advised read is my first Medium post that illustrates some of its capabilities and the paradigm behind it.

We'll be using the http.poll backend configured with one or more RssUpdates objects to poll our RSS sources at regular intervals and create the digests, and either the mail.smtp plugin or the google.mail plugin to send the digests to our email.

You can install Platypush on any device where you want to run your logic - a RaspberryPi, an old laptop, a cloud node, and so on. We will install the base package with the rss module. Optionally, you can install it with the pdf module as well (if you want to export your digests also to PDF) or the google module (if you want to send the newsletter from a GMail address instead of an SMTP server).

The first option is to install the latest stable version through pip:

pip install 'platypush[rss]'
# Or
pip install 'platypush[rss,pdf,google]'

The other option is to install the latest git version:

git clone git@github.com/BlackLight/platypush.git
cd platypush
pip install '.[rss]'
# Or
pip install '.[rss,pdf,google]'

Monitoring your RSS feeds

Once the software is installed, create the configuration file ~/.config/platypush/config.yaml if it doesn't exist already and add the configuration for the RSS monitor:

# Generic HTTP endpoint monitor
backend.http.poll:
    requests:
        # Add a new RSS feed to the pool
        - type: platypush.backend.http.request.rss.RssUpdates
          url: https://www.technologyreview.com/feed/  # URL to the RSS feed
          title: MIT Technology Review                 # Title of the feed (shown in the head of the digest)
          poll_seconds: 86400                          # How often we should monitor this source (24*60*60 secs = once a day)
          digest_format: html                          # Format of the digest (HTML or PDF)

You can also add more sources to the http.poll requests object, each with its own configuration. Also, you can customize the style of your digest by passing some valid CSS to these configuration attributes:

# Style of the body element
body_style: 'font-size: 20px; font-family: "Merriweather", Georgia, "Times New Roman", Times, serif'

# Style of the main title
title_style: 'margin-top: 30px'

# Style of the subtitle
subtitle_style: 'margin-top: 10px; page-break-after: always'

# Style of the article titles
article_title_style: 'font-size: 1.6em; margin-top: 1em; padding-top: 1em; border-top: 1px solid #999'

# Style of the article link
article_link_style: 'color: #555; text-decoration: none; border-bottom: 1px dotted font-size: 0.8em'

# Style of the article content
article_content_style: 'font-size: 0.8em'

The digest_format attribute determines the output format of your digest - you may want to choose html if you want to deliver a summary of the articles in a newsletter, or pdf if you want instead to deliver the full content of each item as an attachment to an email address. Bonus point: since you can send PDFs to a Kindle if you configured an email address, this mechanism allows you to deliver the full digest of your RSS feeds to your Kindle's email address.

The RssUpdates object also provides native integration with the Mercury Parser API to automatically scrape the content of a web page - I covered some of these concepts in my past article on how to parse RSS feeds and send the PDF digest to your e-reader. The same mechanism works well for newsletters too. If you want to parse the content of the newsletter as well, all you have to do is configure the http.webpage Platypush plugin. Since the Mercury API doesn't provide a Python binding, this requires a couple of JavaScript dependencies:

# Install Node and NPM, e.g. on Debian:
apt-get install nodejs npm

# Install the Mercury Parser API
npm install [-g] @postlight/mercury-parser

# Make sure that the Platypush PDF module dependencies
# are installed if you plan HTML->PDF conversion
pip install 'platypush[pdf]'

Then, if you want to parse the full content of the items and generate a PDF digest out of them, change your http.poll configuration to something like this:

backend.http.poll:
    requests:
        - type: platypush.backend.http.request.rss.RssUpdates
          url: https://www.technologyreview.com/feed/
          title: MIT Technology Review
          poll_seconds: 86400
          digest_format: pdf     # PDF digest format
          extract_content: True  # Extract the full content of the items

WARNING: Extracting the full content of the articles in an RSS feed has two limitations - a practical one and a legal one:

Some websites may require user login before displaying the full content of an article. Some websites perform such checks client-side - and the parser API can usually circumvent them, especially if the full content of an article is actually just hidden behind a client-side paywall. Some websites, however, implement their user checks server-side too before sending the content to the client - and in those cases the parser API may return only a part of the content or no content at all.
Always keep in mind that parsing the full content of an article behind a paywall may represent a violation of intellectual property under some jurisdictions.

Configuring the mail delivery

When new content is published on a subscribed RSS feed Platypush will generate a NewFeedEvent and it should create a copy of the digest under ~/.local/share/platypush/feeds/cache/{date:time}_{feed-title}.[html|pdf]. The NewFeedEvent in particular is the link you need to create your custom logic that sends an email to a list of addresses when new content is available.

First, configure the Platypush mail plugin you prefer. When it comes to sending emails you primarily have two options:

The mail.smtp plugin - if you want to send emails directly through an SMTP server. Platypush configuration:

mail.smtp:
    username: you@gmail.com
    password: your-pass
    server: smtp.gmail.com
    port: 465
    ssl: True

The google.mail plugin - if you want to use the native GMail API to send emails. If that is the case then first make sure that you have the dependencies for the Platypush Google module installed:

pip install 'platypush[google]'

In this case you'll also have to create a project on the Google Developers console and download the OAuth credentials:

Click on “Credentials” from the context menu > OAuth Client ID.
Once generated, you can see your new credentials in the “OAuth 2.0 client IDs” section. Click on the “Download” icon to save them to a JSON file.
Copy the file to your Platypush device/server under e.g. ~/.credentials/google/client_secret.json.
Run the following command on the device to authorize the application:

python -m platypush.plugins.google.credentials \
    "https://www.googleapis.com/auth/gmail.modify" \
    ~/.credentials/google/client_secret.json \
    --noauth_local_webserver

Copy the link in your browser, log in with your Google account, and authorize the application.

At this point the GMail delivery is ready to be used by your Platypush automation.

Connecting the dots

Now that both the RSS parsing logic and the mail integration are in place, we can glue them together through the NewFeedEvent event. The new advised way to configure events in Platypush is through native Python scripts - the custom YAML-based syntax for events and procedure was becoming too cumbersome to maintain and write, and I feel like going back to a clean and simple Python API may be a better option.

Create and initialize the Platypush scripts directory, if it doesn't exist already:

mkdir -p ~/.config/platypush/scripts
cd ~/.config/platypush/scripts
touch __init__.py  # Initialize the root Python module

Then, create a new hook on NewFeedEvent:

$EDITOR rss_news.py

import os
from typing import List

from platypush.event.hook import hook
from platypush.message.event.http.rss import NewFeedEvent
from platypush.utils import run

# Path to your mailing list - a text file with one address per line
maillist = os.path.expanduser('~/.mail.list')

def get_addresses() -> List[str]:
    with open(maillist, 'r') as f:
        return [addr.strip() for addr in f.readlines()
                if addr.strip() and not addr.strip().startswith('#')]


# This hook matches:
# - event_type=NewFeedEvent
# - digest_format='html'
# - source_title='MIT Technology Review'
@hook(NewFeedEvent, digest_format='html', source_title='MIT Technology Review')
def send_mit_rss_feed_digest(event: NewFeedEvent, **_):
    # The digest output file is stored in event.args['digest_filename']
    with open(event.args['digest_filename'], 'r') as f:
        run(action='mail.smtp.send',
            from_='you@yourdomain.com',
            to=get_addresses(),
            subject=f'{event.args.get("source_title")} feed digest',
            body=f.read(),
            body_type='html')

If you opted for the native GMail plugin you may want to go for:

@hook(NewFeedEvent, digest_format='html', source_title='MIT Technology Review')
def send_mit_rss_feed_digest(event: NewFeedEvent, **_):
    # The digest output file is stored in event.args['digest_filename']
    with open(event.args['digest_filename'], 'r') as f:
        run(action='google.mail.compose',
            sender='you@gmail.com',
            to=get_addresses(),
            subject=f'{event.args.get("source_title")} feed digest',
            body=f.read())

If instead you want to send the digest in PDF format as an attachment:

@hook(NewFeedEvent, digest_format='html', source_title='MIT Technology Review')
def send_mit_rss_feed_digest(event: NewFeedEvent, **_):
    # mail.smtp plugin case
    run(action='mail.smtp.send',
        from_='you@yourdomain.com',
        to=get_addresses(),
        subject=f'{event.args.get("source_title")} feed digest',
        body='',
        attachments=[event.args['digest_filename'])

    # google.mail case
    run(action='google.mail.compose',
        sender='you@gmail.com',
        to=get_addresses(),
        subject=f'{event.args.get("source_title")} feed digest',
        body='',
        files=[event.args['digest_filename'])

Finally, create your ~/.mail.list file with one destination email address per line and start platypush either from the command line or as a service. You should receive your email with the first batch of articles shortly after startup, and you'll receive more items if a new batch is available after the poll_seconds configured period.

One browser extension to rule them all

Fabio Manganiello — Wed, 08 Jul 2020 09:17:25 +0000

The current WebExtensions API has made the world of browser extensions safer and more cross-compatible, but it has also introduced its share of limitations and compromises in order to achieve better security.

Something I miss from the "first wave" of add-ons/browser apps is the possibility to customise every single aspect of the browser (UI, input events, background scripts etc.) through tweaks and custom scripts. And I've always wanted the ability to connect these tweaks/custom browser actions to my increasing (and increasingly fragmented) network of smart devices around - perform actions like turn on the lights, cast a video or action a switch directly from the browser, without grabbing the phone and opening an app, and without switching tabs in the browser. Moreover, I wanted the ability to run any kind of simple browser actions (simplify/translate page, share to Twitter/Facebook, play on Chromecast/Kodi, send link to mobile device...) as simple JavaScript snippets within the same extension - I've always considered the idea of having a separate extension, and a separate icon in the browser toolbar, just to do one specific little thing as pure madness.

So I got my plan together and finally developed an extension that could fill the gap. An extension that makes e.g. writing a browser action to cast a YouTube URL to your Chromecast something as simple as a JavaScript snippet:

async (app, args) => {
  const url = await app.getURL();
  if (!url.startsWith('https://www.youtube.com/watch?v=')) {
    return;
  }

  const response = await app.run({
    action: 'media.chromecast.play',
    args: {
      resource: url,
    },
  }, args.host);

  if (response.success) {
    app.notify('YouTube video now playing on Chromecast');
  }
}

Medium article: https://medium.com/@automationguru/one-browser-extension-to-rule-them-all-3118dc7f9c9b

Firefox link: https://addons.mozilla.org/en-US/firefox/addon/platypush/

Chrome link: https://chrome.google.com/webstore/detail/platypush/aphldjclndofhflbbdnmpejbjgomkbie?hl=en-GB&authuser=0

Build your own open dashboard to keep track of COVID-19

Fabio Manganiello — Sun, 03 May 2020 17:39:18 +0000

We’re living in relatively unprecedented times, as none of those alive today (except the few that were around in 1918) have gone through a…

Continue reading on Towards Data Science »

Building Custom Voice Assistants

Fabio Manganiello — Sun, 08 Mar 2020 18:41:33 +0000

An overview of current solutions and integrations

Continue reading on Better Programming »