Forem: Florimond Manca

Introduction to ASGI: Emergence of an Async Python Web Ecosystem

Florimond Manca — Tue, 31 Dec 2019 13:55:17 +0000

Originally posted at florimond.dev/blog on Nov. 8th, 2019.

There's a lot of exciting stuff happening in the Python web development ecosystem right now — one of the main drivers of this endeavour is ASGI, the Asynchronous Standard Gateway Interface.

I already mentioned ASGI several times here, in particular when announcing Bocadillo and tartiflette-starlette, but I never really took the time to write a thorough introduction about it. Well, here we are!

This post is targeted at people interested in recent trends of Python web development. I want to take you on a guided tour about what ASGI is and what it means for modern Python web development.

Before we begin, I'd like to announce that I recently created awesome-asgi, an awesome list to help folks keep track of the ever-expanding ASGI ecosystem. 🙌

florimondmanca / awesome-asgi

A curated list of awesome ASGI servers, frameworks, apps, libraries, and other resources

awesome-asgi

A curated list of awesome ASGI servers, frameworks, apps, libraries, and other resources.

This list should help you keep yourself up to date with the most awesome ASGI projects and resources. You can watch releases on this repo to be notified about new entries. If you find anything missing, please contribute. ❣️

ASGI is a standard interface positioned as a spiritual successor to WSGI. It enables communication and interoperability across the whole Python async web stack: servers, applications, middleware, and individual components.

Born in 2016 to power the Django Channels project, ASGI and its ecosystem have been expanding ever since, boosted by the arrival of projects such as Starlette and Uvicorn in 2018.

Contents

Application frameworks

Frameworks for building ASGI web applications.

Asgineer - A…

View on GitHub

You can watch releases to be notified of new entries to the list. 👀

Alright, let's dive in!

It all started with async/await

Contrary to JavaScript or Go, Python is not a language that had asynchronous execution baked in from the start. For a long time, executing things concurrently in Python could only be achieved using multithreading or multiprocessing, or resorting to specialized networking libraries such as eventlet, gevent, or Twisted. (Back in 2008, Twisted already had APIs for asynchronous coroutines, e.g. in the form of inlineCallbacks and deferredGenerator.)

But this all changed in Python 3.4+. Python 3.4 added asyncio to the standard library, adding support for cooperative multitasking on top of generators and the yield from syntax.

Later, the async/await syntax was added in Python 3.5. Thanks to this, we now had native coroutines independent of the underlying implementation, which opened the gold rush towards Python concurrency.

And what a rush it was, indeed! Since 3.5 was released, the community has been literally async-ifying all the things. If you're curious, a lot of the resulting projects are now listed in aio-libs and awesome-asyncio.

Well, you guessed it — this also means that Python web servers and apps are moving towards async. In fact, all the cool kids are doing it! (Even Django.)

An overview of ASGI

Now, how does ASGI fit in all of this?

From a 1000-foot perspective, ASGI can be thought of as the glue that allows Python asynchronous servers and applications to communicate with each other. It shares a lot of design ideas with WSGI, and is often presented as its spiritual successor with async built-in.

Here's what this mental model looks like in a diagram:

At a very high-level, ASGI is a communication interface between apps and servers.

But in reality, it's a bit more complex than that.

To find out how ASGI really works, let's take a look at the ASGI specification:

ASGI consists of two different components:

A protocol server, which terminates sockets and translates them into connections and per-connection event messages.

An application, which lives inside a protocol server, is instanciated once per connection, and handles event messages as they happen.

So according to the spec, what ASGI really specifies is a message format and how those messages should be exchanged between the application and the protocol server that runs it.

We can now revise our diagram into a more detailed version:

How ASGI *really* works.

There are many more interesting details to look at, obviously. For example, you can take a look at the HTTP and WebSocket specification.

Besides, although the spec focuses a lot on server-to-application communication, ASGI turns out to encompass much more than that.

We'll get to this in a minute, but first…

ASGI basics

Now that we've seen how ASGI fits in the Python web ecosystem, let's take a closer look at what it looks like in code.

ASGI relies on a simple mental model: when the client connects to the server, we instanciate an application. We then feed incoming bytes into the app and send back whatever bytes come out.

"Feed into the app" here really means call the app as if it were a function, i.e. something that takes some input, and returns an output.

And in fact, that's all an ASGI app is — a callable. The shape of this callable is, again, defined by the ASGI spec. Here's what it looks:

async def app(scope, receive, send):
    ...

The signature of this function is what the "I" in ASGI stands for: an interface which the application must implement for the server to be able to call it.

Let's take a look at the 3 arguments:

scope is a dictionary that contains information about the incoming request. Its contents vary between HTTP and WebSocket connections.
receive is an asynchronous function used to receive ASGI event messages.
send is an asynchronous function used to send ASGI event messages.

In essence, these arguments allow you to receive() and send() data over a communication channel maintained by the protocol server, as well as know in what context (or scope) this channel was created.

I don't know about you, but the overall look and shape of this interface fits my brain really well. Anyway, time for code samples.

Show me the code!

To get a more practical feel of what ASGI looks like, I created a minimal Glitch project which showcases a raw ASGI HTTP app served by uvicorn:

Feel free to remix and interact with the code above!

Here, we use send() to send an HTTP response to the client: we send headers first, and then the response body.

Now, with all these dictionaries and raw binary data, I'll admit that bare-metal ASGI isn't very convenient to work with.

Luckily, there are higher-level options — and that's when I get to talk about Starlette.

Starlette is truly a fantastic project, and IMO a foundational piece of the ASGI ecosystem.

In a nutshell, it provides a toolbox of higher-level components such as requests and responses that you can use to abstract away some of the details of ASGI. Here, take a look at this Starlette hello world:

# app.py
from starlette.responses import PlainTextResponse

async def app(scope, receive, send):
  assert scope["type"] == "http"
  response = PlainTextResponse("Hello, world!")
  await response(scope, receive, send)

Starlette does have everything you'd expect from an actual web framework — routing, middleware, etc. But I decided to show this stripped-down version to hint you at the real power of ASGI, which is…

Turtles all the way down

The interesting and downright game-changing bit about ASGI is the concept of "Turtles all the way down", an expression originally coined (I think?) by Andrew Godwin, the person behind Django migrations and now the Django async revamp:

But what does it mean, exactly?

Well, because ASGI is an abstraction which allows to tell in which context we are and to receive and send data at any time, there's this idea that ASGI can be used not only between servers and apps, but really at any point in the stack.

For example, the Starlette Response object is an ASGI application itself. In fact, we can strip down the Starlette example app from earlier to just this:

# app.py
app = PlainTextResponse("Hello, world!")

How ridiculous is that?! 😍

But wait, there's more.

The deeper consequence of "Turtles all the way down" is that we can build all sorts of applications, middleware, libraries and other projects, and ensure that they will be interoperable as long as they all implement the ASGI application interface.

(Besides, from my own experience building Bocadillo, embracing the ASGI interface very often (if not always) results in much cleaner code.)

For example, we can build a ASGI middleware (i.e. an app that wraps another app) to display the time a request took to be served:

# app.py
import time

class TimingMiddleware:
    def __init__(self, app):
        self.app = app

    async def __call__(self, scope, receive, send):
        start_time = time.time()
        await self.app(scope, receive, send)
        end_time = time.time()
        print(f"Took {end_time - start_time:.2f} seconds")

To use it, we simply wrap it around an app…

# app.py
import asyncio
from starlette.responses import PlainTextResponse

async def app(scope, receive, send):
    await asyncio.sleep(1)
    response = PlainTextResponse("Hello, world!")
    await response(scope, receive, send)

app = TimingMiddleware(app)

…and it will magically just work.

$ uvicorn app:app
INFO: Started server process [59405]
INFO: Waiting for application startup.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
...
INFO: ('127.0.0.1', 62718) - "GET / HTTP/1.1" 200
Took 1.00 seconds

The amazing bit about this is that TimingMiddleware can wrap any ASGI app. The inner app here is super simple, but it could be a full-blown, real-life project (think hundreds of API and WebSocket endpoints) — it doesn't matter, as long as it's ASGI-compatible.

(There's a more production-ready version of such a timing middleware: timing-asgi.)

Why should I care?

While I think interoperability is a strong selling point, there are many more advantages to using ASGI-based components for building Python web apps.

Speed: the async nature of ASGI apps and servers make them really fast (for Python, at least) — we're talking about 60k-70k req/s (consider that Flask and Django only achieve 10-20k in a similar situation).
Features: ASGI servers and frameworks gives you access to inherently concurrent features (WebSocket, Server-Sent Events, HTTP/2) that are impossible to implement using sync/WSGI.
Stability: ASGI as a spec has been around for about 3 years now, and version 3.0 is considered very stable. Foundational parts of the ecosystem are stabilizing as a result.

In terms of libraries and tooling, I don't think we can say we're there just yet. But thanks to a very active community, I have strong hopes that the ASGI ecosystem reaches feature parity with the traditional sync/WSGI ecosystem real soon.

Where can I find ASGI-compatible components?

In fact, more and more people are building and improving projects built around ASGI. This includes servers and web frameworks obviously, but also middleware and product-oriented applications such as Datasette.

Some of the non-web framework components that I got the most excited about are:

Mangum: ASGI support for AWS Lambda
datasette-auth-github: GitHub authentication for ASGI apps
tartiflette-starlette (I wrote this one!): ASGI support for Tartiflette, an async GraphQL engine.

While seeing the ecosystem flourish is great, I've personally been having a hard time keeping up with everything.

That's why, as announced at the beginning of this article I created awesome-asgi. My hope is that it helps everyone keep up with all the awesome things that are happening in the ASGI space. (And seeing that it almost reached 100 stars in a few days, I have a feeling there was indeed a need to colocalize ASGI resources.)

Wrapping up

While it might look like an implementation detail, I truly think that ASGI has laid down the foundations for a new era in Python web development.

If you want to learn more about ASGI, take a look at the various publications (articles and talks) listed in awesome-asgi. To get your hands dirty, try out any of the following projects:

uvicorn: ASGI server.
Starlette: ASGI framework.
TypeSystem: data validation and form rendering
Databases: async database library.
orm: asynchronous ORM.
HTTPX: async HTTP client w/ support for calling ASGI apps (useful as a test client).

These projects were built and are maintained by Encode, which mostly means Tom Christie. There are open discussions on setting up an Encode maintenance team, so if you were looking for an opportunity to help advance an open source niche, there you go!

Enjoy your ASGI journey. ❣️

HTTPX: Help Build The Future Of Python HTTP

Florimond Manca — Wed, 02 Oct 2019 11:47:37 +0000

encode / httpx

A next generation HTTP client for Python. 🦋

HTTPX - A next-generation HTTP client for Python.

HTTPX is a fully featured HTTP client library for Python 3. It includes an integrated command line client, has support for both HTTP/1.1 and HTTP/2, and provides both sync and async APIs.

Install HTTPX using pip:

$ pip install httpx

Now, let's get started:

>>> import httpx
>>> r = httpx.get('https://www.example.org/')
>>> r
<Response [200 OK]&gt
>>> r.status_code
200
>>> r.headers['content-type']
'text/html; charset=UTF-8'
>>> r.text
'<!doctype html>\n<html>\n<head>\n<title>Example Domain</title>...'

Or, using the command-line client.

$ pip install 'httpx[cli]'  # The command line client is an optional dependency.

Which now allows us to use HTTPX directly from the command-line...

Sending a request...

Features

HTTPX builds on the well-established usability of requests, and gives you:

A broadly requests-compatible API.
An integrated command-line client.
HTTP/1.1 and HTTP/2 support.
Standard synchronous…

View on GitHub

HTTPX 🦋 is a next generation HTTP client for Python. It is Requests-compatible, supports HTTP/2 and HTTP/1.1, has async support, and an ever-expanding community of 35+ awesome contributors. ❤

HTTPX is part of a growing ecosystem of async-capable Python projects lead by the Encode organization (which backs the Django REST Framework). Some of the most notable projects include Uvicorn, Starlette, and Databases.

Hacktoberfest started yesterday, and on behalf of the HTTPX maintenance team I can say we're very excited to be part of it this year!

Before getting to the contributing guide, a short preview of what HTTPX can do…

Your first request to dev.to using HTTPX:

>>> import httpx
>>> r = httpx.get('https://dev.to')
>>> r.status_code
200
>>> r.headers['content-type']
'text/html; charset=utf-8'
>>> r.text[:50]
'<!DOCTYPE html>\n<html lang="en">\n  <head>\n    <met'
>>> r.http_version
'HTTP/1.1'

Does DEV support HTTP/2?

>>> import httpx
>>> # The top-level API only supports HTTP/1.1 for efficiency reasons,
>>> # but 'Client' uses HTTP/2 by default.
>>> with httpx.Client() as client:
...    r = client.get('https://dev.to')
...
>>> r.http_version
'HTTP/2'  # YAY! 🎉

HTTPX can even call into Python web apps (WSGI + ASGI). A Flask example…

>>> from flask import Flask
>>> import httpx
>>> app = Flask(__name__)
>>> @app.route("/")
... def hello():
...     return "Hello, world"
...
>>> with httpx.Client(app=app, base_url='http://testserver') as client:
...     r = client.get('/')
...
>>> r.status_code
200
>>> r.text
'Hello, world'  # Tada! 🦋
>>>

And it supports async/await! Let's fire an IPython REPL to try this out:

In [1]: import httpx
In [2]: import asyncio
In [3]: # We're going to fetch tag pages concurrently...
In [4]: async def fetch(tag, client):
   ...:     return await client.get(f'https://dev.to/t/{tag}')
   ...:

In [5]: async with httpx.AsyncClient() as client:
   ...:     responses = await asyncio.gather(
   ...:         fetch('hacktoberfest', client),
   ...:         fetch('python', client),
   ...:         fetch('opensource' client),
   ...:     )

In [6]: responses
Out[6]: [<Response [200 OK]>, <Response [200 OK]>, <Response [200 OK]>]
In [7]: [r.url for r in responses]
Out[7]:
[URL('https://dev.to/t/hacktoberfest'),
 URL('https://dev.to/t/python'),
 URL('https://dev.to/t/opensource')]

(This example uses asyncio, but trio is also supported!)

HTTPX already has a lot of features built-in, but it's still in an alpha stage as some of the areas still need more work.

So enough of an intro, here's how YOU can help us build the future of Python HTTP.

How you can help

Try it out!

The most straight-forward way to help move HTTPX forward is by trying it out for yourself.

For example:

Run the sample code in the introduction to this post to see HTTPX in action!
Build a simple application that calls a public API using HTTPX.
Migrate an application that uses Requests to make HTTP calls over to HTTPX.
Use HTTPX as a test client for your web project powered by Flask, Starlette, FastAPI, etc. More on this in Calling into Python web apps.
Build a web spider that fetches web pages in parallel using HTTPX.
... And any other idea you feel like trying!

However you decide to try HTTPX out, we'd love to hear about it! Send us a link to a repo, blog post, or anything else you'd like to share via the comments section. :-)

Contribute!

As of writing this post, there's a whopping list of 10 hacktoberfest-labelled issues, all up for grabs! Expect more to come!

Below are some highlights of what's available at the moment.

Documentation

Documentation is critical to any software project. It helps new users get started quickly, and existing users find the information they need on how to use a particular feature or piece of API.

We know that the HTTPX documentation needs refining in several places. Feel free to walk through it and submit PRs for anything you think can be improved — be it typos, copy, or missing information. :-)

.post(), .put(), and .patch() functions in API docs missing 'files' parameter #409

sethmlarson posted on Sep 28, 2019

Our API docs are missing the files parameter on .post(), .patch(), and .put() for Client and AsyncClient

View on GitHub

Environment Variables docs should justify each usage case. #404

tomchristie posted on Sep 27, 2019

Our environment variable documentation should include some justification or context for each case, by doing one of the following for each envvar:

Make it clear that an environment variable is specific to HTTPX. (Probably only relevant to anything prefixed by HTTPX_)
Make it clear that an environment variable is used as a convention by various tooling, with at least one relevant hyperlink.

Eg.

SSLKEYLOGFILE could give some more context about usage - "The keylog file is designed for debugging purposes only." doesn't make it clear how or when to use it, or which debug tooling it's usful with.

SSL_CERT_FILE and SSL_CERT_DIR can include a link to some docs about their convention. Perhaps give an example of other tooling that supports them?

HTTP_PROXY, HTTPS_PROXY, ALL_PROXY can include a link to some docs about their convention. Perhaps give an example of other tooling that supports them?

View on GitHub

Host header contains userinfo component #391

sethmlarson posted on Sep 26, 2019

BaseRequest.prepare() incorrectly sets the default Host header to userinfo@example.com:123 if given the following URL: http://userinfo@example.com:123.

We should be removing the userinfo component from the authority before assigning to Host.

View on GitHub

…But most of them still need to be discovered, tracked down, and fixed!

This is also why it's super important that you give HTTPX a try. Your use case may not be one we have thought about yet, or it might be in conditions that haven't be accounted for yet.

If you find about something that doesn't look right, please open an issue and we'll look into it together!

New features

HTTPX already has a lot of features built-in, but we've got ideas on how to push things even further.

Below are some features you can tackle. If you try out HTTPX and feel like something's missing, please open an issue! We'll discuss it together and will gladly help you get started if you'd like to implement the feature.

Add support for the NO_PROXY environment variable #360

sethmlarson posted on Sep 19, 2019

The NO_PROXY environment variable is described here:

In order to accomplish this task we'd need:

Implementing NO_PROXY, a list of hosts that will bypass the proxy if found within the list.
Only enabled if trust_env=True
Documentation updates for the new environment variable.
Test cases for all code branches.

View on GitHub

[Feature Request]: raise httpx exception instead of buildin ConnectionResetError and ConnectionError #349

Trim21 posted on Sep 16, 2019

This is a feature requests instead of bug report.

I'm using requests in my web server to request upstream api. So I can handler all requests.RequestException in my middleware and return a 502 upstream error.

But after I migrated to httpx, httpx doesn't have such exception when issuing a http(s) request, it will raise httpx.ConnectTimeout, buildin ConnectionError and ConnectResetError. And last 2 exception are not only raised by httpx.

Maybe httpx should also wrap ConnectionError and other buildin connection exception?

View on GitHub

Closing thoughts

There are many ways to contribute to HTTPX, and all contributions are welcome! Once you've selected an issue, documentation improvement, feature request or bug report you'd like to work on, read the contributing guide to get started, and hack away!

Thank you so much for reading through this post, and we look forward to seeing the amazing contributions you'll make. 💖🦋

— The HTTPX maintenance team

How To Upgrade Your VuePress Site To v1.0

Florimond Manca — Sat, 15 Jun 2019 00:50:39 +0000

Originally published at blog.florimond.dev

Disclaimer: I am not a VuePress maintainer, and this is not an official guide.

VuePress is a Vue-powered static site generator. It's great for writing technical documentation, and I've been using it in production for the Bocadillo docs site since December 2018.

I recently learnt that VuePress v1.0 was now out of beta! There are many new exciting features. The one I'm most excited about is the new plugin architecture. There's also a brand new blog theme!

Anyway, this means it's time to upgrade! But… how? Well, you're in luck! Today we will explore together how to upgrade your VuePress site from 0.x to 1.x. 🚀

This post is based on my experience upgrading VuePress from 0.14 to 1.0.1 for the Bocadillo documentation.

1. Read the official announcement

Ulivz (the current maintainer of VuePress) has published a thorough blog post: Intro to VuePress 1.x. It contains a lot of interesting information on what's new in 1.x, so I highly recommend you skim through it first. Hopefully it will give you even more reasons to upgrade!

Intro to VuePress 1.x. TL;DR: This post is a speech record of… | by ULIVZ | Medium

ULIVZ ・ Jun 13, 2019 ・
Medium

2. Read the migration guide

The VuePress team also put up a migration guide on the official documentation site. I also recommend you skim through it, although we'll be covering the main changes later on.

3. Prepare your repo

Assuming your project is checked in Git, it's a good idea to create a new branch to explore the upgrading of VuePress. You never know whether things could go wrong, but more importantly you'll be able to review the changes by opening a PR.

So, go ahead and:

git checkout -b upgrade/vuepress-1.0

4. Upgrade VuePress

Enough already, it's time to upgrade! You can install VuePress 1.x using:

npm install vuepress@^1.0

It should update package.json with "vuepress": "^1.0.1" or similar, and update package-lock.json as well.

5. Use the new plugins

One of the main new features of VuePress 1.x is the plugin architecture. A lot of built-in features were refactored as plugins, so we'll need to update the VuePress configuration.

Google Analytics

Google Analytics could previously be configured via a ga option in the configuration object (see 0.x docs). It is now handled by the google-analytics plugin:

Install the plugin:

npm install @vuepress/plugin-google-analytics

Edit .vuepress/config.js with:

module.exports = {
- ga: 'UA-12345678-9'
+ plugins: [
+   [
+     '@vuepress/google-analytics',
+     { ga: 'UA-12345678-9' }
+   ]
+ ]
}

(Instructions adapted from the migration guide).

`markdown.config`

If you were using the markdown.config config option to customize the markdown-it instance, you should now use the extendMarkdown option:

Edit .vuepress/config.js with:

module.exports = {
- markdown: {
-   config(md) {
-     // ...
-   }
- }
+ extendMarkdown(md) {
+   // ...
+ }
}

(Instructions adapted from the migration guide).

All other Markdown options are still valid, though.

Service Worker

One killer feature of VuePress is the built-in Service Worker support. It allows users to access the website offline after they visited it for the first time.

Previously this was enabled via the serviceWorker option, but there is now a dedicated pwa plugin:

Install the plugin:

npm install @vuepress/plugin-pwa

Edit .vuepress/config.js with:

module.exports = {
- serviceWorker: true,
+ plugins: ['@vuepress/pwa']
}

(Instructions adapted from the migration guide).

There are many new features that come with the pwa plugin, which you can learn about in the pwa plugin docs.

6. Update styles

VuePress allows you to apply custom styles to your website, which is great to use on-brand colors and global CSS.

Previously, this could be done using .vuepress/override.styl (for custom colors) and .vuepress/style.styl (for arbitrary CSS).

VuePress 1.x now looks for these in .vuepress/styles/palette.styl and .vuepress/styles/index.styl respectively (see migration guide). Upgrading is merely a matter of moving and renaming the existing files!

Note: custom themes can now also have their own copy of these, as described in Writing a theme.

The points above were already covered in the official migration guide. The rest of this post focuses on extra steps I had to follow to successfully upgrade the Bocadillo documentation to use VuePress 1.x.

7. Update component imports

Another great feature of VuePress is that it allows you to use Vue components in Markdown, including registering your own components in .vuepress/components.

The default theme comes with a bunch of these components which I had been reusing in custom components of mine.

The file structure of the default theme changed, so I needed to fix the imports of these built-in components. For example:

- import NavLink from "@theme/NavLink.vue";
+ import NavLink from "@theme/components/NavLink.vue";

I now keep an eye on the theme-default folder to see everything that can be imported using @theme/*.

8. Fix custom layouts

The default theme in VuePress allows you to use custom layout for specific pages, i.e. replace the Vue component responsible for rendering a page with a custom one. To do this, you create a Vue component in .vuepress/components and specify the layout option in the frontmatter.

I had been using this for the layout of blog posts in the news section:

# docs/news/some-post.md
---
layout: Post
---

When I upgraded to 1.x, the layout was broken. I had various layout issues related to changes in the default CSS, but more specifically, the navbar wouldn't show up anymore!

Ugh! The layout is off, and… where is my navbar?

This is because the component given to layout now completely replaces the page (including the base Layout component which contains the navbar, sidebar, etc.). Here, look at what the docs say:

If you wish to use a completely custom component in place of the page, you can again specify the component to use using YAML front matter:

If we want the base <Layout> to be included again, we need to explicitly wrap the custom layout component in it, and use one of the available slots (see Layout.vue).

So this is how Post.vue now looks like:

<template>
  <Layout>
    <template slot="page-top">
      <!-- Page content… -->
    </template>
  </Layout>
</template>

<script>
import Layout from "@theme/layouts/Layout.vue";

export default {
  components: { Layout }
  // ...
};
</script>

Once I fixed this, the navbar was back!

9. Consider theme inheritance

VuePress comes with a built-in default theme, but previously customizing it was hard and you often had to eject. It was impractical, because you now had a lot of files which wouldn't be updated with new versions of VuePress anymore.

To fix this, VuePress 1.x comes with a brilliant new feature called Theme inheritance.

In practice, this means we can build a custom theme that extends the default theme by creating a .vuepress/theme/index.js file with:

module.exports = {
  extends: "@vuepress/default-theme"
};

and then override a particular set of components by placing them in .vuepress/theme/components. They will be available under the @theme alias just as if they came from the default theme!

Note: in addition to this, Vue itself also allows you to extend components, which can be very useful to create a custom component based on a component included in the default theme (or other component, really).

I didn't use this myself in the upgrade of the Bocadillo docs, so I can't go into more detail, but I thought this new feature was definitely worth sharing!

10. Final checks

Once you've been through the above steps, you should make sure your site runs and builds correctly before pushing it to production.

For example, you can:

Run vuepress dev to start the dev server, and then go around to see if anything looks odd or broken.
Run vuepress build to make sure that the build runs smoothly.
Serve the built website to make sure there are no final cracks. I like to use Python for this: $ python -m http.server -d path/to/.vuepress/dist.

Conclusion

This is it! Your VuePress website should now be upgraded to 1.x. Hopefully this was useful to you in upgrading from 0.x, and maybe you've learnt about the new features that landed in 1.x.

If you're interested in what the upgrade looked like for me, you can take a look at the Pull Request for the Bocadillo docs site.

Happy upgrading!

Entropic: federated JS package registry, announced at JSConfEU 2019

Florimond Manca — Wed, 05 Jun 2019 08:10:21 +0000

Hey everyone! I don't think I've seen this discussed on DEV yet. I'm not the most aware about the JS community (I'm a Pythonista at the core), but I thought it'd be interesting to have a chat together about this.

C J Silverio gave a talk at JSConfEU 2019 a few days ago:

The talk is super interesting and very well crafted — I highly recommend you watch it! But here's a modest tl;dr.

In this talk, Silverio discusses the reasons behind the Entropic project.

The first reason is that a part of the "JavaScript commons" (the package registry) is owned by a private company (npm Inc), whose sole purpose is to "take money and turn it into more money".

She makes the point that having another private company (Microsoft) run the Node package ecosystem isn't going to be the solution either.

(We previously discussed the GitHub Package Registry on DEV:)

GitHub announces GitHub Package Registry

Peter Kim Frank ・ May 10 '19

#news #npm #github #javascript

Plus, still according to her, building any other centralized package registry — even owned by a foundation like OpenJS — wouldn't work either, because the flow of package downloads in the Node.js ecosystem is humongous. It simply costs way too much to host and serve all packages from a central location.

With all the reasons in mind, at the very end of the talk Silverio introduces Entropic, a "federated package registry for anything, but mostly JavaScript". It comes with a CLI called ds (like "delta entropy" — a of bit of a physics joke).

From the words of C J Silverio herself, Entropic is still very young ("don't use it yet!"), but a handful of people are working on the Entropic project already. The main repository is here:

entropic-dev / entropic

🦝 📦 a package registry for anything, but mostly javascript 🦝 🦝 🦝

Entropic: a federated package registry for anything

A new package registry with a new CLI, designed to be easy to stand up inside your network. Entropic features an entirely new file-centric API and a content-addressable storage system that attempts to minimize the amount of data you must retrieve over a network. This file-centric approach also applies to the publication API. See the API section of the manifesto for more details about the API offered.

Entropic assumes many registries co-existing and interoperating as a part of your normal workflow. All Entropic packages are namespaced, and a full Entropic package spec also includes the hostname of its registry.

The legacy node package manager is treated as a read-only archive. You may install legacy packages through your Entropic home instance.

See docs/README.md for the manifesto.

Are you interested in contributing? Do you have some feedback to share? Come talk with us in our…

View on GitHub

So, what are your reactions to this announcement?

Attending Tech Conferences: A Beginner’s Guide

Florimond Manca — Wed, 29 May 2019 16:02:55 +0000

PyConWeb is a conference focusing on Python, web development, and related technologies. The 2019 edition took place on the 25th and 26th of May in Munich — and I was there! 🙌

Going to this conference was very special for me. It was the first time I had attended a tech conference, but also given a tech talk!

The talk I gave was about Bocadillo, async Python for web development, and open source, and lasted 30 minutes. It was a great opportunity to explain the concepts behind ASGI, highlight some key Bocadillo features, and share what I'v learnt about maintaining open source projects. All in a 10-question long "Q&A with myself" format. If you're interested, the slides are available on GitHub. ✨

Title slide!

In this post, I want to share with you what I've learnt about attending tech conferences, and give away actionable tips if you plan to attend yourself!

Basics

What is a tech conference?

A conference is an event in which professionals and hobbyists interested in a particular topic gather to learn, train, and share ideas about this topic.

There are conferences for many different types of audiences, such as researchers, business people, engineers, etc. Many different activities take place, but talks and workshops are the most common.

Even among tech conferences though, a vast amount of topics exist. Conferences are often organised around a technology (a programming language, a framework, …) or field (frontend development, security, etc.). For example, PyConWeb is "a conference about web tools in Python".

Why should I attend?

There are probably as many reasons for going to tech conferences as there are people attending them. So my answer would be that it depends on you!

Do you want to learn about a particular topic, get involved in networking (i.e. meeting and connecting with new people), find a job, or simply have a good time?

Most of the time though, I'd say it's a combination of these.

Personally, I went to PyConWeb first and foremost because I was giving a talk there (very good reason on its own, right?), but also because I like Python and web development and I was curious to know what a conference on this topic would look like. To be honest, I didn't really know what to expect, but in the end I learnt a lot and got to meet a lot of new friendly people.

Do I need to be an expert to attend conferences?

Absolutely not!

Let me reiteriate, as I think this is a very important point: everyone belongs to conferences!

There's no such thing as "qualifying" or "not qualifying" for a conference. Everyone is welcome regardless of their technical level or knowledge of the field. You're here to learn, after all!

I met people at PyConWeb 2019 who were getting started with or transitioning into Python and/or web development, and also experts in more specific domains.

The people is what makes attending conferences so uniquely fun, so join and be part of it!

How do I find conferences?

At first, it may not be obvious where you should look for the upcoming events.

One easy way to find them is to simply ask a search engine. For example, making a search for "python conferences" yields results about conferences around me (e.g. PyConFR), but also websites that list such conferences, such as python.org, pycon.org or confs.tech.

Twitter is also a great way to hear about upcoming conferences. Just search for "conf", and you'll find tons of official conferences accounts to look at!

Deciding on a conference to attend

Since there is a lot of possible conferences you can attend, which one(s) should you choose?

I'd say that the following criteria are enough to help you decide:

Topic: it may be obvious, but select a conference which you will be interested in.
Location: finding a conference near your place is probably best at first, but you may also want to take the opportunity to travel in another city or country.
Size / popularity: both smaller and larger conferences have their pro's and con's. Small to medium conferences (like PyConWeb) feel more like a family reunion, while larger conferences obviously tend to attract very notorious speakers and shiny sponsors, but can be harder to navigate.
Price: some conferences are more expansive than others — even though it does not necessarily reflect the quality of the event.
Dates: again, it may be obvious, but be sure that you will be available to attend. If you're not, no worries — conferences happen at all times of the year!

You may also want to take a look at testimonials, and what people said about the previous editions.

It's now time to dig into more specific tips about what to do before, during and after the conference!

Before the conference

Getting your ticket

So you've chosen a conference, and you're excited to go — nice! It is now time to get your ticket.

Indeed, in general conferences are not free. From what I've seen, ticket prices range from 60-70€ for the smaller conferences up to a few hundred € for the largest events.

There are actually many costs that come with organizing a conference, including paying for the venue, logistics, or security. Many conferences have sponsors though, which helps them reduce the price.

If you can't afford the price yourself, it's probably a good idea to investigate whether your employer could help, e.g. via a yearly self-learning budget.

If you're a student, many conferences have special student tickets which are often less expensive.

At PyConWeb, I paid about 45€ for the entry, which is a 90€ student ticket with a 50% speaker discount. The entry ticket was all-inclusive: talks, workshops, and food and beverages. This may not be the case for all conferences, though. In particular, some conferences may charge extra for workshops.

Logistics

Depending on the location, you may need to book accommodation and transportation, all of which come at a price as well.

PyConWeb took place in Munich, so I had to travel from Paris and book a place to stay. This represents about 200€ on transportation (train and metro fares) and about 150€ on accommodation.

There are often many possible options, and I recommend taking the time to compare prices.

It's best to figure out the logistics one week before the conference, so that you arrive at the conference as relaxed as possible, ready to enjoy your time.

Preparation

Even though you're not giving a talk, in my opinion attending a conference requires some preparation in order to get the most out of it.

A few days before the conference, it's a good idea to skim through the list of talks and workshops, and see which ones you may want to attend.

The day before the conference, the PyConWeb team had setup a mobile web app, with the schedule, speakers bios and practical information listed. I could fav the talks there, and learn about the other speakers' backgrounds. Not all conferences provide an app, but if they do I highly recommend using it!

The PyConWeb team also had a discussion chat. If the conference you go to has one as well, join and say hi! It's a good way to introduce yourself and see who else will be there.

During the conference

Now that you've arrived at the conference, what should you do? Let's take a closer look!

Talks

Talks are without a doubt the highlight of any conference. Speakers come from all around the world to talk about their experience, share ideas, and make sure the audience enjoys their time. Talks are selected by conference organisers to be as diverse and interesting as possible.

Printed schedule for the morning of May 25th at PyConWeb 2019.

All that said, attending all the talks is generally impossible.

Most conferences have more than one track of talks, which means that two or more talks take place at the same time. As a result, you'll have to decide which talk you want to attend.

Some conferences record talks and make them available online afterwards — if that's the case, you can be looser on which talks to decide to see.

So, which talks should you prioritise?

I think you should primarily attend talks about topics you don't know, or topics you know about that are presented in an original way. After all, you're most likely here to learn, right?

Also, and this may be counter-intuitive, I think you should go to at least one talk that you think won't be interesting to you, or that doesn't sound very sexy. You may be surprised about what you discover, and will help fill what could have otherwise been an empty room (which is probably a speaker's worst fear).

During a talk, I highly recommend you take notes about key points or questions you may want to ask. Personally, it helps me memorise key learning points better. If you do so, prefer using pen and paper instead of your laptop, as it is less intrusive for both the speaker and other attendees.

If the talk turns out to be a bit boring, refrain from zoning out and starting browsing your phone. It can be disheartening to the speaker, make them even less at ease and worsen the experience for everyone. Instead, take it as an opportunity to observe what makes the talk boring.

After the talk, you can (should?) reach out to the speaker if you enjoyed the talk, and tell them why you found it interesting. Speakers are always up for feedback and ways they can improve! Obviously, you can also ask them anything unclear or you'd like more information about.

In the hallway, you can also discuss the talk you've just seen with other attendees, or ask those who went to other talks what they thought about it.

Other activities

Although talks are the main activity at most conferences, there are many other activities going on.

One of these activities is workshops. They're typically longer than talks (2 to 3 hours) and consist in a hands-on session about a tool or technology.

For example, I decided to attend a workshop by Ben Parsons about real-time asynchronous messaging with Matrix, and I enjoyed it a lot!

Another activity organised at PyCon events is lightning talks. They consist of very short 5-minutes talks which anyone can give, and they're often very fun to attend.

There may also be sponsor booths in the hall. It can be a good idea to go and say hi, e.g. if you're interested in career opportunities or just to get some sponsor swag (stickers, bookmarks, etc).

Lastly, be sure to attend the conference opening and conference announcements. They're definitely not as boring as they seem (bad jokes inside!), and organizers may announce fun special contests or events which you can participate in. For example, PyConWeb had a selfie contest going on, as well as an open brainstorming session for the name of The Cat, the conference's mascot (which was finally elected to be CharField!).

PyConWeb 2019 swag, featuring CharField himself. 🐈

Social media

If you're into social media, conferences are an incredible source of inspiration for posts and Tweets. As a matter of fact, people are typically more active than usual during a conference, so don't be shy!

During my talk, some attendees tweeted — I had placed my Twitter handle in the footer of each slide — and it was great to see, because it probably meant that they thought the talk was worth sharing. I also tweeted a lot about other people's talks!

By the way, it's definitely okay to get your phone out, take a picture, and publish something about the talk. But be sure to do it in a way that doesn't prevent other attendees from enjoying the talk. Try to do it quickly, or you risk zoning out and missing on some interesting content. To avoid this entirely, you can also take a picture and write the post after the talk.

Networking

Attending a conference is a fantastic opportunity to meet and connect with new people. In fact, it's probably the main motivation for many attendees. I was very impressed myself about how many new people I met at PyConWeb!

Meeting new people turned out to be very easy:

Identify someone or a group of people you don't know yet.
Say hi, tell your name, shake hands.
Strike a conversation!

I think this was even easier since I came by myself at the conference. I know I would've been tempted to stick with my team or colleagues if I came with other people.

Once you're in, remember to encourage the PacMan shape. In other words, leave a small gap open instead of forming a closed circle. It shows to people looking for a group that they are welcome to join.

If you're unsure about what to talk about, there are actually many possible topics. For example, you can ask other people:

What motivated them to come to the conference.
How they relate to the topic of the conference. At PyConWeb, I enjoyed asking other people what web frameworks they used, or what they used Python for and how they got into it.
Where they're traveling from, how long they plan to stay.
Which conferences or events they've been to in the past.
…

The list is endless! Just listen and bounce off the answers. Also, be ready to answer those questions yourself!

Another great thing about conferences is that a lot of them are international. I obviously met attendees who were from Munich or Germany, but also people from Canada, Portugal, the UK, Singapore, India, and Slovenia!

Note that if you're a local but the conference is international, the language for talks and hallway chit-chat will probably be English.

As a final note, be aware that networking and socialising can be exhausting, especially if, like me, you are a bit of an introvert.

Actually, you shouldn't feel obliged to be constantly talking to people. It's okay to take a break, sit somewhere alone, and recharge your batteries.

Everyone has their limits, so remember to take care of yourself so that the conference remains a memorable experience.

Behavior

A lot of conferences have a code of conduct, which is a short document that frames how people should behave during the conference.

The PyConWeb code of conduct is a few pages long, but the tl;dr which was given to us at the conference opening was:

We're all adults. Be nice to people, and expect the same in return.

After all, a conference is a professional event, which means you're supposed to behave as a professional.

That doesn't mean it has to be formal — you can totally have informal and fun discussions with speakers and other attendees. The Saturday night party was a good example of this!

But while doing so, be sure to stay respectful, open-minded and inclusive.

After the conference

When the official program is over, there are still many ways you can keep your conference experience going.

First and foremost, say thanks to the organizers and the crew! They worked hard to organize the conference and make sure everything goes smoothly. You know they succeeded in doing so when you can't even notice the work they've done before or during the conference. So reach out to them, and say thanks!

For example: did you like the selection of talks? Was the schedule well-managed? Did you enjoy the extra activities? Tell them! You can do it in many ways: in person, on social media, in the chat, etc.

Another thing you can do is reach out to people you met during the conference. Thank them for your discussions, offer to meet again sometime, and follow up on conversations you've had. A nice time to do this is after the final round of talks.

If you've exchanged contact info (Twitter, LinkedIn, phone, email…) you can also stay in touch after everyone went home.

Outside of extended networking, there are things you can do on your own.

For example, try to apply one or two things you've learnt during the conference to a practical project. If you do, remember to share it with the speaker!

For example, I created fetch-metadata-asgi, a PoC ASGI middleware for the upcoming Fetch Metadata specification. This was a follow up on the talk given by Lukas Weichselbaum about new web security features.

Lastly, it's definitely a good idea to take the time to reflect on your experience. If you have a blog, you can write a blog post and share it with attendees and organizers. Examples from PyConWeb 2019:

This very blog post!
Came for the talks, stayed for the raffles by Jürgen Gmach — great summary with links to slides and source code for the talks.
PyConWeb 2019 by Jesse Stippel.

Wrapping up

PyConWeb 2019 was full of first times for me: first time in Germany and in Munich, first time attending a tech conference, and first time giving a talk at a conference!

I genuinely enjoyed it. It made me consider getting more into public speaking, and attending more conferences — from the Python community in particular. I just learnt that PyConFR 2019 was confirmed for October, so that might be my next stop!

I will be writing up my experience of speaking at PyConWeb in an upcoming blog post, in a similar format: "Speaking at Tech Conferences: A Beginner's Guide". Stay tuned, and in the meantime you can view the slides for my talk on GitHub.

Building a real-time chatbot server in Python with WebSocket, ChatterBot and Bocadillo

Florimond Manca — Tue, 19 Mar 2019 09:15:18 +0000

This post is an adaptation of the official Bocadillo tutorial.

Hi everyone! Today's post is going to be a little special. Some of you may remember this post I wrote some months back: How I Built A Python Web Framework And Became An Open Source Maintainer.

Since then, I kept working on Bocadillo, and it's been a great time! In fact, I learnt just last week that I will be flying to Munich at the end of May to give a talk at PyConWeb 2019! This will be my first conference and talk ever, so needless to say that I'm SUPER EXCITED! 🙌🤩

Another great news is that Bocadillo v0.13 has just been released:

// Detect dark theme var iframe = document.getElementById('tweet-1107029038923743232-902'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1107029038923743232&theme=dark" }

With all these good vibes in the air, I finally decided to go ahead and publish a thorough tutorial.

Without further ado, here's the plot: we're going to try and build a chatbot server!

Bocadillo has many features built-in, so this is a great opportunity to go through some aspects of building web services with Bocadillo.

In this tutorial, you'll get to play with chatbots but also learn how to:

Use WebSocket to handle multiple connections in real-time.
Create REST endpoints.
Use providers to inject reusable resources into views.
Test a Bocadillo application using pytest.

Still wondering how we'll build something seemingly as complex as a chatbot? Well, you probably know that Python has a gigantic data science ecosystem. I would've bet actual money there would be a chatbot framework somewhere out there.

Turns out — there was! After some research, I stumbled upon ChatterBot. It looks pretty solid and popular, so we'll use it to build Diego, a friendly conversational agent. Don't worry, this won't require any background in data science nor chatbot technology!

Sounds exciting? Alright, let's dive in! 🙌

Setting up the project

First things first: let's set up our project:

Open up a terminal, and create an empty directory somewhere on your computer, then cd to it:

mkdir ~/dev/bocadillo-chatbot
cd ~/dev/bocadillo-chatbot

Install Bocadillo and ChatterBot. We're using pipenv to install dependencies here, but you can also use plain ol' pip + virtualenv too.

# Note: pytz is required by chatterbot.
pipenv install bocadillo chatterbot pytz

Create an empty app.py script. This is where we'll create the application later on:

touch app.py

We should now have the following directory structure:

$ tree
.
├── Pipfile
├── Pipfile.lock
└── app.py

Bootstrapping the application

Now, let's write the app skeleton in app.py. Hang tight — first decent bit of code incoming:

# app.py
from bocadillo import App

app = App()

if __name__ == "__main__":
    app.run()

If you've ever worked with Flask or, well, nearly any Python web framework really, this should look oddly familiar. Nearly boring. Who cares? It works! Check for yourself:

python app.py

If you go to http://localhost:8000 and get a 404 Not Found response, you're all good! Enter Ctrl+C in your terminal to stop the server.

Writing the WebSocket endpoint

We're now ready to get to the meat of it! The first thing we'll build is the WebSocket endpoint.

If you're not familiar with WebSocket, don't worry — here's a 10-word summary: it allows a server and a client to exchange messages in a bidirectional way. It's good old sockets reinvented for the web.

Due to their bidirectional nature, they're very suitable for the kind of application we're building here — some sort of conversation between a client and a server (i.e. our chatbot).

If you're interested in learning more about WebSockets in Python, I strongly recommend this talk: A beginner's guide to WebSockets.

Alright, so we won't plug the chatbot in yet. Instead, let's make the server send back any message it receives — a behavior also known as an "echo" endpoint.

Add the following between the app object declaration and the app.run() block in app.py:

@app.websocket_route("/conversation")
async def converse(ws):
    async for message in ws:
        await ws.send(message)

A few minimal explanations here, for the curious:

This defines a WebSocket endpoint which will be accessible at the ws://localhost:8000/conversation location.
The async for message in ws: line iterates over messages received over the WebSocket.
Lastly, await ws.send(message) sends the received message as-is back to the client.

Trying out the WebSocket endpoint

How about we try this out by creating a WebSocket client? Fear not — we won't need to write any JavaScript. We'll stick to Python and use the websockets library, which comes installed with Bocadillo.

Create a client.py file and paste the following code there. What it does is connect to the WebSocket endpoint and run a simple REPL:

# client.py
import asyncio
from contextlib import suppress
import websockets

async def client(url: str):
    async with websockets.connect(url) as websocket:
        while True:
            message = input("> ")
            await websocket.send(message)
            response = await websocket.recv()
            print(response)

with suppress(KeyboardInterrupt):
    # See asyncio docs for the Python 3.6 equivalent to .run().
    asyncio.run(client("ws://localhost:8000/conversation"))

Run the server-side application with python app.py and, in a separate terminal, start the client.py script. You should be greeted with a > prompt. If so, start chatting!

$ python client.py
> Hi!
Hi!
> Is there anyone here?
Is there anyone here?
>

Pretty cool, isn't it? 🤓

Type Ctrl+C to exit the session and close the WebSocket connection.

Hello, Diego!

Now that we're able to make the server and a client communicate, how about we replace the echo implementation with an actual, intelligent and friendly chatbot?

This is where ChatterBot comes in! We'll create a chatbot rightfully named Diego — a chatbot speaking the asynchronous salsa. 🕺

Go ahead and create a chatbot.py file, and add Diego in there:

# chatbot.py
from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTrainer

diego = ChatBot("Diego")

trainer = ChatterBotCorpusTrainer(diego)
trainer.train(
    "chatterbot.corpus.english.greetings",
    "chatterbot.corpus.english.conversations"
)

(ChatterBot's chatbots are quite dumb out of the box, so the code above trains Diego on an English corpus to make him a bit smarter.)

At this point, you can try out the chatbot in a Python interpreter:

$ python
>>> from chatbot import diego  # Be patient — this may take a few seconds to load!
>>> diego.get_response("Hi, there!")
<Statement text:There should be one-- and preferably only one --obvious way to do it.>

(Hmm. Interesting response! 🐍)

Let's now plug Diego into the WebSocket endpoint: each time we receive a new message, we'll give it to Diego and send his response back.

# app.py
from chatbot import diego

...

@app.websocket_route("/conversation")
async def converse(ws):
    async for message in ws:
        response = diego.get_response(message)
        await ws.send(str(response))

If you run the server/client setup from earlier, you can now see that Diego converses with us over the WebSocket!

$ python client.py
> Hi there!
I am a chat bot. I am the original chat bot. Did you know that I am incapable of error?
> Where are you?
I am on the Internet.
>

Looks like Diego is a jokester. 😉

Refactoring the chatbot as a provider

Clients are now able to chat with Diego over a WebSocket connection. That's great!

However, there are a few non-functional issues with our current setup:

Loading Diego is quite expensive: it takes about ten seconds on a regular laptop.
Because of the import at the top of the script, we'd load Diego every time we import the app module. Not great!
Diego is injected as a global dependency into the WebSocket endpoint: we can't swap it with another implementation (especially useful during tests), and it's not immediately clear that the endpoint depends on it at first sight.

If you think about it, Diego is a resource — ideally, it should only be made available to the WebSocket endpoint at the time of processing a connection request.

So, there must be a better way… and there is: providers. ✨

Providers are a unique feature of Bocadillo. They were inspired by pytest fixtures and offer an elegant, modular and flexible way to manage and inject resources into web views.

Let's use them to fix the code, shall we?

First, let's move Diego to a providerconf.py script:

# providerconf.py
from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTrainer
from bocadillo import provider

@provider(scope="app")
def diego():
    diego = ChatBot("Diego")

    trainer = ChatterBotCorpusTrainer(diego)
    trainer.train(
        "chatterbot.corpus.english.greetings",
        "chatterbot.corpus.english.conversations",
    )

    return diego

The code above declares a diego provider which we can now inject into the WebSocket view. All we have to do is declare it as a parameter to the view.

Let's do just that by updating the app.py script. Here, you get it in full:

from bocadillo import App

app = App()

@app.websocket_route("/conversation")
async def converse(ws, diego):  # <-- 👋, Diego!
    async for message in ws:
        response = diego.get_response(message)
        await ws.send(str(response))

if __name__ == "__main__":
    app.run()

No imports required — Diego will automagically get injected in the WebSocket view when processing the WebSocket connection request. ✨

Alright, ready to try things out?

Run the app.py script. You should see additional logs corresponding to Bocadillo setting up Diego on startup:

$ python app.py
INFO: Started server process [29843]
INFO: Waiting for application startup.
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /Users/Florimond/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package punkt to /Users/Florimond/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to
[nltk_data]     /Users/Florimond/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
Training greetings.yml: [####################] 100%
Training conversations.yml: [####################] 100%
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

Run the client.py script, and start chatting! You shouldn't see any difference from before. In particular, Diego responds just as fast.

$ python client.py
> Hello!
Hi
> I would like to order a sandwich
Yes it is.
>

There you go! Beautiful, modular and flexible dependency injection with Bocadillo providers.

Keeping track of clients

Let's go one step further. True, we have quite elegantly implemented conversation with a chatbot over WebSocket. Now, how about we keep track of how many clients are currently talking to the chatbot?

If you were wondering — yes, we can implement this with providers too!

Let's add a clients provider to providerconf.py:

# providerconf.py
from bocadillo import provider

...

@provider(scope="app")
def clients():
    return set()

Now, let's add another provider which returns a context manager that takes care of registering the ws connection to the set of clients. FYI, this is an example of a factory provider, but you don't really need to understand the whole code at this point.

# providerconf.py
from contextlib import contextmanager
from bocadillo import provider
...

@provider
def save_client(clients):
    @contextmanager
    def _register(ws):
        clients.add(ws)
        try:
            yield ws
        finally:
            clients.remove(ws)

    return _register

In the WebSocket view, use the new save_client provider to register the WebSocket client:

# app.py

...

@app.websocket_route("/conversation")
async def converse(ws, diego, save_client):
    with save_client(ws):
        async for message in ws:
            response = diego.get_response(message)
            await ws.send(str(response))

That's it! While the client is chatting with Diego, it will be present in the set of clients.

How about we do something with this information?

Exposing client count via a REST endpoint

As a final feature, let's step aside from WebSocket for a moment and go back to the good old HTTP protocol. We'll create a simple REST endpoint to view the number of currently connected clients.

Go back to app.py and add the following code:

# app.py

...

@app.route("/client-count")
async def client_count(req, res, clients):
    res.media = {"count": len(clients)}

Again, if you've ever worked with Flask or Falcon, this code shouldn't come as a surprise. All we do here is send the number of clients (obtained from the clients provider) in a JSON response.

Go ahead! Run python app.py and run a few python client.py instances. Check out how many clients are connected by opening http://localhost:8000/client-count in a web browser. Press Ctrl+C for one of the clients, and see the client count go down!

Did it work? Congrats! ✨

Testing

We're mostly done in terms of the features we wanted to cover together. I've got some ideas you can explore as exercises, of course, but before getting to that let's write some tests.

One of Bocadillo's design principles is to make it easy to write high-quality applications. As such, Bocadillo has all the tools built-in to write tests for this chatbot server.

You can write those with your favorite test framework. We'll choose pytest for the purpose of this tutorial. Let's install it first:

pipenv install --dev pytest

Now, let's setup our testing environment. We'll write a pytest fixture that sets up a test client. The test client exposes a Requests-like API as well as helpers to test WebSocket endpoints. Besides, we don't actually need to test the chatbot here, so we'll override the diego provider with an "echo" mock — this will have the nice side effect of greatly speeding up the tests.

So, go ahead and create a conftest.py script, and place the following in there:

# conftest.py
import pytest
from bocadillo import provider
from bocadillo.testing import create_client

from app import app

@provider
def diego():
    class EchoDiego:
        def get_response(self, query):
            return query

    return EchoDiego()

@pytest.fixture
def client():
    return create_client(app)

Now is the time to write some tests! Create a test_app.py file at the project root directory:

touch test_app.py

First, let's test that we can connect to the WebSocket endpoint, and that we get a response from Diego if we send a message:

# test_app.py

def test_connect_and_converse(client):
    with client.websocket_connect("/conversation") as ws:
        ws.send_text("Hello!")
        assert ws.receive_text() == "Hello!"

Now, let's test the incrementation of the client counter when clients connect to the WebSocket endpoint:

# test_app.py
...

def test_client_count(client):
    assert client.get("/client-count").json() == {"count": 0}

    with client.websocket_connect("/conversation"):
        assert client.get("/client-count").json() == {"count": 1}

    assert client.get("/client-count").json() == {"count": 0}

Run these tests using:

pytest

And, well, guess what?

==================== test session starts =====================
platform darwin -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
rootdir: ..., inifile: pytest.ini
collected 2 items

test_app.py ..                                         [100%]

================== 2 passed in 0.08 seconds ==================

Tests pass! ✅🎉

Wrapping up

If you've made it so far — congratulations! You've just built a chatbot server powered by WebSocket, ChatterBot and Bocadillo.

In this article, we've seen how to:

Setup a Bocadillo project.
Write a WebSocket endpoint.
Write an HTTP endpoint.
Use providers to decouple resources and their consumers.
Test WebSocket and HTTP endpoints.

The complete code for this tutorial is available on the Bocadillo repo on GitHub: get the code! All in all, the server and providerconf.py only add up to about 60 lines of code — pretty good bang for the buck!

Obviously, we've only scratched the surface of what you can do with Bocadillo. The goal of this tutorial was to take you through the steps of building a Minimum Meaningful Application.

You can iterate upon this chatbot server we've built together very easily. I'd be interested to see what you come up with!

Want to challenge yourself? Here are a few ideas:

Add a home page rendered with templates. The web browser should connect to the chatbot server via a JavaScript program. You'll probably also need to serve static files to achieve this.
Train Diego to answers questions like "How many people are you talking to currently?"
Currently, all clients talk to the same instance of Diego. Yet, it would be nice if each client had their own Diego to ensure a bespoke conversation. You may want to investigate cookie-based sessions and factory providers to implement this behavior.

I hope you enjoyed this tutorial! If you'd like to support the project, be sure to star the repo. If you don't want to miss on new releases and announcements, feel free to follow @bocadillopy on Twitter!

Consuming APIs in Angular: Displaying Data In Components

Florimond Manca — Fri, 01 Mar 2019 10:35:53 +0000

Welcome back! This article is a follow up to a previous article:

Consuming APIs in Angular – The Model-Adapter Pattern

Florimond Manca ・ Sep 5 '18 ・ 8 min read

#webdev #angular #typescript #devtips

If you haven't read it yet — go check it out! This post will be referencing it quite often.

I recently had a question from @milankovach about how the CourseService could be used in a component, say, to display the list of courses.

Milan Kovac • Feb 22 '19

Hi Florimond,
this is great but can you add short description (maybe here in comment section) how to use your services in other components for the n00bs? Thanks! :)

This is exactly what we'll cover in this beginner-friendly article.

This post should be helpful to anyone wondering how to retrieve and display data fetched from an external API. 😊

Quick refresher

In the original post, we discussed a design pattern that I use to standardise how my Angular apps communicate with REST APIs: the Model-Adapter pattern.

Given the GET /courses API endpoint, we built a Course model, a CourseAdapter and a CourseService that can help us fetch the list of courses from the API.

Here's what the project structure looks like for now:

src/app
├── app.component.css
├── app.component.html
├── app.component.ts
├── app.module.ts
└── core
    ├── course.model.ts
    └── course.service.ts

The goal here is to build a CourseListComponent that fetches the list of courses using the CourseService, and displays them in the form of a simple unordered list.

Let's build!

Generating the `CourseListComponent`

Okay, let's get started. First, we will generate the component using the Angular CLI:

ng generate component CourseList

The TypeScript (TS), HTML and CSS files for the component will be generated under src/app/course-list/. Here's what the TS file looks like so far:

// course-list/course-list.component.ts
import { Component, OnInit } from '@angular/core';

@Component({
  selector: 'app-course-list',
  templateUrl: './course-list.component.html',
  styleUrls: ['./course-list.component.css']
})
export class CourseListComponent implements OnInit {

  constructor() { }

  ngOnInit() {
  }

}

Setting up attributes and templates

As a first step, let's add an empty list of courses on the component:

  // course-list/course-list.component.ts
  import { Component, OnInit } from '@angular/core';

  @Component({
    selector: 'app-course-list',
    templateUrl: './course-list.component.html',
    styleUrls: ['./course-list.component.css']
  })
  export class CourseListComponent implements OnInit {

+   courses: Courses[];

-   constructor() { }
+   constructor() {
+     this.courses = [];
+   }

    ngOnInit() {
    }

}

Next, let's set up the template. Nothing fancy here, we're just using an *ngFor to display each course in its own list item, as well as the DatePipe to format the date.

<!-- course-list/course-list.component.html -->
<ul>
  <li *ngFor="let course of courses">
    <p>
      {{ course.name }} • {{ course.code }} • Created
      {{ course.created | date }}
    </p>
  </li>
</ul>

While we're at it, let's update the AppComponent's template to display the list of courses:

<!-- app.component.html -->
<h1>Courses</h1>

<app-course-list></app-course-list>

Alright! Let's fire up the browser, and we should be greeted with the "Courses" title and… an empty list. Why? Well, we haven't fetched any course yet!

Implementing the API endpoint

Before we go and plug the CourseService in, remember that for now it refers to https://api.myapp.com/courses — and that API doesn't exist!

That said, it would be nice to test the CourseService against a live server, wouldn't it?

So, let's build a quick backend API for this exact purpose. I'll be using Python and Bocadillo (shameless plug here: I am the maintainer of Bocadillo!) to provide the GET /courses endpoint we need to have access to from the browser.

You don't need to know about Python nor understand the code below, but I'm displaying it here for those interested:

# app.py
# Install: `pip install bocadillo`
from bocadillo import App

app = App(
    enable_cors=True,
    cors_config={"allow_origins": ["*"], "allow_methods": ["*"]},
)

COURSES = [
    {
        "id": 1,
        "code": "adv-maths",
        "name": "Advanced Mathematics",
        "created": "2018-08-14T12:09:45",
    },
    {
        "id": 2,
        "code": "cs1",
        "name": "Computer Science I",
        "created": "2018-06-12T18:34:16",
    },
]

@app.route("/courses")
async def courses_list(req, res):
    res.media = COURSES

if __name__ == "__main__":
    app.run()

As you can see, the GET /courses endpoint will just return a hardcoded list of courses.

We can fire the API up in a terminal using $ python app.py, and leave it running.

Integrating the API with the `CourseService`

As a last integration step, we need to update the URL which the CourseService uses to fetch the courses:

// core/course.service.ts
// ...

@Injectable({
  providedIn: 'root'
})
export class CourseService {

  private apiUrl = 'http://localhost:8000/courses';

  constructor(
    private http: HttpClient,
    private adapter: CourseAdapter,
  ) { }

  list(): Observable<Course[]> {
    return this.http.get(this.apiUrl).pipe(
      // Adapt each item in the raw data array
      map((data: any[]) => data.map(item => this.adapter.adapt(item))),
    );
  }
}

Fetching courses with the `CourseService`

We're now ready to plug the CourseService into the CourseListComponent!

Here are the steps we'll take to do it:

Import the service.
Inject it in the component using Angular's dependency injection.
In the component's ngOnInit() method, get the RxJS observable to the list of course and subscribe to it.
Store the fetched list of course on the component so that it gets rendered in the template.

Wondering how that translates into code? Take a look below — I added landmarks for each of the steps above:

import { Component, OnInit } from "@angular/core";
import { Course } from "../core/course.model";
// (1) Import
import { CourseService } from "../core/course.service";

@Component({
  selector: "app-course-list",
  templateUrl: "./course-list.component.html",
  styleUrls: ["./course-list.component.css"]
})
export class CourseListComponent implements OnInit {
  courses: Course[];

  // (2) Inject
  constructor(private courseService: CourseService) {
    this.courses = [];
  }

  ngOnInit() {
    // (3) Subscribe
    this.courseService.list().subscribe((courses: Course[]) => {
      // (4) Store
      this.courses = courses;
    });
  }
}

Celebrate

There you go! If we open the browser at http://localhost:8000, we see the list of courses displayed in sexy Times New Roman.

Styling is, indeed, out of the scope of this blog post.

Wrapping up

Alright, let's see what we've achieved here:

We generated the CourseListComponent using Angular CLI.
We set up the component's courses attribute and its [template].
We used Python and Bocadillo to build the API endpoint to test our component against.
We used the CourseService and RxJS to fetch the list of course.

In fact, this is quite a typical workflow for me when I build web apps using Angular — I start by stubbing out the components, then implement the backend endpoints I need, and integrate them with the services to finally display the data.

If you're interested in the code, I uploaded it to a GitHub repo: ng-courses.

Stay in touch!

If you enjoyed this post, you can find me on Twitter for updates, announcements and news. 🐤

How I Built A Python Web Framework And Became An Open Source Maintainer

Florimond Manca — Sun, 23 Dec 2018 19:21:42 +0000

This post was originally published at blog.florimond.dev.

It's been a while since I've written a blog post. Nearly two months, actually. So, where have I been?

For one, the final year of my engineering studies has been taking up more of my time than I thought it would. I'm not complaining, though — I've been learning a ton of interesting and useful things, both theoretical and practical. Plus, I want to take the opportunity to live this final year fully.

The other side of the story is — I've been working on building Bocadillo, an open source asynchronous Python web framework. The adventure, which started as a way for me to learn about the internals of a web framework, has been thrilling so far.

This means that, because a day is only 24 hours, I had to put blogging aside for a bit. But now that I'm on holiday, I can finally take the time to reflect on what's been going on. 🥳

In this blog post, I want to put on paper my thoughts about the whole process of building a web framework and especially launching an open source project. I've already learnt a lot in the process, from programming techniques to project management and development tooling, so I wanted to share my experience with you!

What's on the menu?

What is Bocadillo?: introducing Bocadillo in order to set up some context.
The story behind it all: I'll tell you the story of how it became my first open source project!
Tips to start your own open source project: a series tips I gathered from experience.

First things first — let me introduce you to Bocadillo! (This will be short and sweet, I promise.)

What is Bocadillo?

bocadilloproject / bocadillo

(UNMAINTAINED) Fast, scalable and real-time capable web APIs for everyone

NOTICE

Bocadillo is now UNMAINTAINED. This repository should be archived soon. We recommend users to migrate to other well-supported alternatives, such as Starlette or FastAPI. Please see #334 for more information.

Documentation: https://bocadilloproject.github.io

Bocadillo is a Python async web framework that makes building performant and highly concurrent web APIs fun and accessible to everyone.

Requirements

Python 3.6+

Installation

pip install bocadillo

Example

from bocadillo import App, configure

app = App()
configure(app)

@app.route("/")
async def index(req, res):
    res.json = {"hello": "world"}

Save this as app.py, then start a uvicorn server (hot reload enabled!):

uvicorn app:app --reload

Say hello!

$ curl http://localhost:8000
{"hello": "world"}

Ready to dive in? Visit the documentation site.

Changelog

All changes to Bocadillo are recorded in the…

View on GitHub

Elevator pitch

Bocadillo is a modern Python web framework that provides a sane toolkit for building performant web applications and services using asynchronous programming. It is compatible with Python 3.6+ and MIT-licensed.

Want to get started? Read the documentation!

Key figures

Bocadillo's initial commit was on November 3rd, 2018. As of December 21st, i.e. a month and a half later, here's where Bocadillo stands:

500+ commits and 31 stars on the bocadilloproject/bocadillo repo — feel free to add yours!
10 releases to PyPI — latest being v0.7
8k+ downloads as measured by PePy
About 30 pages of docs!
2 open source contributors — and many others welcome! 🥳

These figures are of course very modest, but I'm already very happy with them. Of course, if you want to support Bocadillo, you can:

Also, if you've already tried Bocadillo out, please get in touch via Twitter or the Gitter chat room — I'd LOVE to hear from you!

Philosophy

In terms of philosophy, Bocadillo is beginner-friendly, but it aims at giving power-users the flexibility they need. It focuses on developer experience while encouraging best practices.

Besides, Bocadillo is not meant to be minimalist (but not a mastodon either). The idea is to include a carefully chosen set of included batteries, with sensible defaults, so that you can solve common problems and be productive right away.

Goals

My goal is that people embrace the new possibilities of async Python and that Bocadillo becomes a tool that helps people solve real-world problems more easily and efficiently.

There's still a lot of work ahead to get there, but the marathon has started!

Speaking of async Python, let me address a question you may have already asked yourself…

What with async?

Generally, a web app instance spends a lot (if not most) of its request processing time waiting for I/O to complete — API calls, database queries, filesystem operations. etc. Most of these operations are blocking, which typically limits performance if multiple clients are requesting the server.

The idea with async frameworks like Bocadillo is to build apps that do not block on I/O operations. To achieve this, we leverage asynchronous programming and recent additions to the Python language such as asyncio and async/await — available from Python 3.4+ and 3.6+ respectively. This allows us to consider processing a request as a task which is "scheduled" to run in a near future, i.e. when the CPU is avaiable.

As a result, beyond making better use of the CPU, this architecture has a very interesting advantage — we can now handle multiple requests concurrently.

(Note: I didn't write in parallel, as async still uses a single thread. Concurrency is not parallelism.)

This property ultimately results in more stable throughput and performance as the number of concurrent clients increases. From my own (yet to be published) benchmarks, Bocadillo keeps a steady processing rate whether it talks to 10 or 10,000 clients. On the other hand, "sync" frameworks like Flask or Django show a significant drop in reqs/sec in high concurrency settings.

Eager for more on Python asynchronous programming? Here are a few talks I recommend, perhaps to be watched in this order:

Asynchronous Python for the Complete Beginner, Miguel Grinberg, Pycon 2017.
Async/await in Python 3.5 and why it is awesome, Yuri Selivanov, EuroPython 2016.
Fear and Awaiting in Async: A Savage Journey to the Heart of the Coroutine Dream, David Beazley, PyOhio 2016.

Implementation

Bocadillo is built on Uvicorn, the lightning-fast ASGI web server, and Starlette, a handy ASGI toolkit. Both were created by Tom Christie, a core contributor to the Django REST Framework (among other things).

Note: ASGI is the asynchronous equivalent to WSGI, i.e. a specification for how web servers should communicate with asynchronous Python web applications.

Features

What Bocadillo already has: requests, responses, views (function-based and classed-based), routes and route parameters, media types, redirections, templates, static files, background tasks, CORS, HSTS, GZip, "recipes" (a.k.a. blueprints), middleware, hooks, and even a CLI. There are more in the release pipeline!

One thing that Bocadillo does not have (yet), though, is a database layer. Most web apps or APIs I've built needed to persist data in some way, so I believe this (or at least an official recommendation for how to integrate an async database layer such as Tortoise ORM) should land into the framework at some point.

Hello, world!

Let's finish with the traditional "Hello, World" script!

# api.py
from bocadillo import API

api = API()

@api.route("/")
async def hello(req, res):
    res.text = "Hello, World!"

if __name__ == "__main__":
    api.run()

The story behind it all

Alright, enough of pitching Bocadillo! Now that you know what it is, I want to share with you the story that led me to write this very blog post.

How and why did it begin? What were some of the most meaningful events? Let's figure this out.

A learn-by-doing project

Bocadillo started as a way for me to learn more about the internals of a web framework. I wanted to get behind the scenes after nearly 2 years of using various Python and JS web frameworks. I wanted to know how it actually all worked.

To be clear, Bocadillo didn't start with a very detailed plan. Heck, I didn't even think about whether there was a need for an(other) Python async web framework. I wanted to learn more than anything else.

Let's reinvent the wheel, and release it ASAP

So there I was, on the 3rd of November, implementing features so common it felt like reinventing the wheel. These were features like requests, responses, views, routes or the application server. "Hundreds of web frameworks out there already solved these problems before", I thought…

But I didn't really care. As @funkybob kindly twitted to me:

Reinventing the wheel is an awesome way to learn… and sometimes what you learn is just how much your existing frameworks are doing for you.

A solid argument in favour of reinventing the wheel which made me realize how much Django is an absolute massive beast.

Anyway, this initial endeavour led me to release v0.1 on PyPI on the 4th of November. Just two days after the initial commit, people could already pip install bocadillo and build a minimal async web app. (Who said Python packaging was a pain? 🐍)

First signs of potential

After v0.1 was released, I carried on with implementing more features such as new types of responses or error handling.

On November 6th, v0.2.1 was out. That's when I began to realize that Bocadillo was a good candidate for my first full-blown open source project. The idea seemed appealing to me, so I went for it!

At that point, I hadn't disclosed anything about Bocadillo yet, not even to friends, so I wanted to make a first announcement. I chose to do it on Twitter.

Because Bocadillo's initial code design and implementation took heavy inspiration from Responder, Kenneth Reitz's own async framework, I decided to give a shoutout.

The first announcement about Bocadillo on Twitter, and Kenneth Reitz's answer. ❣️

Kenneth's answer and the forthcoming reactions after he retweeted the announcement made me think that Bocadillo actually had potential.

In just a day, the repo got 20 stars (a personal record already!) and, although it may look trivial, I thought it was really cool.

Psst: if you want to help get Bocadillo known, you can star it too and spread the news!

So, after v0.2 was released, I felt motivated to keep working on Bocadillo, and add more features.

Up until I realized something…

Where are the docs? Like, real docs?

It was clear for me: I wanted Bocadillo to be my first open source project. I wanted to take it seriously in order to learn as much as I can from the process.

So, right from the start, I wrote an informative README, curated a CHANGELOG (with the help of keep a changelog) and added CONTRIBUTING guidelines. As more and more releases went out between November 6th and November 18th, I updated the changelog and documented new features in the repo's README.

Quickly though, this became impractical. The README was growing in size and it became hard to navigate, even with a table of contents.

The table of contents for the repo on November 17th. See that "Usage" section growing to an astronomical size?

That's when I realized I needed proper documentation.

If you think about it, good documentation is a sine qua non condition to having people use what you've built. And that lengthy README was not good documentation considering the size that Bocadillo was heading at.

Then, it hit me — a lot of large-scale open source tools, libraries or frameworks I use and love have a documentation site.

This is what led me to release v0.5 on November 18th with a major addition: a brand new docs site, which I built with VuePress and hosted on GitHub Pages.

Bocadillo's documentation site home page (end of November, 2018).

On the necessity of good documentation — Joe Mancuso, creator of the Masonite framework, once shared with me this great piece of advice:

If it's not documented, it doesn't exist.

That's why I'm taking documentation very seriously and strive to make it as good as I can — and so should you with every project you're working on.

Then, while building the docs site, I made what I now consider as a very important move for any open source project…

Letting Bocadillo stand on its own feet

Before releasing the docs, I moved Bocadillo from a personal repo to its own GitHub organization, namely BocadilloProject.

The main motivation at the time was that I could use the organization's GitHub Pages domain bocadilloproject.github.io for the documentation. It is definitely way cleaner and more accessible than florimondmanca.github.io/bocadillo. 🙃

However, this had the positive effect of giving Bocadillo an online space of its own. It was not tied to my personal GitHub account anymore — the organisation was the new home for Bocadillo's source code.

Later, as I realized that Bocadillo announcements was taking over my personal Twitter account, I created a dedicated Twitter account.

The point is: it's important that an open source project lives outside of its creators.

Now, back to the story — on November 18th, I had a docs site up and running that people could visit. What next?

Opening up the development process

Up to November 20th, the way I kept track of the backlog and progress was via a private Trello board.

This was very practical to me: I use Trello for a variety of things. But I realized that people visiting the repo had no visibility on what was coming next or possible ways they could contribute.

In fact, from a visitor's perspective, I think the repo looked like just any other personal project — no issues, no PRs, just a ton of commits from a single person — and not a community-driven effort, i.e. what I'd like Bocadillo to become.

So, per the advice of a close friend of mine and inspired by this article by Dhanraj Acharya, I decided to open up the development process.

I converted all my Trello cards to GitHub issues, and added meaningful labels (see Sane GitHub labels by Dave Lunny) and description to them. I think the repo now provides better visibility on the project and feels more encouraging for newcomers.

I've learnt from this that open source is not only about opening the source code. You've got to be open about the development process, too.

I now hoped that, with the repo filled with issues and public PRs, it would attract its first open source contributors.

Spoiler alert: it did!

Yay! First contributors!

About at the same time, as Bocadillo grew in size, I began to feel the need for external advice. I was afraid that I might be taking bad design decisions or that the code could have been better. In short, I needed contributors.

Luckily, still on November 20th, I had the great pleasure of welcoming the first contribution to the repo, in the form of Alin Panaitiu commenting on PR #3 for the new "hooks" feature.

Alin helped me fix a few things about the feature which I was initially unsure about myself, and suggested ways in which it could be made even more useful. He went as far as forking the repo and sending me a diff showing off a fix.

Later, on November 23rd, Alin got his first PR merged. Bocadillo had officially gained its first contributor! 🎉

Screenshot of PR #18.

As emotional as I can be, I was moved.

Even more so that Alin actually sticked around. In the v0.7 release, Alin contributed 2 new features (GZip and ASGI middleware), with a code-to-merged time of probably less than a few hours. Thanks Alin, great stuff!

Entering maintenance mode

From the end of November and onwards, the pace of the project slowed down a bit. On one hand, I was in a bit of a rush with school projects and exams being on the agenda, but that didn't explain everything. I was experiencing something else.

You see, in the beginning, committing code to Bocadillo felt very easy. There was barely any legacy so it was exciting and I had tons of ideas. Everything remained to be done.

But as more features were added in, it started feeling heavier. Releases took longer to get out — now taking a week instead of days. Testing, refactoring and documentation became major parts of the development process. Plus, I now managed Bocadillo's online presence too.

Don't get me wrong — I'm not complaining. In fact, I like that I got to go past the initial excitation and entered maintenance mode. Besides, it's definitely a normal shift for a project that's trying to gain momentum and reach out to the community.

I am now enjoying that working on Bocadillo doesn't have to take me nights like it used to in the beginning. Which leads me to the next point…

It's not a sprint, it's a marathon

I recently noticed that my attitude towards the project has changed.

Instead of rushing in to get features out as quickly as possible, and hoping that a burst of users would pop by, send stars en masse, love the framework and beg for more, I now felt more in peace with the idea that the project growth would be slow.

This stems from the fact that maintaining an open source project is a marathon, not a sprint.

Put differently, success should always be a by-product, not a goal.

You'll have noticed that Bocadillo's goal statement doesn't mention fame, nor a threshold number of users. It only states that I hope Bocadillo can help some people solve problems. If that is the case for at least one person, I'll consider it a win. If it becomes the case for a lot of people, I shall only consider it as a side effect.

Two person standing on gray tile paving. @goian, unsplash.com

This is it for the story behind building Bocadillo! As you may have noted from these discussions, I have been loving the experience so far, and I'm actually now convinced that I made the right decision when I decided to go past the fear of judgement and build my own web framework. Which leads me to the last section of this blog post.

Tips to start your own open source project

If there is one thing I want you to take away from reading this article, it's got to be this:

Open source is an awesome way to learn. You should start your own project today!

"Sounds great", you think, "but how should I proceed? Do you have any advice?"

Well, I do have some. 😋

First and foremost, learn as much as you can and make sure to enjoy yourself. Open source should never become a burden, and if it does, try to find ways in which you can start delegating or relying on the community to drive the project forward.

For the rest, brace yourselves — categorized bullet points ahead!

Note: if you're unsure about how to implement any of the following items and/or want to see how I used or configured the tools in practice, feel free to check out Bocadillo's repo and just copy the bits you're interested in — it's open source, after all!

Project definition

Decide what you want to build.
Decide why you want to build it — although it doesn't have to be deep or abstract, having a clear source of motivation helps.
Think about scope and design philosophy: this will help make informed design decisions, and prevent feature creep.
Decide who you are targetting: are your users web developers, sys admins, project managers…?
Explicitly define user skills expectations: what should your users be familiar with, and how much?
Decide how you will distribute your project (e.g. a PyPI package).

Marketing & Communication

Build an identity: a name and a tagline at least. Make them short, catchy, and memorable. Showcase them everywhere (repo, PyPI page, docs site…).
Create a visual identity: this is your logo and graphic charter. It's better to have an okay temporary logo than no logo at all.
Decide on an entry point, i.e. a natural online location where people can look up your project. It can be the GitHub repo, a docs site, or whatever seems fit, but it should exist. (For Bocadillo, I believe this is the docs site.)
Give your project a life of its own: it's a good idea to create a separate GitHub organisation or social media account.
Use social media to communicate news, annoucements and tips, and start gathering people around the project (I use Twitter for this).
Provide ways for people to see where you're heading at — a roadmap, a list of issues, an "unreleased" section in the CHANGELOG, etc.

Community

Implement open source best practices: a proper README, contributing guidelines, a code of conduct, issue/PR templates, etc (use GitHub's checklist!). This will make the repo more welcoming to potential contributors, and show that you care about the community.
Be supportive and kind to others. Thank them for their questions. Provide helpful resources.
It might be a good idea to create a place for informal discussions. I recently decided to experiment with a Gitter chat room.

Project management

Use GitHub issues to list your TODOs. That way, when wondering what you should work on next, just pick up a ticket and work on it!
Set up meaningful issue labels (see Sane GitHub Labels).
You can also set up a GitHub project to display your issues and PRs in a kanban board.

Code quality

Set up a CI/CD pipeline.
Be unforgiving on tests — besides ensuring your software works as it is intended to, they'll help you and everyone catch regressions and be confident when making changes (I use pytest as a testing framework).
Measure test coverage (I use pytest-cov for pytest/coverage.py integration, and CodeCov for coverage reports).
Enforce that PRs pass tests before merging.
Every PR should contain all 3 items: code, tests and docs.
Use a code formatter to reduce syntax/code style noise in code reviews (I use the opinionated Black formatter along with a pre-commit hook).
If you don't have other reviewers, make PRs to yourself, let them settle and come back later. It's easier to see if the code is a mess after a few days.

Documentation

Write a clear and informed README with at least a project description, install instructions, a quick start example and a link to the docs or somewhere users can learn more.
Keep a changelog, you'll thank yourself later.
Build a docs site if you're building more than a simple library (I use VuePress as a static site generator).
Structure your docs: tutorials, discussions, how-to's, reference (tip: I use PydocMd to generate Markdown API reference straight from my Python docstrings).
Remember: if it's not documented, it doesn't exist.
Add pretty badges to your README (e.g. with shields.io).

Versioning and releasing

Use semantic versioning (see SemVer).
Automate version bumping with tools such as bumpversion.
Automate the release pipeline. I use TravisCI to release tagged commits to PyPI.
Set up special release branches. For example, I use release/docs for docs deployment and release/test for releasing to Test PyPI.

Achievement unlocked?

This blog post is now coming to an end, so let's wrap up!

Although Bocadillo started as a way for me to learn more about the internals of a web framework, it has turned into a full-blown open source project.

With all the effort that already went into building the framework, documenting it, configuring the repo and managing releases, I now start to think of myself as an open source maintainer — which I think is a very enriching experience!

Even though I do take pride for what I have achieved so far, all of this is also very humbling. I now realize how challenging managing an open source project can be, let alone building a community around it. It's tough!

That said, I do believe that you should go for it and start your own open source project. It could be a simple tool or library, or an entire application framework — either way, you'll learn a lot in the process.

Of course, feel free to check out the Bocadillo repo if you seek ideas on structuring an open source project or setting up tooling. There are also tons of great resources on opensource.guide, so check this website out, too!

Thanks for reading through this article! As always, feedback is much appreciated. In particular, I'd love to hear your own stories on maintaining open source projects. Plus, if by any chance this article has inspired you, be sure to drop a comment! ❣️

Best wishes for this holiday season to you all. ✌️

Stay in touch!

If you enjoyed this post, you can find me on Twitter for updates, announcements and news. 🐤

From Angular To Vue: Feeling Like A Beginner Again

Florimond Manca — Thu, 25 Oct 2018 13:44:05 +0000

Update (30 Oct, 2018): a few days after writing this post, I've already built a good understanding of Vue and its ecosystem. Thanks to everyone who shared useful resources about Vue; it helped me a ton!

I've worked on a couple of web projects in the past year or so. I now feel at ease and productive with a few select technologies. In the realm of frontend development, that means Angular. And to be honest, it feels great.

Yet, I find that I've begun resting on my laurels, and boredom even began to take the upper hand. As a result, I started learning Vue.

In this short post, I'll expose my state of mind, the difficulties I am currently encountering and what I do to cope with them in the process of learning Vue after a year of working with Angular.

How did we get there?

I've worked with Angular quite extensively lately. Angular is the first modern web framework I've learnt, and I've built at least three or four different apps of various sizes. I also learnt a ton about modern concepts such as Progressive Web Apps and Server Side Rendering along the way.

But there's a catch. Every time I am given the opportunity to choose a frontend framework for a new project, I go for Angular. Why? Simply put, because it's the only one I know and I feel comfortable with.

Yet, immediately after taking the decision, a little voice sneaks into my head, and it whispers: "Angular, again? Meh…".

Don't get me wrong — it does feel rewarding to have become knowledgeable enough at a technology that I can build a whole system without barely looking at the documentation. I feel proud.

There's also no need per se to be knowledgeable at every frontend framework out there — there are way too many of them, plus specialising also has its advantages.

But for me, there's a problem:

Where is the challenge? Where is the difficulty? What is there left to learn?

You see, I do love learning new things. I don't particularly enjoy feeling like a beginner — that's rather uncomfortable — but I always dig the outcome: I can do things I couldn't do before, and it's thrilling.

That's what motivated me to try something else. Also, some circumstances have helped.

Take your chance

I boarded on a major school project a few weeks ago. We had the possibility to choose between Angular and Vue.

I didn't know much about Vue — only a few things I've read here and there, and perhaps a toy project from a year ago. Going for Angular would have been so much easier because I've built up a lot of experience and know-how.

But guess what? We went for Vue.

Sometimes, leaving your comfort zone is as simple as asking: "why not?".

It won't be easy

I dived in just today, and let me tell you — I'm excited, but it feels very uncomfortable right now.

Actually, I feel frustrated. I have to re-learn everything. Why does it all look so similar, yet so different? I felt like I was a master, but I am now a total n00b. Duh.

So, in this new and unknown ecosystem, I need a survival strategy not to drown under the seemingly huge amount of new things to learn.

Build on what you know

Luckily, I now have a better understanding of the world of frontend development than I had when I first tried Vue. A lot of stuff is already out of my way, and I can focus on the framework and its ecosystem. (Also, Vue has matured a lot, which is very good news.)

Quite naturally, I am looking for what is familiar. I have already noticed that:

Both Angular and Vue are component-driven — great!
Both have a CLI — fantastic! (but I miss ng generate)
Both have a templating system based on directives — sweet!
Both have bidirectional data binding through inputs (@Input() vs @Prop()) and outputs/events (@Output() vs. $emit()) — nice!
The template syntax is quite similar: *ngFor becomes v-for, [foo]="bar" becomes :foo="bar", etc — stellar!
Both have extensive and precise documentation, and a vibrant community — brilliant!
I can also use TypeScript with Vue — yay! 🎉
I've been digging Angular Material, and there seems to be even more Vue component frameworks out there, like Vuetify — amazing!

At least, this is not a cold start. There are indeed a lot of aspects I can relate to and compare. Phew!

Me generating a project using Vue CLI: "TypeScript! There you are!"

Identify the differences

Yet, I couldn't help but notice big differences:

Vue components are single-file, while Angular separates HTML, CSS and TypeScript into their own files. (I'll probably get used to it.)
Vue focuses on the view layer, while Angular comes batteries included: where are my beloved Router and HttpClient?!
Vue doesn't have modules, and instead it has some other concepts such as mixins, filters and transitions.
There is no concept of services — how am I supposed to abstract logic from components? Is that even a thing in the Vue philosophy?
If there are no services, then what about state management? Am I forced to resort to Redux/Flux/similar even for smaller apps?
Where are my Observables? It took me so much time to get to know them. What should I use instead of RxJS?

All of these differences gave me the general intuition that Vue imposes much less on the developer.

As someone with an Angular background, I find this a a bit daunting.

Angular's conventions and ways of doing felt secure. They also facilitated working with other devs — because we all shared the same practices. How is it going to be with Vue? Does everyone have a different workflow?

Look for best practices and popular solutions

So what am I left with? Some stuff looks familiar, other stuff looks better (single-file components look quite slick after all), but there's also some stuff that I miss. And so I am looking for replacements.

Take HTTP requests: two popular solutions I've seen are axios and vue-resource. Looks like a good substitue for HttpClient.

The issue of routing has also been solved — there's no built-in Router in Vue, but vue-router is a standard plugin that's even suggested when creating a new project via Vue CLI.

Now, what about state management? Angular has services and I've been lucky enough that, when data binding becomes insufficient, I could use services to store some shared state.

Vue has no such concept of services, so what's left? A popular option seems to be vuex, a Flux-inspired state management Vue plugin. I heard about state management before (although mostly through Redux — see Understanding Redux), so I'm interested to dig deeper and see how that turns out. I just hope it won't be too much of a burden.

The Vuex state management pattern. Looks sensible, but I hope it won't be too heavy. (Source: Vuex docs.)

Lastly — RxJS and event streaming. That's probably the thing I like the most in Angular. Observables make working with streams of events so pleasant and maintainable.

Well, I haven't found a substitute for that yet. My guess is — there won't be one. Actually, I suppose Vue won't get in my way and I could use RxJS, but there seems to be many built-in features already to ensure reactivity — which I also find quite exciting. We'll see!

Enjoy the journey

I used to feel very knowledgeable about Angular, yet Vue has got me back to the position of a beginner. It's not easy, and sure as hell feels uncomfortable.

But — and this is a message to all striving beginners out there — experience also tells me that the journey will be worth it.

At the end of the day, you'll have learnt yet another technology, and be able to do things you couldn't do before. It may take weeks or months, but it will happen, and you'll feel proud. 💪

As for me, I'm not giving up on Angular just yet, but I'm excited to learn more about Vue. I will focus on building a mental model of how everything fits together in Vue. I look forward to the day when it all just "clicks" — because there's nothing more exhilarating than pushing your boundaries to learn something new. 💻

Have you already been in this position? How have you coped with having to re-learn everything? I'd be happy to hear your thoughts!

Stay in touch!

If you enjoyed this post, you can find me on Twitter for updates, announcements and news. 🐤

Inbox Zero: How To Keep A Clean Email Inbox (And Mind)

Florimond Manca — Thu, 11 Oct 2018 14:01:19 +0000

This post was originally published at blog.florimondmanca.com.

I am a final year student in systems and software engineering at CentraleSupélec. My school has an extremely active campus life. The main channel for communication here is email. Like, mass email. I mean, 30 emails per day, every day. I'm not even joking. So when I arrived in first year back in 2015, I quickly had to learn how to manage my inbox efficiently.

As a former software engineering intern, I had to deal with dozens of email per day too: JIRA notifications, company-wide announcements, CI/CD build alerts, etc.

I adopted a simple workflow to keep a clean inbox:

Is the email irrelevant? Unsubscribe if need be, then delete it.
Does it require taking action? Do it now, or keep the email until that's done.
Archive it!

Then I discovered this had a name: Inbox Zero.

Most of my posts are technical, but I wanted this one to be non-technical and aim at anyone who has to deal with high-volume inbound email.

I will share how this method helps me keep a clean inbox and a peaceful mind. I'll show you how easy it is to implement and hopefully inspire you to adopt it as well!

What others do

My school of engineering has about 3,000 students, about 500 of which are in my promotion. As a result, I've been able to compare how other people manage large numbers of daily emails.

I have told friends about how I manage my email. I try to keep my inbox as empty as possible, essentially using it as a todo list. They also told me about their workflow.

Well, I was shocked to discover that some of them never cleaned their inbox. I've seen inboxes containing 7000+ emails, some of them from years ago! Downright horrifying. These days, I feel overwhelmed as soon as I've got more than 10 emails in my inbox. How can they bear having thousands of them?

A few reasons they gave me:

I don't have the time to clean my inbox.
I don't need to clean it — I get away with inbox folders.
I don't know how to clean it.

After digging deeper, I also learnt something incredibly surprising: some people don't know about the "Archive" button! They either keep the email in their inbox or delete it.

Yet, Inbox Zero is all about the archive button.

It's as easy as that.

Email is just like regular mail

Have you ever received mail? Like, actual physical letters? I'm pretty sure you have.

At least, I have, and I've got a very simple way of dealing with it:

I check the mailbox once per day and take the new mail out.
If it's garbage, I gladly throw it away.
If it's interesting and I have to do something about it (like fill in an online form, or notify someone):
- Can I do it within 2 minutes? If so, I do it right away.
- Otherwise, I keep the mail front and center on a shelf so I am reminded to deal with it when I've got the time.
When I'm done, I either keep the letter in my records for future reference or toss it away into the waste paper basket of oblivion (how poetic is that?).

I'm pretty sure that's mostly how you do it too. You don't even have to think about it too hard. It's natural.

Yet, I have a question:

Would you let letters, postcards and ads pile up in your physical mailbox forever, even after they've been been read and dealt with?

Sounds absurd, right? Yet, this is exactly what you do when you keep email in your inbox.

I have good news

The ideas behind Inbox Zero are simple. It's no rocket science. There's no black magic. Even better:

Inbox Zero isn't any different from how you're used to manage physical mail.

Let me repeat — it's the exact same workflow! You've been doing it all your life. You know how it works.

So there is absolutely reason zero (pun intended) for not dealing with electronic email the same way.

Still feeling a bit unsure? Let me show you da wae.

The way to a clean inbox

1. Start by archiving all your inbox. Now.

Here you are with hundreds or thousands of emails in your inbox. How can you make sense of all that mess?

Well, I've already told you: the core of Inbox Zero is two words — just archive!

Contrary to trash, archived emails are not lost. They just go out of your inbox, and into somewhere you can retrieve them in the future. That's why my advice is:

You should feel confident about archiving.

The first step should then be: archive all emails presently in your inbox. Tested and proven by fellow students and friends.

And believe me, it's pretty relieving.

Gladly archive all your inbox. 👌 (Now.)

2. Deal with irrelevant incoming emails

Is your inbox filling up with tons of ads or content you don't even care about? You need to sort this out.

If it's a subscription of some sort, find and click the unsubscribe link. Then only should you delete the email.

If the email is a one-off, then the only thing you can do is delete it.

The important thing is that you should decide immediately whether to toss an incoming email away so that garbage doesn't end up filling up your inbox again.

3. Ponder emails requiring action

Sometimes, an email requires you to take action. It may be forwarding it to someone else, answering a meeting request, filling in an online form — anything.

When that is the case, there are only 2 options:

Can the action be done quickly, e.g. under 2 minutes? Then, just do it!
Otherwise, you should keep the email in your inbox and work it out it later.

The good thing about option 2 is that any read email in your inbox can then be thought of as a todo. That's also why I sometimes call Inbox Zero "Inbox as a Todo".

Yet, if you notice these "todo" emails pile up in your inbox, it might be worth investing in a task management tool, or a todo app of some sort (or get stuff done in the first place 😉). That's an even shorter path to a clean inbox. Plus, if it has reminders and alerts, you'll probably find it easier on your mind as well!

As an example, here is an email I would certainly take action upon right away:

From: elsa@company.biz
To: florimond@company.biz
Subject: Spare paper
---
Hi Florimond,

Do you have any spare paper left in the cupboard?
I need 2 sheets of paper for a presentation this afternoon.

Cheers,
Elsa

That definitely won't need more than 2 minutes. I can just check whether I have some paper left and answer to Elsa that she can pick it up at my desk anytime she wants.

Here's another example:

From: marketing@company.biz
To: everyone-including-me@company.biz
Subject: Rebranding Survey
---
Hi all,

We are working on rebranding our company and would like to gather everyone's input and ideas.

Please fill in this online form when you have some time: [Link]

Best,
Marketing Team

This will take more than 2 minutes to deal with. I probably want to give some actionable input to marketing as to how we can improve our brand and what values it should convey. As a result, I will keep this email in my inbox and anytime I check my email again — which might be when I have some free time — I will be reminded to answer the survey.

4. Just. Archive. It.

In any case, if the email is of any importance to you, you should keep it. By that I mean, archive it.

The reason why is simple: if you ever need that email again, you'll find it in the archives.

On the other hand, if you'll never need it again, there's no reason to waste storage for it and you should throw it away.

Free up your mind

We sometimes think of email as a burden. We need to "deal with it" or "sort it out" multiple times a day. It shouldn't be like that. It doesn't have to be like that.

On a personal level, I strongly believe that Inbox Zero has helped me free up my mind. It helped me reduce stress because I always know that things I have to do are in my inbox, and I can just look them up. It's a matter of reducing cognitive load.

And if you're a developer, there's a chance you're already facing a fair amount of cognitive load in your regular job. It only makes managing your email smoothly even more important.

If you try out Inbox Zero — let me know how it turns out! A few fellow students who went for it told me they love how easy it is to manage email now. Hope you reconcile with email too. 💻

Stay in touch!

If you enjoyed this post, you can find me on Twitter for updates, announcements and news. 🐤

Reconciling Dataclasses And Properties In Python

Florimond Manca — Mon, 08 Oct 2018 06:14:21 +0000

Originally published at blog.florimondmanca.com on Oct 6, 2018.

If you can't wait for the TL;DR, jump to Lessons Learned at the end of this article.

I've been playing around with Python 3.7's dataclasses — and so far they've been super awesome. For the most part, they're quite intuitive and easy to use and customize.

Yet, there's one thing I've been struggling with and couldn't find online help for, which is implementing Python properties on dataclasses. And chances are — I am not alone.

That's why I decided to solve the problem once and for all. I wanted to figure out how to reconcile dataclasses and properties.

So I sat down before my computer, fired up an interpreter and wrote down my thought process as I made my way towards a solution. After a night of trial and error and some wording enhancements, the result is this very blog post!

As a pleasant side effect, you'll get to see many features of dataclasses in action. I'll start with a quick overview of dataclasses, and we'll get to the code right after that.

Of course, you'll find all the code supporting this post on GitHub.

Dataclasses: a 10,000-ft overview

Dataclasses are, simply put, classes made to hold data. Their specification in PEP-557 was motivated by the fact that a lot of classes we write are merely used as editable data containers. When that happens, we spend time writing boilerplate code which most often results in an ugly __init__() method with tons of arguments and just as many lines for storing them as attributes — not to speak about handling default arguments…

Dataclasses, featuring Raymond Hettinger in a GIF.

And there is! The answer is: dataclasses. 🎉

Python implements dataclasses in the well-named dataclasses module, whose superstar is the @dataclass decorator. This decorator is really just a code generator. It takes advantage of Python's type annotations (if you still don't use them, you really should) to automatically generate boilerplate code you'd have to mechanically type yourself otherwise.

As a point of comparison, here's how you would create a Vehicle class with a wheels attribute using a regular class declaration:

class Vehicle:
    def __init__(self, wheels: int):
        self.wheels = wheels

Nothing fancy, really. Now, the @dataclass version:

# 0_initial.py
from dataclasses import dataclass

@dataclass
class Vehicle:
    wheels: int

Believe it or not — these two code snippets are strictly equivalent! It's actually a win, because beyond __init__(), @dataclass generates a bunch of extra stuff for free, including a handsome __repr__():

>>> car = Vehicle(wheels=4)
>>> car
Vehicle(wheels=4)

In short, dataclasses are a simple, elegant, Pythonic way of creating classes that hold data. 🐍

But…

The problem

I sometimes resort to the @property decorator to implement specific logic when getting/setting an attribute. That's really just the Pythonic way of implementing getters and setters.

Building upon the previous Vehicle class, I would make the wheels attribute private and put an @property on top of it:

class Vehicle:

    def __init__(self, wheels: int):
        self._wheels = wheels
        # note the underscore — it's now private! 👻

    @property
    def wheels(self) -> int:
        print('getting wheels')
        return self._wheels

    @wheels.setter
    def wheels(self, wheels: int):
        print('setting wheels to', wheels)
        self._wheels = wheels

Here's what it looks like:

>>> car = Vehicle(wheels=4)
>>> car.wheels = 3
setting wheels to 3
>>> car.wheels
getting wheels
3
>>>

Now the question is — how can I implement such a property on a dataclass?

Wait, so this is the problem?

You may think this to be a trivial question. Mind you, it is not trivial.

Dataclasses generate the __init__() method for you — and that's great. They even provide a __post_init__() hook method in case you want to do some more initialization (see Post-init processing).

However, this means you cannot do the same trick as with normal classes, i.e. storing a public-looking argument (e.g. wheels) into a private attribute (_wheels) that you'll build an @property out of.

That's where the problem comes from. And to be honest, it gave me a bit of a headache.

Because I think the problem solving was interesting, I'll take you through 5 consecutive attempts to correctly implement that property on a dataclass version of the Vehicle class.

Attempt 1: declaring a private field

First, let's keep things simple. We want to store wheels in a private field and use it in the @property, right? So why not simply declare a _wheels field on the dataclass?

# 1_private_field.py
from dataclasses import dataclass

@dataclass
class Vehicle:
    _wheels: int

    # wheels @property as before

Unfortunately, this won't work — otherwise this blog post wouldn't be of much use! 😙

The reason why is because the constructor now expects a _wheels argument instead of wheels.

>>> car = Vehicle(wheels=4)
Traceback (most recent call last)
<ipython-input-3-9c9de8fb1422> in <module>()
----> 1 car = Vehicle(wheels=4)
TypeError: __init__() got an unexpected keyword argument 'wheels'

To be fair, that's just @dataclass doing its job. Still, that's not what we want.

Attempt 2: make use of `InitVar`

If you read through the documentation, you'll learn that InitVar allows you to implement init-only variables. These variables can be passed to the constructor, but won't be stored in an attribute on the class. Instead, the variable is passed as an argument to __post_init__().

Why not use this to create an init-only wheels variable and store that in a _wheels field? We just need to give the latter a default (e.g. None) so that it is not required by the constructor:

# 2_initvar.py
from dataclasses import dataclass, InitVar

@dataclass
class Vehicle:

    wheels: InitVar[int]
    _wheels: int = None  # default given => not required in __init__()

    def __post_init__(self, wheels: int):
        self._wheels = wheels

    # wheels @property as before

Granted, __init__() now expects a wheels argument instead of _wheels, which is what we want.

However, @dataclass now generates other boilerplate code and magic methods using _wheels, which is problematic.

>>> car = Vehicle(wheels=4)
setting wheels to 4
>>> car
Vehicle(_wheels=4)  # 😕

Attempt 3: make use of `field()`

Digging deeper into the docs, I found that one could fine-tune the field generation behavior using the field() function. You can pass it a default value and it accepts a repr argument to control whether the field should be included in the generated __repr__(). Here's how it looks when used on _wheels:

# 3_field.py
from dataclasses import dataclass, field

@dataclass
class Vehicle:

    wheels: InitVar[int]
    _wheels: int = field(default=None, repr=False)

    # __post_init__() as before
    # wheels @property as before

Sweet — we don't have _wheels included in __repr__() anymore. But we still don't have wheels either!

>>> car = Vehicle(wheels=4)
setting wheels to 4
>>> car
Vehicle()  # Where is `wheels=4`? 😕😕😕

Attempt 4: make `wheels` a proper field

In the previous attempts, wheels was an InitVar — not a field. This time, let's declare it as a field in its own right. It will be possible to pass it in the constructor, and it should be included in __repr__() this time around.

The good thing is, the @property definition of wheels declared later in the class will not interfere with @dataclass's generation process — because it is not a type annotation, which is what @dataclass relies on to generate the fields.

That might start to be a bit complicated, so let me show you some code. I'll reproduce the @property for wheels in full this time:

# 4_wheels_field.py
from dataclasses import dataclass, field

@dataclass
class Vehicle:

    wheels: int  # Now a regular dataclass field

    # The rest just as before:

    _wheels: int = field(default=None, repr=False)

    def __post_init__(self):
        # Note: wheels is not passed as an argument
        # here anymore, because it is not an
        # `InitVar` anymore.
        self._wheels = self.wheels  # (1)

    @property
    def wheels(self) -> int:
        print('getting wheels')
        return self._wheels

    @wheels.setter
    def wheels(self, wheels: int):
        print('setting wheels to', wheels)
        self._wheels = wheels

It looks like we're getting there, aren't we?

Unfortunately, not quite. 😞 There's a catch in this implementation.

Indeed, you may think line (1) puts the value of wheels that was given to the constructor (and stored into self.wheels) into _wheels. For example, calling Vehicle(wheels=4) would result in having _wheels == 4. Sadly, that is not the case!

Here's why: when executing __post_init__(), self.wheels is the value returned by the wheels property's getter — not the value initially stored during __init__()! And that getter returns self._wheels, which is None by default.

I know, it's getting all tangled up, but please bear with me:

>>> car = Vehicle(wheels=4)
setting wheels to 4
getting wheels  # hint: this is (1) being executed
>>> print(car.wheels)
getting wheels
None  # nope, nothing in there…

If you think about it, what we're doing in (1) is just replacing _wheels with its own value. Quite useless, if you ask me. We would actually get the same result if we didn't even implement __post_init__().

Duh! So what can we do? 😩

Fortunately, there's hope!

Attempt 5: exclude `_wheels` from the constructor

Let me warn you — this fifth and final attempt will work, and the reason why, which I'll explain in a minute, is outrageous.

At this point, you'd be right to feel sad — I felt sad myself. But fear not! There is one thing from the documentation that we haven't tried yet.

So far, the _wheels attribute has been declared using field(default=None, repr=False). Using default=None here means that we are able to omit passing a value for _wheels in the constructor — it will be given the value of None during __init__(). However, it is still possible to give it a value in the constructor, and everything will work as expected:

>>> car = Vehicle(wheels=4, _wheels=3)
setting wheels to 4
getting wheels
>>> car.wheels
getting wheels
3

Well, how about we find a way to remove the _wheels argument from the constructor? Will it solve our problem? (Spoiler alert: it will.)

Guess what: field() accepts an init argument for that exact purpose. The docs on field() read:

init: If true (the default), this field is included as a parameter to the generated __init__ method.

Sounds trivial, right? Well, let's try using it on _wheels (I removed the __post_init__() hook because we previously showed that it was actually useless):

# 5_init_false.py
from dataclasses import dataclass, field

@dataclass
class Vehicle:

    wheels: int
    _wheels: int = field(init=False, repr=False)

    @property
    def wheels(self) -> int:
        print('getting wheels')
        return self._wheels

    @wheels.setter
    def wheels(self, wheels: int):
        print('setting wheels to', wheels)
        self._wheels = wheels

Well, guess what? This has just solved all of our problems.

Because we used init=False, the constructor generated by @dataclass will not initialize _wheels at all.

However, it will initialize wheels with the value passed to the constructor. If we could extract the generated code, one of the instructions in __init__() would look like this:

self.wheels = wheels

Now, you tell me — what does this execute exactly?

Yep, that's right. It will execute the setter! 🙀

Look! We've actually been seeing the print statement in the setter since attempt 4!

>>> car = Vehicle(wheels=4)
setting wheels to 4  # the `wheels` setter being called

Let me remind you of the code for that setter:

@wheels.setter
def wheels(self, wheels: int):
    print('setting wheels to', wheels)
    self._wheels = wheels

It sets the value of _wheels!

As a result, the value for wheels passed in the constructor is put into _wheels — and nowhere else, because after the class has been generated, wheels only refers to the @property, not to a field on the dataclass.

If you think about it, this is exactly what we were doing when implementing the property on good ol' regular classes. Remember?

class Vehicle:

    def __init__(self, wheels: int):
        self._wheels = wheels
        # This is equivalent to calling:
        # `self.wheels = wheels`
        # which *is* what the __init__() method
        # now generated by @dataclass actually does.

Caveat: this approach only holds because the property's setter is implemented. If we only implemented a getter (i.e. to make a read-only field), the __init__() method wouldn't be able to assign the attribute and would crash. This is intended behavior, though, because dataclasses were designed to be editable data containers. If you really need read-only fields, you shouldn't be resorting to dataclasses in the first place. Perhaps NamedTuples would be a viable alternative — they are the read-only equivalent of dataclasses.

Anyway, long story short…

Success!

We have successfully implemented properties on dataclasses. 🎉

Memes will never let you down.

To be honest, it was surprisingly not easy. We've been through five different attempts, navigating through the documentation and painstakingly coding our way towards dataclass properties.

So after all this hassle, can we at least derive a quick recipe for implementing properties on dataclasses?

The answer is: yes, we can. ✌️

Lessons learned

Have you noticed a pattern between using an @property on a regular class vs. on a dataclass?

Look, here's the regular class version:

class Vehicle:

    def __init__(self, wheels: int):
        self._wheels = wheels

    @property
    def wheels(self) -> int:
        return self._wheels

    @wheels.setter
    def wheels(self, wheels: int):
        self._wheels = wheels

And the dataclass version, using a diff syntax to highlight the differences:

+ from dataclasses import dataclass, field

+ @dataclass
  class Vehicle:

+     wheels: int
+     _wheels: field(init=False, repr=False)

-     def __init__(self, wheels: int):
-         self._wheels = wheels

      @property
      def wheels(self) -> int:
          return self._wheels

      @wheels.setter
      def wheels(self, wheels: int):
          self._wheels = wheels

Written in plain words, for you litterature freaks:

How to implement a property on a dataclass:

Remove __init__()

Declare the property as a field

Add an associated private variable using field(init=False, repr=False).

Way to go!

If you managed to read up to here, congratulations! This blog post dealt with a highly specific and technical topic, yet I've had a lot of fun writing it and figuring out how to implement dataclass properties.

For those wondering — the series on Apache Kafka still goes on! I figured a small break never hurts, and I felt like it gave me the opportunity to write something spontaneous.

I hope you enjoyed this post and, as mentioned in introduction, you can find all the code on GitHub. See you next time! 💻

Stay in touch!

If you enjoyed this post, you can find me on Twitter for updates, announcements and news. 🐤

Breaking News: Everything Is An Event! (Streams, Kafka And You)

Florimond Manca — Mon, 01 Oct 2018 19:51:36 +0000

Welcome back! Here's the third article in my series on building streaming apps with Apache Kafka.

The previous article had us build our first real-world streaming application — a real-time fraud detection system.

This time, we'll take a deeper look at some core concepts behind Apache Kafka.

More specifically, we'll try to perform a mind shift. We are used to think in terms of state, but thinking in terms of events allows us to enter the realm of streaming data.

Then, after introducing streams and the streaming data paradigm in greater detail, we'll take a moment to think about why it might be important you become knowledgeable about it.

Spoiler alert: because it's fun to learn, reusable, marketable, and useful to build reactive applications businesses.

Without further ado, let me start your introduction to streaming data.

State and requests

As developers who build apps, data pipelines and data-driven systems, we are used to think in terms of state. We often ask questions such as:

"Hi, Mailer microservice! Frontend app speaking. Could you send me the list of processed emails for User A so I can update their dashboard?"
"Hi, REST API! Mailer microservice here. Would you send me the current list of posts, so I can diff it with last week's list and send an update to newsletter subscribers?"
"Hi, SQL database! I'm the application server. What rows do I have in the posts table?"

We build complex systems made of many different components and try to wire them up through requests, (or queries, or any communication fitting the request/response paradigm).

Even when we use the latest and shiniest architecture design principles (hint: microservices), we run the risk of ending up with a tightly coupled architecture because components need to know about each other's API in great detail, or need to keep polling one another to detect changes.

Your typical microservices web application. See that soup of requests? Can we do better?

Introducing events

There must be a better way

Database queries, API HTTP requests, cache lookups — as far as this article is concerned, all of these are requests that aim at retrieving state. But what do you think the following are?

User created
Tag followed
Post published
Email processed
Draft submitted
Profile updated

These are not requests. These also have nothing to do about state.

This is a plain and simple list of events — mere facts about what just happened in our application or in the outside world. 🏷

We don't always realise that the data we retrieve from storage (like a database or a cache) is only there because something happened in the first place.

That something is an event. But we'll get to it in a moment.

What does an event look like?

An event is a general, abstract concept. One helpful dictionary definition is:

Event: a thing that happens or takes place, especially one of importance.

It may be useful at this point to give you a concrete intuition.

In software, an event is essentially made of two things:

An action (or label, name, code…) used to clearly identify what happened. For example, 'user_created' or 'email_processed'.
Context that provide details and fully determines the event. For example, the first and last name of a newly created user, or the ID of a processed email.

For example, here's one of many possible representations (in JSON) of an event stating that "User 1343 had they first name updated to 'Bob'":

{
  "action": "user_updated",
  "context": {
    "id": 1343,
    "field": "first_name",
    "previous": "Robert",
    "new": "Bob"
  }
}

As you can see, an event is simply a fact describing something that happened in the universe.

While seemingly trivial, this idea is at the core of a entirely new paradigm.

The brutal truth

There's a lot to say about events and how they mean and imply different things depending on the field of computing you're working in.

But in my opinion, here's the hard and brutal truth.

Everything is an event, and our ability to harness them is critical to building reactive and performant applications.

If you're not convinced yet, let me give you some examples based on real-world apps you probably already know about.

Arguments from the real world

What do you think applications such as LinkedIn and Slack (no advertising intended) have in common?

All of them produce, harness and process events.

Take LinkedIn for example:

New job offer? That's an event.
A notification? That's an event too.
A profile view? Still an event.
You updated a skill or your resume? Yet another event!

How about Slack? Well…

A new direct message? Yup, that's an event.
Someone created a channel in a workspace? Another event.
Logged into Slack? Heck — even that is an event!

Anything that LinkedIn or Slack do is process those events in order to create value (whatever that means) to their users. Notice how events can also form a chain: for example, a 'direct_message_received' event will probably trigger a 'notification_created' event.

Anyway, the core idea is that every single thing that happens within an app is an event.

At the beginning of this article, I told you that requests (in the general sense) allow to retrieve state. So if everything is an event, where does state come from?

State, a.k.a persisted events

Events occur at a given point in time. They are ephemeral by nature.

This means that events need to be persisted so you can access them later, i.e. after they happened.

For example, notifications need to be stored in database so you can read them again by the click of a menu.

It turns out that once events are processed and persisted, they become state. From then on, you can retrieve them (or some transformation or aggregation of them) using regular requests.

Let's take an example. For those of you who use Twitter, have you ever noticed how a "Like" you received via a notification may not be counted on the tweet's page until after a few seconds have passed? That's a 'tweet_liked' event being persisted into state — a tweet's counter of likes!

At this point, you may start to realise that, in a sense, state is the consequence of events. An event occurs, you catch it, process it, store it and update state. That's all.

Say "Hello" to streaming data!

When I accepted the truth that everything can be considered as an event, and state simply derives from them, I began to realise there is a completely different paradigm at play here.

This paradigm is that of streaming data.

Instead of thinking about systems exchanging state through requests and response, I started to think about systems exchanging state changes through streams of events.

What is a stream?

Streams have a particular definition in computing:

Stream: a continuous flow of data or instructions, especially one having a constant or predictable rate.

However, I find this definition to be too general. In the context of event streams, we can boil it down to a more specific and more concise form. Here's what I ended up with:

A stream is an unbounded, time-ordered collection of events.

Graphical representations always help, don't they? Here's how you can represent a stream of events as a set of points arranged in a timeline:

Simple representation of a stream.

To give you an even better intuition of what streams are and what you can do with them, I'd like to take you through an analogy.

The river analogy

Although streams have their own definition in computer science, in everyday language, a stream generally refers to a small river, or at least a flow of liquid.

Time-lapse photography of river. @the_bracketeer, unsplash.com

Image you walk down the riverside. You stop somewhere and watch the water flow past you. There are a lot of observations you can make that have an event stream equivalent:

Water flows at a rather steady rate → streams are a continuous flow of events.
Yet, rock formations may locally vary the flow rate → processing rate can vary at various stages of a stream processing pipeline.
You can count the number of fish swimming in the river → events can be persisted and become state.
The river results from mulitple, small upstream brooks combined together → streams can have multiple inputs and outputs.
Large tree trunks retain debris, leaving a clean water downstream → streams can be manipulated through transformations (such as filtering, mapping, etc.).
etc.

I like this analogy because it gives a general, lively overview of event streams.

But wait a second. So far, we've been talking about streams but didn't say much about how it all fitted within the Apache Kafka ecosystem.

How are streams implemented in Apache Kafka?

Kafka calls itself a distributed streaming platform and implements the streaming data paradigm perfectly. It provides simple yet extremely powerful abstractions to deal with streams of events in a reliable, fault-tolerant manner.

In Kafka, the equivalent of a stream of events is a topic containing messages. A topic is, just like an event stream, a time-ordered collection of messages, with virtually no bounds in the future nor the past — although there is always a first message in a topic and they are generally cleaned up after some time.

Kafka also provides two APIs implementing the publish/subscribe — a.k.a observable/observer — design pattern. They allow topics to have multiple "inputs" and "outputs":

The Producer API is used to publish events to topics.
The Consumer API is used to subscribe to topics and process their messages on the fly.

If you're into handy comparison tables, here you go:

Generic term	Kafka equivalent
Event	Message
Stream	Topic
Publisher	Producer
Subscriber	Consumer

As far as stream transformation (a.k.a. stream processing) is concerned, there's ways to do that in Kafka. 🎉 There are at least two tools for that:

Kafka Streams: this API provides high-level stream processing capabilities, including mapping, filtering, reducing, windowing, branching, and more. Unfortunately, it is only available for Java and Scala (for now).
KSQL by Confluent: use plain SQL (and nothing else) to join, aggregate and do windowing on streams in real-time.

However, stream processing is a rather broad area of interest. We might be able to dig into stream processing in Kafka in future posts. I think sticking to a high-level overview is probably enough for the sake of this article.

Why should I care?

We've talked about state and requests vs. events and streams, how everything is an event, what streams actually are and how Kafka implements the streaming data paradigm.

For you, the developer, but also for teams and businesses, why is any of this important anyway?

New possibilities

Streaming data is new paradigm that can open up ideas for new systems, concepts and architectures. It enables an asynchronous, event-driven streaming approach that is entirely different from a more classical synchronous, request-driven batch processing approach.

I also believe it is an exciting opportunity to learn something new and ramp up your skills. When I learnt about Apache Kafka and began to build software with it, I was flabbergasted by how powerful it was — and I loved that it allowed me to build systems communicating via something else than REST APIs and rigid message queues.

Reusable core concepts

Streaming data and event streams are concepts that you can — and may eventually have to — reapply in other areas of software development.

For example, frontend development is going crazy about event streams at the moment. The whole Redux ecosystem (very fashionable among React developers it seems) is based on events triggering a chain of reactions, eventually updating state.

You can also reapply streaming data concepts outside of frontend development. For example, the ReactiveX API and its implementations in various languages — RxJS for JavaScript, RxPy for Python, and more — bring streams and events to many popular languages in the form of "observable streams". Although Rx is mostly used in web development, you're free to use it for projects in other areas too.

The ReactiveX logo. What a cute… eel?

Anyway, put in a quotable form:

As tools and frameworks change ever more rapidly, core concepts are here to stay; streaming data is one such core concept.

Advance your career

Being knowledgeable about distributed streaming platforms is actually super marketable.

On January 2nd, 2018, LinkedIn published a report entitled "The Skills Companies Need Most In 2018". Based on job offers activity, they ranked "Cloud and Distributed Computing" and "Middleware and Integration Software" as respectively top 1 and top 3 most needed skills for companies.

Guess what? Distributed streaming platforms like Apache Kafka fall in both of these top skill categories.

Put more casually, Apache Kafka can help you land a job — no less.

I also think that knowing how to use, deploy and design systems interacting with a distributed streaming platform like Apache Kafka is a great skill to have in your toolbox. You'll be able to work across multiple projects and with a lot of different people, which I've always found downright thrilling. 🔥

Scalable systems and teams

Streaming applications with Kafka are multi-tenant, in the sense that a Kafka topic can have multiple producers and multiple consumers.

In particular, this means that increasing throughput generally boils down to a) increasing the number of producers and/or b) increasing the number of consumers. Streaming applications are inherently horizontally scalable.

But there's more to it.

Have you noticed how producers do not have to know anything about who the downstream consumers are? Producers simply push messages to topics, and whoever is interested in them can subscribe to them and do their own processing.

This means that you can have loosely coupled teams working on separate parts of an application. Their interface is exactly defined by the contents of messages — the data is the contract.

Scaling systems and teams with Kafka Octopus! (I've been waiting for this moment for too long.)

Even better — a new application (say, developed by the BI team) can come and tap into an existing topic without disrupting the flow of other stream processors. Beyond encouraging loose coupling, this property means that streaming applications allow better scaling of teams.

There are many more such benefits of streaming data for human organisations. Here's a last one before we wrap up.

Reactive businesses

As a company, you want your product or services to be able to react to changes in their environment.

A typical way of doing so is batch processing. You build up changes in buffering applications, whose task is to prepare batches to be processed by other systems. However, this way of doing is typically slow both in duration (batches being processed over hours) and frequency (a batch being processed twice or three times a day, but no more).

In contrast, streaming data allows you to process a continuous stream of data. Whatever happens in your system, you can catch it and react to it rapidly. It is then very easy to achieve minute-level reactivity. In the end, streaming data decreases your response time allowing you to wave the killer argument of real-time processing. Boom! 💥

Welcome to a world of streams and events

As we build systems ever more complex, it turns out that the usual request-based approach is not enough to harness events. Instead, a stream is a time-ordered collection of events and an intuitive and efficient way to build reactive applications.

The streaming data paradigm is implemented in Apache Kafka through a set of simple yet powerful abstractions, and I strongly believe you need to become knowledgeable about it.

Wrapping up

Updated on December 15, 2018.

I initially planned to write up a final blog post on Apache Kafka's APIs (and perhaps more on cluster deployment), but for various reasons I unfortunately won't have the time to publish them.

This means that this series is coming to an end! Of course, there is way much more to Kafka than I have covered here! We've only scratched the surface. To continue your journey, I've gathered a few great resources for you to enjoy:

How to use Apache Kafka to transform a batch pipline into a real time one: a thorough example of designing a streaming pipeline, also with a fraud detection example.
Introduction to Schemas in Apache Kafka with the Confluent Schema Registry: using Avro as a safeguard for your events' formats.
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push: example of using Kafka for real-time user interfaces.
Lessons Learned From Kafka in Production: useful advice and rules of thumbs for configuring a live production Kafka cluster.
ETL is Dead, Long Live Streams: a case for moving from batch to stream processing, and an experience report of building and scaling a streaming platform.

I also highly recommend you O'Reilly's "Apache Kafka: the Definitive Guide". It helped me a lot in the beginning as it contains a lot of useful background and great guidance on how to set up and configure a cluster, how to monitor Kafka as well as more configuration details on consumers and producers.

Take some time to see how stream processing may be applied to systems you know or you've built, and I'd be happy to hear about the outcome! 🖖💻

We have now covered some of the core concepts behind streams in Apache Kafka. The next blog post will be more technical as we'll try to make sense of Apache Kafka's APIs and abstractions by looking at their actual implementation.

Until then, take some time to see how stream processing may be applied to systems you know or you've built, and I'd be happy to hear about the outcome! Stay tuned for more Kafka goodness. 🖖💻

Stay in touch!

If you enjoyed this post, you can find me on Twitter for updates, announcements and news. 🐤

Forem: Florimond Manca

Introduction to ASGI: Emergence of an Async Python Web Ecosystem

florimondmanca / awesome-asgi

A curated list of awesome ASGI servers, frameworks, apps, libraries, and other resources

awesome-asgi

Application frameworks

It all started with async/await

An overview of ASGI

ASGI basics

Show me the code!

Turtles all the way down

Why should I care?

Where can I find ASGI-compatible components?

Wrapping up

HTTPX: Help Build The Future Of Python HTTP

encode / httpx

A next generation HTTP client for Python. 🦋

Features

How you can help

Try it out!

Contribute!

Documentation

.post(), .put(), and .patch() functions in API docs missing 'files' parameter #409

Environment Variables docs should justify each usage case. #404

More explicit documentation that `httpx.Client` is equivelent to `requests.Session`. #403

Bug fixes

Host header contains userinfo component #391

New features

Add support for the NO_PROXY environment variable #360

[Feature Request]: raise httpx exception instead of buildin ConnectionResetError and ConnectionError #349

Closing thoughts

How To Upgrade Your VuePress Site To v1.0

1. Read the official announcement

Intro to VuePress 1.x. TL;DR: This post is a speech record of… | by ULIVZ | Medium

ULIVZ ・ Jun 13, 2019 ・ Medium

2. Read the migration guide

3. Prepare your repo

4. Upgrade VuePress

5. Use the new plugins

Google Analytics

markdown.config

Service Worker

6. Update styles

7. Update component imports

8. Fix custom layouts

9. Consider theme inheritance

10. Final checks

Conclusion

Entropic: federated JS package registry, announced at JSConfEU 2019

GitHub announces GitHub Package Registry

Peter Kim Frank ・ May 10 '19

entropic-dev / entropic

🦝 📦 a package registry for anything, but mostly javascript 🦝 🦝 🦝

Entropic: a federated package registry for anything

Attending Tech Conferences: A Beginner’s Guide

Basics

What is a tech conference?

Why should I attend?

Do I need to be an expert to attend conferences?

How do I find conferences?

Deciding on a conference to attend

Before the conference

Getting your ticket

Logistics

Preparation

During the conference

Talks

Other activities

Social media

Networking

Behavior

After the conference

Wrapping up

Building a real-time chatbot server in Python with WebSocket, ChatterBot and Bocadillo

Setting up the project

Bootstrapping the application

Writing the WebSocket endpoint

Trying out the WebSocket endpoint

Hello, Diego!

Refactoring the chatbot as a provider

ULIVZ ・ Jun 13, 2019 ・
Medium

`markdown.config`

Generating the `CourseListComponent`

Integrating the API with the `CourseService`

Fetching courses with the `CourseService`