Forem: Alessandro Cuppari

Getting your user's country and name in a Whatsapp bot using Fastapi and FaunaDB - Part 3

Alessandro Cuppari — Sat, 23 May 2020 22:22:40 +0000

We're back! So a bit recap of the last part: we now have our Fauna database set up, as well as our API with our greeting endpoint and the functionality to search for a user via their phone number. But before we continue, let me tell you I lied to you: I told you we would fetch the user's country from the phone number, but we did not. So let's do that right away!

Getting a country from a phone number

For this, we'll use the great phonenumbers library, which is the Python fork of Google's libphonenumber. Install it like so:

pip install phonenumbers

Now, let's go back to our greet.py file:

from fastapi import APIRouter, HTTPException, Form
import phonenumbers
from phonenumbers import geocoder # For geographical data

from src.models import User

user_greeting = APIRouter()

def phone_to_country(number: str) -> str:
    parsed_number = phonenumbers.parse(number)

    if not phonenumbers.is_possible_number(parsed_number):
        raise HTTPException(status_code=400, detail="Invalid phone number")

    return geocoder.country_name_for_number(parsed_number, "en")

@user_greeting.post("/greeting")
def greet_user(UserIdentifier: str = Form(...)):
    """
    User greet endpoint
    :param: UserIdentifier: user's phone number from Twilio
    """

    user = User.get_by_phone(phone_number=UserIdentifier)

# Greeting the user since already exists
    if user is not None:
        return {
            "actions": [
                {"remember": {"name": user.name, "country": user.country}},
                {"say": "Hi there {name}! Welcome back, how are things in {c}?".format(name=user.name, c=user.country)},
            ]
        }
    country = phone_to_country(UserIdentifier)

    return {
        "actions": [
            {
                "say": "Hello there! Looks like you are writing from {c}".format(c=country)
            },
            {"redirect": "task://can-have-name"},
        ]
    }

See that, besides importing phonenumbers we also import geocoder, this adds the ability to obtain the country (among other geographical information) from a phone number; it is a separated import because it adds a large amount of metadata.

Next, we write a little helper function to parse a phone number and throw a Bad Request error in case the number it's not valid. This would only happen if this endpoint it's hit outside of Twilio, but let's add it anyway. Then, it's just a matter of using that helper function in case the user its not in the database.

Cool, with that out of the way, let's do the rest of the API.

Saving users to FaunaDB

Ok, so the first thing we need its a method to store a user in Fauna. Lets quickly add one to our User class:

from typing import Optional, Any

from pydantic import BaseModel

from faunadb import query as q
from faunadb.objects import Ref

from src.core import session

class User(BaseModel):
  _collection_name: str = 'users'
  ref: Optional[Ref]
  ts: Optional[str]
  phone_number: str
  name: str
  country: str


  class Config:
    arbitrary_types_allowed = True

  def __init__(self, **data: Any):
    super().__init__(**data)

  def save(self):
    """
    Saves the document in the collection,
    the attributes are serialized utilizing Pydantic's dict method, which
    traverses trough the child class attributes
    :return: An instance of the newly saved object.
    """
    attributes = self.dict()
    result = session().query(
        q.create(q.collection(self._collection_name), {"data": attributes})
    )

    return self.__class__(ref=result["ref"], ts=result["ts"], **result["data"])
 # The code from the previous post

Thanks to Pydantic, this is pretty straightforward:

First, I added an extra attribute (_collection_name) to get the collection's name; it matches the name we set on the Fauna dashboard
We get the data we want to save as a dict, we use the BaseModel's dict instance method; which does exactly this.
Next is the saving process itself:
- We then use the session object to create a query
- We pass a collection object using the collection name attribute
- Finally, we pass the attributes dict as a second argument. Remember we said that non-standard Fauna objects need to be inside a data dict? Here we do exactly that.
- Finally, we create a new User object using the class's __class__ magic method; which it's basically like calling the constructor. The result object is nothing more than the object returned by Fauna's saving operation. I know that this may not be the best approach (perhaps we should just set the new attributes instead of returning a new instance), but hey that's what I did

Parsing `collect` actions

Now, let's finally go to the reason for this entire series: asking the user for their name.

Asking the user if they want to give us their name

Remember the first post, when we defined our schema? We said that our endpoint's route would be can-have-name, the task's name would be the same, the action's name would be ask-for-name and the answer from the action would be can_have_name. Recall that action and tasks are two different things: a task is a series of one or more actions that a bot performs, while the action its the specific thing the bot will do in that task; in this case, asking a question.

Now, the code:

import json
# The other imports

# The code we added at the beginning

@user_greeting.post("/can-have-name")
def can_have_name(Memory: str = Form(...)):
    """
    Asks the user if he/she wants to give us their name
    :param: Memory: JSON Stringified object from Twilio
    """
    memory = json.loads(Memory)

    answer = memory["twilio"]["collected_data"]["ask-for-name"]["answers"][
        "can_have_name"
    ]["answer"]
    if answer == "Yes":
        return {"actions": [{"redirect": "task://store-user"}]}
    return {
        "actions": [
            {
                "say": "😭"
            }
        ]
    }

Recall that our collect question had a type of Twilio.YES_NO, that means that Autopilot's natural language processing engine has done it's magic to convert whatever the user wrote into a Yes or No answer (those two are literal). So we just need to parse that response.

Here, you need to take into account the thing that had me hitting the keyboard with my head a couple of days: Autopilot sends all requests with an encoding of application/x-www- form-urlencoded and the Memory object it's in JSON, but, because of the encoding, this JSON is not parsed; it is just a plain string. Because of that, we set the Memory parameter to be of type str and we then use Python's standard library json.loads to parse the Memory string into a dict. Now, what's that intricate dictionary right there? Well, the Memory object contains other things besides parsed responses, the parsed responses are saved under collected_data. After that key: the format is action_name.answers.question_name.answer, as we said above, our action's name is ask-for-name and the question is can_have_name.

So after that confusing prop drilling, we have our answer. If the user said yes we just redirect to the task tasked (no pun intended) to get the name; if not well we just can do whatever we want. In this case let's just return an emoji, why? Because we are using Whatsapp and we can!

Finally getting the name

For the last endpoint, let's get the user's name. Recall that we
wrote in our schema that the endpoint's route is store-user:

# Rest of greet.py up here

@user_greeting.post("/store-user")
def store_user(UserIdentifier: str = Form(...), Memory: str = Form(...)):
    """
    Stores a user in the database, fields stored are: country, name, and phone number
    :param: UserIdentifier: Phone number from Twilio
    :param: Memory: JSON Stringified object from Twilio
    """
    memory = json.loads(Memory)
    name: str = memory["twilio"]["collected_data"]["collect-name"]["answers"][
        "first_name"
    ]["answer"]

    country = phone_to_country(UserIdentifier)
    # This needs error handling right here ;)
    new_user = User(
        name=name.capitalize(), phone_number=UserIdentifier, country=country
    ).save()

    return {
        "actions": [
            {"remember": {"name": new_user.name, "country": new_user.country}},
            {
                "say": "Hi, {name} from {c}!".format(
                    name=new_user.name, c=country
                )
            },
            {
                "say": "This is a WIP ok bye"
            }
        ]
    }

Besides the Memory object, we also need the UserIdentifier; see how FastAPI allows us to pass as a parameter just the fields we need from the request.

The next couple of lines are the same as with the last endpoint, the only thing that changes is the keys for the memory dict: in this case, the name of the action is collect-name and the name of the question is first_name. If you check our schema, you'll realize that the type of question is Twilio.FIRST_NAME; was does this mean? You guessed it! Autopilot's natural language processing engine will try to parse the answer as a first name.

Now, the saving process:

First, we parse the country just like we did in the /greeting endpoint
Then, we pass instantiate a User object with the parameters we have: name (we sanitize the input a bit via capitalizing the string), country, and the phone; we also call the save method right away to have a full instance
Then, like in the first endpoint, we tell the bot to remember the name and the country and return whatever action we want

Testing all out

So now comes the part we've waited for: testing this bad boy out. For that, we first need to deploy our API to get an endpoint to put in our schema. There are several ways to do this, for example, you could deploy this on Heroku, Digital Ocean, or just write the code directly on Glitch to get a live URL (which is what I did for this series); whatever you choose is up to you! Just remember to set the DEBUG flag to false and to fill the FAUNA_SERVER_KEY env variable with a server key.

After the API is live, let's go to the schema file and replace all the occurrences of our_super_yet_to_be_endpoint with the live URL of our API. Now is the time to create the bot, go create a Twilio account if you already haven't; after that, copy the SID and the auth token you'll find in the dashboard.

There are two ways of creating an Autopilot bot: the first one is through the console and the second one is using the CLI. Since we already have the schema for the bot ready, let's use the CLI; you need either yarn or npm for that, I'll use the former:

yarn global add twilio-cli
twilio plugins:install @dabblelab/plugin-autopilot

After that's installed, to a twilio login in your terminal and paste the SID and auth key when prompted.

Now, set the terminal on the path where the finished schema is stored and do:

twilio autopilot:create -s=./schema.json

That will create a bot named almighty-bot with the account you used to log in. Now, for the final part, we need to request a phone number from Twilio and enable WhatsApp integration. The process for doing that is out of the scope of this post, so check here for enabling WhatsApp and connecting that number to our bot.

Texting the bot

Now, let's add our bot to our contact list a send a message!

We can greet the bot again:

Finally, let's go to the Fauna dashboard and erase our user; just to say no and see what happens:

Great! We see that our bot reacts correctly to "yes" and "no".

"But wait a minute", I hear you say "Didn't you tell me that a session lasts for 24 hours? Why did the conversation reset the second time I greeted the bot". Well, that was a half-truth actually: a session lasts for 24 hours if we include a listen action at the end or if we redirect to another action. If you check the schema and the actions we returned from our endpoints, you'll realize that we did not return that action: we just drove the users to a pre-defined path, so to speak. So, if the returned action neither a redirection nor a listen action, the session will finish.

Are we done?

Of course not! You probably already thought about this: this does not seem too secure, does it? How do I know that the requests really come from Twilio? If you did, you're right, this is completely insecure: everyone could figure our API out and make requests to it. So let's change that.

Twilio has a package for Python that has, among other things, a class to validate that incoming requests come from Twilio; let's install it:

pip install twilio

Now, how do we implement this for every request? Well, if you are like me and already have worked with Python in the past, you probably are thinking of a decorator. And, if you're like me, you went directly to implement said decorator just to have it fail miserably when deployed live.

You see, Starlette (and FastAPI, by extension), don't quite work like this. Like I've mentioned a million times before, they are asynchronous; thus, various properties of the request object (like the body or the form) are implemented as asynchronous streams: if you consume them in, say, a decorator, they will complete and will not be available to the next handler in the chain. So, in mortal words: accessing the async parts of the request in any other part that it's not its final destination will make that data unavailable in that final destination; therefore an error will occur.

So, if not a decorator, how do we implement this? Well, we need to make a custom router. What's this? A router is a class whose methods get called before the request (or the response) hits the handler function. Because of this, we have, in a router, complete access to the request (or the response). Let's write the code:

from typing import Callable

from fastapi import Request, Response, HTTPException
from fastapi.routing import APIRoute
from starlette.datastructures import FormData
from twilio.request_validator import RequestValidator

from src.core import config

class AutopilotRequest(Request):
    """
    This class serves two purposes. First one, know that Starlette (framework on which Fastapi is built upon)
    is an ASGI framework. That means that parts of the request (like the body) are async. So, if we await those streams in middleware they will be consumed and will not be available to the final route.
    For that, this class consumes the steam (in this case the form) does what it needs to do with the data,
    and creates a new FormData object to pass to the final route.
    """

    async def form(self):
        if not config.DEBUG:
            return await self.__handle_non_dev_env()
    return super().form()

    async def __handle_non_dev_env(self):
        """
        In production or staging, validate that the request comes from Twilio
        """
        validator = RequestValidator(config.TWILIO_AUTH_TOKEN)

        params = await super().form()
        x_twilio_signature = super().headers.get("X-Twilio-Signature", "no-header")
        is_valid = validator.validate(str(super().url), params, x_twilio_signature)
        if not is_valid:
            raise HTTPException(status_code=403)
        return FormData(dict(params.items()))

class AutopilotRoute(APIRoute):
    """
    The custom route to route requests through our AutopilotRequest object
    """

    def get_route_handler(self) -> Callable:
        original_route_handler = super().get_route_handler()

        async def custom_route_handler(request: Request) -> Response:
            new_request = AutopilotRequest(request.scope, request.receive)
            return await original_route_handler(new_request)

        return custom_route_handler

Ok, what's going on here? First, for a custom router, we need two things: the custom router and the custom request object. For this, we subclass ApiRoute and Request (from FastApi) respectively.

Now, for the request:

We are only interested in the form property since that is where Twilio data is
We check if we're on debug mode if we are then we just return the original form
Now, in the __handle_non_dev_env is where the magic happens
- We create a RequestValidator object with the TWILIO_AUTH_TOKEN (remember that?) property that is already in our configuration object
- We consume the form stream, see that wee need to await it. We need to do this because it contains data the validator needs
- Then, we extract the header from the request. The signature is in the X-Twilio-Signature header
- Now, we use the RequestValidator's validate method to perform the validation
- Next, we need to check if that method returned True, if not we raise a 403 Forbidden error
- Finally, if the request did indeed come for Twilio, then we need to construct a new form since the old one was consumed; for this we just do: FormData(dict(params.items())) which will return a new FormData with the original data from the request

Now, the router is simpler: it just needs to convert the incoming request to an AutopilotRequest and pass it to the original handler. That way, when the handler calls the form method under the hood, the validation will occur.

How do we use this? Easily, let's go to our greet.py file:

# routes/greet.py

# Original imports
from .custom_router import AutopilotRoute

user_greeting = APIRouter()
user_greeting.route_class = AutopilotRoute # <- here

That's it, now just remember to fetch your Twilio auth token and set the TWILIO_AUTH_TOKEN with it.

Bonus

Now you're thinking: "Wait a minute, do I need to use my phone to debug this? Doesn't this like, cost money?" Yes! You're also right, texting a number provided by Twilio costs money. For Autopilot, they provide a simulator; which is great for debugging. But, unfortunately, the simulator does not send the UserIdentifier property; since is not a phone.

But fear not! I have you covered as well. Remember that FAKE_NUMBER property? Let's use that now, back our AutopilotRoute:

# The imports

class AutopilotRequest(Request):
    # The huge docstring

    async def form(self):
        if not config.DEBUG:
            return await self.__handle_non_dev_env()
        return await self.__handle_dev_env()

    async def __handle_dev_env(self):
        """
        Here we just inject a fake number for testing, this so we can test from
        the Twilio Autopilot Simulator through an SSH tunnel.
        """
        form = await super().form()
        new_form = FormData(dict(form.items()), UserIdentifier=config.FAKE_NUMBER)
        return new_form

    # __handle_non_dev_env goes here
# Our router goes here

Now, we have added a __handle_dev_env method that gets executed when DEBUGis set to True. And whats does it do? Well, it takes whatever the request was and injects the UserIdentifier property as whatever number we specified in FAKE_NUMBER. Now behold!

So, for debugging, just set DEBUG to True and set FAKE_NUMBER to a valid phone number. Pro-tip: you can use this with something like Ngrok to debug your API locally.

Wrapping up

That's all folks! I hope you liked the series. I certainly did learn a lot while doing this. Before finishing up let me state that this is all very basic, it could be improved in so many ways:

Needs error handling
While the Twilio default types do a good job in recognizing common language, the bot still needs to be trained using samples from real conversations
Needs better automation for deployment
So much more!

If you want to check out how I solved this things (and also check a slightly more complex bot) you can check out the original bot I was developing before writing this post.

PS: Oh, also, here's the code for this post, did you think I would forget? ;)

Getting your user's country and name in a Whatsapp bot using Fastapi and FaunaDB - Part 2

Alessandro Cuppari — Wed, 20 May 2020 02:55:25 +0000

Hello there! Here we will continue when we left off in the previous part. So right now we have our shiny static actions all set up; now, in this post, we will scaffold the FastApi project and write the code needed to query FaunaDB.

Setting up our project

Ok so first I will assume you have both Python 3.8.2 and virtualenv in your system; if not, please set them up according to your operating system before continuing.

Next, let's create a folder and a virtual environment for our project; for the remaining of this article I will assume a UNIX-based system:

mkdir our-awesome-api && cd our-awesome-api
virtualenv venv
source venv/bin/activate

Now, we will install the packages we need, and what do we need? Just FastApi and Fauna! Well no, we'll need also a couple more:

pip install uvicorn fastapi faunadb python-dotenv

Ok, the two in the middle we recognize, but what are the other two? Well, python-dotenv it's a package FastApi (or more precisely, Pydantic) uses to parse .env files; just like Javascript's dotenv.

On the other hand, uvicorn it's an ASGI server; remember those? It's the layer that allows FastApi to talk to HTTP servers like Nginx or Apache, it also has a nice dev server built-in along with hot reload when the code changes.

The file structure

Now, let's create a nice file structure for our project:

└───src
    ├───core
    ├───models
    └───app.py

What do we have here?

src it's the source root of our project
In core lives all the code that's critical for the start of our application
In models are all the data models (like users, for example)
In app.py is where we create our application instance

Now that we have our packages installed and our file structure all set up, let's figure out what we need first.

Querying FaunaDB

The core of this project is, basically, storing and retrieving users. For that, we need to query our database; now, how do we do that? First, let's see how Fauna stores data: the records fauna stores are called documents (like in MongoDB), and documents are stored into collections. Like I mentioned in the previous article, Fauna does not enforce any type of data schema at the database level, and because of that, we can find inconsistencies between documents in the same collection. Nevertheless, all the documents have two common properties:

ref, which is a unique identifier given automatically to all documents (like an old fashioned id)
ts, which is a timestamp that changes every time an event occurs to this particular document (like creation or update). Fauna automatically stores the events in the lifetime of an object, and you can query it, but that is beyond the scope of this post.
data, this is a dictionary containing all the data we store for the object. So, for example, is this was a users collection this would contain information like the username or email of our users.

Cool, now that we know how the data is stored, we need to know how to query it. Because no data schema is enforced at the database level, Fauna does not know out of the box how to query our data (we can't for example, query by username like we would do in a relational database); to tell Fauna how to query our data, we need to create an index. Indexes are a set of instructions, associated with a particular collection, that tells fauna how to query that collection; at the same time, we can tell indexes what data we need to be retrieved. So, for example, maybe we need to list all our users, this list will only show their username and the date they signed up; it would be a waste of bandwidth to fetch all the additional data we won't show in the list. So, for this, we can create an index that does exactly this: paginate through all the users and retrieve only their username and the document creation date.

Ok, enough theoretical talking, let's recall our flowchart from the last post and see what indexes we need:

Now, we see that we need to check if the user already exists. For that, we'll need an index that queries our collection via a unique index. What property of our users is unique? That's right, their phone number! So, let's go to the Fauna website.

Creating an index

Well, that title is a bit misleading. We can't create an index for a collection that does not exist and we cannot create a collection for a database that does not exist. So let's go to FaunaDB and sign up using our GitHub credentials.

Then, in the dashboard, let's click in "New Database" and create our shiny new instance:

Next, let's create our collection, I named mine users for the sake of originality:

Now, with the database and collection created, we can finally make our index:

Ok, we have several things going on here:

First of all, we need to select to which collection this index belongs. We say that this is for querying our users collection.
Then, the index name. Pretty self-explanatory.
Now we have this thing called "terms", what's this? Well, indexes can be used to paginate over a collection (that is, to list a collection) or they can be used to query a collection using a special field (for example, via the phone number, as we have here). Terms are exactly that: the terms that this index will use to query our data against. We write phone_number here and we see the dashboard formats this to data.phone_number, that is because all non-standard fields are considered to be inside of the data dict I mentioned before. By standard fields, I mean fields that Fauna generated, like ref and ts.
The values are all those fields we want the index to fetch, in this case, we want to fetch the name, the country, and the ref. Notice how the dashboard put ref and no data.ref.

Cool! Now let's click save and leave the dashboard for now.

Querying Fauna from Python

Ok, now we need to write code in order to query our database from python. For that, I adopted an OOP approach, kinda like a traditional ORM:

# src/models/user_document.py
from typing import Optional, Any

from pydantic import BaseModel

from faunadb import query as q
from faunadb.objects import Ref

class User(BaseModel):
  ref: Optional[Ref]
  ts: Optional[str]
  phone_number: str
  name: str
  country: str

  class Config:
    arbitrary_types_allowed = True

  def __init__(self, **data: Any):
    super().__init__(**data)

 @classmethod
  def get_by_phone(cls, phone_number: str):
      result = session().query(
          q.paginate(q.match(q.index("get_by_phone_number"), phone_number))
      )
      if len(result["data"]) == 0:
          return None
      # The result is a list with the values ordered as the index we defined
      ref, name, country = result["data"][0]

      return User(
          ref=ref,
          name=name,
          country=country,
          phone_number=phone_number,
      )

Now, what the heck do we have here? First of all, what is BaseModel parent class? Well, FastApi uses Pydantic under the hood. Pydantic it's a validation library that uses the type hints introduced in Python 3.6 for doing its validation. For a class to be validated with Pydantic, we have to do two things:

Inherit from the BaseModel class
Defined our class attributes with type hints

And that's what we have done here:

ref and ts its Fauna's reference type as discussed above, the Python driver comes with its own type hints for the former, so we use those; for tss we use int, because it is just a UNIX timestamp. You'll see why are they marked as Optional in a moment.
We add the same attributes we defined in our index, so the phone number, the country, and the name.

After that, we have just defined a constructor to init the base class.

Now, we come to the heart of this class, the get_by_phone static method:

First, we create a query using the FaunaDB client (we will define the session object in a moment)
- Inside the query we pass a paginate object
- Then we say we want to match against and index, in this case, is the "get_by_phone_number" index we created above
- As a param to that index, we pass the phone_number parameter passed to the function
If the query returns an empty list, we say that we did not find the user, so we return None
If not, then we create an instance of User with the first element of the list the query returned. Notice that this is a static method, because of that we use User directly instead of self

Cool, now two things:

The order in which the document's properties are returned is the same order that we defined in our index
The returned data is a list, why is that? Well, remember that Fauna does not make any assumptions about our data. It does not know that the phone_number must be unique, that is for us to enforce at the application level. So for all Fauna knows, there could be a thousand documents with the same number, what we're doing here is a very educated guess; so we need to be careful to not allow duplicate phone numbers at the application level.

Populating that `core` folder

You noticed above that I imported a magical session object, you also probably guessed that was the Fauna client initialized somewhere. If you did, you are correct; let's write that initialization code now:

For this, we need to first write a model to hold the configuration of our app.

# src/models/config.py
from typing import Optional

from pydantic import BaseSettings, AnyHttpUrl, validator, HttpUrl
from pydantic.types import SecretStr

class Config(BaseSettings):
  DEBUG: bool = False
  FAUNA_SERVER_KEY: SecretStr = "your_server_key_here"
  FAKE_NUMBER: str = "+15555555"
  TWILIO_AUTH_TOKEN: Optional[str]

  def __init__(self, **kwargs):
        super().__init__(**kwargs)

  @validator("TWILIO_AUTH_TOKEN")
  def check_for_twilio_token(cls, v, values, field):
    if (
        "DEBUG" in values
        and not values["DEBUG"]
        and v is None
    ):
        raise RuntimeError("{f} must be set".format(f=field))
    return v

Now, what's this? Well, Pydantic comes with a very handy class to handle settings (and validation of those settings) for your application. Let's explain what's going on here:

First, the base class is BaseSettings instead of BaseModel
Then, we defined the configuration variables our application will need:
- DEBUG a handy flag to, well, enable debug mode. Defaults to False
- FAUNA_SERVER_KEY, the API key four our database. It is of type SecretStr, which is a helper type to avoid printing the value to stdout: it will print a bunch of asterisks instead of the real value
- FAKE_NUMBER, more about that later ;)
- TWILIO_AUTH_TOKEN our Twilio API Token, we'll need it later.
After that, there is a Pydantic decorator, this function acts as a custom validator for the property that was passed as a parameter to the decorator. In this case, if we are running in debug mode, we don't care if the Twilio API key is not present so we jus return; on the other hand, if DEBUG is set to false, we need to check that the Twilio API key is present, otherwise the application would raise an exception

Great, now let's put this model into use:

# src/core/config.py
import sys
from os import environ

from src.models.config import Config

if environ.get("DEBUG", True):
    config = Config(_env_file=".env")
else:
    config = Config()

Not much going on here: we are just checking if DEBUG is set. If it is, we populate our config via a .env file; if not the config will be taken from environment variables.

Now, the Pydantic base setting has an order of priority to load variables:

Higher priority goes to environment variables
The next higher priority goes to the .env file, so if the same variable is defined in the file and in the environment, the one in the environment will take precedence.
Defaults defined in the model

So if an attribute not marked as Optional which does not have a default and is not defined either in the .env nor in the environment, will cause Pydantic to throw an error.

Next, the misterious session object:

# src/core/engine.py
from faunadb.client import FaunaClient

_session = None


def session():
    """
    Helper function to get faunadb session, if there is one already it just returns
    """
    from .config import config

    global _session
    if _session is None:
        _session = FaunaClient(secret=config.FAUNA_SERVER_KEY.get_secret_value())
    return _session

Nothing crazy here (and sorry for the global), we are just checking if Fauna client is initialized: if it is, we just return it, if it is not, then we initialize it with the key loaded from our configuration. Check that we are calling the get_secret_value method, remember that helper class from Pydantic? Well, we have to call this method in order to get the real value, otherwise, we would get a bunch of asterisks.

Our first endpoint

Now finally! Let's write our first endpoint. What does this endpoint do? Well, according to our flowchart, it will be responsible to check if our user is in the database and will react accordingly. Let's create a routes folder inside our source root:

mkdir src/routes

Inside this is of course our __init__.py and our new greet.py file which will contain our routes. Next, let's first modify our bot's schema to redirect to our endpoint when the user greets:

"uniqueName": "greeting",
"actions": {
  "actions": [
    {
      "redirect": {
        "method": "POST",
        "uri": "our_super_yet_to_be_endpoint/greeting"
      }
    }
  ]
},
// Other stuff down here

Here, we just changed our default action to be a redirect instead of a say action. Also, we said the Autopilot should send this as a POST request.

Now, in order to write our endpoint, we need to take a look at the Autopilot request reference. There, we learn that autopilot sends the data using the application/x-www-form-urlencoded encoding. For FastApi to parse that, we need yet to add another library:

pip install python-multipart

Now, finally, our endpoint:

# src/routes/greet.py
from fastapi import APIRouter, HTTPException, Form

from src.models import User

user_greeting = APIRouter()

@user_greeting.post("/greeting")
def greet_user(UserIdentifier: str = Form(...)):
    """
    User greet endpoint
    :param: UserIdentifier: user's phone number from Twilio
    """

    user = User.get_by_phone(phone_number=UserIdentifier)

   # Greeting the user since already exists
    if user is not None:
        return {
            "actions": [
                {"remember": {"name": user.name, "country": user.country}},
                {"say": "Hi there {name}! Welcome back, how are things in {c}?".format(name=user.name, c=user.country)},
            ]
        }

    return {
        "actions": [
            {
                "say": "Hello there!"
            },
            {"redirect": "task://can-have-name"},
        ]
    }

Not much going on here, if you've worked with Python web frameworks this will be pretty familiar to you. Let's see what's going on:

First we create a router, this is what FastApi uses to bundle routes together. It helps organize code in self-contained routes.
Next, we use that object to create a POST endpoint using it as a decorator, FastApi has decorator for all the HTTP verbs. We pass the endpoint's URI as a parameter.
Next, the function name, it can be anything

Now, what's with the function parameters? Well, FastApi also takes advantage of Python type hints in order to validate our requests. So if, for example, we pass something like: email: str as a parameter, FastApi will look for a parameter called email of type str in the query parameter string of the request. Similarly for all types of request parameters (Body, path parameters, etc). The name of the function parameter needs to be the exactly the parameter we're expecting; otherwise, FastApi will respond with a 422 Unprocessable Entity error.

So, in this case, we're expecting a parameter called UserIdentifier (as per the Twilio docs), which is a string. We also pass Form as a default value: that way we tell FastApi that we are expecting an application/x-www-form-urlencoded request. The ellipsis (the three ...) inside the Form constructor is just telling that this form has no dependencies (dependencies are another FastApi feature beyond the scope of this post). But, after all, what's a UserIdentifier? Well, depends on the channel Autopilot is using to communicate with the user, in Whatsapp and SMS channel it will be the user's phone number.

After that it's all pretty self-explanatory:

We use the Fauna index we created before to check if the user exists.
If it exists, we return a personalized greeting
If not, we redirect to the action that will ask the user for their name; using the task's uniqueName we created in our previous post

By default, if we return a Python dict from a FastApi endpoint, it will be encoded as application/json; which is just what we need. Also, check that we are adhering to Autopilot's Action Schema. We already know about the say and redirect, but what is this remember thingy? Well, Autopilot bots have the concept of "session": sessions are a time frame on which a conversation between the bot and a user is considered active. While the session is active, all data interchanged between the bot and the user will be store and the bot is able to send messages to the user without having to wait for their input; the time the session is considered active varies between channels, for Whatsapp is 24 hours. Now, the bot also has a memory that lasts the whole session; with the remember action; we are telling the bot to store those values in memory. That way, if we need to refer to the user's name or country, we don't have to hit the database again.

Wrapping up

So that's all for this post folks! Initially, this was to be a two-post series, but it overextended a bit. So for the sake of not boring the heck out of you, I decided to make this into a three-post series. In the next post, we'll implement the endpoint to store the name, deploy this thing live and a few more tricks ;)

Getting your user's country and name in a Whatsapp bot using Fastapi and FaunaDB

Alessandro Cuppari — Wed, 06 May 2020 02:45:26 +0000

Hi there you all! Yes I know, the title it's a bit weird: I was supposed to document my project during the Twilio hackathon; but alas I didn't have the time to do it. So, I decided to do a kind of ~~post mortem~~ describing the part that took me the longest to develop. On top of it, celebrating my ~~second~~ first post on DEV I wanted to tackle this with a different angle.

Instead of doing a by-the-book tutorial, Which would make me look super smart, I want to walk through my thought process and document all the gotchas along the way; whilst keeping this as beginner-friendly as possible. So let's start!

What are we going to build?

We are going to build a virtual bot using Twilio Autopilot, In the end, this bot will work via a Whatsapp channel, but Autopilot allows you to connect to several other channels without changing your bot's code. This fancy bot will ask the user for its name and will store it in a database along with the user's phone number and country. Use case for this? Well maybe you are building a virtual assistant or you just want to bot to be polite and greet the user via their name; the possibilities are endless!

The stack

You probably guessed by now the building blocks we need: firstly, we need to persist data; so we need a database. Also, we need something to interact with that database and do some processing, like figuring out the user's country. For the former, I used FaunaDB: a multi-cloud decentralized database service and, for the later, the Fastapi framework: an ASGI framework built for the modern Python.

Wait, what?

"Wait a minute Alessandro", I hear you say, "That's a lot of sophisticated technical mumbo-jumbo you have there, care to elaborate?" Sure, let's see what these technologies are all about and why I choose them.

FaunaDB

Fauna it's a NoSQL database that's "built for the JAMStack", that last part means that it's designed to be accessed directly from front-end applications and, because of that, it offers the possibility to set permissions in a very granular way; so you can control with precision which users can access what data.

The NoSQL part means (you guessed it) that it does not use the Schema Query Language for making queries to the data; but this name can come with a bit of an understatement: it also means that it makes no assumptions about the shape of your data (hold this); so that means that, inside a collection of data (say users for example), we can find inconsistencies in records; for example: maybe a user record has a last name field but other user doesn't.

The decentralized part means that your database it's replicated across multiple instances, that's a fancy way of saying that there are a bunch of copies of your database scattered across several different servers. Why is this good? Because this offers data redundancy; sure, maybe you've heard that data redundancy it's the root of all evil and it must be eradicated from the face of the earth, but that depends on the context. In this case, it's good because it means that, if one of your servers fails, the other replicas are there to ensure that your service does not interrupt and, more importantly, that you don't lose any data; it also means that data will be server from the replica geographically closet to the user, resulting in faster response times. This replication it's harder to achieve when you have to set it up by yourself, but Fauna abstracts away all this hassle for us.

Why I choose it

Well all this explanation sound well and good, but the reasons I chose it are far less romantic:

1) Cost, as in money: as devs, when we embark on a side project that needs to be hosted live, we try to minimize costs as much as possible. And database hosting is one of those things that can be costly, especially if you don't pay much attention to the pricing model: you can end up with a bill that's higher than you expected. Fauna has a great free plan that's more than enough.
2) The nature of the schema: as you have noticed, the data schema it's pretty simple (name, phone number, and country); so spinning up a whole relational database with SQLAlchemy, migrations, and such was a bit overkill.
3) Sheer curiosity: Being a database built for serverless, I was curious to see if Fauna could adapt to a more traditional application, and since they have a Python driver that was a no brainer (Spoiler alert: it does, kinda).

FastAPI

FastaAPI is an Asynchronous Standard Gateway Interface (ASGI) framework built for the modern Python. It's built upon two great projects: Starlette (another ASGI framework) and Pydantic (a validation library with zero dependencies). So, what's ASGI anyways? Well, it's the natural evolution of Web Server Gateway Interface (WSGI).

You see, if you have worked with a WSGI framework (like Flask or Django) you know that they have built-in servers; so you can just test your application in real-time when development, usually with nice logging and live reload built-in. Thing is that, when the project gets deployed, that dev server won't cut it: you'll need a real server like Apache's HTTP Webserver, Nginx or IIS. But the detail it's that those servers are usually written in C/C++ and they handle requests via sockets and all that low-level stuff, they don't speak Python! So you need a "glue" for the servers to be able to call your Python application, that's were WSGI came in.

The problem with WSGI it's that it is synchronous: it cannot take advantage of asynchronous programming, thus it's often "slower" than pure asynchronous applications like those written with the Express framework or the Go programming language.

For that, ASGI was created: it brings a new interface that supports asynchronous code on the server-side with Python's async/await syntax and also brings new features like HTTP/2 support and Websockets.

I won't be diving into more details about this, that would be a post in its own. But you can find plenty of material on the web or here in DEV

Why I choose it

1) Because of Python!
2) I had my eye on FastAPI for a while and was just looking for an excuse to use it

Yeah, a couple of pretty objective well-researched reasons right there; thank you very much.

The bot

So now, let's get into the bot's design. First, let's head to Autopilot's documentation and we'll that bots are composed as follows:

The unique name of the bot
The defaults section, which defines (you guessed it) default behaviors for your bot: the default task, default task to run on errors and the default task to run in case a collect action fails (well's see what this is later on).
The styleSheet defines the "personality", here we define the default messages to say in case of errors or success, the default behavior of collect actions, and the voice for the bot (this in case we use the bot for automatic phone calls).
Tasks: the tasks (duh) they need to perform; each one with a name
- Actions: the individual actions that these tasks trigger.
- Samples: the part of the conversation that triggers this task.
- Fields: pre-defined fields that hold special value and will appear in your samples (we'll also see what's this about later on)

I will only be explained the tasks needed for this bot, so, for the complete list check the docs.

We can also see that task are a simple JSON schema and that there are two types of actions:

1) Static: those defined in the bot's JSON schema.
2) Dynamic: those that are returned from an API, must comply with the JSON schema.

As you probably have guessed by now, static actions cannot perform any kind of logic: they only trigger, perform the actions described in the schema, and end. If we are to do any kind of logic, we must use dynamic actions.

So, with this in mind, let's see which tasks we need. For the sake of making this article less boring, let's throw a flowchart right here:

From the flowchart's decision and process blocks, we can easily determine, not only the number of actions needed but also which actions are dynamic and which are static.

The static actions would be:

Ask the user if they want to give us their name; this question needs to be broken down into two, we'll see why in a moment.
The action that takes place after the user response decision block.

The only dynamic action will be the greet action since we need to check if the user is already in the database to build the appropriate greeting message.

Creating the initial schema

Ok, now that we determined how many actions we need; let's create the actual schema that will describe our bot. In this post, we'll be writing only the static actions; while the dynamic will be written in the next post. First of all, you need to know that there are two ways of developing an Autopilot bot: first one it's through the Twilio Dashboard, and the second one it's by editing the bot's JSON schema directly. In this little project, we'll be using the latter.

Setting up the schema

There are a bunch of templates ready for us to use; we'll be using the HelloWorld template since the other ones don't quite fit our use case. So go to that link and download the schema.json file inside Assistants/HelloWorld.

Let's open up the file with our favorite editor, first of all: we'll change our bot's name. Something like almighty-bot will do, so let's go and change the uniqueName and friendlyName keys:

{
    "friendlyName" : "The almighty bot",
    "logQueries" : true, // This activates logging messages to the Twilio Dashboard
    "uniqueName" : "almighty-bot"
    // The other stuff
}

Now, we'll see that the schema has only four tasks; two of which are fallback actions in case of error. Let's write the static actions we defined above:

Asking for the name

For asking information, we use the collect action. This action takes a name, an array of questions that will be made in the order they are defined, and an on_success callback; which can be an HTTP endpoint or another task.

Remember I said that this action needs to be broken down into two? Well, imagine that our bot asks the users for their name and the user responds with a negative answer, what do we do? As stated above, we cannot perform any logic in static actions; so we need to call an endpoint with our user's response (that is if they want to give us their name or not) and proceed accordingly; just like the flowchart above.

So, let'a add our very first task; inside the schema's actions array add:

{
   "uniqueName":"can-have-name",
   "actions":{
      "actions":[
         {
            "collect":{
               "on_complete":{
                  "redirect":{
                     "method":"POST",
                     "uri":"our_super_yet_to_be_endpoint/can-have-name"
                  }
               },
               "name":"ask-for-name",
               "questions":[
                  {
                     "type":"Twilio.YES_NO",
                     "question":"Do you want to tell me your name?",
                     "name":"can_have_name"
                  }
               ]
            }
         }
      ]
   },
   "fields":[
      {
         "uniqueName":"can_have_name",
         "fieldType":"Twilio.YES_NO"
      }
   ],
   "samples":[
      {
         "language":"en-US",
         "taggedText":"{can_have_name}"
      }
   ]
}

Ok, so, what kind of wizardry did we just do? First of all, the task needs a unique (machine-friendly) name. After that, we write the actions the task will perform. In this case, we are doing a collect. Our collect task has:

A name: ask-for-name
An on_complete key, the action that'll be performed when the collect ends. In this case, we are instructing the bot to do an HTTP call.
An array of questions, a question must have:
- A type.
- The actual question's text.
- A name, this is the key we'll use to parse the answers in our API; so put a programming language friendly name.

Notice that the type is Twilio.YES_NO; Autopilot already has a bunch of pre-defined types for the most common questions you'll need. These types are already trained to do natural language recognition; so we'll use it whenever we can.

Finally, notice the fields and samples keys. Like I stated above, fields are pre-defined, well, fields, that act like variable names and samples are the inputs that will trigger this particular task. All collect actions must have at least one sample (Twilio recommends at least 6) but, since we're expecting only a yes or no, we'll only add one sample, which sample? Well, since we're just expecting a yes or no, let's create a field with that type and add it to the samples enclosing the field's name with curly braces. Yes, I know this part with the samples and fields it's not clear at the moment; but bear with me, it'll be clear with the next task.

Now, let's add the actual question for asking the name:

 {
   "uniqueName":"store-user",
   "actions":{
      "actions":[
         {
            "collect":{
               "on_complete":{
                  "redirect":{
                     "method":"POST",
                     "uri":"our_super_yet_to_be_endpoint/store-user"
                  }
               },
               "name":"collect-name",
               "questions":[
                  {
                     "type":"Twilio.FIRST_NAME",
                     "question":"Great! What's your name?",
                     "name":"first_name"
                  }
               ]
            }
         }
      ]
   },
   "fields":[
      {
         "uniqueName":"first_name",
         "fieldType":"Twilio.FIRST_NAME"
      }
   ],
   "samples":[
      {
         "language":"en-US",
         "taggedText":"I'm {first_name}"
      },
      {
         "language":"en-US",
         "taggedText":"{first_name} it's what I'm called"
      },
      {
         "language":"en-US",
         "taggedText":"{first_name} is my name"
      },
      {
         "language":"en-US",
         "taggedText":"people call me {first_name}"
      },
      {
         "language":"en-US",
         "taggedText":"people know me as {first_name}"
      },
      {
         "language":"en-US",
         "taggedText":"{first_name}"
      },
      {
         "language":"en-US",
         "taggedText":"my name is {first_name}"
      },
      {
         "language":"en-US",
         "taggedText":"i'm {first_name}"
      }
   ]
}

Now, what's going on here it's pretty much the same we did on the previous task: the only changes are the type of the question (now we tell Autopilot to try to recognize first names), the fields and samples and the URI of the endpoint. Remember that about fields being like variable names? So, imagine that the answers you're waiting for it's in the middle of a broader sentence, like: "I'd like to make an appointment at 3 o'clock" or "My name is John"; that's where the fields come in: by defining a field, we can write samples and tell Autopilot on which part of the sample we expect our answer to be. If we take a look at the samples above, we'll see the field name ({first_name}) we defined in the part of the sentence we expect our answer to be; the more samples the better. So now, If our user texts something like "My name is John"; that will trigger the "my name is {first_name}" sample and Autopilot will parse it as the answer of the current active question.

The final action

This is the action that'll take place after the user has (or not) give us their name. For the sake of simplicity we'll thow a placeholder:

{
   "uniqueName":"placeholder",
   "actions":{
      "actions":[
         {
            "say":"Almighty bot under construction"
         }
      ]
   },
   "fields":[],
   "samples":[]
}

Here, we're introducing the say it (you guessed it again) simply sends a text to the user.

Wrapping up

That's all for this post! Not a lot of writing code I know, we'll be writing the API that wires all this up in the next post; so stay tuned😉

PS: I know, I need to work on post-titling skills

COVID-19 Helper Whatsapp Bot - A Twilio Hackathon Submission

Alessandro Cuppari — Wed, 29 Apr 2020 05:23:06 +0000

What I built

Category: Interesting Integrations

I set up to build a Whatsapp Bot using Twilio Autopilot which would help the user with several tasks in his daily quarantined life; several ones stayed just as ideas (hows that for feature-creep huh?) and my bot got the following function:

Statistical data of the COVID19 pandemic by country:
- "How are they doing in Spain?"
- "I'd like information on the USA"
Historical data of the COVID19 pandemic by country:
- "How was Ireland doing on march 15th"
A self-screening process (with a little help from AI):
- Here, the bot will ask the users if they are feeling the most common symptoms of COVID-19
- Then, along with other variables (do they live in a country with confirmed cases? have they had contact with someone that looks ill?), would advise the users if they should seek medical attention
- Of course, this self checker is merely informative, by no means replaces the diagnosis of a health professional

But first, show me dah GIFS!

Link to Code

You can join the sandbox by sending a Whatsapp message to: +1 (415) 523-8886, first text join using-similar and then say hi!

alessandrojcm / covid19-helper-bot

A helper Whatsapp Bot for COVID-19 related information

COVID19 Helper Bot

A Whatsapp Bot to help get information about COVID19, aiming to participate on the Twilio & Dev 2020 Hackathon.

Using Twilio Autopilot and Python 3.8

Stack

Python3.8.2
Poetry
Fastapi
- Pydantic
Fauna DB
Hosted on Heroku
Pytest
Loguru
Requests
Sentry
Click

Kickstarted with: https://github.com/Dectinc/cookiecutter-fastapi

Project structure

Files related to application are in the app or tests directories Application parts are:


├── app
│   ├── api - Api routes
│   │   └── routes
│   │       └── autopilot - Twilio Autopilot Dynamic actions
│   ├── core - Critical configuration (sessions, logging, etc)
│   ├── custom_routers - FastAPI custom routes
│   ├── error_handlers - Custom error handlers
│   ├── middlewares - FastAPI/Starlette middleware configuration
│   ├── models - Pydatic models
│   ├── scripts - Helper scripts (mainly cli stuff)
│   ├── services - Utilites for interaction with external apis and logic too heavy for the routes
│   └── utils

…

View on GitHub

The project

The one million dollar ideas

So first of all, when I saw the hackathon announcement I knew I had to participate. Right away I also knew that I wanted to make a Whastapp Bot and I wanted to do it with Python. But alas, what should the bot do? I'd started brainstorming ideas like crazy, here are some of then:

Query a specific statistic (ie: how many recoveries in Spain)
Query by more specific location (ie: how many cases in San Francisco)
Take the user's location and show open grocery stores near then
Send the user the latest news for their country daily
And there are a couple more left out

So, pretty feature creep huh? Some of those features were left out because of lack of time and others for technical reasons (for example, the Whatsapp API does not let you take the user's location; at least not yet). But I knew that I had to implement the self-screening process.

The stack, oh the stack

well, I knew more or less what I wanted to do, now with the stack. I wanted to it in Python and I wanted to do it with Fastapi, since Twilio has a helper library for Python that was a no brainer.

Now, the fun part was my database of choice: FaunaDB. Why did I choose it? Well, first of all, it has a generous free tier, and second, they advertise heavily for being a solution for the JAM stack; so I was curious to see how it fitted with a more traditional use case (spoiler: it can, but with some gotchas).

So I set sail with my shiny stack, and boy it was a ride. First, because it was my first time with an ASGI framework; and thus I had to wrap my head around Python coroutines and async routes. Things that you would normally do in WSGI frameworks are different here (middlewares, for example); but, overall, I had a blast working with Fastapi: I learn a lot in the process and was nice to work with a framework designed for the modern Python.

But Fauna, well that was another beast. I come mainly from a traditional relational database background, so jumping to NoSQL was a challenge of its own; there were a lot of gotchas in the process: where are my primary keys? If this value is unique why does my query return an array? Where are my migrations? How can this record be repeated if all its fields are the same? And all kinds of surprises.

Last but not least, the bot

At this point, I was sure developing the bot would be the bigger challenge. At first, it seemed like so: I couldn't get the Autopilot requests to hit my API; it turned out it was a case of me skipping a little of pretty important documentation (specifically how to parse application/x-www-form-urlencoded requests with FastAPI). But, once I had that sorted out, the bot itself turned out to be a breeze to develop.

Additional Resources/Info

The APIS I used are:

Novelcovid API, an API that aggregates data from several sources. I choose the one from the John Hopkins Universite
Endless Medical API, AI-powered trained with real records and diagnosis that suggests outcomes based on a patient's symptoms. Thanks to Lukasz Kiljanek for kindly answering my inquiries!

Conclusion

Overall, this was a great project and not only got me to learn several things in the process; at first, I was a little disappointed that I couldn't squeeze more features in, but seeing all the new concepts I managed to grasp in only a month made it a worthy ride.

It also got my first post on DEV published! So that alone was worth it (I may be following up with more in-depth articles about the code 😉). So that's all, happy coding and keep safe!

Forem: Alessandro Cuppari

Getting your user's country and name in a Whatsapp bot using Fastapi and FaunaDB - Part 3

Getting a country from a phone number

Saving users to FaunaDB

Parsing collect actions

Asking the user if they want to give us their name

Finally getting the name

Testing all out

Texting the bot

Are we done?

Bonus

Wrapping up

Getting your user's country and name in a Whatsapp bot using Fastapi and FaunaDB - Part 2

Setting up our project

The file structure

Querying FaunaDB

Creating an index

Querying Fauna from Python

Populating that core folder

Our first endpoint

Wrapping up

Getting your user's country and name in a Whatsapp bot using Fastapi and FaunaDB

What are we going to build?

The stack

Wait, what?

FaunaDB

Why I choose it

FastAPI

Why I choose it

The bot

Creating the initial schema

Setting up the schema

Asking for the name

The final action

Wrapping up

COVID-19 Helper Whatsapp Bot - A Twilio Hackathon Submission

What I built

Category: Interesting Integrations

But first, show me dah GIFS!

Link to Code

alessandrojcm / covid19-helper-bot

A helper Whatsapp Bot for COVID-19 related information

COVID19 Helper Bot

Stack

Project structure

The project

The one million dollar ideas

The stack, oh the stack

Last but not least, the bot

Additional Resources/Info

Conclusion

Parsing `collect` actions

Populating that `core` folder