Scontreeno - Your AI-empowered expense manager companion

Marco T. — Sun, 23 Jun 2024 16:21:38 +0000

This is a submission for Twilio Challenge v24.06.12

What I Built

Scontreeno - whose name is taken from "Scontrino", wich is the italian term for "Recipt" - is a Twilio and AI powered expense manager companion.
Scontreeno is offered as a Whatsapp bot in order to simplify the user experience. You can simply chat with it and upload an image (or a PDF) or your receipt. Scontreeno will analyze it, by using Microsoft Azure AI Services, and will ingest it into the system, creating an AI-enriched search index.

Scontreeno is capable of understanding where, when, what you purchased and how much you paid (both for the single items and for the total). By ingesting these informations, it's quite easy to enrich the bot with a conversational-based search engine in order to let the user ask:
"Hey, how much did I spend for vegetables last week?" or
"Hey, can you please tell me if I spent more for meat or fish during the last month?"

Scontreeno could also provide insights and suggestions to users (even extracting informations from other users' receipt). For example:

you purchase a lot of alcohol / sodas
your weekly vegetable expense is too low
you can find your favourite brands at a lower price at shop XYZ this week

Architectural analysis of the Scontreeno ecosystem

Scontreeno's demo is composed by two Microsoft Azure functions:

TwilioInput, which receives messages from Twilio
ReceiptInput, which receives uploaded media

TwilioInput has the responsability of understanding if the message contains a media and, if's true, to upload this content to Azure Blob Storage.

As soon as a file is uploaded an Event Grid event is raised and the second function, ReceiptInput, is triggered. Uploaded media is analyzed using Azure AI Document Intelligence, a powerful cognitive service provided by Microsoft, which can interpret what's inside the document.

Results are then sent back to the user, using Twilio APIs.

Azure AI Document Intelligence output can be easilly linked to Azure AI Search, out of the scope of this demo, to index document and process them.

Code

Find the public repository here: https://github.com/marconline-scontreeno/Scontreeno

Demo

You can try it - only document upload (jpg / pdf) and receipt analysis - by following these steps:

Add Twilio Whatsapp Sandbox number to your contacts: +1(415)523-8886
Send him a "join save-famous" message
As soon as you are accepted into the sandbox, kindly reply with a "hello" message
After you have been greeted, upload a picture of a receipt (JPG / PDF version). You can also try adding something that is not a receipt, in order to verify that Scontreeno is smart enough!

Twilio and AI

Programmable Messaging is a great Twilio feature and being integrated with Whatsapp let you create incredible user experiences. Processing sent messages using AI - like in this Scontreeno demo - let you create powerful services with almost no effort.

Scontreeno is capable of extracting informations from pictures of receipt like this one, even if sentences are skewed and folded.

Additional Prize Categories

Impactful Innovators: in my opinion, the real power of Scontreeno is to let user understand how money is spent and when, so the first positive impact is a money management educations for users. But I won't underestimate the power of health suggestions: being able to understand what the user purchased, we can provide him with great insights useful for his health (like: you purchases too much sodas / alcohol, you are not buying vegetables or fruit, your meat consumption is too high and so on). By using Twilio and Whatsapp, tracking expenses is so easy!