Forem: Vladimir

From Amazon Lex to GPT-4: How to make a bot with your own data?

Vladimir — Tue, 04 Apr 2023 17:58:42 +0000

The ChatGPT and other OpenAI models is currently on hype. But they are not the only solution available. Or are they? Let's try to figure it out today without code, only general concepts, pain, and suffering.

Imagine that you need to create a chatbot that can answer user questions based on your own data: products in an online store, a knowledge base of a support service, marketing articles, etc. Or a list of cafes and coworking spaces, in my case.

Until recently, such dialogues came down to choosing from "Are you interested in this? - Click here. Interested in something else? - Click there. Didn't understand anything? - Wait for the operator". Not very friendly, but sometimes quite predictable and understandable, only creating such a complex logic takes a long time.

Ok, let's add some friendliness and create a "natural language interface" like "Show me 10 cafes with sockets close to me in London" (if marketers are to be believed, people will write even crazier things, just to find what they're looking for).

One of the first "human language recognizers" like this was the service Amazon Lex (as well as Google Dialogflow and a dozen others)

Amazon Lex V2 is an AWS service for building conversational interfaces for applications using voice and text. Amazon Lex V2 provides the deep functionality and flexibility of natural language understanding (NLU) and automatic speech recognition (ASR) so you can build highly engaging user experiences with lifelike, conversational interactions, and create new categories of products.

Amazon Lex V2 enables any developer to build conversational bots quickly. With Amazon Lex V2, no deep learning expertise is necessary—to create a bot, you specify the basic conversation flow in the Amazon Lex V2 console. Amazon Lex V2 manages the dialog and dynamically adjusts the responses in the conversation. Using the console, you can build, test, and publish your text or voice chatbot. You can then add the conversational interfaces to bots on mobile devices, web applications, and chat platforms (for example, Facebook Messenger).

Sounds great: it can extract entities from plain text and pass to the data API.

For my example above, I had to write an utterance "Show me {count} {type} with {sockets} close to me in {region}" and describe the entities. After that, the JSON {count: 10, type: cafe, sockets: many, region: London} was obtained from the original phrase.

But here's the problem, for a similar phrase "Give me 10 coworking in Riga", a completely different utterance is needed, and for the simplest query "5 workplaces nearby", a third one is needed. 🤷‍♂️ In general, I stopped after several hundred utterances of stupid word permutations. Running all the tests took about an hour.

Another pain is dialogues; for example, the seeker of coworking spaces' second request may be "What about cafes?". In Lex, context can be passed in three ways, but a limited number of times and only through code (Amazon Lambda functions).

Lex also has another way of use: training on a dataset of hundreds of thousands of questions and answers, and further automatic responses. Probably suitable for call centers with similar queries.

Well, let's move on to ChatGPT and the capabilities that are rumored about

Debunking myths: models available through API are "dumber" than a chat web interface because they don't have memory or context 🤦‍♂️

So, to work with a product catalog, you need to transmit the entire catalog in each request, which won't even fit in the 32k token limit of the most expensive gpt-4-32k model. And with each message, you need to transmit all previous requests and responses to maintain context.

This is how about 99.9(9)% of typical bots that start charging users from the 10th response (for example) work. The curtain falls.

In general, the second option for implementing the idea is tokenization and vectorization of the source texts, articles, or product catalogs; tokenization and vectorization of the user query and finding several closest suitable vectors based on cosine similarity.

Simplified, this is how semantic search works, and it doesn't require chat models. However, for the "friendliness" of the dialogue and responses, chat models can be used and the found vectors and their corresponding original texts can be transmitted along with the query.

Here are a few articles about it and a little more.

All actions can be performed on your own models, OpenAI models (such as text-embedding-ada-002) via API, and on any publicly available models, such as NLP cloud. Vectors can be stored in CSV files, but it's better to use specialized vector databases like Qdrant.

Nowadays, this approach is becoming a de facto standard, and most search plugins for ChatGPT are based on it.

There are several advantages:

low cost of vectorization (it's only performed when the source data is uploaded/modified) and storage, especially in a local database
some possibility to maintain context by carrying the entire dialogue with each request to the chat model

There are also some disadvantages:

source data must be text-based, or more precisely, descriptive (my cafe catalog made of enum parameters in JSON format didn't work)
there must be a lot of them

Services have already appeared that simplify the implementation of the entire stack from data preparation to obtaining chat code for your website. For example, Databerry and Spellbook. And there are also good alternative models, such as Vicuna.

After vector experiments, I switched to the third option - translating human requests into JSON using a chat model

It turned out to be the easiest, cheapest, and fastest way to implement my initial idea 😎

The model is accompanied by instructions that are transmitted together with the user's request.

Convert the question below to JSON data.
Mostly questions are related to cafes and coworkings with different amenities.
Use only following parameters.
Skip unknown parameters and parameters that not in question.
Just output JSON data without explanation, notes or error messages!
Parameters
"""
- count: integer from 0 to 5
- type: one of "Cafe", "Coworking" and "Anticafe"
- region: any city
- sockets: one of "None", "Few" and "Many"
- noise: one of "Quiet", "Medium" and "Noisy"
- size: one of "Small", "Average" and "Big"
- busyness: one of "Low", "Average" and "High"
- view: one of "Street", "Roofs" and "Garden"
- cuisine: one of "Coffee & snacks" and "Full"
- roundclock: one of true and false

In most cases, a perfectly normal JSON response like {count: 5, type: cafe, sockets: many, region: London} is returned, which can be passed on to a microservice API.

But a text generation model wouldn't be a text generation model if it always responded the same way (even when frozen with temperature=0). Approximately 10% of the time, it "gets stuck" and adds non-existent parameters or forgets to close the JSON, and a similar repeated request is processed normally.

It's pointless to fight this, but you can remove non-existent parameters or invalid values by validating the response against the JSON schema, and also suggest that the user ask the bot again.

By the way, the latest gpt-4 model turned out to be "smarter", more predictable, and doesn't add random parameters, but it costs 6 times more. We're waiting for gpt-4-turbo.

You can ask the resulting bot here @WorkplacesDigitalBot, its budget is $10/month until it starts earning on its own, and there's no context saving.

Workplaces for digital nomads: the frontend

Vladimir — Tue, 01 Nov 2022 13:59:17 +0000

A continuation of the story about the development of a pet project about cafés and co-working spaces in sunny Cyprus. Workplaces for digital nomads ヽ(。_°)ノ

I talked the API microservice in the first part, the frontend site in the second part.

The project code is open, website https://workplaces.cy/.

Architecture

The site is built as a PWA using the Vue 3 Composition API and the Vuetify UI framework. Both tools are ideal for a quick start from scratch and contain less redundant code than earlier versions.

The major portion of the site is Google Maps in its always-free tier, which displays data from the microservice's REST API, as well as a panel with filters and a list of workplaces.

The entry component src/components/Home.vue:

manages the display of sections for mobile and desktop versions;
manages data from the API, which is retrieved from src/composable/api.js;
handles them with computed filters from src/components/Filters.vue;
passes the filtered data to component src/components/Places.vue to display a list of locations;
handles a 404 error;
sets meta tags via component src/components/PlaceHead.vue.

There is no separate "place card" in the project, a simple scrolling of the place list to the required place is sufficient for now:

watch(
    () => props.selectedPlaceId,
    id => {
      scrollToPlace(id)
    }
)

The filters are integrated in src/components/Filters.vue and stylized further for a more compact appearance.

Meta tags are handled by the vueuse/head package.

The fawmi/vue-google-maps library, version 0.9.72, is used to display the map. The author broke the map in later versions and did not publish the source code on GitHub (however, these versions are available via npm).

You may also see local points of interest by displaying your location on a map.

Swiper/vue and a number of components are used to display large and small photographs of locations in a carousel.

In the HSL colour space, a gradient colour rating of places is calculated using a simple function in src/composable/colors.js. Only the hue changes depending on the rating value, while the saturation and brightness remain fixed.

On the place cards, there is a Complain button to report any inaccuracies - it's all serious:-)

A simple src/router/index.js router based on vue-router allows you to avoid utilising the store for now (with the current project capabilities) and helps with 404 error handling.

Most components are loaded asynchronously to speed up page display, for example, const Navigation = defineAsyncComponent() => import("/Navigation.vue")).

When opening/closing the panel, a portion of the template is wrapped into <KeepAlive> to cache components. However, it did not work with Suspense for displaying stubs :-(

Instead of using Vuetify's default font icons, the project imports SVG icons from the mdi/js package at https://materialdesignicons.com. This saves approximately 1 MB in the final package.

Builds the project with Vite and some magic in vite.config.js to optimise the final code, including CSS and HTML minification and the creation of a lighter version of Sentry.

PWA is created with vite-pwa/vite-plugin-pwa, and its parameters are set in vite.config.js. In summary, the implementation of PWA was not a project aim, but it did provide good caching of all project components and resulted in a very quick reopening of the site.

About Vuetify

Overall, I preferred Vuetify over Quazar and other UI frameworks. However, it has its own drawbacks:

The difficulty in finely customising visual components. Vuetify is built for Material Design, therefore you can't just remove the indents between checkboxes or make mobile buttons smaller. Because of the high DOM depth, you may need to use :deep or wrap components in additional divs in some circumstances, lowering your Google PageSpeed score.
Unable to get rid of unused styles. Importing components individually and setting styles in SASS can greatly simplify builds, but several common unused styles can't be deleted using PurgeCSS and analogues due to dynamic class names.
Some components, such as the slider, have few features.

But I'm sure, this framework is good enough for many projects.

Deployment

The same Fly.io platform with managed micro VM Firecracker. It's much easier than hosting the microservice API here, and a four-line Dockerfile suffices:

FROM pierrezemb/gostatic
COPY docker/config/headerConfig.json /config/
COPY dist/ /srv/http/
ENTRYPOINT ["/goStatic", "-fallback", "/index.html"]

CI/CD

This is slightly more complicated: first, Github Action creates a build with Vite, then flyctl creates a container from it and deploys it to the production VM. All secrets are stored in the GitHub production environment in this case.

Monitoring

The same Sentry and Honeybadger.

At this stage, the website is live, hosted in a production and available to all users. The project is fully completed :-)

Frontend repository, website https://workplaces.cy/

Workplaces for digital nomads: the API

Vladimir — Mon, 31 Oct 2022 16:03:06 +0000

Another pet project: cafés and co-working spaces in sunny Cyprus. Workplaces for digital nomads ヽ(。_°)ノ

I like to share my development process. The overall development approach is pragmatic and similar to the previous project.

The project's goals

There are numerous cafes, coffee houses, taverns, restaurants, and bars on the island, but not everyone is good to work for at least a couple of hours.
There are well-known Starbucks, Costa Coffee, Gloria Jeans Coffee, and so on, but there are also very cosy and totally underrated local places.
That is why it was decided:

Classify locations based on factors relevant to remote working, such as kitchen, outlets, noise, size, occupancy, view, and so on.
Filter locations by the parameters you've chosen.
Display a map with relevant locations.
Create a desktop and mobile web app.

Overall, everything was successful and the project code is open. Website: https://workplaces.cy/

To achieve the objectives, it was decided to build a REST API microservice on Laravel, with an admin panel on Twill and a frontend web application on Vue. Deploy on Fly.io as before.

REST API microservice

The platform chosen is the familiar and lightweight Laravel and PHP 8.1 with promoted- and readonly- properties and strict typing.

The composer.json and project configuration is lightened as much as possible: unused packages and classes are removed, platform-check is disabled, and classmap-authoritative is enabled.

As a result, the number of classes to be loaded decreased 4.5 times from 28247 to 6230, the vendor directory was "thinned" almost 1.5 times, and tests were slightly faster.

Architecture

The main Place model is a typical Laravel model that extended Twill model (A17\Twill\Models\Model).

Properties for filtering - native PHP enum's with a few common methods from EnumValues trait to obtain values for the admin panel. These are cast in the properties of the model.

In addition, each property has a coefficient and a weight to calculate an place's rating. For example, the availability of outlets is more important than the view.

enum Sockets: string implements PropertyEnum
{
    use EnumValues;

    case None = 'None';
    case Few = 'Few';
    case Many = 'Many';

    public const WEIGHT = 3;


    public static function default(): self
    {
        return self::Few;
    }


    /** @inheritDoc */
    public function coefficient(): int
    {
        return match ($this) {
            self::None => 1,
            self::Few => 3,
            self::Many => 5,
        };
    }
}

API queries are handled by single-action controllers, validated by Request including enum matching. For example, IndexRequest.

#[OA\Parameter(name: 'busyness', in: 'query', allowEmptyValue: false, schema: new OA\Schema(type: 'string'))]
#[OA\Parameter(name: 'city', in: 'query', allowEmptyValue: false, schema: new OA\Schema(type: 'string'))]
#[OA\Parameter(name: 'size', in: 'query', allowEmptyValue: false, schema: new OA\Schema(type: 'string'))]
#[OA\Parameter(name: 'sockets', in: 'query', allowEmptyValue: false, schema: new OA\Schema(type: 'string'))]
#[OA\Parameter(name: 'noise', in: 'query', allowEmptyValue: false, schema: new OA\Schema(type: 'string'))]
#[OA\Parameter(name: 'type', in: 'query', allowEmptyValue: false, schema: new OA\Schema(type: 'string'))]
#[OA\Parameter(name: 'view', in: 'query', allowEmptyValue: false, schema: new OA\Schema(type: 'string'))]
#[OA\Parameter(name: 'cuisine', in: 'query', allowEmptyValue: false, schema: new OA\Schema(type: 'string'))]
#[OA\Parameter(name: 'vRate', in: 'query', allowEmptyValue: false, schema: new OA\Schema(type: 'string', format: 'float', maximum: 0, minimum: 5))]
final class IndexRequest extends FormRequest
{
    /** @return array{busyness: string, city: string, size: string, sockets: string, noise: string, type: string, view: string} */
    public function rules(): array
    {
        return [
            'busyness' => ['sometimes', 'required', new Enum(Busyness::class)],
            'city' => ['sometimes', 'required', new Enum(City::class)],
            'size' => ['sometimes', 'required', new Enum(Size::class)],
            'sockets' => ['sometimes', 'required', new Enum(Sockets::class)],
            'noise' => ['sometimes', 'required', new Enum(Noise::class)],
            'type' => ['sometimes', 'required', new Enum(Type::class)],
            'view' => ['sometimes', 'required', new Enum(View::class)],
            'cuisine' => ['sometimes', 'required', new Enum(Cuisine::class)],
            'vRate' => ['sometimes', 'required', 'float', 'numeric', 'between:0,5'],
        ];
    }
}

Native PHP attributes allowed OpenAPI markup to be much more compact than in DocBlocks. The resulting openapi.yaml is created with swagger-php and used to test the API.

Apart from the validators, requests are passed through EloquentFilter-based filters - a very expressive solution instead of a bunch of if's and when's.

Some places have photos that are transparently uploaded to AWS S3 from the admin and processed by the Imgix service. There is nothing on the API side to handle the pictures.

To get detailed geodata for the place from Google Maps, GooglePlacesService and alexpechkarev/google-maps package are used. In API service all the places are added only with the name, city and properties for rating. The rest data - coordinates, business ID, address and link are obtained from Google Places API.

To calculate the rating of an place VRateService is used.

Both services are wrapped in their respective actions and are available through console commands and events after recording the place.

The finished data is wrapped in PlaceResource and PlaceCollection. Excessive fields are also removed there. The middleware JsonResponse.php is used to force response in JSON format

final class JsonResponse
{
    /** @param Closure(Request): (BaseJsonResponse) $next */
    public function handle(Request $request, Closure $next): BaseJsonResponse
    {
        $request->headers->set('Accept', 'application/json');

        return $next($request);
    }
}

Administrative control panel

I've worked with Twill before, so I decided to use it for my project: an open, free system with rich features and good support. Why not? :-)

Installed via composer requires area17/twill, adds some migrations and communicates transparently with existing models. In some cases, you need to add service fields like `published and activity start/stop dates to them. However, the documentation describes everything in detail.

Now I recommend to try version 3-beta: it has much more options to programmatically manage the data on pages instead of separate widgets in the blade templates.

Example of a controller, repository and template.

Database

Simple and fast SQLite ¯_(ツ)_/¯
Hosted on a persistent volume. No configuration required.

Tests

For the tests, Pest is used with Laravel support, parallel execution of tests, and with disabled throttling ($this->withoutMiddleware(ThrottleRequests::class).

The endpoints are used to check datasets responses and their consistency with the OpenAPI specification.

For manual check there is Rector with some exceptions.

Found one downside: laravel/dusk and php-webdriver/webdriver are nailed to Twill and require mandatory installation, although not used in my tests :-(

Deployment

The server is hosted on the Fly.io platform with managed micro VM Firecracker. It never sleeps, has a good free tier, and allows you to host both static and any application server, unlike the popular Heroku. There are also different deployment and rollback strategies, health checks, and hosting geographies to choose from.

The runtime can be set up automatically with the flyctl launch command from the application directory or you can write your own config and Dockerfile.

I used my Dockerfile and run API microservice in the easiest way via php artisan serve.

Static distribution (admin assets and robots.txt & Co) can be delegated to the Fly platform by configuring fly.toml
[[statics]] guest_path = "/var/www/html/public/assets" url_prefix = "/assets"

CI/CD

It's simple here, GitHub Action with one workflow and the same flyctl.

Monitoring

Sentry is used for errors tracking and Honeybadger is used for uptime and availability checks.

At this point, the microservice API is live, hosted in production, and accessible to all users. MVP is complete :-)

API repository, website https://workplaces.cy/

I'll tell about creating the frontend with Vue 3 Composition API in the second part.

Pragmatic development 3: Telegram bot

Vladimir — Fri, 14 Oct 2022 13:17:22 +0000

The conclusion of a simple project about specialty coffee shops in Cyprus. The first part focused on the API microservice, the second on the frontend site, and the third on the Telegram bot.

The bot began with simple tasks:

/map - display map of coffee shops
/list - show list of coffee shops
show coffee shop details
/random - show random coffee shop
search for a coffee shop by the name
search for the nearest coffee shop by location or by /nearest command

During the implementation, it was discovered that Telegram cannot display an embedded map with multiple markers, nor does it send the location in the web version. As a result, instead of a real map, I had to show a link to a website with a map and a stub message instead of a response on an empty location. Everything else has been done.

The project code is open, bot https://t.me/SpecialtyCoffeeCyBot.

Architecture

Nutgram was chosen as the bot's base after much deliberation: it is the most lightweight, simple, and modern library. A fully configured DI container comes as a bonus, allowing you to avoid manual service initialization and delivery to customers.

Using an actual version of PHP 8.1 allowed me to write slightly less code while achieving slightly better performance. Development is made much easier by promoted properties, read-only properties, and strict typing.

The composer settings are simple as the API. The final composer.json file.

Telegram updates are received at the webhook endpoint and transmitted to handlers of commands and message types. Handlers can respond on their own or request data from the REST API. For the unexpected, there are Fallback, Exception, and ApiError handlers.

The use of short, single-action invokable handlers allowed the bot's logic to be condensed into only 32 lines!

$bot = new Nutgram($_ENV['BOT_TOKEN'], [
    'timeout' => $_ENV['CONNECT_TIMEOUT'],
    'logger' => ConsoleLogger::class
]);
$bot->setRunningMode(Webhook::class);

$bot->middleware(AuthMiddleware::class);

$bot->fallback(FallbackHandler::class);
$bot->onException(ExceptionHandler::class);
$bot->onApiError(ApiErrorHandler::class);

$bot->onText(NearestCommand::SEND_TEXT, NotSupportedHandler::class);

$bot->onMessageType(MessageTypes::TEXT, SearchHandler::class)->middleware(SearchRequirementsMiddleware::class);
$bot->onMessageType(MessageTypes::LOCATION, LocationHandler::class);

$bot->onMessageType(MessageTypes::NEW_CHAT_MEMBERS, NullHandler::class);
$bot->onMessageType(MessageTypes::LEFT_CHAT_MEMBER, NullHandler::class);

$bot->onCommand(ListCommand::getName(), ListCommand::class)->description(ListCommand::getDescription());
$bot->onCommand(MapCommand::getName(), MapCommand::class)->description(MapCommand::getDescription());
$bot->onCommand(NearestCommand::getName(), NearestCommand::class)->description(NearestCommand::getDescription());
$bot->onCommand(RandomCommand::getName(), RandomCommand::class)->description(RandomCommand::getDescription());
$bot->onCommand(StartCommand::getName(), StartCommand::class)->description(StartCommand::getDescription());

$bot->registerMyCommands();

$http = new Client(['base_uri' => $_ENV['API_URL']]);
$bot->getContainer()->addShared(Client::class, $http);

$bot->run();

A /nearest command example:

final class NearestCommand extends BaseCommand
{
    public const SEND_TEXT = 'Send location';


    public static function getName(): string
    {
        return 'nearest';
    }


    public static function getDescription(): string
    {
        return 'Show nearest specialty coffee shop';
    }


    public function getAnswer(): Answer
    {
        return new TextAnswer('Send your location to find the nearest coffee shop', [
            'reply_markup' => ReplyKeyboardMarkup::make(resize_keyboard: true)->addRow(KeyboardButton::make(self::SEND_TEXT, request_location: true)),
        ]);
    }
}

Location handler example:

final class LocationHandler extends BaseHandler
{
    private Location $location;


    public function __construct(Sender $sender, private readonly ApiService $api)
    {
        parent::__construct($sender);
    }


    public function __invoke(Nutgram $bot): ?Message
    {
        $this->location = $bot->message()->location;

        return $this->sender->send($this->getAnswer());
    }


    /** @inheritDoc */
    public function getAnswer(): Answer|array
    {
        $cafe = $this->api->getNearest((string)$this->location->latitude, (string)$this->location->longitude);

        return [
            new TextAnswer(Formatter::item($cafe), ['parse_mode' => ParseMode::HTML]),

            new VenueAnswer((float)$cafe->latitude, (float)$cafe->longitude, $cafe->name, '', [
                'google_place_id' => $cafe->placeId,
                'reply_to_message_id' => 0,
                'reply_markup' => ['remove_keyboard' => true],
            ]),
        ];
    }


    public function getLocation(): Location
    {
        return $this->location;
    }


    public function setLocation(Location $location): LocationHandler
    {
        $this->location = $location;

        return $this;
    }
}

The same short and concise middleware is used to validate search data as well as check the legitimacy of messages.

Configuration

Common options and secret names are stored in the .env file, while local overrides are stored in the .env.local file.

Tests

I'm pleased that the Nutgram library provides enough possibilities for writing tests. Unlike simple Guzzle mocks, project's tests use the mocked ApiClient, which returns predefined responses an infinite number of times.

Monitoring

The same Sentry as in the API microservice and the frontend, in the .env just specify an empty value of SENTRY_DSN (for clarity), and write the actual value to secret.

Deployment

Still the same Fly.io platform, but now with 300ms-load-time Machines. Generally, it's FaaS (serverless), but in my case, with the PHP server, it's still a regular VM.

I used an embedded PHP server instead of the usual combination of PHP-FPM + Nginx/Caddy + Supervisor to speed up the project's launch. The Docker image of course got smaller, but I had to use a separate router:

pass only POST requests to bot handlers
redirect the dev domain like .fly.dev to the main domain
distribute statics (robots.txt, favicon.ico, etc.)
block all other requests

Final router and Dockerfile (same layered as in API part).

CI/CD

Github Action is simple enough: update the machine flyctl deploy and update the webhook registration curl -sS ${{ secrets.APP_URL }}/setup.php.

All secrets are stored on the hosting platform and partially duplicated in the GitHub production environment for webhook registration.

At this stage, the bot is live, hosted in a production and available to all users. The project is fully completed :-)

Bot repository, website https://specialtycoffee.cy/

TODO

The final non-critical tasks for the entire project:

health checks for real service response, not just port 'survivability'.
optimize the build with Caddy
try a Buildpack or Nixpack
replace the built-in PHP server with something more secure
add strict typing (Typescript)
add API usage stats
add bot usage statistics
expand link and event tracking in Google Analytics
replace Google Analytics with something lighter and more GDPR compliant.

Pragmatic development 2: frontend

Vladimir — Thu, 13 Oct 2022 15:45:35 +0000

Development of a simple project about specialty coffee shops in Cyprus continues. I talked the API microservice in the first part, the frontend site in the second part, and the Telegram bot in the final article.

The project code is open, website https://specialtycoffee.cy/.

Architecture

The website has only two pages: a map with coffee shop markers and an About page. You don't need anything else for a startup, but for feedback, I have Telegram and email.

The implementation is modular vanilla JavaScript and Google Maps in a free tier, which can always be replaced by Mapbox, Mapquest, and other services.

The map and data from the REST API are loaded concurrently, and when ready, markers from the GeoJson data array are added to the map and collected in clusters. A click on the marker sends a request to the Google Places API for a company id (cid) and opens a small description (Info Window) with a link to the company card on the larger Google Maps with the ability to build a route, view reviews, and so on. This link is generated using the cid.

If Google Maps is unavailable, a stub with an error message is displayed, but the website navigation remains functional.

Everything is simple and quick thanks to asynchronous execution.

I use Vite to build the project, which is also very fast and simple.

I use Google Analytics with a direct connection instead of Google Tag Manager for analytics and insights about site visitors (a bit less traffic this way). Info Window openings are recorded in analytics as reaching goals in order to calculate site conversion. The plugin used is vite-plugin-radar.

Two additional plugins, vite-plugin-html-purgecss and vite-plugin-minify, enable the removal of all unused code from the final build. It took 15 minutes to set them up, which is fine.

Configuration

Common options and secret names are stored in the .env file, while local overrides are stored in the .env.local file.

Monitoring

The same Sentry as in the API microservice, in the .env just specify an empty value of VITE_SENTRY_DSN (for clarity), and write the actual value to secret.

Deployment

The same Fly.io platform with managed micro VM Firecracker. It's much easier than hosting the microservice API here, and a four-line Dockerfile suffices:

FROM pierrezemb/gostatic
COPY docker/config/headerConfig.json /config/
COPY dist/ /srv/http/
ENTRYPOINT ["/goStatic", "--fallback", "/index.html"]

CI/CD

This is slightly more complicated: first, Github Action creates a build with Vite, then flyctl creates a container from it and deploys it to the production VM). All secrets are stored in the GitHub production environment in this case.

At this point, the website is live, hosted in a production, and accessible to all users. In fact, the project is completed :-)

Frontend repository, website https://specialtycoffee.cy/.

In the third section, I'll go over how to make a Telegram bot.

Pragmatic development

Vladimir — Thu, 13 Oct 2022 12:07:26 +0000

I recently had some free time, so I created a simple project about specialty coffee shops in Cyprus: a website and a Telegram chatbot that adhered to "big" enterprise standards. I adore good coffee.

Now I like to share my development process as well as tips on how to get things done quickly.

The project's goals are simple:

Display a map of coffee shops on the website.
Examine a coffee shop's details
Use Google Maps for directions, reviews, and so on.
List of coffee shops in the Telegram chatbot
Search for local coffee shops in the bot
Search for the near coffee shop in the bot
Show a random coffee shop in the bot
Everything has a minimal and clear style.

Also, avoid perfectionism and never-ending development.

That is actually quite difficult because you may want to outperform others, wear badges of various linters, and express all of your abilities in the code.

Although it is sufficient for the service to simply work, metrics are tracked and errors are recorded. It is worth 20% of the effort to get 80% of the result.

So break everything down into small tasks and discard anything that will take more than a few hours or that cannot be estimated.

To be honest, I failed a few times: the first time, I got hooked on the idea of running a Caddy server without config from the console, but it can only run a reverse proxy and/or file server; I spent two days on it in total.

Another day was wasted on a bad choice of library for the Telegram chatbot.

Overall, everything was successful and the project code is open. Website: https://specialtycoffee.cy/

To achieve my objectives I've decided to build a REST API microservice, a frontend website (more on that in the second part), a backend for the bot (more on that in the third part) and deploy it to something modern and managed with a free tier, rather than VPS or shared hosting. The bot is generally compatible with the Serverless/FaaS ideology.

My first step was to register a domain name, which cost me 20 euros out of my own pocket - a good incentive not to squander them.

Speaking of registration, it is difficult to predict the project's future: it may turn out to be profitable to sell, and you may want to get rid of it quickly. As a result, it is preferable to separate all external services into separate accounts: mail, domain, hosting, analytics, monitoring, and so on. You can also use free tiers and trials.

REST API microservice

I have previous experience with Laravel and Symfony, so I chose a familiar and simple-to-use technology for quick implementation. ~~Then I'll almost certainly rewrite it in Go.~~ Using an actual version of PHP 8.1 allowed me to write slightly less code while achieving slightly better performance. Development is made much easier by promoted properties, read-only properties, and strict typing.

To make things easier, I remove some unused packages and services from Laravel: it's almost like Lumen.

Packages can be marked as "installed" in composer.json, but they will not be installed. This is very useful for removing redundant polyfills like this:

"replace": {
    "symfony/polyfill-ctype": "*",
    "symfony/polyfill-iconv": "*",
    "symfony/polyfill-intl-grapheme": "*",
    "symfony/polyfill-intl-idn": "*",
    "symfony/polyfill-mbstring": "*",
    "symfony/polyfill-php72": "*",
    "symfony/polyfill-php73": "*",
    "symfony/polyfill-php80": "*",
    "symfony/polyfill-php81": "*",
    "dragonmantank/cron-expression": "*",
    "egulias/email-validator": "*",
    "league/commonmark": "*",
    "league/flysystem": "*",
    "symfony/mime": "*",
    "symfony/var-dumper": "*",
    "tijsverkoyen/css-to-inline-styles": "*"
}

You can also disable platform-check so that you don't have to check the PHP version on every request and limit your check to installing packages. It's also useful to enable classmap-authoritative so that classes are only loaded from the map created by the composer, not from each use, but this would interfere with development, so it's sufficient to enable it on deployment.

There are final composer.json and config/app.php. This kind of optimization took less than an hour, so ok. But deeper optimization will require much more time, so not now.

Architecture

The service is made up of single-action controllers that retrieve data from models. There are no repositories because I consider them unnecessary for simple queries with no additional logic.

Separate Requests validate input data, and GeoJson Resources wrap output data. One class, one responsibility.

When I was developing the frontend, there was only one endpoint /cafes that returned a list of all the coffee shops: this allowed me to quickly get the API up and running without affecting other parts of the project. I added a few more endpoints during bot development.

Database

To begin, SQLite is used as a database, which allowed me to avoid the time spent on the traditional MySQL/PostgreSQL deployment. Furthermore, I'm confident that SQLite is an excellent choice for microprocessing with a load of 100 hits per day and a few dozen or hundreds of table entries.

During the deployment process, data is seeded from a regular array in database/seeders/CafeSeeder.php. I intend to write 1-2 console commands to edit data in the future because they are much faster than any visual admin panel.

Search

Scout with a "collection" driver makes the API full-text searchable: it allows the search of each model's fields with a simple "LIKE%smth%" query and does not require any full-text indexes in the database. It only took 15 minutes to implement.

Static

There are a few static files that must be present in the service:

robots.txt, which doesn't allow indexing.
favicon.ico, which is loved by many services
humans.txt
etc.

Tests

First, swagger-php generates openapi.yaml based on code attributes, and then the spectator checks API responses to match openapi specification. The popular L5-Swagger is redundant in this case, as it is based on the same swagger-php with the addition of Swagger UI.

Configuration

Unlike Symfony, Laravel does not read the.env.local file to override/add to the.env configuration and does not recommend storing the.env in a repository. This is a good approach, but it is ineffective when there are many configuration parameters.

You can do it a little differently: put the local parameters in.env (but don't put them in the repository) and all the production and secret parameter names in.env.production (but put them in the repository). The hosting and/or deployment tools must write the APP_ENV=production parameter as well as the secrets themselves.

In this case, .env.local will replace (not complement!) the configuration from.env.production, and listing all used parameters (even without values) in.env.production will help with project understanding. In this case, remove .env.example.

Monitoring

When the first stage of development is complete, add Sentry to the project: in .env.production just specify an empty value of SENTRY_LARAVEL_DSN (for clarity) and record the actual value in the secret.

Deployment

The flyctl launch command from the application directory will detect the required components and build the fly.toml and Dockerfile configurations automatically. You can also create your own configuration and Dockerfile.

I already had a Dockerfile for similar projects, so I used that. As an added bonus, I could run all services as an unprivileged user.

Because layer caching has been implemented in relatively recent Docker versions, it makes no sense to write all instructions in a single RUN command. On the contrary, it is preferable to arrange the "thin" RUN and COPY layers in the order of how frequently the data changes.

Because OS distributions and packages are rarely changed, the RUN apk add... command may appear at the beginning of the Dockerfile.

Because composer packages are updated more frequently than project source code, the layers COPY composer.* . and RUN composer install --no-autoloader --no-dev --no-interaction --no-scripts can be specified in the middle of a Dockerfile and fetched from cache.

COPY --chown=www-data:www-data . ., RUN composer dump-autoload --classmap-authoritative --no-interaction, and possibly other commands affecting the project source code can be placed at the end and executed only if the project code itself changes, rather than the OS packages or composer dependencies.

I used an embedded PHP server instead of the usual combination of PHP-FPM + Nginx/Caddy + Supervisor to speed up the project's launch.

Final Dockerfile.

The Fly.io platform itself terminates https and manages certificates, so the container application only needs to handle normal http traffic.

CI/CD

It's simple here, GitHub Action with one workflow and the same flyctl.

At this point, the microservice API is live, hosted in production, and accessible to all users. MVP is complete :-)

API repository, website https://specialtycoffee.cy

I'll tell about creating the frontend in the second part and the chatbot for Telegram in the third part.

Demo Symfony Currency Converter

Vladimir — Mon, 25 Jul 2022 09:44:00 +0000

I recently did a test task on Symfony - currency converter with direct and cross conversion. So I want to share the result with the community as an example of a simple console application according to Symfony's rules: DI, autowiring, service tagging, flexible configuration, that's all. I hope this will be useful for beginners.

Source code

The application calculates "currency exchange" at direct rates (for example, USD -> EUR), as well as through "intermediate" currencies (for example, BTC -> USD -> EUR). There are also fake rates for tests.

Rates are taken from ecb.europa.eu (major world currencies against EUR) and coindesk.com (BTC against USD).

Triangulation is based on the principles from http://www.dpxo.net/articles/fx_rate_triangulation_sql.html.

The data storing in the SQLite database.

The application can be used through local PHP or in Docker.
PHP requirements: version 8.1+, bcmath, ctype, iconv, intl, pdo_sqlite, simplexml, sqlite3 extensions.

I had little experience with Symfony (I worked with Laravel mostly), so there may be some flaws.
In addition, SQLite imposed some limitations due to the lack of real decimal and numeric formats, and INSERT IGNORE, the calculation accuracy of 16.8 had to be hardcoded.
There was trouble with the ECB rates' dates, so the application uses the last available day from each source.

Highlights

Commands

The application has two console commands: "currency:update" - updating exchange rates (\App\Command\CurrencyUpdateCommand) and "currency:exchange" - exchangу currencies (\App\Command\CurrencyExchangeCommand).

The commands accept parameters, validate data, pass them to services, catch exceptions and beautifully print the result to the console with the appropriate exit status.

All services and providers are passed through constructor injection. Rate providers have been tagged with the "app.rates_provider" tag in config/services.yaml and passed through an iterator to \App\Services\RatesUpdater by this tag. Very convenient, I think.

App\Providers\CoinDeskRatesProvider:
    tags: [ 'app.rates_provider' ]

App\Providers\EcbRatesProvider:
    tags: [ 'app.rates_provider' ]

App\Services\RatesUpdater:
    arguments:
        - !tagged_iterator app.rates_provider

class RatesUpdater
{
    public function __construct(private readonly iterable $ratesProviders, ...)
    {
    }
...
}

Data exchange and validation

Data for currency exchange and saving rates are sent via DTO: \App\Dto\Exchange and \App\Dto\Rate, respectively.
Validation of "AmountRequirements" - quantity requirements and "ExchangeCurrencyRequirements" - currency requirements are imposed on DTO for currency exchange.
In addition, validation applies to the \App\Entity\Pair and \App\Entity\Rate entities.

All validators are custom to hide unnecessary details from consumers. Validators locate in the src/Validator/ classes. Most of them are compounds of simple rules. For example, the quantity requirements are "Non-empty string", "Numeric type", and "Positive value".

class AmountRequirements extends Compound
{
    protected function getConstraints(array $options): array
    {
        return [
            new Assert\NotBlank(),
            new Assert\Type(type: 'numeric', message: 'The value {{ value }} is not a valid {{ type }}'),
            new Assert\Positive(),
        ];
    }
}

There is also a more complex currency existence validator \App\Validator\PairCurrencyExistValidator. It accesses the currency pair repository and checks the database for SELECT COUNT(1) FROM pair WHERE base = <passed currency ticker>. Its realized via Doctrine Query Builder.

Exchange rates update

Everything is quite simple here: \App\Services\RatesUpdater receives an iterator of currency rate providers in the constructor and calls them one by one (via __invoke, so you don't need to invent a method name). All providers inherit the \App\Providers\RatesProvider abstract class and implement their data transformation methods in the \App\Dto\Rate DTO.

The abstract provider asks for rates at the address specified in the configuration and .env, which is embedded in the constructor and the base currency's name. Then the provider parses rates from JSON or XML into a simple array and passes them to the provider-specific transformer.
Parsers locate in src/Parsers/.

For tests, \App\Providers\FakeRatesProvider is used with an overridden fetch method and a couple of rates wired into it.

The rates received in the form of DTO are stored in the database in direct and reverse form, after which the triangulator \App\Services\RatesTriangulator is put into operation. It creates all possible combinations of rates through intermediate currencies (so-called cross rates) and records them in the \App\Entity\Pair entity.

Triangulation is based on the principles of http://www.dpxo.net/articles/fx_rate_triangulation_sql.html. It is much easier to get one pair of currencies for conversion from a separate table with currency pairs than calculate rates for each conversion.

If something goes wrong, then the providers or the triangulator throw exceptions.

Using

If you have PHP installed locally, you need to clone the repository, install packages, create a database, perform migrations and update exchange rates.

git clone https://github.com/vladimirmartsul/symfony-exchange-demo.git
cd symfony-exchange-demo
composer install --no-dev --no-interaction
php bin/console doctrine:database:create
php bin/console doctrine:migrations:migrate --no-interaction
php bin/console currency:update

Calculate exchange

php bin/console currency:exchange <amount> <from> <to>

For example
php bin/console currency:exchange 2 EUR BTC
should output
[OK] 2 EUR is 0.00005254 BTC

You can also build and run the application in Docker

git clone https://github.com/vladimirmartsul/symfony-exchange-demo.git
cd symfony-exchange-demo
docker compose up --build

The exchange rates are loaded during the build.

Calculate exchange

docker compose run symfony-exchange-demo currency:exchange <amount> <from> <to>

For example
docker compose run symfony-exchange-demo currency:exchange 2 EUR BTC
Should output the same result as the local PHP run.

Testing

A couple of tests have been written for the application to make sure that the main functionality works correctly. The tests use a mocked provider of exchange rates.

\App\Tests\Command\CurrencyUpdateCommandTest - a simple check for messages about successful download, triangulation and update of courses.

\App\Tests\Command\CurrencyExchangeCommandTest - a little more complicated: checking the real conversion using a dataProvider with several currency pairs and the expected result. Each time the test is run, the exchange rates are updated.

You can run tests locally by installing additional dev packages.

cd symfony-exchange-demo
echo APP_ENV=test > .env.local
composer install --no-interaction
php bin/console doctrine:database:create
php bin/console doctrine:migrations:migrate --no-interaction
php bin/phpunit

Or similar with the Docker.

cd symfony-exchange-demo
echo APP_ENV=test > .env.local
docker compose run symfony-exchange-demo composer install --no-interaction
docker compose run symfony-exchange-demo doctrine:database:create
docker compose run symfony-exchange-demo doctrine:migrations:migrate --no-interaction
docker compose run symfony-exchange-demo bin/phpunit

Welcome to comments and pull requests :-)