Forem: Gabriel Sena

Meet Munio: Improve your English conversation skills with AI!

Gabriel Sena — Thu, 15 Aug 2024 23:58:05 +0000

Hi everyone! I created a very cool web app called Munio. The goal is to improve your English conversation skills using AI! This version of the app is fully open source, and you can check it using link below.

Motivation
How the app works
- The session mode
- The conversation mode
Open Source
Thanks

Motivation

Google recently launched a competition to develop something using Gemini AI. I decided to create something useful with tools that piqued my curiosity. I combined all of these tools and products to develop this web app.

Here are the main tools that I have used:

NestJS
Angular 18
Knex
WebSockets
Google Products (Gemini, GCS, TTS and Google Infra solutions)
various others...

How the app works

Here is a showcase video:

the app's goal is to enhance your conversation skills through speaking exercises. Currently, there are two modes: session and conversation.

The session mode

In this mode, you choose preferences such as context and level. These preferences are used to generate phrases with AI.

After starting a session, your goal is to repeat the requested phrase. AI analyzes your audio and provides feedback.

At the end of the session, AI generates overall feedback, helping you understand what to improve and encouraging you to try again!

The conversation mode

In this mode, like in session mode, you choose preferences, and AI uses them to generate a simulated conversation. To enhance the user experience, we use WebSockets to ensure fluid interaction.

After starting a conversation, AI decides whether you or the AI will speak first.

If it's your turn to speak, you need to say the requested phrase. AI analyzes your audio and generates feedback.
If it's AI's turn to speak, we use Google Cloud Speech To Text to generate audio with the requested phrase, which you can listen to.
The conversation alternates between steps 1 and 2 until the end. Once the conversation is complete, AI generates overall feedback, highlighting areas for improvement.

Open Source

This version of the app is fully open source and free. You can collaborate or self-host it! =)

Me:

https://github.com/gabrielsenadev

Thanks

Thank you for reading this post. If you have any feedback, please let me know in the comments or at feedback@gabrielsena.dev

Gallery Spaces - Create your own gallery and share

Gabriel Sena — Sun, 12 May 2024 01:22:35 +0000

This is a submission for the Netlify Dynamic Site Challenge: Visual Feast.

This is a submission for the Netlify Dynamic Site Challenge: Build with Blobs.

What I Built

I've developed a gallery platform that empowers users to create their own galleries, manage them (uploading and deleting images), share their work with others, and customize images before downloading.

Key features include:

Gallery: Leveraging Netlify Image CDN to showcase our image galleries.
Image Customization: Utilizing Netlify Image CDN for on-the-fly image customization prior to download.
Image Uploader System: Employing Netlify Blobs to store user-uploaded images.
Authentication System: Utilizing Netlify Blobs to store user authentication data.

Demo

Demo Link
User Gallery Example

Go ahead and create your own gallery and share to everyone too 😸

Some pictures

Platform Primitives

To submit my entry, I utilized Netlify Image CDN and Netlify Blobs Platform Primitives.

For enhanced user image experiences, Netlify Image CDN proved invaluable, enabling flexible image customization:

All Gallery Images (Example Gallery) are loaded using Netlify Image CDN.
Gallery Image customization (such as clicking on an image) is facilitated by Netlify Image CDN, allowing for image transformations before download.
Throughout the website, Netlify Image CDN, integrated with NuxtImg, ensures optimized image customization.

To store datas, such as authentication, images, and gallery data, I employed Netlify Blobs with separate stores for authentication, gallery, and images, each dedicated to user data:

The authentication system, including user credentials and JWT tokens, is stored with Netlify Blobs, utilizing mainly directories.
Image uploads, including profile pictures and gallery images, leverage Netlify Blobs, eliminating the need for additional file storage systems.
User gallery data is organized and stored using directories provided by Netlify Blobs, ensuring efficient data management.

If you're curious to delve deeper, the submission code is open source on my GitHub. Feel free to check it out!

Github Repository-

Frontend Challenge: Animated Globe CSS Art: Earth Day

Gabriel Sena — Mon, 29 Apr 2024 06:18:39 +0000

This is a submission for Frontend Challenge v24.04.17, CSS Art: Earth Day.

Inspiration

The word 'earth' represents to me a globe, and, on current days, the news about climate changes calls our attention. So, I decided to use this challenge to challenge myself by developing an animated globe with climate change representation using only CSS. 🌍 🤯

Demo

Link to Demo

Link to Code - Repository

Screenshot

Journey

To start this challenge, I decided to brainstorm my ideas using Asesprite to create globe illustrations and it's states.

After that, I ran to code and created the first state: the globe. Then, I developed the others states: fires and water overflow.

To help me a develop these states, I decided to use the power of css variables to I am not get lost and keep my code readable to keep going.

The major problem to develop this illustration pixel by pixel, is because is really hard to work with it. To make more easier, I used the box-shadow tecnique to help me a create something like a "canvas" to point these points like a editor.

After making these states, I started building my CSS animation to sync everything and tried to make more smooth possible.

And there you have it! I'm proud to have developed this CSS art =)

Bot Innovator: Converting images with Image Wizard AI Assistant

Gabriel Sena — Sun, 28 Apr 2024 19:55:29 +0000

This is a submission for the Coze AI Bot Challenge: Bot Innovator.

What I Built

Sometimes, we need to convert an image from one format (X) to another (Y). Perhaps you require an optimized image format, such as JPEG, or the website where you intend to upload doesn't accept your current format.

To address this issue, the Image Wizard AI Assistant helps by converting images from both common and uncommon formats to another desired format. It also assists in selecting the best format for a specific case. There's no need to search online; simply upload the image and specify the desired format. The bot will handle the conversion and provide a URL for the new image.

Demo

Bot link on Coze Bot Store

Video showcase

Your Configuration

Initially, I chose to concentrate this bot solely on image processing contexts, defining specific roles and behaviors.

On Skills, I exclusively utilize the Image Wizard plugin to convert images into various images formats. Making possible accepts gif and converting to JPG, for example.

To improve the user experience, I used the Dialog to display helpful options for starting a chat. Additionally, I utilized Auto-Suggestions to generate some useful questions related to the context.

Journey

As described in my Coze Trailblazer submission, developing this bot helped me learn and experiment with many things to make it possible.

Working with the Coze ecosystem was a gratifying experience because everything worked well and helped me improve further in AI-based Assistant development.

In the future, my expectation is to develop more tools for image processing and provide them to users through this bot.

Coze Trailblazer: Adding image processing capability with Image Wizard

Gabriel Sena — Sun, 28 Apr 2024 18:37:40 +0000

This is a submission for the Coze AI Bot Challenge: Trailblazer.

What I Built

So, users are increasingly relying on AI-based tools to solve their problems. One common issue with images is file formats. Some websites don't accept certain formats or something similar. To address this problem, my plugin adds the capability for the bot/workflow to convert a image in another format. Example: PNG to JPG.

With this tool, users will not waste time searching for a solution; they simply need to upload their image and specify the desired format, and we'll take care of the rest.

Demo

Image Wizard - Coze Plugin Store

To preview, you can review my bot submission that utilizes the Image Wizard plugin in the background to convert an image to another format.

Bot link that uses Image Wizard plugin

To use this plugin for now, you just need to add the Convert tool to your bot or workflow. In the video below, I demonstrate a simple bot image converter.

Journey

To complete this project, I decided to develop an application for image conversion. In the background, I worked with NestJS, MongoDB, Image Uploader Provider, Logging System, Operational System Setup, and Image libraries setup to offer a wide range of format conversions. Additionally, I learned other things to make this tool possible.

After that, I gained an understanding of the Coze ecosystem and developed my plugin and bot.

It's a great opportunity to explore new things and keep learning, so thank you!

Cloudflare AI Challenge: Audio Interaction with AudioInsight

Gabriel Sena — Sun, 14 Apr 2024 21:32:35 +0000

This is a submission for the Cloudflare AI Challenge.

What I Built

I built AudioInsight, an app that processes audio, transcribes it, summarizes it, generates a title for the content, and allows users to ask questions about the related audio.

The chat and audio are stored remotely, so the user can access later and ask new questions or rewatch the audio.

To make this app, I used some products from the Cloudflare catalog, such as: Cloudflare D1, R2, Workers AI, Cloudflare Pages, and AI Models: Automatic Speech Recognition, Summarization, and Text Generation. These will be explained in the Journey section.

Demo

Demo Link
Original Cloudflare Pages Demo Link

My Code

gabrielsenadev / audioinsight

AudioInsight is a web application that processes audio, generates transcriptions, and allows users to ask questions about the related audio.

AudioInsight

AudioInsight is a full-stack application that processes audio, generates transcriptions, and allows users to ask questions about the related audio.

Its creation was motivated by participation in a dev.to challenge.

How to Install

Start by cloning this repository:

git clone git@github.com:gabrielsenadev/audioinsight.git

Install dependencies:

npm ci

Configure your environment

See Environment Variables.

Run application

npm run dev

Environment Variables

This application depends on some providers to work with ai and database. It has been developed with minimal provider dependency. So, if you prefer a different provider, you can easily switch.

Cloudflare AI:

This application integrates with the Cloudflare AI ecosystem to utilize AI Models.

CLOUDFLARE_ACCOUNT_ID
CLOUDFLARE_API_TOKEN

Netlify Blobs:

For storing audio data, this application relies on Netlify Blobs. You will need a Netlify Site and Account.

NETLIFY_SITE_ID
NETLIFY_TOKEN

MongoDB:

MongoDB is used to store chats and chat messages.

View on GitHub

Journey

Working with AI is a curious thing to me. I was thinking about developing something using AI, and this challenge is enough to motivate me to do this. So, one of the incredible AI features for me is the capability to transform voice into text. Therefore, I decided to follow this path.

After some time thinking about what to do, I decided to process the audio, generate the content of the audio, summarize the content, and allow the user to ask questions about the uploaded audio.

I also decided to explore more of the Cloudflare ecosystem. Thus, one of my personal requirements is the capability to store chat and audio remotely and provide a way for the user to go back later.

After defining my requirements and goals, I started learning about the AI, how it works, and how Workers AI works. In this process, I decided to use these AI models: a audio to text (whisper), a content summarization (bart-large-cnn) and text generation to answer questions and generate chat title (neural-chat-7b-v3-1-awq).

In the Multiple Models and/or Triple Task Types section, I explain how I use these models and show the application flow, which explains how I combine these AI models to participate in the Additional Prize Category.

After developing the main idea, I began to understand how Cloudflare databases and Cloudflare R2 work. Then, I implemented the capability to store user's chats and audio.

Multiple Models and/or Triple Task Types

To create this app, I utilized three different AI model types to generate its content.

whisper is responsible for converting audio to text.
bart-large-cnn is tasked with generating a summary of the related audio content.
neural-chat-7b-v3-1-awq handles generating the chat title and answering questions about the related content.

When the user uploads an audio, start the chat creation process. Here, I combine all three models to generate a piece of content for each: audio transcription, summarization, and chat title.

When the user asks a question, I only use the text generation AI model to answer the user's question.

Application Flow
Follow the flow below to understand how it works.

In this flow, we understand how I use these different AI models and how I utilized Cloudflare storage solutions to develop this app.

Final words

Developing this entry helped me understand more about how the AI ecosystem works and how I could use the Cloudflare ecosystem to empower my ideas into products.

Looking ahead, I'm considering incorporating private chats and additional chat features to enhance user interaction with audio.

Thank you for this challenge!

Forem: Gabriel Sena

Meet Munio: Improve your English conversation skills with AI!

Motivation

How the app works

The session mode

The conversation mode

Open Source

Thanks

Gallery Spaces - Create your own gallery and share

What I Built

Demo

Platform Primitives

Frontend Challenge: Animated Globe CSS Art: Earth Day

Inspiration

Demo

Journey

Bot Innovator: Converting images with Image Wizard AI Assistant

What I Built

Demo

Your Configuration

Journey

Coze Trailblazer: Adding image processing capability with Image Wizard

What I Built

Demo

Journey

Cloudflare AI Challenge: Audio Interaction with AudioInsight

What I Built

Demo

My Code

gabrielsenadev / audioinsight

AudioInsight is a web application that processes audio, generates transcriptions, and allows users to ask questions about the related audio.

AudioInsight

Table of Contents

How to Install

Environment Variables

Cloudflare AI:

Netlify Blobs:

MongoDB:

Journey

Multiple Models and/or Triple Task Types