The latest articles on Forem by Abhishek Taneja (@abhishektaneja). https://forem.com/abhishektaneja https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3355796%2F5dc6b156-ea08-4e96-80e8-21a58e16b47b.jpg https://forem.com/abhishektaneja en Abhishek Taneja Sun, 27 Jul 2025 15:53:08 +0000 https://forem.com/abhishektaneja/hows-my-day-a-voice-first-mood-tracker-using-assemblyais-speech-understanding-real-time-tts-c5i https://forem.com/abhishektaneja/hows-my-day-a-voice-first-mood-tracker-using-assemblyais-speech-understanding-real-time-tts-c5i <p><em>This is a submission for the <a href="https://dev.to/challenges/assemblyai-2025-07-16">AssemblyAI Voice Agents Challenge</a></em></p> <h2> What I Built </h2> <p>Domain Expert Voice Agent</p> <p><strong>How’s My Day?</strong> is a one-shot voice check-in app that helps users feel heard — emotionally, not just functionally.</p> <p>In just one tap:</p> <ol> <li>The user speaks how they’re feeling</li> <li>The app transcribes their voice in real-time using <strong>AssemblyAI Universal Streaming</strong> </li> <li>It detects the <strong>emotional tone</strong> of their voice using <strong>AssemblyAI’s Speech Understanding</strong> </li> <li>It matches that emotion to a hand-curated emotional tip using <strong>Algolia MCP Server</strong> </li> <li>Finally, it reads the tip aloud using <strong>AssemblyAI’s Text-to-Speech</strong> — or falls back to <strong>ElevenLabs</strong> if needed</li> </ol> <p>✨ The experience feels like being heard by someone who cares — not a chatbot.</p> <h2> 🧠 How I Used AssemblyAI </h2> <p>We used <strong>AssemblyAI’s Universal-Streaming API</strong> to:</p> <ul> <li>Capture and transcribe voice input with &lt;300ms latency</li> <li>Show live transcript to the user while they speak</li> <li>Handle punctuation and natural pauses beautifully</li> </ul> <h2> What I Learned </h2> <ul> <li><p>AssemblyAI’s emotion detection is shockingly accurate — tone alone can reveal so much more than words</p></li> <li><p>Transcription feels like magic when done right — and AssemblyAI nailed it</p></li> <li><p>Using voice as input and output feels more natural than a chatbot for mental wellness apps</p></li> <li><p>People want calm, 1-shot interactions — not 20-message bots</p></li> </ul> <h2> Challenges </h2> <ul> <li><p>Browser-based mic streaming + latency management was tricky</p></li> <li><p>Emotion ↔ tip mapping needed thoughtful writing</p></li> <li><p>Not all users want to hear their feelings read back — we added a toggle</p></li> <li><p>AssemblyAI TTS is clean, but fallback was needed for broader support</p></li> </ul> <h2> Demo </h2> <p> <iframe src="https://www.youtube.com/embed/4w0usY0G3Jc"> </iframe> </p> <h2> GitHub Repository </h2> <div class="ltag-github-readme-tag"> <div class="readme-overview"> <h2> <img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"> <a href="https://github.com/abhishektaneja09" rel="noopener noreferrer"> abhishektaneja09 </a> / <a href="https://github.com/abhishektaneja09/how-is-your-day" rel="noopener noreferrer"> how-is-your-day </a> </h2> <h3> </h3> </div> <div class="ltag-github-body"> <div id="readme" class="md"> <div class="markdown-heading"> <h1 class="heading-element">How's My Day? - AI-Powered Voice Mood Tracker</h1> </div> <p>A sophisticated, voice-powered mood tracking web application that listens to your feelings and provides supportive, human-like responses using cutting-edge AI technology.</p> <div class="markdown-heading"> <h2 class="heading-element">✨ Features</h2> </div> <ul> <li> <strong>🎤 Professional voice recording</strong> - File-based audio capture with high quality</li> <li> <strong>🎯 AI-powered transcription</strong> - AssemblyAI integration for accurate speech-to-text</li> <li> <strong>🧠 Enhanced mood detection</strong> - Local algorithm with scoring and emotion mapping</li> <li> <strong>🤖 GPT-4o-mini responses</strong> - Human-like, empathetic AI-generated support messages</li> <li> <strong>🔊 High-quality TTS</strong> - OpenAI text-to-speech with natural voice synthesis</li> <li> <strong>⌨️ Real-time typing animation</strong> - Text appears character-by-character during speech</li> <li> <strong>🎨 Modern UI</strong> - Clean, responsive design with smooth animations</li> <li> <strong>🚀 Full-stack architecture</strong> - Node.js backend with Express server</li> </ul> <div class="markdown-heading"> <h2 class="heading-element">🚀 Quick Setup Guide</h2> </div> <div class="markdown-heading"> <h3 class="heading-element">Prerequisites</h3> </div> <ul> <li> <strong>Node.js</strong> (v14 or higher) - <a href="https://nodejs.org/" rel="nofollow noopener noreferrer">Download here</a> </li> <li> <strong>Git</strong> (optional) - For cloning the repository</li> <li> <strong>Modern web browser</strong> - Chrome, Firefox, Safari, or Edge</li> </ul> <div class="markdown-heading"> <h3 class="heading-element">1. Installation</h3> </div> <div class="highlight highlight-source-shell notranslate position-relative overflow-auto js-code-highlight"> <pre><span class="pl-c"><span class="pl-c">#</span> Clone or download the project</span> git clone <span class="pl-k">&lt;</span></pre>… </div> </div> </div> <div class="gh-btn-container"><a class="gh-btn" href="https://github.com/abhishektaneja09/how-is-your-day" rel="noopener noreferrer">View on GitHub</a></div> </div> <p><a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyzm3x530fk8cmxhfap32.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyzm3x530fk8cmxhfap32.png" alt=" "></a></p> devchallenge assemblyaichallenge ai api Abhishek Taneja Sun, 27 Jul 2025 15:47:03 +0000 https://forem.com/abhishektaneja/hows-my-day-a-one-shot-voice-check-in-that-listens-understands-and-responds-1kgo https://forem.com/abhishektaneja/hows-my-day-a-one-shot-voice-check-in-that-listens-understands-and-responds-1kgo <p><em>This is a submission for the <a href="https://dev.to/challenges/algolia-2025-07-09">Algolia MCP Server Challenge</a></em></p> <h2> What I Built </h2> <p><strong>How’s My Day?</strong> is a voice-first emotional wellness app that helps people reflect on their mood — all through a single spoken sentence.</p> <p>Users simply tap a mic button, speak freely, and the app:</p> <ul> <li>Transcribes their voice in real time using AssemblyAI</li> <li>Detects the emotion behind their voice using Speech Understanding</li> <li>Searches Algolia MCP Server for a warm, human-written emotional tip</li> <li>Reads the tip back using AssemblyAI’s or ElevenLabs’ Text-to-Speech</li> </ul> <p>The result is a <strong>calm, empathetic micro-interaction</strong> that helps people feel heard — no chatbot, no complexity.</p> <h2> Demo </h2> <blockquote> <p>📺 <strong>Video Walkthrough</strong><br><br> <iframe src="https://www.youtube.com/embed/4w0usY0G3Jc"> </iframe> </p> <p>📦 <strong>GitHub Repo</strong><br><br> [🔗 <a href="https://github.com/abhishektaneja09/how-is-your-day" rel="noopener noreferrer">https://github.com/abhishektaneja09/how-is-your-day</a>]</p> </blockquote> <h2> How I Utilized the Algolia MCP Server </h2> <p>Algolia’s <strong>MCP Server</strong> powers the backend of emotional intelligence in our app.</p> <p>Here's how we used it:</p> <ul> <li>We created a static <code>moods.json</code> index with handcrafted emotional responses for moods like <code>anxious</code>, <code>burnt out</code>, <code>grateful</code>, <code>lonely</code>, etc.</li> <li>When the emotion is detected (via AssemblyAI), we query Algolia MCP with that mood keyword</li> <li>MCP returns a <strong>personalized, human-style tip</strong>, emoji, and short comforting message</li> <li>This response is instantly delivered to the user, giving the illusion of true emotional understanding</li> </ul> <p>The fast response time of MCP and its simple API made it the perfect choice for a real-time emotional UX.</p> <h2> Key Takeaways </h2> <ul> <li>Voice-first interactions can feel personal, if designed with empathy</li> <li>AssemblyAI’s <strong>Speech Understanding</strong> feature can infer emotional tone from voice — not just text</li> <li>Algolia MCP makes it easy to build “domain brains” — in our case, an <strong>emotional advice brain</strong> </li> <li>Text-to-Speech should be slow, warm, and natural — we used <strong>AssemblyAI</strong>, and <strong>ElevenLabs</strong> as a fallback</li> <li>People don’t want a chatbot to solve their feelings — they want to be <em>heard</em> </li> </ul> devchallenge algoliachallenge webdev ai