Forem: Jérôme M

How to create a pronunciation assessment App (Part 2)

Jérôme M — Mon, 26 Aug 2024 15:36:59 +0000

The purpose of this tutorial is to create an application to control the user's pronunciation.

To follow it, you must have knowledge of javascript and more ideally Vue.js 3.

What we are going to do

In the previous article we set up the skeleton of the application with the Quasar framework based on VueJS

We will now set up the first of the two main components of the application:

The component responsible for entering the word or sentence to be pronounced. The latter will be able to read the word to allow the user to listen to what he must pronounce
The voice acquisition component. This is a button to record his voice and then send the recording to the API.

We will start by creating the skeleton of the component and positioning it in the main page of the application.

To do this, create the WordInputField.vue file in the /src/components directory. You can take this opportunity to delete the EssentialLink.vue file which is no longer useful.

Creating a vue component

<template>
  <q-input
    class="input-text q-mt-lg"
    v-model="sentence"
    type="textarea"
    :lines="2"
    autogrow
    hint="Input a word or a sentence"
    clearable
  />
</template>

<script setup>
import { ref } from 'vue'

// Reference on the word or sentence to be pronounced
const sentence = ref('')
</script>

<style scoped>
.input-text {
  width: 80vw;
  max-width: 400px;
}
</style>

Once this is done we can use the component in the main page: /src/pages/IndexPage.vue

<template>
  <q-page class="column wrap content-center items-center">
    <sentence-input-field />
    <div>
      <q-btn
        class="q-mt-lg"
        icon="mic"
        color="primary"
        round
        size="30px"
        @click="record"
      />
    </div>
  </q-page>
</template>

<script setup>
import SentenceInputField from '../components/SentenceInputField'

function record () {
  console.log('Record')
}
</script>

As you can see, this allows us to remove the style section and the script code declaring the sentence reference

We will now be able to focus on the design of this component

Component design

We will implement the following component.

With its q-input component, Quasar has already done almost all the work for us!

Here is how we will modify our html template

<template>
  <q-input
    class="input-text q-mt-lg"
    v-model="sentence"
    type="textarea"
    :lines="2"
    autogrow
    hint="Input a word or a sentence"
    clearable
  >
    <template v-slot:after>
      <q-btn
        round
        dense
        flat
        icon="record_voice_over"
      >
        <q-tooltip>Listen the word or the sentence</q-tooltip>
      </q-btn>
    </template>
  </q-input>
</template>

Speech synthesis

Now we have to pronounce what was entered, that is to say the content of the sentence reference to our device.

We will use the SpeechSyntesis API. To simplify the use of the API we will use a wrapper around it: Artyom.js which is available as an npm module.

In a console, install the module:

npm i --save artyom.js

Then add the initialization code in the javascript part of the component

<script setup>
import { ref } from 'vue'
import Artyom from 'artyom.js'

// Reference on the word or sentence to be pronounced
const sentence = ref('')

const artyom = new Artyom()
artyom.initialize({
  debug: false,
  continuous: false,
  listen: false,
  lang: 'de-DE'
})

async function speak () {
  artyom.say(sentence.value)
}
</script>

Note that we are using the German language by initializing Artyom with: lang: 'de-DE'
We could use another language like:

French: 'fr-FR'
Spanish: 'es-ES'

All that's left is to add our speak function to the button:

<template v-slot:after>
  <q-btn
    round
    dense
    flat
    icon="record_voice_over"
    @click="speak"
  >
    <q-tooltip>Listen the word or the sentence</q-tooltip>
  </q-btn>
</template>

Small improvements to the component

We will make some improvements to our component.

First of all, if the API is not available, the button should not appear.
To do this, we will simply create a reference that will test the availability of the API isSpeachSyntesisAvailable and condition the creation of the button on this:

<script setup>
import { ref } from 'vue'
import Artyom from 'artyom.js'

// Reference on the word or sentence to be pronounced
const sentence = ref('')

const isSpeechSyntesisAvailable = ref(false)

const artyom = new Artyom()
artyom.initialize({
  debug: false,
  continuous: false,
  listen: false,
  lang: 'de-DE'
}).then(() => {
  isSpeechSyntesisAvailable.value = artyom.speechSupported()
})

async function speak () {
  artyom.say(sentence.value)
}
</script>

In the implementation we wait for the end of the initialization to know if the speech synthesis is available, so we must only make the button available once the initialization is done. To do this we will add an isArtyomReady reference and a computed to know if the button is disable cannotSpeak (we prefer to use cannotSpeak than canSpeak because we will associate it with the disable state, so it is better that it is true when the button is disabled)

<script setup>
import { ref, computed } from 'vue'
import Artyom from 'artyom.js'

// Reference on the word or sentence to be pronounced
const sentence = ref('')
const isArtyomReady = ref(false)
const isSpeechSyntesisAvailable = ref(false)
const cannotSpeak = computed(() => {
  return !sentence.value || !isArtyomReady.value || !isSpeechSyntesisAvailable.value
})

const artyom = new Artyom()
artyom.initialize({
  debug: false,
  continuous: false,
  listen: false,
  lang: 'de-DE'
}).then(() => {
  isSpeechSyntesisAvailable.value = artyom.speechSupported()
  isArtyomReady.value = true
})

async function speak () {
  artyom.say(sentence.value)
}
</script>

This point must be taken into account in our template

<template>
  <q-input
    class="input-text q-mt-lg"
    v-model="sentence"
    type="textarea"
    :lines="2"
    autogrow
    hint="Input a word or a sentence"
    clearable
  >
    <template v-slot:after>
      <q-btn
        round
        dense
        flat
        icon="record_voice_over"
        :disable="cannotSpeak"
        @click="speak"
      >
        <q-tooltip>Listen the word or the sentence</q-tooltip>
      </q-btn>
    </template>
  </q-input>
</template>

All that remains is to prohibit the user from clicking the button twice. For this we will use a new ref isSpeaking to know if the speech synthesis is in progress or not, and add its state to the calculation of the computed cannotSpeak

<script setup>
import { ref, computed } from 'vue'
import Artyom from 'artyom.js'

// Reference on the word or sentence to be pronounced
const sentence = ref('')
const isArtyomReady = ref(false)
const isSpeechSyntesisAvailable = ref(false)
const isSpeaking = ref(false)
const cannotSpeak = computed(() => {
  return isSpeaking.value || !sentence.value || !isArtyomReady.value || !isSpeechSyntesisAvailable.value
})

const artyom = new Artyom()
artyom.initialize({
  debug: false,
  continuous: false,
  listen: false,
  lang: 'de-DE'
}).then(() => {
  isSpeechSyntesisAvailable.value = artyom.speechSupported()
  isArtyomReady.value = true
})

async function speak () {
  isSpeaking.value = true
  artyom.say(sentence.value, {
    onEnd: function () {
      isSpeaking.value = false
    }
  })
}

We now have an application that allows you to enter a sentence and listen to it in the chosen language. For this we have isolated an independent view component.

What is left to do?

In a future part we will see how to acquire the audio, and obtain a score via the SpeechSuper API

Part 3: Acquiring the audio and the score via the SpeechSuper API
Part 4: Packaging the application

Conclusion

Feel free to comment on the article! Part 3 is coming soon!

How to create a pronunciation assessment App (Part 1)

Jérôme M — Sat, 17 Aug 2024 18:02:42 +0000

The purpose of this tutorial is to create an application to control the user's pronunciation.

To follow it, you must have knowledge of javascript and more ideally Vue.js 3.

The idea

I recently decided to get back to German. The main difficulty I encounter in this language is pronouncing it correctly. Usually I listen to an example, record myself repeating it and listen to myself again. It's a complicated process and I must admit I don't have a very good ear.

Based on this observation, I wondered if there was an App or an API that could tell me if I was pronouncing a word or a sentence correctly in German! After some investigations and great discoveries, I wanted to code my own App to solve my problem.

Here's how I did it!

Available APIs

After some research, I was able to find Apps that solved my problem. But overall, pronunciation validation was often just an additional function of a paid application (or one that worked with a subscription). I then decided to look for APIs.

Here is the list of APIs that do the job :

These APIs are paid but generally allow you to get 2 weeks of access to test and experiment.

Since I wanted to check my pronunciation of German, I chose to test with the SpeechSuper API because it supports several languages including German. Later in the tutorial we will try the Speechace API to demonstrate how easy it is to switch from one API to another depending on your needs.

Definition of the application's ergonomics

The goal is to implement a simple App that allows you to enter a word, record your voice, send the audio recording to the API and display your score.

This is what the application will look like:

So we will create an application that will present a text field allowing the entry of a word or a sentence. A button will allow you to listen to it.
We then have a button to record our voice, this one will change style when it is in recording mode. Just click on it to stop and send to the API to obtain a pronunciation score.
Once the score is obtained, it is displayed as a tile with a color representing our score, from red to green to orange.

Initialization of the application

The ideal would be to be able to deploy the App as a webapp, but also as a native Android application. For this reason we will use Quasar.

The Quasar framework

Quasar is an open-source Vue.js framework for developing applications with a single codebase. They can be deployed on the web (SPA, PWA, SSR), as a mobile application (Android, iOS) or as a Desktop application (MacOs, Windows, Linux).

Preparation

If this is not already the case, you need to install NodeJS. The better is to use volta as it will allow you to use differents versions of NodeJs depending on your projects.

We will start by initializing our project with the Quasar scaffolding tool.

npm i -g @quasar/cli
npm init quasar

The cli will ask us several questions, choose the following options :

Options List

App with Quasar CLI
Project folder : learn2speak
Quasar v2
Javascript
Quasar App with Vite
Package name : learn2speak
Project product name : Learn to speak
Project description : Assess your pronunciation
Author : yourself
CSS preprocessor : Sass with SCSS syntax
Features needed :
- ESLint
- Axios
ESLint preset : Standard
Install project dependencies : Yes, use npm

Once the command is executed, you can enter the directory and serve the application locally:

cd learn2speak
npm run dev

Your default browser should open the page at the following address http://localhost:9000

Modification of the proposed skeleton to obtain the targeted ergonomics

The example application is available, we will remove the elements we do not need. To do this we will open the source code in VSCode (you can of course use another editor)

code .

Layout modification

Quasar provides us with the notion of Layout and then of page included in the latter. The pages and the layout are chosen via the router. For this tutorial, we do not need to know these notions, but you can learn them here: Quasar layout

We do not need drawer, at least not for now so we will delete it from the src/layouts/MainLayout.vue file. To do this, delete the section of the <template> included between the <drawer>...</drawer> tags, then we will modify the header to display the icon then the name of the application, this gives us the following code for the template:

<template>
  <q-layout view="lHh Lpr lFf">
    <q-header elevated>
      <q-toolbar>
        <q-icon name="interpreter_mode" size="md" />
        <q-toolbar-title>
          Learn2Speak
        </q-toolbar-title>
      </q-toolbar>
    </q-header>

    <q-page-container>
      <router-view />
    </q-page-container>
  </q-layout>
</template>

We can then remove the entire script part and replace it with the following code:

<script>
import { defineComponent } from 'vue'

export default defineComponent({
  name: 'MainLayout',
  setup () {
  }
})
</script>

We don't need more for the layout part because our application will define only one page.

The main page

The implementation of the main page is in the file: src/pages/IndexPage.vue

this is the main page where we will position our text field and the save button.

For this file, we simply remove the Quasar logo from the template (the <img> tag) and modify the script part to use the vueJS 3 composition api, so that the source looks like the following file:

<template>
  <q-page class="flex flex-center">

  </q-page>
</template>

<script setup>

</script>

We will now add the text field using the Quasar component QInput

To do this we add the q-input component to the page template:

<template>
  <q-page class="flex flex-center">
    <q-input
      type="textarea"
      :lines="2"
      autogrow
      hint="Input a word or a sentence"
      clearable
    />
  </q-page>
</template>

You can see that the text field is displayed in the center of the screen, this is due to the Quasar flex and flex-center classes. These classes are defined by Quasar: Flexbox. We will fix this by placing the text field at the top of the screen, we will also take advantage of this to style the component.

Quasar even provides us with a Flex Playground to experiment and find the classes to put.

<template>
  <q-page class="column wrap content-center items-center">
    <q-input
      class="input-text"
      v-model="sentence"
      type="textarea"
      :lines="2"
      autogrow
      hint="Input a word or a sentence"
      clearable
    />
  </q-page>
</template>

<script setup>
import { ref } from 'vue'

// Reference on the word or sentence to be pronounced
const sentence = ref('')

</script>

<style scoped>
.input-text {
  width: 80vw;
  max-width: 400px;
}
</style>

As you can see, we have defined a sentence reference in the script part to store the value entered by the user. It is associated via the v-model directive to the q-input component

We will finish this first part by adding the button allowing the recording of our pronunciation of the word or sentence. For this we will simply use the q-button component of Quasar and position it after our text field.

<template>
  <q-page class="column wrap content-center items-center">
    <q-input
      class="input-text q-mt-lg"
      v-model="sentence"
      type="textarea"
      :lines="2"
      autogrow
      hint="Input a word or a sentence"
      clearable
    />
    <div>
    <q-btn
      class="q-mt-lg"
      icon="mic"
      color="primary"
      round
      size="30px"
      @click="record"
    />
    </div>
  </q-page>
</template>

<script setup>
import { ref } from 'vue'

// Reference on the word or sentence to be pronounced
const sentence = ref('')

function record () {
  console.log('Record')
}
</script>

<style scoped>
.input-text {
  width: 80vw;
  max-width: 400px;
}
</style>

Note that we added the q-mt-lg class to air out the interface a bit by leaving some space above each component. You can refer to the Quasar documentation on spacing.

The application will look like:

What will we do next

We have therefore managed to obtain the skeleton of our application.

In a future part we will see how to acquire the audio, then how to obtain a score via the SpeechSuper API

Part 2: Acquiring the audio
Part 3: Obtaining the score via the SpeechSuper API
Part 4: Packaging the application

Conclusion

Don't hesitate on comments the post ! Part 2 will follow soon !