Forem: Ottomatias Peura

Add Voice Search with a Few Lines of Code (for free)

Ottomatias Peura — Wed, 21 Apr 2021 07:24:45 +0000

Just want to develop? Click here

Voice search means for most optimizing your website to support natural language queries. This is important, because many are searching on Google by using voice instead of typing and typically these queries are somewhat different.

For instance, searching for the population of US by typing the keywords might result in a query such as "us population" whereas searching for the same thing by voice might be natural by asking "what is the population of the United States?". In fact, already 20% of Google App searches are done by using voice.

However, when the user eventually lands to a page they were looking for by using their voice, they are back to the "old-school" way of operating websites. Instead of voice, they'll have to resort to typing, clicking and scrolling.

While most of us are very familiar with those UI patterns, they are nowhere near perfect. A typical eCommerce search with a few category filters (such as blue Nike t-shirts in size large for men) takes easily around half a minute or more, even for a somewhat experienced user.

This is because the order, names, and even the existence of these filters vary widely in eCommerce stores. Some stores don't offer color filters, some categorize t-shirts under shirts and some have a separate category for them and so on. Sometimes the filters are on the sidebar, sometimes on top and sometimes they require opening a hamburger menu.

What if this tedious filtering could be replaced with voice? Let's see an example.

Voice is a very natural way for creating search experiences. While a text-based menu can't include all the synonyms because of lack of screen real estate, voice UI can support as many synonyms and different ways to express things as required. As the filters are updated in real time as the user keeps on speaking, the user gets visual feedback and is encouraged to go on with filtering. On the other hand, if they misspeak or if the system recognizes something wrong, they can naturally correct themselves.

We have created a fully streaming voice UI API that enables building this kind of experiences very easily. Instead of the traditional "voice assistant" style that resembles Microsoft's Clippy, our solution is based on directly manipulating the graphical user interface with voice.

As most of the eCommerce stores have already implemented some kind of a category filtering, adding Speechly enables their users to replace searching and selecting for the correct filters by simply saying out loud their preferences. The user interface stays the same and doesn't require any other updates than adding the microphone button. We even provide a customizable microphone button for easy integration.

We have productized voice search as a very simple solution that enables developers to add voice search to their website in a few lines of code. We have also created a demo to showcase a real-life application using this pattern. You can try out the demo here.

Instructions for integrating voice search to your website can be found here. There's also a simple Codepen example that you can edit yourself and see the full code for a simple implementation.

You can use Speechly for free for up to 20 hours a month.
If you need any help creating your models or integrating it, you can contact our support at hello@speechly.com.

Would be great to hear your thoughts!

Voice search in eCommerce

Ottomatias Peura — Thu, 25 Feb 2021 14:25:13 +0000

Search is a very important component for many apps and services and especially so in eCommerce.

Voice offers many unique features that are not possible by using more traditional approaches, such as:

Learn the language your users are using of your products
Accessibility benefits for people who have difficulties using keyboard and mouse
Support users on the go
Intuitive user experience
No need to clutter the UI with filters

Speechly is a developer tool for improving user experience with voice user interfaces and we created a demo showcasing how voice would work in a fashion eCommerce setting.

You can try the demo here.

Would be great to hear your thoughts! Would you use voice when browsing eCommerce sites? How would you improve the experience?

You can see a demo video of the experience below.

You can read more about what we mean with a voice user interface in this article or see more examples of voice user interfaces.

Voice search in eCommerce

Ottomatias Peura — Tue, 19 Jan 2021 09:01:19 +0000

Search is a very important component for many apps and services and especially so in eCommerce.

Voice offers many unique features that are not possible by using more traditional approaches, such as:

Learn the language your users are using of your products
Accessibility benefits for people who have difficulties using keyboard and mouse
Support users on the go
Intuitive user experience
No need to clutter the UI with filters

Speechly is a developer tool for improving user experience with voice user interfaces and we created a demo showcasing how voice would work in a fashion eCommerce setting.

You can try the demo here.

Would be great to hear your thoughts! Would you use voice when browsing eCommerce sites? How would you improve the experience?

You can see a demo video of the experience below.

Are You Using Web Speech API? (Should You Reconsider?)

Ottomatias Peura — Tue, 12 Jan 2021 16:41:17 +0000

I've seen quite a lot of projects using Web Speech API, the browser API for speech recognition.

While this is great as in we need more people building voice user interfaces in browsers, I'm not sure if it's at all suitable for production use.

First of all, it's severely lacking in browser support, meaning it's pretty much a Chrome (and derivates) specific feature.

Second, as it's only for speech recognition, it's not very easy to leverage it for anything that's not just replacing keyboard.

And if you think about it, speech is not at all like a keyboard with voice. Written and spoken language are pretty much two different languages and even if saying something like "Turn off the lights" is pretty intuitive and fast, typing that to do the same action would not be very good UX.

This is why any voice user interface will anyway need a natural language understanding component to extract meaning out of what the user said. Even with the most simple voice user interfaces, actual users will come up with very many different ways of expressing the same intent and grepping text transcripts is not a viable solution.

So this is my question: Are you using WebSpeech API and for what kind of projects? What do you think are the benefits of the API compared to some other solutions? Have you considered other options and why did you ditch those?

Why hasn’t the iPhone moment happened yet for voice UIs

Ottomatias Peura — Tue, 15 Dec 2020 15:06:27 +0000

What’s wrong with current voice UIs

There is a lot of positive momentum around voice interfaces. We’ve all seen the stats: the number of smart speakers in US households has risen steadily for the past five years. The share of voice searches of all search engine traffic has soared. Yet, the type of revolution that the touch screen gave birth to, after the launch of the iPhone, hasn’t really happened for voice — despite all the hype. Why is that?

Anyone who has a bit of experience from using voice interfaces knows that you can do simple things like setting on the alarm, switching on the lights, or playing your favorite music on Spotify pretty easily. However, if you try to do anything more sophisticated, say, order pizza for your eight best friends, reserve an intercontinental flight for a family of five, or buy a new party dress online, the chances are that you're going to fail miserably. For more demanding and for most real-world tasks, the user experience with voice just isn’t there yet — at least as it is for the touch screen.

Still, contrary to what people might think, the problem is not really anymore in speech recognition or natural language understanding accuracy. Even for fairly open-ended domains, both speech recognition and natural language understanding accuracy is pretty close to human parity.

The problem lies rather in the way these systems give feedback to the user. Typically, when a voice command is uttered, the modern voice assistants wait until the user has stopped talking (using the technique called endpointing) before they start processing it. This works great for short things like “turn on the lights” or “Play X on Spotify”. However, for more complex tasks this is a disaster.

Imagine if you need to express something that requires a longer explanation. When looking for a new t-shirt, a person might be tempted to say something like “I’m interested in t-shirts for men ...in color red, blue or orange, let’s say Boss ...no wait, I mean Hilfiger ...maybe size medium or large ...and something that’s on sale and can be shipped by tomorrow ...and I’d like to see the cheapest options first.”

When uttering something this long and winding to a traditional voice UI, most likely something will go wrong, resulting in the familiar, “Sorry, I didn’t quite get that.” Having just made the extended effort of explaining your intent to the system, this is an extremely frustrating experience. Or even worse, the endpointing might trigger a false positive half way the utterance, causing the voice assistant to prematurely resolve the intent and start a voice synthesis response, interrupting the speaker violently and irritatingly.

In the following sections of this article, we will introduce the powerful techniques of streaming spoken language understanding and reactive multi-modal voice user interfaces that address the very problem the current generation of voice UIs suffer from.

Streaming Spoken Language Understanding

Contrary to the traditional voice systems that rely on endpointing to trigger a response, the systems using streaming spoken language understanding actively try to comprehend the user intent from the very moment the user starts to talk. The idea is that as soon as the user says something actionable, the UI is able to instantly react to it.

The benefit here is that if the system does not understand the user, the UI will instantly signal this back. This way the UI will fail fast but also recover quickly as the user can immediately stop, correct, and continue. On the other hand, if the system does understand the user, also this information is conveyed immediately. This gives reassurance to the user that their message is going through, and that they can continue their expression. As long as the system understands, the user can just go on and on, which results in longer and more complex utterances that are supported.

Moreover, the immediate feedback from both the small failures and successes of the UI can be combined in a way that the users can correct either themselves or the UI in an online manner, e.g., “I’m interested in Boss, no, I mean Tommy Hilfiger”. This ushers a way for the UI to not only support more sophisticated and complex UI workflows but also a more stream-of-consciousness way of expressing the users’ intent. This is more natural for humans and requires much less effort than the very specific way that the current voice UIs require the utterances to be given.

Reactive and non interruptive multi-modal voice user interfaces

The most prominent feedback modality of the current day voice interfaces is voice synthesis. As a feedback mechanism, however, this works poorly as any ongoing user utterance will be abruptly interrupted — a problem exhibited commonly in the contemporary voice UIs. As a feedback mechanism, voice is also a pretty narrow band. Instead, the feedback should be given with a non-interruptive modality. Such modalities include haptic, non-linguistic auditory, and perhaps most naturally and expressively, visual feedback. Using these modalities, the UI can react fast and without interruption to the user. For instance, in the case of “I’m interested in t-shirts,” the UI would swiftly show the most popular t-shirt products, instantly enabling the user to continue with a refining utterance, ”do you have Boss.” This narrows further down the displayed products to show only the Boss branded t-shirts.

This iteration loop reminisces a familiar setting to everybody: human face-to-face communication, which is, in effect, a reactive, multimodal communication setup. In fact, it is a common misperception that human face-to-face conversation is primarily turn-based (or half-duplex) in a similar fashion that chatbots or voice assistance are. Meaning, first I say something, then you say something, then I say something again, and so on. Not exclusively. The human face-to-face conversation is very much full-duplex: as one person talks, the listener gives feedback with nods, facial expressions, gestures, and interjections like aha and mhm. Furthermore, if the person listening doesn't understand what is being said, they are likely to start making more or less subtle facial expressions to signal their lack of comprehension. This is the tight full-duplex feedback loop that makes human face-to-face communication so efficient. The same efficiency is exhibited in the reactive multi-modal voice user computer interfaces!

Voice is not the UI, voice is a modality!

At the height of the voice assistant hype ushered in by Amazon Alexa, many probably heard the flying phrase “Voice is the UI!”. This article disagrees. Voice is a modality! By augmenting a UI with voice in combination with other available modalities, the result can be an extremely efficient UI.

The perfect interface is when you can use touch and voice seamlessly and choose the best option for the context, sometimes interchangeably

This efficiency comes from how the modalities work together, not from voice alone. Voice is, for instance, a very efficient method for inputting rich information. For scrolling, swiping, pointing, or selecting between a couple of valid alternatives, touch is probably better. For displaying complex multidimensional information, the visual display is unbeatable. Combining all of these modalities in a smart way is the killer app.

What was the revolutionary thing in the iPhone?

Circling back to where we started, the iPhone moment. What made the iPhone so powerful? Well, the very intuitive swipe and pinch gestures with which the user could effortlessly control their phone. However, when the iPhone came out, the touch screen wasn't a new thing. There had been prior touch screen devices. They just sucked! You could swipe or press, and a second or two later something would happen.

What the iPhone brought to the mix was the extremely fast feedback that its touch screen could provide to the user, resulting in the very intuitive, fluid, and satisfying user experience of controlling your phone. Voice UIs based on streaming spoken language understanding are a similar type of revolution. The streaming spoken language understanding offers extremely fast, fluid, and intuitive feedback to the user akin to what the iPhone brought to the controlling devices back in 2007. However — this time, voice is in the center, providing user experiences that we haven’t seen before, ushering the iPhone moment at last for voice as well.

Play Chess with Voice

Ottomatias Peura — Mon, 30 Nov 2020 16:26:47 +0000

SpeechToChess tutorial

End result of the tutorial in action

Let’s play chess by using voice! In this tutorial, we’ll build a simple chess game with JavaScript that can be controlled by using voice commands.

After completing this tutorial, you will be able to create a Speechly voice interface for a new project or integrate it to an existing one in Javascript.

Chessboard consists of rows and columns, or in the language of chess ranks and files. The files (rows) are identified by the letters a to h and the ranks (columns) by the numbers 1 to 8. In this tutorial, we will be using the chess game notation: “e4 e5 Nf3 Nc6 Bb5 a6 Ba4 Nf6”. Upper-case letters N and B stand for the knight and bishop.

You say “knight f3" and Speechly will provide you with a javascript object of intent

{
 “intent”: “move”
}

and an array of entities

[
  {
    “type”: “piece”,
    “value”: “KNIGHT”
  },
  {
    “type”: “square”,
    “value”: “F3"
  }
]

Define intents in SAL - Speechly Annotation Language

Building voice user interfaces starts from the declaration of the intents and entities. In our chess example, the most common user intent is to move a piece on the board. This intent has two entities (modifiers for this intent): piece and square where this piece will be moved.

Go to Speechly Dashboard, login and click on the “Create app” button.

Set a name for the app. Now you can see the editor where you can write your SAL code.
Like mentioned,the intents in chess are moves. There is a list of intents on the right side of the screen, type the name of the first intent “move” and click on “Add” button.

There are also two kinds of special moves: “capture” and “castle”, add them to the list, too. And finally we need an intent to restart the game, call it “reset”.

Add three entities: “piece” of the type string and “square” as the type identifier.

SAL uses a asterisk (*) to define an intent. To define an entity in SAL syntax list all the possible values in square brackets followed by the entity name in parenthesis [knight|bishop](piece). We can set the above mentioned list as a variable pieces = [pawn|knight|bishop|rook|queen|king]. Set also variables for the lists of squares.

SAL code sample:

pieces = [pawn|knight|bishop|rook|queen|king]
squares = [
  A8|B8|C8|D8|E8|F8|G8|H8
  A7|B7|C7|D7|E7|F7|G7|H7
  A6|B6|C6|D6|E6|F6|G6|H6
  A5|B5|C5|D5|E5|F5|G5|H5
  A4|B4|C4|D4|E4|F4|G4|H4
  A3|B3|C3|D3|E3|F3|G3|H3
  A2|B2|C2|D2|E2|F2|G2|H2
  A1|B1|C1|D1|E1|F1|G1|H1
]
*move {$pieces(piece)} $squares(square)
*castle castle
*capture $pieces(piece) takes $pieces(piece) on $squares(square)
*reset new game

Curly braces wrap the optional values. You can refer to Cheat Sheet for SAL syntax for more information on the syntax.

You can see the App ID under the application name. You’ll need to use it when integrating the project.

Deploy Speechly app and try

Now you can play around with your configuration. Press “Deploy” and wait for about 2 minutes.

When you see the status “Deployed”, press “Try”. The Playground screen will show up. Press space and hold it to say something like “KNIGHT E4”, “KNIGHT takes PAWN on E5", “castle”.

Create a plain javascript project

Now we can start building our Javascript application.

Create a new folder mkdir MyVoiceApp and run there yarn init. Add Speechly’s client and webpack for bundling the code yarn add @speechly/browser-client webpack webpack-cli. By default webpack looks for the index.js file in the src folder and creates a bundle named main.js in the dist folder.

Add index.js file to the src folder and index.html file to the dist folder.

src/index.js

import { Client } from ‘@speechly/browser-client’;
console.log(‘Hello Speechly!’);

dist/index.html

<html>
<head>
  <style>
    * {font-size: 40px}
    table {border: 1px #000 solid}
    td {text-align: center}
    tr:nth-child(2n+1) td:nth-child(2n), tr:nth-child(2n) td:nth-child(2n+1) {
      background: #DDD
    }
  </style>
</head>
<body>
  <script src=“main.js”></script>
  <table id=“board” cellpadding=0 cellspacing=0></table>
  <br/>
  <button id=“mic”>Microphone</button>
</body>
</html>

Now you can run the bundler yarn run webpack. As a result you will get the main.js file in the dist folder.

Open the index.html file in Chrome browser. You will see the ‘Microphone’ button on the screen and the greeting in the console.

Client is a key

Create a new Client and initialize it:

const client = new Client({
 appId: ‘HereIs-AppId-From-The-Dashbord’,
 language: ‘en-US’,
});
client.initialize();

Bind the ‘Microphone’ button to record and send voice audio:

 window.onload = () => {
 document.getElementById(‘mic’).onmousedown = () => client.startContext();
 document.getElementById(‘mic’).onmouseup = () => client.stopContext();
};

We will connect the game and the client by the onSegmentChange event handler:

client.onSegmentChange((segment) => {
  if (segment.intent && segment.isFinal) {
    // TODO: game logic
  }
}

State of the game, reducer and rendering

Create game object with the default position on the chessboard to store the state. Add reducer function to update the game state by incoming segments. Finally add a function to render the chessboard.

const defaultPosition = [
 [‘r’, ‘n’, ‘b’, ‘q’, ‘k’, ‘b’, ‘n’, ‘r’],
 [‘p’, ‘p’, ‘p’, ‘p’, ‘p’, ‘p’, ‘p’, ‘p’],
 [‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’],
 [‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’],
 [‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’],
 [‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’, ‘.’],
 [‘P’, ‘P’, ‘P’, ‘P’, ‘P’, ‘P’, ‘P’, ‘P’],
 [‘R’, ‘N’, ‘B’, ‘Q’, ‘K’, ‘B’, ‘N’, ‘R’],
];
let game = {
 position: defaultPosition,
 activeColor: ‘w’,
};
/**
 * Creates a new position by changing current file and rank of a piece
 */
function move(position, {file, rank}, dst) {
 const piece = position[rank][file];
 let newPosition = position;
 newPosition[rank][file] = ‘.’;
 newPosition[dst.rank][dst.file] = piece;
 return newPosition;
}
const files = [‘A’, ‘B’, ‘C’, ‘D’, ‘E’, ‘F’, ‘G’, ‘H’];
/**
 * Transforms square string value like ‘E4’ to an object with coordinates
 * 
 * @param {string} square
 * @return {object} file number and rank number combined in an object.
 */
const transformCoordinates = (square) => ({
 file: files.indexOf(square[0]),
 rank: 8 - square[1]
});
const pieces = {
 PAWN: ‘P’,
 KNIGHT: ‘N’,
 BISHOP: ‘B’,
 ROOK: ‘R’,
 QUEEN: ‘Q’,
 KING: ‘K’,
};
/**
 * Transforms array of entities to a key value object
 * @param {array} entities 
 * @return {object} key value object.
 */
const formatEntities = (entities) =>
 entities.reduce((accumulator, currentValue) => ({
  ...accumulator,
  [currentValue.type]: currentValue.value
 }), {});
/**
 * Creates a new game state
 * @return {object} new state of the game.
 */
const reducer = (game, segment) => {
 switch (segment.intent.intent) {
  case ‘reset’:
   const newGame = {
    position: defaultPosition,
    activeColor: ‘w’,
   };
   return newGame;
  case ‘move’:
   let {piece, square} = formatEntities(segment.entities);
   if (piece) {
    piece = pieces[piece];
   } else {
    piece = ‘P’;
   }
   piece = game.activeColor === ‘b’ ? piece.toLowerCase() : piece;  
   const {file, rank} = transformCoordinates(square);
   const selectedPiece = selectPiece(game, piece, file, rank);
   if (!selectedPiece) {
    console.error(`Can’t find out the piece ${piece} for move on ${square}`);
    return game;
   }
   return {
    position: move(game.position, selectedPiece, {file, rank}),
    activeColor: game.activeColor === ‘w’ ? ‘b’ : ‘w’,
   };
  case ‘capture’:
   return game;
  case ‘castle’:
   let newPosition;
   if (game.activeColor === ‘w’) {
    newPosition = move(game.position, transformCoordinates(‘E1’), transformCoordinates(‘G1’));
    newPosition = move(newPosition, transformCoordinates(‘H1’), transformCoordinates(‘F1’));
   } else {
    newPosition = move(game.position, transformCoordinates(‘E8’), transformCoordinates(‘G8’));
    newPosition = move(newPosition, transformCoordinates(‘H8’), transformCoordinates(‘F8’));
   }
   return {
    position: newPosition,
    activeColor: game.activeColor === ‘w’ ? ‘b’ : ‘w’,
   };
  default:
   return game;
 }
}
/**
 * Since user provide us only with a destination square for example ‘E4’,
 * we add a selectPiece function to get the piece coordinates on the chessboard.
 */

function selectPiece(game, piece, newFile, newRank) {
 return game.position.flatMap((rank) => rank)
  .map((piece, i) => ({ piece, rank: Math.floor(i / 8), file: (i % 8) }))
  .find((item) =>
   item.piece === piece
   && isCorrectMove(piece, newRank, newFile, item.rank, item.file));
}
/**
 * Checks correctness of a move
 * @return {boolean} is correct.
 */
function isCorrectMove(piece, rank, file, rankIndex, fileIndex) {
 const dRank = Math.abs(rankIndex - rank);
 const dFile = Math.abs(fileIndex - file);
 switch (piece.toUpperCase()) {
  case ‘P’:
   return file === fileIndex && dRank <= 2;
  case ‘N’:
   return dRank + dFile === 3 && Math.abs(dRank - dFile) == 1;
  case ‘B’:
   return dRank === dFile;
  case ‘R’:
   return rankIndex === rank || fileIndex === file;
  default:
   return false;
 }
};
/**
 * Maps the game position to html table content 
 */
function renderBoard(position) {
 const view = position.map(
  (rank) => `<tr>${
   rank.map((file) => `<td>${file}</td>`).join(‘’)
  }</tr>`,
 ).join(‘’);
 document.getElementById(‘board’).innerHTML = view;
};

Now you can call the game reducer on each event with a segment and render the chessboard.

client.onSegmentChange((segment) => {
  if (segment.intent && segment.isFinal) {
      game = reducer(game, segment);
      renderBoard(game.position);
  }
}

Enjoy the game

Now you can run your application by running ´yarn start´ 9. Oress the ‘Microphone’ button and say ‘E4’ and release the button. Repeat with ‘E5’, ‘knight f3’, ‘knight c6’ etc.

Have a great game!

Real-time Voice Search on iOS

Ottomatias Peura — Mon, 30 Nov 2020 14:27:14 +0000

Introduction

This tutorial will help you to get up and running with Speechly by guiding you through the process of building a simple voice filtering web app with Speechly and SwiftUI.

You can find the source code for this tutorial on GitHub.

Prerequisites

We'll be using Xcode and iOS simulator, so make sure you have those installed.

Xcode 12
iOS Simulator with iOS version 14.0 or later

This tutorial assumes that you know the basics of iOS development and the Swift programming language. Feel free to check the official Apple documentation to get started with Swift or SwiftUI.

Create a new Xcode project

First, we'll need a new iOS app project. Let's create one!

Open Xcode and go to File > New > Project.
Under the set of iOS templates, choose App:

Click Next and fill in your project details. Name your project SpeechlyRepoFiltering, choose SwiftUI as your interface option, SwiftUI App as life cycle and Swift for the language.

Click Next once more and you should have your project created for you.

Adding Speechly client dependency

Next we need to add Speechly client library to our project.

Click File > Swift Packages > Add Package Dependency:

Specify https://github.com/speechly/ios-client.git as the package repository and click Next:

Xcode will fetch the package information for you and will ask you to specify the version. Leave the default rules in and click Next:

After the Xcode fetches the package and its dependencies, it will ask which target you want to add it to. Make sure you have Speechly product added to SpeechlyRepoFiltering and the checkbox checked and then click Finish:

Add repo data and layout

Now that we have the project ready, let's start coding!

Since we are building a filtering app, let's add some data to filter and layout to display it.

To make it simple, our data source will be just a static array with some popular repositories on GitHub.

First, let's add the data model and save it as GithubRepoModel.swift:

import Foundation

struct GithubRepo: Hashable, Identifiable {
    enum Language: String, Hashable {
        case Go = "Go"
        case Python = "Python"
        case TypeScript = "TypeScript"
    }

    let id: Int
    let name: String
    let organisation: String
    let language: Language
    let followers: Int
    let stars: Int
    let forks: Int
}

Now let's add a repository that will return us some sample data. Add the following code to GithubRepoRepository.swift:

import Foundation

class GithubRepoRepository {
    static let shared = GithubRepoRepository()

    func list() -> [GithubRepo] {
        return self.repositories
    }

    private let repositories = [
        GithubRepo(
            id: 1,
            name: "typescript",
            organisation: "microsoft",
            language: .TypeScript,
            followers: 2200,
            stars: 65000,
            forks: 8700
        ),
        GithubRepo(
            id: 2,
            name: "nest",
            organisation: "nestjs",
            language: .TypeScript,
            followers: 648,
            stars: 30900,
            forks: 2800
        ),
        GithubRepo(
            id: 3,
            name: "vscode",
            organisation: "microsoft",
            language: .TypeScript,
            followers: 3000,
            stars: 105000,
            forks: 16700
        ),
        GithubRepo(
            id: 4,
            name: "deno",
            organisation: "denoland",
            language: .TypeScript,
            followers: 1700,
            stars: 68000,
            forks: 3500
        ),
        GithubRepo(
            id: 5,
            name: "kubernetes",
            organisation: "kubernetes",
            language: .Go,
            followers: 3300,
            stars: 70700,
            forks: 25500
        ),
        GithubRepo(
            id: 6,
            name: "moby",
            organisation: "moby",
            language: .Go,
            followers: 3200,
            stars: 58600,
            forks: 16900
        ),
        GithubRepo(
            id: 7,
            name: "hugo",
            organisation: "gohugoio",
            language: .Go,
            followers: 1000,
            stars: 47200,
            forks: 5400
        ),
        GithubRepo(
            id: 8,
            name: "grafana",
            organisation: "grafana",
            language: .Go,
            followers: 1300,
            stars: 37500,
            forks: 7600
        ),
        GithubRepo(
            id: 9,
            name: "pytorch",
            organisation: "pytorch",
            language: .Python,
            followers: 1600,
            stars: 43000,
            forks: 11200
        ),
        GithubRepo(
            id: 10,
            name: "tensorflow",
            organisation: "tensorfow",
            language: .Python,
            followers: 8300,
            stars: 149000,
            forks: 82900
        ),
        GithubRepo(
            id: 11,
            name: "django",
            organisation: "django",
            language: .Python,
            followers: 2300,
            stars: 52800,
            forks: 22800
        ),
        GithubRepo(
            id: 12,
            name: "airflow",
            organisation: "apache",
            language: .Python,
            followers: 716,
            stars: 18500,
            forks: 7200
        )
    ]
}

Now that we have the data, time to visualise it. Let's add some SwiftUI components for that.

Let's start by adding a list view, save the following code as RepoList.swift:

import SwiftUI

struct RepoList: View {
    var repos: [GithubRepo] = []

    var body: some View {
        NavigationView {
            List(repos) { repo in
                RepoRow(repo: repo)
            }
            .navigationBarTitle(Text("Repositories"), displayMode: .inline)
        }
    }
}

struct RepoList_Previews: PreviewProvider {
    static var previews: some View {
        RepoList(repos: GithubRepoRepository.shared.list())
    }
}

Notice the RepoList_Previews - it's a handy way to quickly preview your components without running the app!

As you can see, our list view uses another component, RepoRow. Let's add some code to RepoRow.swift to define it:

import SwiftUI

struct RepoRow: View {
    let repo: GithubRepo

    var body: some View {
        HStack {
            Image(imageName())
                .resizable().frame(width: 32, height: 32, alignment: .center)

            VStack(alignment: .leading, spacing: 7) {
                Text("\(repo.organisation) / ").font(.system(size: 20)).fontWeight(.light) +
                    Text(repo.name).font(.system(size: 20)).fontWeight(.bold)

                HStack(alignment: .lastTextBaseline) {
                    LabeledIcon(name: "arrow.branch", label: formatNumber(repo.forks))
                    LabeledIcon(name: "star", label: formatNumber(repo.stars))
                    LabeledIcon(name: "eye", label: formatNumber(repo.followers))
                }
            }
        }
    }

    private func formatNumber(_ value: Int) -> String {
        if (value < 1000) {
            return String(value)
        }

        if (value < 10000) {
            return "\(String(Double(value) / 1000))K"
        }

        return "\(String(value / 1000))K"
    }

    private func imageName() -> String {
        switch self.repo.language {
        case .Go:
            return "logo-go"
        case .Python:
            return "logo-python"
        case .TypeScript:
            return "logo-ts"
        }
    }
}

OK, now as you can see we are referencing some images with language logos, so let's add those to our project. Download and extract https://docs.speechly.com/client-libraries/ios/tutorial/LanguageLogos.zip, then you should have the following files:

logo-go.png
logo-python.png
logo-typescript.png

Now let's add them to our project:

Select Assets.xcassets in the navigation bar in Xcode, then click the + button in the bottom pane and finally click on the Image Set:

Type logo-go as the name, then drag and drop the logo-go.png file to all sizes (1x, 2x and 3x) placeholders. Repeat the same for logo-python and logo-typescript and you should have the following result:

Lastly, there is one more component to add - the LabeledIcon. Let's add the following code and save it as LabeledIcon.swift:

import SwiftUI

struct LabeledIcon: View {
    let name: String
    let label: String

    var body: some View {
        HStack(spacing: 3) {
            Image(systemName: name).foregroundColor(.gray).imageScale(.small)
            Text(label).font(.system(.caption)).fontWeight(.ultraLight)
        }
    }
}

Voila! Now you can go ahead and try running the preview for RepoList. You should see something like this:

Add microphone button and transcript text

Now that we have some data to work with and we can visualise it, time to add a button for activating the microphone and a simple component to display the transcript of what a user is saying.

Save the following code as Microphone.swift:

import SwiftUI

struct MicrophoneButton: View {
    let onStart: () -> Void
    let onStop: () -> Void

    var body: some View {
        ToggleButton(onDown: startRecording, onUp: stopRecording) {
            Circle()
                .frame(width: 80, height: 80)
                .foregroundColor(.blue)
                .overlay(
                    Image(systemName: "mic.fill")
                        .resizable()
                        .aspectRatio(contentMode: .fit)
                        .frame(width: 40, height: 40)
                        .foregroundColor(.white)
                )
        }.buttonStyle(Style())
    }

    private func startRecording() {
        self.onStart()
    }

    private func stopRecording() {
        self.onStop()
    }

    private struct Style: ButtonStyle {
        func makeBody(configuration: Configuration) -> some View {
            configuration.label
                .scaleEffect(configuration.isPressed ? 1.25 : 1.0)
                .animation(.easeInOut)
        }
    }
}

struct MicrophoneButton_Previews: PreviewProvider {
    static var previews: some View {
        MicrophoneButton(onStart: {}, onStop: {})
    }
}

And since it uses a ToggleButton component, let's add it as ToggleButton.swift:

import SwiftUI

struct ToggleButton<Content: View>: View {
    @State private var isDown = false

    let onDown: () -> Void
    let onUp: () -> Void
    let content: () -> Content

    var body: some View {
        Button(action: {}, label: {
            self.content()
        }).simultaneousGesture(
            DragGesture(minimumDistance: 0, coordinateSpace: .local)
                .onChanged { _ in
                    if self.isDown {
                        return
                    }

                    self.isDown = true
                    self.onDown()
                }
                .onEnded { _ in
                    self.isDown = false
                    self.onUp()
                }
        )
    }
}

And for the trasncript, let's add the following to TranscriptText.swift:

import SwiftUI
import Speechly

struct TranscriptText: View {
    let words: [SpeechTranscript]

    var body: some View {
        words.reduce(Text("")) { (acc, word) in
            acc + Text(" ") + Text(word.value).fontWeight(word.isFinal ? .bold : .light)
        }
    }
}

struct TranscriptText_Previews: PreviewProvider {
    static var previews: some View {
        TranscriptText(words: [
            SpeechTranscript(
                index: 1,
                value: "SHOW",
                startOffset: TimeInterval(0),
                endOffset: TimeInterval(0.1),
                isFinal: true
            ),
            SpeechTranscript(
                index: 2,
                value: "ME",
                startOffset: TimeInterval(0),
                endOffset: TimeInterval(0.1),
                isFinal: true
            ),
            SpeechTranscript(
                index: 3,
                value: "ALL",
                startOffset: TimeInterval(0),
                endOffset: TimeInterval(0.1),
                isFinal: true
            ),
            SpeechTranscript(
                index: 4,
                value: "GO",
                startOffset: TimeInterval(0),
                endOffset: TimeInterval(0.1),
                isFinal: false
            ),
            SpeechTranscript(
                index: 5,
                value: "REPOS",
                startOffset: TimeInterval(0),
                endOffset: TimeInterval(0.1),
                isFinal: false
            )
        ])
    }
}

Notice that we are using a SpeechTranscript here - a type imported from Speechly package. We show final transcripts (those that won't change in the future) as bold and tentative (or non-final, meaning they can still change) as light to differentiate between them.

Finally, let's update our ContentView.swift and add our new components and a preview:

import SwiftUI
import Speechly

struct ContentView: View {
    let repos: [GithubRepo]
    let transcript: [SpeechTranscript]
    let startRecording: () -> Void
    let stopRecording: () -> Void

    var body: some View {
        VStack {
            TranscriptText(words: transcript)
            RepoList(repos: repos)
            MicrophoneButton(onStart: startRecording, onStop: stopRecording)
                .padding(.bottom, 15)
        }
    }
}

struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView(
            repos: GithubRepoRepository.shared.list(),
            transcript: [
                SpeechTranscript(
                    index: 1,
                    value: "SHOW",
                    startOffset: TimeInterval(0),
                    endOffset: TimeInterval(0.1),
                    isFinal: true
                ),
                SpeechTranscript(
                    index: 2,
                    value: "ME",
                    startOffset: TimeInterval(0),
                    endOffset: TimeInterval(0.1),
                    isFinal: true
                ),
                SpeechTranscript(
                    index: 3,
                    value: "ALL",
                    startOffset: TimeInterval(0),
                    endOffset: TimeInterval(0.1),
                    isFinal: true
                ),
                SpeechTranscript(
                    index: 4,
                    value: "GO",
                    startOffset: TimeInterval(0),
                    endOffset: TimeInterval(0.1),
                    isFinal: false
                ),
                SpeechTranscript(
                    index: 5,
                    value: "REPOS",
                    startOffset: TimeInterval(0),
                    endOffset: TimeInterval(0.1),
                    isFinal: false
                )
            ],
            startRecording: {},
            stopRecording: {}
        )
    }
}

To make sure our project can still be built, we also need to update SpeechlyRepoFilteringApp.swift:

import SwiftUI

@main
struct SpeechlyRepoFilteringApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView(
                repos: GithubRepoRepository.shared.list(),
                transcript: [],
                startRecording: {},
                stopRecording: {}
            )
        }
    }
}

Go ahead and build the project and then check out the ContentView preview. You should see something like this:

Configure your Speechly application

Before we proceed with the app, let's take a quick detour and create our Speechly app.

Go to https://api.speechly.com/dashboard and login (or sign up if you haven't yet) and create a new app (you can check our Speechly Dashboard quickstart guide if you feel lost).

Let's add a couple of simple commands for manipulating the data we see in the table:

A command to filter by programming language, e.g., when a user says, "Show me TypeScript repos," the app will only show repos with that specific language.
A command to sort the results in a specific order, e.g., "Sort the results by forks," will sort the repos by the number of forks it has.
A command to reset the filters, e.g., "Reset the filters to default," will remove the language filter and reset the sorting to some default.

Let's add the following to our app configuration:

languages = [
    Go
    TypeScript
    Python
]

sort_fields = [
    name
    language
    followers
    stars
    forks
]

results = [
    items
    results
    repos
    repositories
]

*filter show {me} {[all | only]} $languages(language) {$results}
*filter filter {$results} by $languages(language) {language}

*sort [sort | order] {the} {$results} by $sort_fields(sort_field)

*reset [reset | remove] {[the | all]} {filters} {to default}

Don't forget to add sort, filter, and reset as intents and languages and sort_fields as entities!

As you can see from the comments, this configuration will make our Speechly app understand the commands we need and properly detect entities and intents. Keep in mind that the cool part is that the model will also be able to understand the variations of commands that are not explicitly defined in our configuration. The same also applies to entities - the app won't be limited to only detecting "Go", "TypeScript", and "Python" as options for the language, but other words as well, which will be roughly in the same place in a phrase. For instance, you could try saying, "Show me all Javascript repos"). However, with very domain-specific words, it's always a good idea to list them all in your configuration; otherwise, they might be mistaken for some regular words. For example, the API might not properly detect "Rust" as a programming language if you say, "Show me all Rust repositories," because it would think that you meant "rust" as that thing that destroys metals. You can read more about how to configure Speechly applications in our documentation.

Now that we have our Speechly app deployed, let's integrate it!

Integrate Speechly client

Let's use an observable pattern and create a manager which will handle Speechly connection and provide us with the recognition results. Add the following code to SpeechlyManager.swift:

import Foundation
import SwiftUI
import Speechly

class SpeechlyManager: ObservableObject {
    let client: SpeechClient
    var active: Bool

    @Published var transcript: [SpeechTranscript] = []
    @Published var filter: GithubRepoFilter = GithubRepoFilter.empty

    init() {
        self.active = true
        self.client = try! SpeechClient(
            appId: UUID(uuidString: "your-speechly-app-id")!,
            language: .enUS
        )

        self.client.delegate = self
    }

    func start() {
        if !self.active {
            return
        }

        self.client.start()
    }

    func stop() {
        if !self.active {
            return
        }

        self.client.stop()
    }

    func suspend() {
        if self.active {
            self.client.suspend()
            self.active = false
        }
    }

    func resume() {
        if !self.active {
            try! self.client.resume()
            self.active = true
        }
    }
}

extension SpeechlyManager: SpeechClientDelegate {
    func speechlyClientDidStart(_: SpeechClientProtocol) {
        DispatchQueue.main.async {
            self.transcript = []
        }
    }

    func speechlyClientDidStop(_: SpeechClientProtocol) {
        DispatchQueue.main.async {
            self.transcript = []
        }
    }

    func speechlyClientDidUpdateSegment(_ client: SpeechClientProtocol, segment: SpeechSegment) {
        DispatchQueue.main.async {
            switch segment.intent.value.lowercased() {
            case "filter":
                self.filter = GithubRepoFilter(
                    languageFilter: self.parseLanguageFilter(segment),
                    sortOrder: self.filter.sortOrder
                )
            case "sort":
                self.filter = GithubRepoFilter(
                    languageFilter: self.filter.languageFilter,
                    sortOrder: self.parseSortOrder(segment)
                )
            case "reset":
                self.filter = GithubRepoFilter.empty
            default:
                break
            }

            self.transcript = segment.transcripts
        }
    }

    private func parseSortOrder(
        _ segment: SpeechSegment, defaultOrder: GithubRepoFilter.SortOrder = GithubRepoFilter.empty.sortOrder
    ) -> GithubRepoFilter.SortOrder {
        var order = defaultOrder

        for e in segment.entities {
            if e.type.lowercased() != "sort_field" {
                continue
            }

            switch e.value.lowercased() {
            case "name":
                order = .name
            case "language":
                order = .language
            case "followers":
                order = .followers
            case "stars":
                order = .stars
            case "forks":
                order = .forks
            default:
                continue
            }
        }

        return order
    }

    private func parseLanguageFilter(_ segment: SpeechSegment, initialValue: [GithubRepo.Language] = []) -> [GithubRepo.Language] {
        var languages = initialValue

        for e in segment.entities {
            if e.type.lowercased() != "language" {
                continue
            }

            switch e.value.lowercased() {
            case "go":
                languages.append(.Go)
            case "python":
                languages.append(.Python)
            case "typescript":
                languages.append(.TypeScript)
            default:
                continue
            }
        }

        return languages
    }
}

Make sure you update the language and the app id in the initialiser!

Now, there's a lot to explain here, so here's the gist:

SpeechlyManager creates our client. To initialise the client you need to pass at least appId and language, which you can find from the app page in the dashboard.
To enable the microphone and start sending data to the API you should use client.start() method. Similarly, you use client.stop() for stopping current call.
To receive recognition results (and also some status updates and errors) you need to implement the SpeechClientDelegate pattern and pass the delegate to the client. Our manager uses speechlyClientDidUpdateSegment to react to recognition results and speechlyClientDidStart / speechlyClientDidStop for resetting current transcript. Note also the usage of DispatchQueue.main - this is a requirement of the platform that observable values should only be updated from the main dispatch queue.
To handle the results, you would need to parse them, relying on intent values and entity types / values.
client.suspend() and client.resume() provide functionality for suspending and resuming the client execution - this will come in handy later!
Check out this article in our documentation for more information about how SLU API works!

As you probably noticed, we are using a GithubRepoFilter as a container for our filters, but we haven't defined it yet. Let's do that! Add the following to GithubRepoFilter.swift:

import Foundation

class GithubRepoFilter {
    enum SortOrder {
        case name, language, followers, stars, forks
    }

    static let empty = GithubRepoFilter(languageFilter: [], sortOrder: .name)

    var languageFilter: [GithubRepo.Language]
    var sortOrder: SortOrder

    init(languageFilter: [GithubRepo.Language], sortOrder: SortOrder) {
        self.languageFilter = languageFilter
        self.sortOrder = sortOrder
    }

    func apply(_ input: [GithubRepo]) -> [GithubRepo] {
        var res = input

        if self.languageFilter.count > 0 {
            res = res.filter { repo in
                self.languageFilter.contains(repo.language)
            }
        }

        return res.sorted { (left, right) in
            switch self.sortOrder {
            case .name:
                return left.name < right.name
            case .language:
                return left.language.rawValue < right.language.rawValue
            case .followers:
                return left.followers < right.followers
            case .stars:
                return left.stars < right.stars
            case .forks:
                return left.forks < right.forks
            }
        }
    }
}

Nothing fancy here - just some logic for sorting and filtering an array of GithubRepo structs!

Tying everything together

Now that we have all the pieces ready, let's tie them all together in our SpeechlyRepoFilteringApp.swift! Let's update it with the following code:

import SwiftUI
import Speechly

@main
struct RepoFilteringApp: App {
    @Environment(\.scenePhase) var scenePhase
    @ObservedObject var speechlyManager = SpeechlyManager()

    var body: some Scene {
        WindowGroup {
            ContentView(
                repos: self.speechlyManager.filter.apply(GithubRepoRepository.shared.list()),
                transcript: self.speechlyManager.transcript,
                startRecording: self.speechlyManager.start,
                stopRecording: self.speechlyManager.stop
            )
        }.onChange(of: self.scenePhase) { newPhase in
            switch newPhase {
            case .background:
                self.speechlyManager.suspend()
            case .active:
                self.speechlyManager.resume()
            default:
                break
            }
        }
    }
}

A couple of things going on here:

We use the @ObservedObject to observe the state of SpeechlyManager. This will automatically update our app with new values of filter and transcript properties defined by the manager.
We use the @Environment(\.scenePhase) to track the state of our main (and only) scene phase and react when the scene (and thus the application) enters background state (i.e. it gets suspended by the OS) and leaves it. We use this to suspend and resume our Speechly client - this ensures that we release the microphone and disconnect from the API when those resources are not needed.

Conclusion

And that's it! Now you can build the app and run it in the simulator. Keep in mind that many of the SwiftUI features are only available on iOS 14.0 and above, so make sure to pick the appropriate simulator target.

Go ahead and try out your app - you can filter the repos by language, apply a sorting order, and reset the filters:

If you want to delve into the details, go ahead and check out Speechly documentation and our public GitHub.

You can also find the source code for this tutorial at https://github.com/speechly/ios-repo-filtering - feel free to poke around or use it as a starter for your next Speechly app!

Build voice user interfaces for iOS

Ottomatias Peura — Thu, 26 Nov 2020 12:16:54 +0000

This tutorial is available at https://docs.speechly.com/client-libraries/ios/tutorial/

Introduction

This tutorial will help you to get up and running with Speechly by guiding you through the process of building a simple voice filtering web app with Speechly and SwiftUI.

You can find the source code for this tutorial on GitHub.

Prerequisites

We'll be using Xcode and iOS simulator, so make sure you have those installed.

Xcode 12
iOS Simulator with iOS version 14.0 or later

This tutorial assumes that you know the basics of iOS development and the Swift programming language. Feel free to check the official Apple documentation to get started with Swift or SwiftUI.

Create a new Xcode project

First, we'll need a new iOS app project. Let's create one!

Open Xcode and go to File > New > Project.
Under the set of iOS templates, choose App:
Click Next and fill in your project details. Name your project SpeechlyRepoFiltering, choose SwiftUI as your interface option, SwiftUI App as life cycle and Swift for the language.
Click Next once more and you should have your project created for you.

Adding Speechly client dependency

Next we need to add Speechly client library to our project.

Click File > Swift Packages > Add Package Dependency:
Specify https://github.com/speechly/ios-client.git as the package repository and click Next:
Xcode will fetch the package information for you and will ask you to specify the version. Leave the default rules in and click Next:
After the Xcode fetches the package and its dependencies, it will ask which target you want to add it to. Make sure you have Speechly product added to SpeechlyRepoFiltering and the checkbox checked and then click Finish:

Add repo data and layout

Now that we have the project ready, let's start coding!

Since we are building a filtering app, let's add some data to filter and layout to display it.

To make it simple, our data source will be just a static array with some popular repositories on GitHub.

First, let's add the data model and save it as GithubRepoModel.swift:

import Foundation

struct GithubRepo: Hashable, Identifiable {
    enum Language: String, Hashable {
        case Go = "Go"
        case Python = "Python"
        case TypeScript = "TypeScript"
    }

    let id: Int
    let name: String
    let organisation: String
    let language: Language
    let followers: Int
    let stars: Int
    let forks: Int
}

Now let's add a repository that will return us some sample data. Add the following code to GithubRepoRepository.swift:

import Foundation

class GithubRepoRepository {
    static let shared = GithubRepoRepository()

    func list() -> [GithubRepo] {
        return self.repositories
    }

    private let repositories = [
        GithubRepo(
            id: 1,
            name: "typescript",
            organisation: "microsoft",
            language: .TypeScript,
            followers: 2200,
            stars: 65000,
            forks: 8700
        ),
        GithubRepo(
            id: 2,
            name: "nest",
            organisation: "nestjs",
            language: .TypeScript,
            followers: 648,
            stars: 30900,
            forks: 2800
        ),
        GithubRepo(
            id: 3,
            name: "vscode",
            organisation: "microsoft",
            language: .TypeScript,
            followers: 3000,
            stars: 105000,
            forks: 16700
        ),
        GithubRepo(
            id: 4,
            name: "deno",
            organisation: "denoland",
            language: .TypeScript,
            followers: 1700,
            stars: 68000,
            forks: 3500
        ),
        GithubRepo(
            id: 5,
            name: "kubernetes",
            organisation: "kubernetes",
            language: .Go,
            followers: 3300,
            stars: 70700,
            forks: 25500
        ),
        GithubRepo(
            id: 6,
            name: "moby",
            organisation: "moby",
            language: .Go,
            followers: 3200,
            stars: 58600,
            forks: 16900
        ),
        GithubRepo(
            id: 7,
            name: "hugo",
            organisation: "gohugoio",
            language: .Go,
            followers: 1000,
            stars: 47200,
            forks: 5400
        ),
        GithubRepo(
            id: 8,
            name: "grafana",
            organisation: "grafana",
            language: .Go,
            followers: 1300,
            stars: 37500,
            forks: 7600
        ),
        GithubRepo(
            id: 9,
            name: "pytorch",
            organisation: "pytorch",
            language: .Python,
            followers: 1600,
            stars: 43000,
            forks: 11200
        ),
        GithubRepo(
            id: 10,
            name: "tensorflow",
            organisation: "tensorfow",
            language: .Python,
            followers: 8300,
            stars: 149000,
            forks: 82900
        ),
        GithubRepo(
            id: 11,
            name: "django",
            organisation: "django",
            language: .Python,
            followers: 2300,
            stars: 52800,
            forks: 22800
        ),
        GithubRepo(
            id: 12,
            name: "airflow",
            organisation: "apache",
            language: .Python,
            followers: 716,
            stars: 18500,
            forks: 7200
        )
    ]
}

Now that we have the data, time to visualise it. Let's add some SwiftUI components for that.

Let's start by adding a list view, save the following code as RepoList.swift:

import SwiftUI

struct RepoList: View {
    var repos: [GithubRepo] = []

    var body: some View {
        NavigationView {
            List(repos) { repo in
                RepoRow(repo: repo)
            }
            .navigationBarTitle(Text("Repositories"), displayMode: .inline)
        }
    }
}

struct RepoList_Previews: PreviewProvider {
    static var previews: some View {
        RepoList(repos: GithubRepoRepository.shared.list())
    }
}

Notice the RepoList_Previews - it's a handy way to quickly preview your components without running the app!

As you can see, our list view uses another component, RepoRow. Let's add some code to RepoRow.swift to define it:

import SwiftUI

struct RepoRow: View {
    let repo: GithubRepo

    var body: some View {
        HStack {
            Image(imageName())
                .resizable().frame(width: 32, height: 32, alignment: .center)

            VStack(alignment: .leading, spacing: 7) {
                Text("\(repo.organisation) / ").font(.system(size: 20)).fontWeight(.light) +
                    Text(repo.name).font(.system(size: 20)).fontWeight(.bold)

                HStack(alignment: .lastTextBaseline) {
                    LabeledIcon(name: "arrow.branch", label: formatNumber(repo.forks))
                    LabeledIcon(name: "star", label: formatNumber(repo.stars))
                    LabeledIcon(name: "eye", label: formatNumber(repo.followers))
                }
            }
        }
    }

    private func formatNumber(_ value: Int) -> String {
        if (value < 1000) {
            return String(value)
        }

        if (value < 10000) {
            return "\(String(Double(value) / 1000))K"
        }

        return "\(String(value / 1000))K"
    }

    private func imageName() -> String {
        switch self.repo.language {
        case .Go:
            return "logo-go"
        case .Python:
            return "logo-python"
        case .TypeScript:
            return "logo-ts"
        }
    }
}

OK, now as you can see we are referencing some images with language logos, so let's add those to our project. Download and extract LanguageLogos.zip, then you should have the following files:

logo-go.png
logo-python.png
logo-typescript.png

Now let's add them to our project:

Select Assets.xcassets in the navigation bar in Xcode, then click the + button in the bottom pane and finally click on the Image Set:
Type logo-go as the name, then drag and drop the logo-go.png file to all sizes (1x, 2x and 3x) placeholders. Repeat the same for logo-python and logo-typescript and you should have the following result:

Lastly, there is one more component to add - the LabeledIcon. Let's add the following code and save it as LabeledIcon.swift:

import SwiftUI

struct LabeledIcon: View {
    let name: String
    let label: String

    var body: some View {
        HStack(spacing: 3) {
            Image(systemName: name).foregroundColor(.gray).imageScale(.small)
            Text(label).font(.system(.caption)).fontWeight(.ultraLight)
        }
    }
}

Voila! Now you can go ahead and try running the preview for RepoList. You should see something like this:

Add microphone button and transcript text

Now that we have some data to work with and we can visualise it, time to add a button for activating the microphone and a simple component to display the transcript of what a user is saying.

Save the following code as Microphone.swift:

import SwiftUI

struct MicrophoneButton: View {
    let onStart: () -> Void
    let onStop: () -> Void

    var body: some View {
        ToggleButton(onDown: startRecording, onUp: stopRecording) {
            Circle()
                .frame(width: 80, height: 80)
                .foregroundColor(.blue)
                .overlay(
                    Image(systemName: "mic.fill")
                        .resizable()
                        .aspectRatio(contentMode: .fit)
                        .frame(width: 40, height: 40)
                        .foregroundColor(.white)
                )
        }.buttonStyle(Style())
    }

    private func startRecording() {
        self.onStart()
    }

    private func stopRecording() {
        self.onStop()
    }

    private struct Style: ButtonStyle {
        func makeBody(configuration: Configuration) -> some View {
            configuration.label
                .scaleEffect(configuration.isPressed ? 1.25 : 1.0)
                .animation(.easeInOut)
        }
    }
}

struct MicrophoneButton_Previews: PreviewProvider {
    static var previews: some View {
        MicrophoneButton(onStart: {}, onStop: {})
    }
}

And since it uses a ToggleButton component, let's add it as ToggleButton.swift:

import SwiftUI

struct ToggleButton<Content: View>: View {
    @State private var isDown = false

    let onDown: () -> Void
    let onUp: () -> Void
    let content: () -> Content

    var body: some View {
        Button(action: {}, label: {
            self.content()
        }).simultaneousGesture(
            DragGesture(minimumDistance: 0, coordinateSpace: .local)
                .onChanged { _ in
                    if self.isDown {
                        return
                    }

                    self.isDown = true
                    self.onDown()
                }
                .onEnded { _ in
                    self.isDown = false
                    self.onUp()
                }
        )
    }
}

And for the trasncript, let's add the following to TranscriptText.swift:

import SwiftUI
import Speechly

struct TranscriptText: View {
    let words: [SpeechTranscript]

    var body: some View {
        words.reduce(Text("")) { (acc, word) in
            acc + Text(" ") + Text(word.value).fontWeight(word.isFinal ? .bold : .light)
        }
    }
}

struct TranscriptText_Previews: PreviewProvider {
    static var previews: some View {
        TranscriptText(words: [
            SpeechTranscript(
                index: 1,
                value: "SHOW",
                startOffset: TimeInterval(0),
                endOffset: TimeInterval(0.1),
                isFinal: true
            ),
            SpeechTranscript(
                index: 2,
                value: "ME",
                startOffset: TimeInterval(0),
                endOffset: TimeInterval(0.1),
                isFinal: true
            ),
            SpeechTranscript(
                index: 3,
                value: "ALL",
                startOffset: TimeInterval(0),
                endOffset: TimeInterval(0.1),
                isFinal: true
            ),
            SpeechTranscript(
                index: 4,
                value: "GO",
                startOffset: TimeInterval(0),
                endOffset: TimeInterval(0.1),
                isFinal: false
            ),
            SpeechTranscript(
                index: 5,
                value: "REPOS",
                startOffset: TimeInterval(0),
                endOffset: TimeInterval(0.1),
                isFinal: false
            )
        ])
    }
}

Finally, let's update our ContentView.swift and add our new components and a preview:

import SwiftUI
import Speechly

struct ContentView: View {
    let repos: [GithubRepo]
    let transcript: [SpeechTranscript]
    let startRecording: () -> Void
    let stopRecording: () -> Void

    var body: some View {
        VStack {
            TranscriptText(words: transcript)
            RepoList(repos: repos)
            MicrophoneButton(onStart: startRecording, onStop: stopRecording)
                .padding(.bottom, 15)
        }
    }
}

struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView(
            repos: GithubRepoRepository.shared.list(),
            transcript: [
                SpeechTranscript(
                    index: 1,
                    value: "SHOW",
                    startOffset: TimeInterval(0),
                    endOffset: TimeInterval(0.1),
                    isFinal: true
                ),
                SpeechTranscript(
                    index: 2,
                    value: "ME",
                    startOffset: TimeInterval(0),
                    endOffset: TimeInterval(0.1),
                    isFinal: true
                ),
                SpeechTranscript(
                    index: 3,
                    value: "ALL",
                    startOffset: TimeInterval(0),
                    endOffset: TimeInterval(0.1),
                    isFinal: true
                ),
                SpeechTranscript(
                    index: 4,
                    value: "GO",
                    startOffset: TimeInterval(0),
                    endOffset: TimeInterval(0.1),
                    isFinal: false
                ),
                SpeechTranscript(
                    index: 5,
                    value: "REPOS",
                    startOffset: TimeInterval(0),
                    endOffset: TimeInterval(0.1),
                    isFinal: false
                )
            ],
            startRecording: {},
            stopRecording: {}
        )
    }
}

To make sure our project can still be built, we also need to update SpeechlyRepoFilteringApp.swift:

import SwiftUI

@main
struct SpeechlyRepoFilteringApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView(
                repos: GithubRepoRepository.shared.list(),
                transcript: [],
                startRecording: {},
                stopRecording: {}
            )
        }
    }
}

Go ahead and build the project and then check out the ContentView preview. You should see something like this:

Configure your Speechly application

Before we proceed with the app, let's take a quick detour and create our Speechly app.

Go to https://api.speechly.com/dashboard and login (or sign up if you haven't yet) and create a new app (you can check our Speechly Dashboard quickstart guide if you feel lost).

Let's add a couple of simple commands for manipulating the data we see in the table:

A command to filter by programming language, e.g., when a user says, "Show me TypeScript repos," the app will only show repos with that specific language.
A command to sort the results in a specific order, e.g., "Sort the results by forks," will sort the repos by the number of forks it has.
A command to reset the filters, e.g., "Reset the filters to default," will remove the language filter and reset the sorting to some default.

Let's add the following to our app configuration:

languages = [
    Go
    TypeScript
    Python
]

sort_fields = [
    name
    language
    followers
    stars
    forks
]

results = [
    items
    results
    repos
    repositories
]

*filter show {me} {[all | only]} $languages(language) {$results}
*filter filter {$results} by $languages(language) {language}

*sort [sort | order] {the} {$results} by $sort_fields(sort_field)

*reset [reset | remove] {[the | all]} {filters} {to default}

Don't forget to add sort, filter, and reset as intents and languages and sort_fields as entities!

Now that we have our Speechly app deployed, let's integrate it!

Integrate Speechly client

Let's use an observable pattern and create a manager which will handle Speechly connection and provide us with the recognition results. Add the following code to SpeechlyManager.swift:

import Foundation
import SwiftUI
import Speechly

class SpeechlyManager: ObservableObject {
    let client: SpeechClient
    var active: Bool

    @Published var transcript: [SpeechTranscript] = []
    @Published var filter: GithubRepoFilter = GithubRepoFilter.empty

    init() {
        self.active = true
        self.client = try! SpeechClient(
            appId: UUID(uuidString: "your-speechly-app-id")!,
            language: .enUS
        )

        self.client.delegate = self
    }

    func start() {
        if !self.active {
            return
        }

        self.client.start()
    }

    func stop() {
        if !self.active {
            return
        }

        self.client.stop()
    }

    func suspend() {
        if self.active {
            self.client.suspend()
            self.active = false
        }
    }

    func resume() {
        if !self.active {
            try! self.client.resume()
            self.active = true
        }
    }
}

extension SpeechlyManager: SpeechClientDelegate {
    func speechlyClientDidStart(_: SpeechClientProtocol) {
        DispatchQueue.main.async {
            self.transcript = []
        }
    }

    func speechlyClientDidStop(_: SpeechClientProtocol) {
        DispatchQueue.main.async {
            self.transcript = []
        }
    }

    func speechlyClientDidUpdateSegment(_ client: SpeechClientProtocol, segment: SpeechSegment) {
        DispatchQueue.main.async {
            switch segment.intent.value.lowercased() {
            case "filter":
                self.filter = GithubRepoFilter(
                    languageFilter: self.parseLanguageFilter(segment),
                    sortOrder: self.filter.sortOrder
                )
            case "sort":
                self.filter = GithubRepoFilter(
                    languageFilter: self.filter.languageFilter,
                    sortOrder: self.parseSortOrder(segment)
                )
            case "reset":
                self.filter = GithubRepoFilter.empty
            default:
                break
            }

            self.transcript = segment.transcripts
        }
    }

    private func parseSortOrder(
        _ segment: SpeechSegment, defaultOrder: GithubRepoFilter.SortOrder = GithubRepoFilter.empty.sortOrder
    ) -> GithubRepoFilter.SortOrder {
        var order = defaultOrder

        for e in segment.entities {
            if e.type.lowercased() != "sort_field" {
                continue
            }

            switch e.value.lowercased() {
            case "name":
                order = .name
            case "language":
                order = .language
            case "followers":
                order = .followers
            case "stars":
                order = .stars
            case "forks":
                order = .forks
            default:
                continue
            }
        }

        return order
    }

    private func parseLanguageFilter(_ segment: SpeechSegment, initialValue: [GithubRepo.Language] = []) -> [GithubRepo.Language] {
        var languages = initialValue

        for e in segment.entities {
            if e.type.lowercased() != "language" {
                continue
            }

            switch e.value.lowercased() {
            case "go":
                languages.append(.Go)
            case "python":
                languages.append(.Python)
            case "typescript":
                languages.append(.TypeScript)
            default:
                continue
            }
        }

        return languages
    }
}

Make sure you update the language and the app id in the initialiser!

Now, there's a lot to explain here, so here's the gist:

SpeechlyManager creates our client. To initialise the client you need to pass at least appId and language, which you can find from the app page in the dashboard.
To enable the microphone and start sending data to the API you should use client.start() method. Similarly, you use client.stop() for stopping current call.
To receive recognition results (and also some status updates and errors) you need to implement the SpeechClientDelegate pattern and pass the delegate to the client. Our manager uses speechlyClientDidUpdateSegment to react to recognition results and speechlyClientDidStart / speechlyClientDidStop for resetting current transcript. Note also the usage of DispatchQueue.main - this is a requirement of the platform that observable values should only be updated from the main dispatch queue.
To handle the results, you would need to parse them, relying on intent values and entity types / values.
client.suspend() and client.resume() provide functionality for suspending and resuming the client execution - this will come in handy later!
Check out this article in our documentation for more information about how SLU API works!

As you probably noticed, we are using a GithubRepoFilter as a container for our filters, but we haven't defined it yet. Let's do that! Add the following to GithubRepoFilter.swift:

import Foundation

class GithubRepoFilter {
    enum SortOrder {
        case name, language, followers, stars, forks
    }

    static let empty = GithubRepoFilter(languageFilter: [], sortOrder: .name)

    var languageFilter: [GithubRepo.Language]
    var sortOrder: SortOrder

    init(languageFilter: [GithubRepo.Language], sortOrder: SortOrder) {
        self.languageFilter = languageFilter
        self.sortOrder = sortOrder
    }

    func apply(_ input: [GithubRepo]) -> [GithubRepo] {
        var res = input

        if self.languageFilter.count > 0 {
            res = res.filter { repo in
                self.languageFilter.contains(repo.language)
            }
        }

        return res.sorted { (left, right) in
            switch self.sortOrder {
            case .name:
                return left.name < right.name
            case .language:
                return left.language.rawValue < right.language.rawValue
            case .followers:
                return left.followers < right.followers
            case .stars:
                return left.stars < right.stars
            case .forks:
                return left.forks < right.forks
            }
        }
    }
}

Nothing fancy here - just some logic for sorting and filtering an array of GithubRepo structs!

Tying everything together

Now that we have all the pieces ready, let's tie them all together in our SpeechlyRepoFilteringApp.swift! Let's update it with the following code:

import SwiftUI
import Speechly

@main
struct RepoFilteringApp: App {
    @Environment(\.scenePhase) var scenePhase
    @ObservedObject var speechlyManager = SpeechlyManager()

    var body: some Scene {
        WindowGroup {
            ContentView(
                repos: self.speechlyManager.filter.apply(GithubRepoRepository.shared.list()),
                transcript: self.speechlyManager.transcript,
                startRecording: self.speechlyManager.start,
                stopRecording: self.speechlyManager.stop
            )
        }.onChange(of: self.scenePhase) { newPhase in
            switch newPhase {
            case .background:
                self.speechlyManager.suspend()
            case .active:
                self.speechlyManager.resume()
            default:
                break
            }
        }
    }
}

A couple of things going on here:

We use the @ObservedObject to observe the state of SpeechlyManager. This will automatically update our app with new values of filter and transcript properties defined by the manager.
We use the @Environment(\.scenePhase) to track the state of our main (and only) scene phase and react when the scene (and thus the application) enters background state (i.e. it gets suspended by the OS) and leaves it. We use this to suspend and resume our Speechly client - this ensures that we release the microphone and disconnect from the API when those resources are not needed.

Conclusion

Go ahead and try out your app - you can filter the repos by language, apply a sorting order, and reset the filters:

If you want to delve into the details, go ahead and check out our documentation and our public GitHub.

You can also find the source code for this tutorial at https://github.com/speechly/ios-repo-filtering - feel free to poke around or use it as a starter for your next Speechly app!

Real-time voice search with React

Ottomatias Peura — Thu, 22 Oct 2020 12:10:43 +0000

Introduction

This tutorial will help you to get up and running with Speechly by guiding you through the process of building a simple voice filtering web app with Speechly and React.

You can find the source code for this tutorial on GitHub and you can also try out the final result hosted on GitHub Pages.

Prerequisites

Since we'll be using create-react-app for this tutorial, we'll need the following tools:

Node.js 8.10+
npm 5.2+

Note that this tutorial also uses TypeScript, so feel free to check out TypeScript documentation if you're not familiar with it.

1. Creating an app

Let's get started by creating an app and installing its dependencies:

npx create-react-app speechly-voice-filter --typescript
cd speechly-voice-filter
npm i

Now that you've created the app, you can check it out by running npm start - it should open a browser tab with your app running in it.

2. Adding data and layout

Since we are building a filtering app, let's add some data to filter and layout to display it.

To make it simple, our data source will be just a static array with some popular repositories on GitHub. Let's add the following code and save it as src/data.ts:

export type Repository = {
  name: string;
  description: string;
  language: string;
  followers: number;
  stars: number;
  forks: number;
};

export const repositories: Repository[] = [
  {
    name: "microsoft/typescript",
    description:
      "TypeScript is a superset of JavaScript that compiles to clean JavaScript output",
    language: "TypeScript",
    followers: 2200,
    stars: 65000,
    forks: 8700,
  },
  {
    name: "nestjs/nest",
    description:
      "A progressive Node.js framework for building efficient, scalable, and enterprise-grade server-side applications on top of TypeScript & JavaScript (ES6, ES7, ES8)",
    language: "TypeScript",
    followers: 648,
    stars: 30900,
    forks: 2800,
  },
  {
    name: "microsoft/vscode",
    description: "Visual Studio Code",
    language: "TypeScript",
    followers: 3000,
    stars: 105000,
    forks: 16700,
  },
  {
    name: "denoland/deno",
    description: "A secure JavaScript and TypeScript runtime",
    language: "TypeScript",
    followers: 1700,
    stars: 68000,
    forks: 3500,
  },
  {
    name: "kubernetes/kubernetes",
    description: "Production-Grade Container Scheduling and Management",
    language: "Go",
    followers: 3300,
    stars: 70700,
    forks: 25500,
  },
  {
    name: "moby/moby",
    description:
      "Moby Project - a collaborative project for the container ecosystem to assemble container-based systems",
    language: "Go",
    followers: 3200,
    stars: 58600,
    forks: 16900,
  },
  {
    name: "gohugoio/hugo",
    description: "The world’s fastest framework for building websites",
    language: "Go",
    followers: 1000,
    stars: 47200,
    forks: 5400,
  },
  {
    name: "grafana/grafana",
    description:
      "The tool for beautiful monitoring and metric analytics & dashboards for Graphite, InfluxDB & Prometheus & More",
    language: "Go",
    followers: 1300,
    stars: 37500,
    forks: 7600,
  },
  {
    name: "pytorch/pytorch",
    description:
      "Tensors and Dynamic neural networks in Python with strong GPU acceleration",
    language: "Python",
    followers: 1600,
    stars: 43000,
    forks: 11200,
  },
  {
    name: "tensorflow/tensorflow",
    description: "An Open Source Machine Learning Framework for Everyone",
    language: "Python",
    followers: 8300,
    stars: 149000,
    forks: 82900,
  },
  {
    name: "django/django",
    description: "The Web framework for perfectionists with deadlines",
    language: "Python",
    followers: 2300,
    stars: 52800,
    forks: 22800,
  },
  {
    name: "apache/airflow",
    description:
      "Apache Airflow - A platform to programmatically author, schedule, and monitor workflows",
    language: "Python",
    followers: 716,
    stars: 18500,
    forks: 7200,
  },
];

We can display this data in a simple table, so let's add a component for that under src/RepoList.tsx:

import React from "react";

import { Repository } from "./data";

type Props = {
  repos: Repository[];
};

export const RepoList = ({ repos }: Props): JSX.Element => {
  return (
    <div className="block">
      <table>
        <thead>
          <tr>
            <th>Name</th>
            <th>Language</th>
            <th>Description</th>
            <th>Followers</th>
            <th>Stars</th>
            <th>Forks</th>
          </tr>
        </thead>
        <tbody>
          {repos.map((repo) => (
            <RepoRow repo={repo} key={repo.name} />
          ))}
        </tbody>
      </table>
    </div>
  );
};

const RepoRow = React.memo(
  ({ repo }: { repo: Repository }): JSX.Element => {
    return (
      <tr>
        <td>{repo.name}</td>
        <td>{repo.language}</td>
        <td>{repo.description}</td>
        <td>{repo.followers}</td>
        <td>{repo.stars}</td>
        <td>{repo.forks}</td>
      </tr>
    );
  }
);

In order to show the table, we'll need to render it. We could render our table right in our top-level App component, but let's instead use a top-level component for our app under src/SpeechApp.tsx - it will come in handy later on:

import React from "react";

import { repositories } from "./data";

import { RepoList } from "./RepoList";

export const SpeechApp: React.FC = (): JSX.Element => {
  return (
    <div>
      <RepoList repos={repositories} />
    </div>
  );
};

Now let's add it to our top-level component:

import React from "react";
import { SpeechProvider } from "@speechly/react-client";

import "./App.css";

import { SpeechApp } from "./SpeechApp";

function App(): JSX.Element {
  return (
    <div className="App">
      <SpeechApp />
    </div>
  );
}

export default App;

3. Adding Speechly client and a microphone button

Before we proceed with the app, let's take a quick detour and train a very simple and not very useful Speechly app, so that we can use it to test our integration later on.

Go to https://www.speechly.com/dashboard and login (or sign up if you haven't yet) and create a new app (you can check our Speechly Dashboard quickstart guide if you feel lost). Feel free to use any configuration you want, even an almost empty configuration with just "Hello world" will suffice, but make sure your app is deployed!

Once you have your Speechly app deployed, let's integrate it. Start by installing Speechly React client:

npm i --save @speechly/react-client

The client exposes a context provider and a hook that allows you to consume that context. Let's add the context provider to src/App.tsx. Make sure you provide the App ID of your Speechly app as a property for SpeechProvider!

import React from "react";
import { SpeechProvider } from "@speechly/react-client";

import "./App.css";

function App(): JSX.Element {
  return (
    <div className="App">
      <SpeechProvider appId="your-app-id-here" language="en-US">
        <SpeechApp />
      </SpeechProvider>
    </div>
  );
}

export default App;

Next, let's add some code to act as the microphone button. Also, it would be nice to see what we are saying, so let's also render the transcript next to the button for some feedback. Let's make that a separate component and save it as src/Microphone.tsx:

import React from "react";
import {
  Word as SpeechWord,
  SpeechSegment,
  SpeechState,
} from "@speechly/react-client";

type Props = {
  segment?: SpeechSegment;
  state: SpeechState;
  onRecord: () => Promise<void>;
};

export const Microphone = React.memo(
  ({ state, segment, onRecord }: Props): JSX.Element => {
    let enabled = false;
    let text = "Error";

    switch (state) {
      case SpeechState.Idle:
      case SpeechState.Ready:
        enabled = true;
        text = "Start";
        break;
      case SpeechState.Recording:
        enabled = true;
        text = "Stop";
        break;
      case SpeechState.Connecting:
      case SpeechState.Loading:
        enabled = false;
        text = "Loading...";
        break;
    }

    return (
      <div>
        <button onClick={onRecord} disabled={!enabled}>
          {text}
        </button>
        <Transcript segment={segment} />
      </div>
    );
  }
);

const Transcript = React.memo(
  ({ segment }: { segment?: SpeechSegment }): JSX.Element => {
    if (segment === undefined) {
      return (
        <div>
          <em>Waiting for speech input...</em>
        </div>
      );
    }

    return (
      <div>
        {segment.words.map((w) => (
          <Word word={w} key={w.index} />
        ))}
      </div>
    );
  }
);

const Word = React.memo(
  ({ word }: { word: SpeechWord }): JSX.Element => {
    if (word.isFinal) {
      return <strong>{`${word.value} `}</strong>;
    }

    return <span>{`${word.value} `}</span>;
  }
);

As you can see, this component renders a button that calls the onRecord callback passed in the properties and uses the state of Speechly client to determine when to enable the button and which text to use as its label. In addition to that, the component also renders the transcript of the phrase by assembling individual transcripted words from a segment (check out this article in our documentation for more information about how SLU API works). Since a word can be either tentative (i.e., its value can change as the API receives more audio data) or final, we use bold text to highlight final words.

One more step - we'd need to render our component and hook it up to the API. Let's add it to our SpeechApp component:

import React from "react";
import { useSpeechContext } from "@speechly/react-client";

import { repositories } from "./data";

import { RepoList } from "./RepoList";
import { Microphone } from "./Microphone";

export const SpeechApp: React.FC = (): JSX.Element => {
  const { toggleRecording, speechState, segment } = useSpeechContext();

  return (
    <div>
      <Microphone
        segment={segment}
        state={speechState}
        onRecord={toggleRecording}
      />
      <RepoList repos={repositories} />
    </div>
  );
};

Here we use the other main part of Speechly React client - a custom hook that consumes the state preserved in SpeechProvider. Feel free to check the API documentation of React client to see what other properties are returned by the hook.

Now you can go ahead and try talking to the app and see what you get back in the transcript. Congratulations, you've just integrated Speechly into the app.

However, we still need to implement the filtering functionality, so let's go ahead and update our Speechly app configuration to support that.

4. Configuring Speechly app

Now that we've integrated the API into the app, it's time to make our Speechly app useful. Let's add a couple of simple commands for manipulating the data we see in the table:

A command to filter by programming language, e.g., when a user says, "Show me TypeScript repos," the app will only show repos with that specific language.
A command to sort the results in a specific order, e.g., "Sort the results by forks," will sort the repos by the number of forks it has.
A command to reset the filters, e.g., "Reset the filters to default," will remove the language filter and reset the sorting to some default.

Let's go back to Speechly dashboard and update the configuration of our app with the following:

# Which languages we can filter by
languages = [
  Go
  TypeScript
  Python
]

# Which fields we can sort by
sort_fields = [
  name
  description
  language
  followers
  stars
  forks
]

# Synonyms for "repo"
results = [
  items
  results
  repos
  repositories
]

# A couple of commands for filtering.
#
# This will expand into e.g. following examples (not exhaustive):
# "Show all Go repos"
# "Show me only TypeScript repositories"
# "Show Python results"
# etc.
#
# Words in curly brackets ("{me}") are optional.
# Square brackets are for lists (e.g. one option from the list may be used)
*filter show {me} {[all | only]} $languages(language) {$results}
*filter filter {$results} by $languages(language) {language}

# A command for sorting, e.g.:
# "Sort the repos by name"
# "Order results by forks"
# etc.
*sort [sort | order] {the} {$results} by $sort_fields(sort_field)

# A command for resetting the filters, e.g.:
# "Reset all filters to default"
# "Remove the filters"
# "Reset to default"
# etc.
*reset [reset | remove] {[the | all]} {filters} {to default}

Don't forget to add sort, filter, and reset as intents and languages and sort_fields as entities!

Once you've deployed your new version of the Speechly app, let's continue to parsing the results.

5. Parsing intents and entities

Now that we've trained a version of Speechly app with proper entities and intents, let's parse the results. First let's add our parsing logic to src/parser.ts:

import { SpeechSegment } from "@speechly/react-client";

export enum IntentType {
  Unknown = "unknown",
  Sort = "sort",
  Filter = "filter",
  Reset = "reset",
}

export enum EntityType {
  Language = "language",
  SortField = "sort_field",
}

export enum SortEntityType {
  Unknown = "unknown",
  Name = "name",
  Description = "description",
  Language = "language",
  Followers = "followers",
  Stars = "stars",
  Forks = "forks",
}

const SpeechIntentValues = Object.values(IntentType) as string[];
const SortTypeValues = Object.values(SortEntityType) as string[];

export function parseIntent(segment: SpeechSegment): IntentType {
  const { intent } = segment;

  if (SpeechIntentValues.includes(intent.intent)) {
    return intent.intent as IntentType;
  }

  return IntentType.Unknown;
}

export function parseLanguageEntity(segment: SpeechSegment): string[] {
  const langs: string[] = [];

  for (const e of segment.entities) {
    if (e.type === EntityType.Language) {
      langs.push(e.value.toLowerCase());
    }
  }

  return langs;
}

export function parseSortEntity(segment: SpeechSegment): SortEntityType {
  let s = SortEntityType.Unknown;

  for (const e of segment.entities) {
    const val = e.value.toLowerCase();

    if (e.type === EntityType.SortField && SortTypeValues.includes(val)) {
      s = val as SortEntityType;
    }
  }

  return s;
}

Here we define a couple of functions to parse intents and different entity types from a SpeechSegment, which is returned by useSpeechContext. As you can see, the code is pretty straightforward. Most of it is actually just listing which intents and entities we expect and defining them as enumerations, since it's always a good idea to check the results returned from API against a pre-defined list of allowed values to avoid bugs. Another good idea is to make sure we use consistent case (in this case by casting the results to lower case) to avoid false negatives when, for example, comparing STARS to stars.

Now that we have our code for parsing the results from a segment, it's time to use it. Let's update our SpeechApp and add some code that calls our parser:

import React, { useEffect } from "react";
import { SpeechSegment, useSpeechContext } from "@speechly/react-client";

import { repositories } from "./data";
import {
  IntentType,
  SortEntityType,
  parseIntent,
  parseLanguageEntity,
  parseSortEntity,
} from "./parser";

import { RepoList } from "./RepoList";
import { Microphone } from "./Microphone";

export const SpeechApp: React.FC = (): JSX.Element => {
  const { toggleRecording, speechState, segment } = useSpeechContext();

  useEffect(() => {
    if (segment === undefined) {
      return;
    }

    parseSegment(segment);
  }, [segment]);

  return (
    <div>
      <Microphone
        segment={segment}
        state={speechState}
        onRecord={toggleRecording}
      />
      <RepoList repos={repositories} />
    </div>
  );
};

function parseSegment(segment: SpeechSegment) {
  const intent = parseIntent(segment);

  switch (intent) {
    case IntentType.Filter:
      const languages = parseLanguageEntity(segment);
      console.log("Filtering by languages", languages);
      break;
    case IntentType.Sort:
      const sortBy = parseSortEntity(segment);
      if (sortBy !== SortEntityType.Unknown) {
        console.log("Sorting by field", sortBy);
      }
      break;
    case IntentType.Reset:
      console.log("Resetting the filters");
      break;
  }
}

Here we define a parseSegment function that is called every time a segment changes by using React's useEffect hook. Since segment might come as undefined (this happens after the user stops speaking and the API sends its final response), we want to check for that before trying to parse it. The function checks for the intent and then calls the appropriate entity parser (or no entity parser at all if the intent was to reset the filters). For now, we are just going to log the results of the parser, but to use them, we'll have to add some filters. Let's continue with that!

6. Adding and applying filters

In order to apply filters, we'd need to implement some filtering logic, so let's do just that and add it as src/filter.ts:

import { Repository } from "./data";
import { SortEntityType } from "./parser";

export type Filter = {
  languages?: string[];
  sortBy?: SortEntityType;
};

export function filterRepos(
  input: Repository[],
  filters: Filter
): Repository[] {
  let output = input;

  const languages = filters.languages ?? [];
  if (languages.length > 0) {
    output = input.filter((repo) =>
      languages.includes(repo.language.toLowerCase())
    );
  }

  if (filters.sortBy === undefined) {
    return output;
  }

  return output.sort((left, right) => {
    switch (filters.sortBy) {
      case SortEntityType.Name:
        return left.name.localeCompare(right.name);
      case SortEntityType.Description:
        return left.description.localeCompare(right.description);
      case SortEntityType.Language:
        return left.language.localeCompare(right.language);
      case SortEntityType.Followers:
        return compareNumber(left.followers, right.followers);
      case SortEntityType.Stars:
        return compareNumber(left.stars, right.stars);
      case SortEntityType.Forks:
        return compareNumber(left.forks, right.forks);
    }

    return 0;
  });
}

function compareNumber(left: number, right: number) {
  if (left < right) {
    return -1;
  }

  if (left > right) {
    return 1;
  }

  return 0;
}

Here we define a Filter type that contains a list of languages to display and the field to sort by. We also define a function filterRepos that takes a list of repositories and a filter and returns a new list of repositories filtered and sorted according to that filter.

Now we need to call the filtering function when we get new results from the API, so let's also update our SpeechApp to do that:

import React, { useEffect, useState } from "react";
import { SpeechSegment, useSpeechContext } from "@speechly/react-client";

import { repositories, Repository } from "./data";
import { Filter, filterRepos } from "./filter";
import {
  IntentType,
  SortEntityType,
  parseIntent,
  parseLanguageEntity,
  parseSortEntity,
} from "./parser";

import { RepoList } from "./RepoList";
import { Microphone } from "./Microphone";

export const SpeechApp: React.FC = (): JSX.Element => {
  const [filter, setFilter] = useState<Filter>(defaultFilter);
  const [repos, setRepos] = useState<Repository[]>(repositories);

  const { toggleRecording, speechState, segment } = useSpeechContext();

  useEffect(() => {
    if (segment === undefined) {
      return;
    }

    const nextFilter = {
      ...filter,
      ...parseSegment(segment),
    };

    setFilter(nextFilter);
    setRepos(filterRepos(repositories, nextFilter));
    // eslint-disable-next-line react-hooks/exhaustive-deps
  }, [segment]);

  return (
    <div>
      <Microphone
        segment={segment}
        state={speechState}
        onRecord={toggleRecording}
      />
      <RepoList repos={repos} />
    </div>
  );
};

const emptyFilter: Filter = {};
const defaultFilter: Filter = {
  languages: [],
  sortBy: SortEntityType.Name,
};

function parseSegment(segment: SpeechSegment): Filter {
  const intent = parseIntent(segment);

  switch (intent) {
    case IntentType.Filter:
      const languages = parseLanguageEntity(segment);

      if (languages.length === 0) {
        return emptyFilter;
      }

      return {
        languages,
      };
    case IntentType.Sort:
      const sortBy = parseSortEntity(segment);
      if (sortBy !== SortEntityType.Unknown) {
        return {
          sortBy,
        };
      }

      return emptyFilter;
    case IntentType.Reset:
      return defaultFilter;
    default:
      return emptyFilter;
  }
}

Here we use React's useState hook to create a couple of stateful variables for storing filtered results and last filters (you can append them by saying "Show me all Go repos" first and then following up with "Sort by start"). Every time we get a new state of segment from the API, we call our parseSegment to parse the filters from it and then append those filters to the ones we've saved in the state. Then we also apply new filters to the list of repositories before passing them on to rendering.

Conclusion

And that's it! Now you can go ahead and try out your app - you can filter the repos by language, apply a sorting order, and reset the filters.

If you want to delve into the details, go ahead and check out our documentation and our public GitHub.

You can also check the source code for this tutorial at https://github.com/speechly/react-example-repo-filtering. Feel free to navigate through individual commits - they refer to each section of this tutorial.

Build real-time voice search with React

Ottomatias Peura — Thu, 15 Oct 2020 18:43:59 +0000

Introduction

This tutorial will help you to get up and running with Speechly by guiding you through the process of building a simple voice filtering web app with Speechly and React.

You can find the source code for this tutorial on GitHub and you can also try out the final result hosted on GitHub Pages.

Prerequisites

Since we'll be using create-react-app for this tutorial, we'll need the following tools:

Node.js 8.10+
npm 5.2+

Note that this tutorial also uses TypeScript, so feel free to check out TypeScript documentation if you're not familiar with it.

1. Creating an app

Let's get started by creating an app and installing its dependencies:

npx create-react-app speechly-voice-filter --typescript
cd speechly-voice-filter
npm i

Now that you've created the app, you can check it out by running npm start - it should open a browser tab with your app running in it.

2. Adding data and layout

Since we are building a filtering app, let's add some data to filter and layout to display it.

To make it simple, our data source will be just a static array with some popular repositories on GitHub. Let's add the following code and save it as src/data.ts:

export type Repository = {
  name: string;
  description: string;
  language: string;
  followers: number;
  stars: number;
  forks: number;
};

export const repositories: Repository[] = [
  {
    name: "microsoft/typescript",
    description:
      "TypeScript is a superset of JavaScript that compiles to clean JavaScript output",
    language: "TypeScript",
    followers: 2200,
    stars: 65000,
    forks: 8700,
  },
  {
    name: "nestjs/nest",
    description:
      "A progressive Node.js framework for building efficient, scalable, and enterprise-grade server-side applications on top of TypeScript & JavaScript (ES6, ES7, ES8)",
    language: "TypeScript",
    followers: 648,
    stars: 30900,
    forks: 2800,
  },
  {
    name: "microsoft/vscode",
    description: "Visual Studio Code",
    language: "TypeScript",
    followers: 3000,
    stars: 105000,
    forks: 16700,
  },
  {
    name: "denoland/deno",
    description: "A secure JavaScript and TypeScript runtime",
    language: "TypeScript",
    followers: 1700,
    stars: 68000,
    forks: 3500,
  },
  {
    name: "kubernetes/kubernetes",
    description: "Production-Grade Container Scheduling and Management",
    language: "Go",
    followers: 3300,
    stars: 70700,
    forks: 25500,
  },
  {
    name: "moby/moby",
    description:
      "Moby Project - a collaborative project for the container ecosystem to assemble container-based systems",
    language: "Go",
    followers: 3200,
    stars: 58600,
    forks: 16900,
  },
  {
    name: "gohugoio/hugo",
    description: "The world’s fastest framework for building websites",
    language: "Go",
    followers: 1000,
    stars: 47200,
    forks: 5400,
  },
  {
    name: "grafana/grafana",
    description:
      "The tool for beautiful monitoring and metric analytics & dashboards for Graphite, InfluxDB & Prometheus & More",
    language: "Go",
    followers: 1300,
    stars: 37500,
    forks: 7600,
  },
  {
    name: "pytorch/pytorch",
    description:
      "Tensors and Dynamic neural networks in Python with strong GPU acceleration",
    language: "Python",
    followers: 1600,
    stars: 43000,
    forks: 11200,
  },
  {
    name: "tensorflow/tensorflow",
    description: "An Open Source Machine Learning Framework for Everyone",
    language: "Python",
    followers: 8300,
    stars: 149000,
    forks: 82900,
  },
  {
    name: "django/django",
    description: "The Web framework for perfectionists with deadlines",
    language: "Python",
    followers: 2300,
    stars: 52800,
    forks: 22800,
  },
  {
    name: "apache/airflow",
    description:
      "Apache Airflow - A platform to programmatically author, schedule, and monitor workflows",
    language: "Python",
    followers: 716,
    stars: 18500,
    forks: 7200,
  },
];

We can display this data in a simple table, so let's add a component for that under src/RepoList.tsx:

import React from "react";

import { Repository } from "./data";

type Props = {
  repos: Repository[];
};

export const RepoList = ({ repos }: Props): JSX.Element => {
  return (
    <div className="block">
      <table>
        <thead>
          <tr>
            <th>Name</th>
            <th>Language</th>
            <th>Description</th>
            <th>Followers</th>
            <th>Stars</th>
            <th>Forks</th>
          </tr>
        </thead>
        <tbody>
          {repos.map((repo) => (
            <RepoRow repo={repo} key={repo.name} />
          ))}
        </tbody>
      </table>
    </div>
  );
};

const RepoRow = React.memo(
  ({ repo }: { repo: Repository }): JSX.Element => {
    return (
      <tr>
        <td>{repo.name}</td>
        <td>{repo.language}</td>
        <td>{repo.description}</td>
        <td>{repo.followers}</td>
        <td>{repo.stars}</td>
        <td>{repo.forks}</td>
      </tr>
    );
  }
);

import React from "react";

import { repositories } from "./data";

import { RepoList } from "./RepoList";

export const SpeechApp: React.FC = (): JSX.Element => {
  return (
    <div>
      <RepoList repos={repositories} />
    </div>
  );
};

Now let's add it to our top-level component:

import React from "react";
import { SpeechProvider } from "@speechly/react-client";

import "./App.css";

import { SpeechApp } from "./SpeechApp";

function App(): JSX.Element {
  return (
    <div className="App">
      <SpeechApp />
    </div>
  );
}

export default App;

3. Adding Speechly client and a microphone button

Before we proceed with the app, let's take a quick detour and train a very simple and not very useful Speechly app, so that we can use it to test our integration later on.

Go to https://www.speechly.com/dashboard and login (or sign up if you haven't yet) and create a new app (you can check our Speechly Dashboard quickstart guide if you feel lost). Feel free to use any configuration you want, even an almost empty configuration with just a "Hello world" will suffice, but make sure your app is deployed!

Once you have your Speechly app deployed, let's integrate it. Start by installing Speechly React client:

npm i --save @speechly/react-client

The client exposes a context provider and a hook that allows you to consume that context. Let's add the context provider to src/App.tsx - make sure you provide the App ID of your Speechly app as a property for SpeechProvider!

import React from "react";
import { SpeechProvider } from "@speechly/react-client";

import "./App.css";

function App(): JSX.Element {
  return (
    <div className="App">
      <SpeechProvider appId="your-app-id-here" language="en-US">
        <SpeechApp />
      </SpeechProvider>
    </div>
  );
}

export default App;

Next let's add some code to act as the microphone button. Also, it would be nice to see what we are saying, so let's also render the transcript next to the button for some feedback. Let's make that a separate component and save it as src/Microphone.tsx:

import React from "react";
import {
  Word as SpeechWord,
  SpeechSegment,
  SpeechState,
} from "@speechly/react-client";

type Props = {
  segment?: SpeechSegment;
  state: SpeechState;
  onRecord: () => Promise<void>;
};

export const Microphone = React.memo(
  ({ state, segment, onRecord }: Props): JSX.Element => {
    let enabled = false;
    let text = "Error";

    switch (state) {
      case SpeechState.Idle:
      case SpeechState.Ready:
        enabled = true;
        text = "Start";
        break;
      case SpeechState.Recording:
        enabled = true;
        text = "Stop";
        break;
      case SpeechState.Connecting:
      case SpeechState.Loading:
        enabled = false;
        text = "Loading...";
        break;
    }

    return (
      <div>
        <button onClick={onRecord} disabled={!enabled}>
          {text}
        </button>
        <Transcript segment={segment} />
      </div>
    );
  }
);

const Transcript = React.memo(
  ({ segment }: { segment?: SpeechSegment }): JSX.Element => {
    if (segment === undefined) {
      return (
        <div>
          <em>Waiting for speech input...</em>
        </div>
      );
    }

    return (
      <div>
        {segment.words.map((w) => (
          <Word word={w} key={w.index} />
        ))}
      </div>
    );
  }
);

const Word = React.memo(
  ({ word }: { word: SpeechWord }): JSX.Element => {
    if (word.isFinal) {
      return <strong>{`${word.value} `}</strong>;
    }

    return <span>{`${word.value} `}</span>;
  }
);

As you can see this component renders a button that calls the onRecord callback passed in the properties and uses the state of Speechly client to determine when to enable the button and which text to use as its label. In addition to that the component also renders the transcript of the phrase by assembling individual transcripted words from a segment (check out this article in our documentation for more information about how SLU API works). Since a word can be either tentative (i.e. its value can change as the API receives more audio data) or final, we use bold text to highlight final words.

One more step - we'd need to render our component and hook it up to the API. Let's add it to our SpeechApp component:

import React from "react";
import { useSpeechContext } from "@speechly/react-client";

import { repositories } from "./data";

import { RepoList } from "./RepoList";
import { Microphone } from "./Microphone";

export const SpeechApp: React.FC = (): JSX.Element => {
  const { toggleRecording, speechState, segment } = useSpeechContext();

  return (
    <div>
      <Microphone
        segment={segment}
        state={speechState}
        onRecord={toggleRecording}
      />
      <RepoList repos={repositories} />
    </div>
  );
};

Now you can go ahead and try talking to the app and see what you get back in the transcript. Congratulations, you've just integrated Speechly into the app.

However, we still need to implement the filtering functionality, so let's go ahead and update our Speechly app configuration to support that.

4. Configuring Speechly app

Now that we've integrated the API into the app it's time to make our Speechly app useful. Let's add a couple of simple commands for manipulating the data we see in the table:

A command to filter by programming language, e.g. when a user says "Show me TypeScript repos" the app will only show repos with that specific language
A command to sort the results in a specific order, e.g. "Sort the results by forks" will sort the repos by the amount of forks it has.
A command to reset the filters, e.g. "Reset the filters to default" will remove the language filter and reset the sorting to some default.

Let's go back to Speechly dashboard and update the configuration of our app with the following:

# Which languages we can filter by
languages = [
  Go
  TypeScript
  Python
]

# Which fields we can sort by
sort_fields = [
  name
  description
  language
  followers
  stars
  forks
]

# Synonyms for "repo"
results = [
  items
  results
  repos
  repositories
]

# A couple of commands for filtering.
#
# This will expand into e.g. following examples (not exhaustive):
# "Show all Go repos"
# "Show me only TypeScript repositories"
# "Show Python results"
# etc.
#
# Words in curly brackets ("{me}") are optional.
# Square brackets are for lists (e.g. one option from the list may be used)
*filter show {me} {[all | only]} $languages(language) {$results}
*filter filter {$results} by $languages(language) {language}

# A command for sorting, e.g.:
# "Sort the repos by name"
# "Order results by forks"
# etc.
*sort [sort | order] {the} {$results} by $sort_fields(sort_field)

# A command for resetting the filters, e.g.:
# "Reset all filters to default"
# "Remove the filters"
# "Reset to default"
# etc.
*reset [reset | remove] {[the | all]} {filters} {to default}

Don't forget to add sort, filter and reset as intents and languages and sort_fields as entities!

As you can see from the comments, this configuration will make our Speechly app understand the commands we need and properly detect entities and intents. Keep in mind, that the cool part is that the model will also be able to understand the variations of commands that are not explicitly defined in our configuration. The same also applies to entities - the app won't be limited to only detecting "Go", "TypeScript" and "Python" as options for the language, but other words as well, which will be roughly in the same place in a pharse (e.g. you could try saying "Show me all Javascript repos"). However, with words that are very specific to domain like programming language names it's always a good idea to list them all in your configuration, otherwise they might be mistaken for some regular words, e.g. the API might not properly detect "Rust" as a programming language if you say "Show me all Rust repositories", because it would think that you meant "rust" as that thing that destroys metals. You can read more about how to configure Speechly applications in our documentation.

Once you've deployed your new version of the Speechly app, let's continue to parsing the results.

5. Parsing intents and entities

Now that we've trained a version of Speechly app with proper entities and intents, let's parse the results. First let's add our parsing logic to src/parser.ts:

import { SpeechSegment } from "@speechly/react-client";

export enum IntentType {
  Unknown = "unknown",
  Sort = "sort",
  Filter = "filter",
  Reset = "reset",
}

export enum EntityType {
  Language = "language",
  SortField = "sort_field",
}

export enum SortEntityType {
  Unknown = "unknown",
  Name = "name",
  Description = "description",
  Language = "language",
  Followers = "followers",
  Stars = "stars",
  Forks = "forks",
}

const SpeechIntentValues = Object.values(IntentType) as string[];
const SortTypeValues = Object.values(SortEntityType) as string[];

export function parseIntent(segment: SpeechSegment): IntentType {
  const { intent } = segment;

  if (SpeechIntentValues.includes(intent.intent)) {
    return intent.intent as IntentType;
  }

  return IntentType.Unknown;
}

export function parseLanguageEntity(segment: SpeechSegment): string[] {
  const langs: string[] = [];

  for (const e of segment.entities) {
    if (e.type === EntityType.Language) {
      langs.push(e.value.toLowerCase());
    }
  }

  return langs;
}

export function parseSortEntity(segment: SpeechSegment): SortEntityType {
  let s = SortEntityType.Unknown;

  for (const e of segment.entities) {
    const val = e.value.toLowerCase();

    if (e.type === EntityType.SortField && SortTypeValues.includes(val)) {
      s = val as SortEntityType;
    }
  }

  return s;
}

Here we define a couple of functions to parse intents and different entity types from a SpeechSegment, which is returned by useSpeechContext. As you can see, the code is pretty straightforward, most of it is actually just listing which intents and entities we expect and defining them as enumerations, since it's always a good idea to check the results returned from API against a pre-defined list of allowed values to avoid bugs. Another good idea is to make sure we use consistent case (in this case by casting the results to lower case) to avoid false negatives when e.g. comparing STARS to stars.

Now that we have our code for parsing the results from a segment, time to use it. Let's update our SpeechApp and add some code that calls our parser:

import React, { useEffect } from "react";
import { SpeechSegment, useSpeechContext } from "@speechly/react-client";

import { repositories } from "./data";
import {
  IntentType,
  SortEntityType,
  parseIntent,
  parseLanguageEntity,
  parseSortEntity,
} from "./parser";

import { RepoList } from "./RepoList";
import { Microphone } from "./Microphone";

export const SpeechApp: React.FC = (): JSX.Element => {
  const { toggleRecording, speechState, segment } = useSpeechContext();

  useEffect(() => {
    if (segment === undefined) {
      return;
    }

    parseSegment(segment);
  }, [segment]);

  return (
    <div>
      <Microphone
        segment={segment}
        state={speechState}
        onRecord={toggleRecording}
      />
      <RepoList repos={repositories} />
    </div>
  );
};

function parseSegment(segment: SpeechSegment) {
  const intent = parseIntent(segment);

  switch (intent) {
    case IntentType.Filter:
      const languages = parseLanguageEntity(segment);
      console.log("Filtering by languages", languages);
      break;
    case IntentType.Sort:
      const sortBy = parseSortEntity(segment);
      if (sortBy !== SortEntityType.Unknown) {
        console.log("Sorting by field", sortBy);
      }
      break;
    case IntentType.Reset:
      console.log("Resetting the filters");
      break;
  }
}

Here we define a parseSegment function that is called every time a segment changes by using React's useEffect hook. Since segment might come as undefined (this happens after the the user stops speaking and the API sends it's final response), we want to check for that before trying to parse it. The function checks for the intent and then calls the appropriate entity parser (or no entity parser at all if the intent was to reset the filters). For now we are just going to log the results of the parser, but to use them we'll have to add some filters. Let's continue with that!

6. Adding and applying filters

In order to apply filters, we'd need to implement some filtering logic, so let's do just that and add it as src/filter.ts:

import { Repository } from "./data";
import { SortEntityType } from "./parser";

export type Filter = {
  languages?: string[];
  sortBy?: SortEntityType;
};

export function filterRepos(
  input: Repository[],
  filters: Filter
): Repository[] {
  let output = input;

  const languages = filters.languages ?? [];
  if (languages.length > 0) {
    output = input.filter((repo) =>
      languages.includes(repo.language.toLowerCase())
    );
  }

  if (filters.sortBy === undefined) {
    return output;
  }

  return output.sort((left, right) => {
    switch (filters.sortBy) {
      case SortEntityType.Name:
        return left.name.localeCompare(right.name);
      case SortEntityType.Description:
        return left.description.localeCompare(right.description);
      case SortEntityType.Language:
        return left.language.localeCompare(right.language);
      case SortEntityType.Followers:
        return compareNumber(left.followers, right.followers);
      case SortEntityType.Stars:
        return compareNumber(left.stars, right.stars);
      case SortEntityType.Forks:
        return compareNumber(left.forks, right.forks);
    }

    return 0;
  });
}

function compareNumber(left: number, right: number) {
  if (left < right) {
    return -1;
  }

  if (left > right) {
    return 1;
  }

  return 0;
}

Now we need to call the filtering function when we get new results from the API, so let's also update our SpeechApp to do that:

import React, { useEffect, useState } from "react";
import { SpeechSegment, useSpeechContext } from "@speechly/react-client";

import { repositories, Repository } from "./data";
import { Filter, filterRepos } from "./filter";
import {
  IntentType,
  SortEntityType,
  parseIntent,
  parseLanguageEntity,
  parseSortEntity,
} from "./parser";

import { RepoList } from "./RepoList";
import { Microphone } from "./Microphone";

export const SpeechApp: React.FC = (): JSX.Element => {
  const [filter, setFilter] = useState<Filter>(defaultFilter);
  const [repos, setRepos] = useState<Repository[]>(repositories);

  const { toggleRecording, speechState, segment } = useSpeechContext();

  useEffect(() => {
    if (segment === undefined) {
      return;
    }

    const nextFilter = {
      ...filter,
      ...parseSegment(segment),
    };

    setFilter(nextFilter);
    setRepos(filterRepos(repositories, nextFilter));
    // eslint-disable-next-line react-hooks/exhaustive-deps
  }, [segment]);

  return (
    <div>
      <Microphone
        segment={segment}
        state={speechState}
        onRecord={toggleRecording}
      />
      <RepoList repos={repos} />
    </div>
  );
};

const emptyFilter: Filter = {};
const defaultFilter: Filter = {
  languages: [],
  sortBy: SortEntityType.Name,
};

function parseSegment(segment: SpeechSegment): Filter {
  const intent = parseIntent(segment);

  switch (intent) {
    case IntentType.Filter:
      const languages = parseLanguageEntity(segment);

      if (languages.length === 0) {
        return emptyFilter;
      }

      return {
        languages,
      };
    case IntentType.Sort:
      const sortBy = parseSortEntity(segment);
      if (sortBy !== SortEntityType.Unknown) {
        return {
          sortBy,
        };
      }

      return emptyFilter;
    case IntentType.Reset:
      return defaultFilter;
    default:
      return emptyFilter;
  }
}

Here we use React's useState hook to create a couple of stateful variables for storing filtered results and last filters (since you can append them by saying "Show me all Go repos" first and then following up with "Sort by start"). Every time we get new state of segment from the API, we call our parseSegment to parse the filters from it and then append those filters to the ones we've saved in the state. Then we also apply new filters to the list of repositories before passing them on to rendering.

Conclusion

And that's it! Now you can go ahead and try out your app - you can filter the repos by language, apply some sorting order and reset the filters.

If you want to delve into the details, go ahead and check out our documentation and our public GitHub.

What are voice user interfaces and how you can start adding them to your application?

Ottomatias Peura — Tue, 08 Sep 2020 06:47:24 +0000

As long as general artificial intelligence doesn't exist (and it won't happen soon), any computer system needs a way of interacting with its users. With the user interface the user gives commands to the computer system.

If you want an application to calculate what is the sum of two arbitrary numbers, you will need a way to get these numbers from the user to your application. A typical calculator solves this problem by presenting a user with digits from 0 to 9 either in physical form or as touchable and clickable buttons.

User interfaces can be either good or bad. Good user interfaces are the ones that you don't think that much and the bad ones – well, they are the ones that make your brains melt when you even think of them. They are the ones that even if you've used them a million times, you still don't know how they work and why they work like this. In my experience, every system used for invoicing is among these.

Good user interfaces are intuitive, concise, simple, and aesthetically pleasing. They give enough feedback to the user, tolerate errors such as misclicks or typos, and use familiar elements so that they are easy to use even for new users.

Voice user interfaces (VUI) are simply user interfaces that use voice or speech as the primary means for interacting with the system. Instead of touching and swiping through menus to find the action the user wants to do, the user can say what they want to do. Typical examples of voice user interfaces are voice assistants such as Google Assistant, Alexa, or Siri.

Voice user interfaces are typically highly intuitive and don't require a lot of training to use. For example, if you know that you are interacting with an alarm clock and you know it has a voice user interface, you can be pretty confident commands such as "set an alarm for 8am" works as you'd suppose it works.

In addition to being intuitive, the benefits of a voice user interface include hands-free action and the ability to use them from a distance. Voice user interfaces can also convey emotion in a way text-based interfaces can't.

The first applications of voice user interfaces were interactive voice response (IVR) systems that came into existence already back in the 80s. These were systems that understood simple commands through a telephone call and were used to improve efficiency in call centers. Last time I encountered one was a customer support call center for home appliances where a synthesized voice asked me to say the name of the manufacturer of the device I needed help with. Once it recognized correctly my utterance "Siemens", it directed me to a correct person.

Current voice user interfaces can be a lot smarter and can understand complex sentences and even combinations of them. For example, Google Assistant is perfectly fine with something like "Turn off the living room light and turn on the kitchen light".

While user interfaces that use voice are intuitive once you know what they are supposed to do, typical problems with voice user interfaces are to do with exploration. Let's say you find yourself closed in a small room that has a mirror on one side and seemingly similar metal panels on every other wall. That might be a cause for panic – or it's a voice-enabled elevator.

If the user doesn't really know what the system is supposed to do as is the case with almost every new mobile application you install, the best way to learn it is by clicking through menus and dialogs (or by reading the documentation!). While you do it, you may find buttons that you were not expecting to find but that might seem interesting or helpful. Then you try them out and hopefully learn how the system works.

A system that only has a voice user interface (for example smart speakers), on the other hand, doesn't support this kind of usage. You can only try saying things and it might or might not do what you were expecting. You can try saying "can you reserve me a table in a seafood restaurant" for your smart speaker, but the answer is either along the lines of "I don't understand" or in the best case something like "You'll need to login to do that. Please open application XYZ on your mobile phone and set up voice."

And even if that worked, it doesn't necessarily mean that the smart speaker can also get you an Uber or flight tickets. You'll have to try each of these individually and learn. If you are building a voice user interface to your application, you should really think about how you support feature discoverability in your application.

The second issue with voice-only user interfaces is slowness. Let's say you want to know what happened in NBA in the last round. Saying out loud results for each of the 15 games will take some time. While voice is the most natural way for us humans to interact, it's nowhere near the fastest way of consuming information. On the other hand, voice is faster than typing when inputting information.

Voice user interfaces should support multi-modality. Multi-modality means that a user interface supports more than one user interaction modality, for example, vision, touch, and voice. Now getting those NBA results will become a lot nicer: say "show me the latest NBA results" give you a screen that shows the results. It's easy to skim them through, a sudden loud sound from distance doesn't break your experience and you don't have to wait until you hear the result for your favorite team.

While adding a voice user interface can improve the user experience, it's not a silver bullet that'll turn a bad application into a good one. Just like any user interface, it needs designing and thinking through.

We have successfully solved issues with voice user interfaces with many of our customers. If you are interested in building voice user interfaces, sign up to our Dashboard or send us a message through the Intercom widget on the bottom left!

Why you have not added voice functionalities to your app?

Ottomatias Peura — Wed, 26 Aug 2020 10:40:42 +0000

Hi! I'm Ottomatias from Speechly We are building developer tools that enable developers to add voice user interfaces to their apps.

I'd be interested to hear from you why you haven't already added voice to your app? Have you tried that but ran into some issues? Don't you think it's beneficial in your use case?

Would be great to hear your thoughts! Tell me what you are building, too and on which platform.