Forem: Erik Hoffman

Contextual metadata just in time

Erik Hoffman — Tue, 02 Mar 2021 06:37:53 +0000

One of the biggest hurdles when building enhanced experiences on top of video streaming has historically been to deliver the data needed in the correct time for it to show up. You either rely on the end user's clock to be correctly set up, or you try to deliver something from your server, ending up not knowing how far behind the live edge your user is due to buffering and other circumstances.

Example

Let's bring up the example of showing the ongoing program on a linear channel.
Historically you would have some kind of API delivering the metadata on the side, probably in an EPG format such as XMLTV, causing you as the client consuming the video and the EPG to try to keep the timed sync between those. Match the current position of the user with the current program in the EPG according to the start and end times of the programs.

The Problem

This creates the need for doing the same implementation in all your clients. Having the correct client clock in sync on every end user's device to match correctly against the EPG. Handle the timestamps correct whether it is UTC or with any timezone applied. You probably don't want to check against the EPG data at every tick either, leading you to implement some logic to check either every X seconds, which would cause a mismatch for some time during the stream or some logic to check once again when the ongoing program is ending. Yet another risk area to implement accurately on all clients. Yet another function dependent on the end user's device clock to be correct.

The Solution

A solution to this "sidecar" solution, is of course to deliver your metadata inside your manifests as timed metadata, in context with your content. This is possible both with MPEG Dash as well as HLS. For this example, we will go through HLS.
In HLS you are expected to deliver the metadata in your manifest through the tag EXT-X-DATERANGE, on which you then probably want to apply a list of keys and their values.
An example in the context of this problem described earlier, would be to deliver your program and its start and end time as well as the title.

#EXT-X-DATERANGE:TITLE=“Lorem Ipsum Dolor Site Amet”, START-DATE=“2021-03-02T11:00:00Z”, END-DATE=“2021-03-02T12:00:00Z”, DURATION=3600

Playing this stream in a native HLS player in Safari you would continuously get all this metadata applied as a metadata track to the video element, nicely split up on their keys and values, simple to read and act on just in time.

videoElement.textTracks.addEventListener("addtrack", (evt) => {
  if (evt.track.kind === "metadata") {
    evt.track.mode = "hidden";
    evt.track.addEventListener("cuechange", (evt) => {
      const cues = evt.target.activeCues;
      for (let i = 0; i < cues.length; i++) {
        if (!cue[i] || !cue[i].value) return;
        const cueObject = cue[i].value;
        // act on your data
        console.log(`${cueObject.key}: ${cueObject.data}`);
      }
    });
  }
});

which would print

TITLE: Lorem Ipsum Dolor Site Amet
START-DATE: 2021-03-02T11:00:00Z
END-DATE: 2021-03-02T12:00:00Z
DURATION: 3600

Simple and nice to act further on.

Looking for a solution in the other browsers not supporting HLS natively, we're looking towards the common MSE player hls.js which do as well expose this metadata in a fairly simple and reachable event, though not as structured data-wise.

hls.on(Hls.Events.FRAG_CHANGED, (evt, data) => {
  const tags = data.frag.tagList;
  tags.forEach((tag) => {
    if (
      Array.isArray(tag) &&
      tag.length > 1 &&
      tag[0] === "EXT-X-DATERANGE"
    ) {
      // tag[1] will include our entire metadata string, titles and values all together
      const data = tag[1].split(",");
      if (!data || !Array.isArray(data)) return;
      for (let i = 0; i < data.length; i++) {
        const dataPair = data[i].split("=");
        // act on your data
        console.log(`${dataPair[0]}: ${dataPair[1]}`);
      }
    }
  });
});

which will end up in the same output as the earlier example. As shown you will get pretty obvious pairs of the keys and values to act upon and you might from there dispatch events for the rest of your application to act upon, whether it is to show the metadata in the skin or to handle some logic.

Conclusion

As this data will appear in the stream just in time, in the context, of the stream content which it relates to - you will have the possibility to act on all your clients in the correct time without having to rely on the device clock nor having the need of doing any implementations concerning dates and timestamps to request the correct data at the correct moment. You will always only get the data needed, no need to search through an EPG or any other list of data for the correct object.

The insertion of metadata is supported on all the major platforms and if you are rather building your own virtual channel, which you might do through our open source channel engine library, we do have support for adding metadata in the vodtolive library through the method addMetadata.

If you need assistance in the development and implementation of this our team of video developers are happy to help out. If you have any questions or comments just drop a line in the comments section to this post.

HTML Video Events done right

Erik Hoffman — Mon, 30 Nov 2020 07:03:03 +0000

Since the move to HTML from Flash a few years back there is one single discussion that tends to appear in each and every player and analytics implementation project there is - the strange event flow that you get from an HTML video player.

Though it’s of course amazing to have it standardised and you more or less can do even all MSE player tracking implementation based on these, without the need of player engine specific APIs which rather may enhance on top of it, there are quite a few questions and issues that appears.

Why is there not a Buffering event?
Why can there be pause events triggered during seek?
How to differ a resume from paused vs an initial play event?

The events that are triggered may make sense from a purely technical perspective, though the analytics that you want to sample should most often sample the experience the end user haves, and the behaviour that they brings.

Solution

The solution as such has always been fairly simple, though repetitive. You have to filter the events, putting them into the context of the current state of the player.

To avoid this repetetive task, we've built a small JavaScript library, Video Event Filter, which applies this kind of filtering on top of the video element events.

So instead of the technical approach, that the standard defines, as mentioned earlier - we do now filter and expose events according to what you want to get into your insights, according to our experiences.

The events that are exposed is now rather these

loading
loaded, video have loaded, but not started
play, video have started to play
pause
resume, video have started to play after a pause
seeking
seeked, video is done seeking. Continue in the state that existed before.
buffering
buffered, video is done buffering. Continue in the state that existed before.
timeupdate
ended
error

As you can see, and as mentioned we do now have proper terminology for when the player is loaded, we will trigger resume rather then play when the user continues after a pause. We do of course have proper buffering and buffered events.

Implementation

As we do only filter events on the video element, the implementation is as simple as if you would listen to the video element directly.

  import { VideoEventFilter } from "@eyevinn/video-event-filter";

  const videoElement = document.querySelector("video");

  const videoEventFilter = new VideoEventFilter(videoElement);
  videoEventFilter.addEventListener("*", (event, data) => {
    console.log("EVENT:", event);
  });

The events available, as mentioned above, are exposed in a variable PlayerEvents for simpler implementation.

Conclusion

This means that we can now track a playback session in a way that reflects the experience from the end user perspective (when it comes to buffering ratio etc) as well as the actual behavior of the user - rather than what the player engine technically do (which of course may be valuable in other contexts).

As mentioned this is a quite small library and a fairly simple task, though repetetive. Abstracting this into a library means that we can focus on building value, we can point at the issues there is with analytics out of the box and not least iterate our opinion based filtering centralized over time and evolve how a playback session should be measured.!

To build the Consuo component ecosystem

Erik Hoffman — Tue, 26 May 2020 04:54:46 +0000

At Eyevinn we’ve always valued platform and product independence as well as taking a neutral stand when it comes to languages and frameworks. Therefore, when building components for Consuo, our vod-to-live stitching product, it was a quite obvious choice for us to not only build a React component, which may have been the simplest decision, but also a web component for vanilla JavaScript projects with no framework being used.

A lot of JavaScript developers these days thinks that you need a framework such as React or Vue to be able to create encapsulated components to be shareable. Though over the last few years standards as been moving quite fast and the web component standard as such give us all the features that we expects and wants out of modern JavaScript solution for building a component to act isolated as well as being shared in a simple way, both internally as well as externally (e.g. webcomponents.org).

Since before we already had an Up Next component for Consuo, built and shared as a React component. As soon as it had been shared internally I took the challenge to create an exact replica as an web component, as I as a developer loves standard vanilla, no framework, solutions.

Why do we need an ecosystem for Consuo?

Consuo as a product, or rather as a base platform together with its open source engine, is the core of a wider end user experience. We want to bring the laid back experience of classic linear tv into the new era of streaming, as well as bringing the unused potential back to the streaming companies backlog of clips and vod videos in terms of consumption and monetization.

Consuo as a core handles the scheduling as well as the stitching of the channel, creating the core experience. Though we want to help you evolve that experience by bringing more functionality to the table by building a base plate of open source components to be used. This starts with our schedule related components to show what’s currently on air and what’s next up.

Framework decisions and framework neutrality?

We started to build this component as a React component, as this is currently the framework of choice for our product web as well as demo solutions. But as mentioned earlier, we have a history and pride of being language and framework agnostic at Eyevinn. And we do really want this experience and these components to be available for everyone to implement and to use.

With the evolvement of the quite recent web component standard it was a simple decision to take, to develop such alternative as well. As this can be used in all frameworks, not isolated to a single one.

The state of web components

As mentioned earlier The state of web components in style JavaScript standard is quite exciting these days. The api as such might not be as simplified and fine tuned as in React or Polymer, but all the functionality as expected is there, and to avoid the overhead of an framework as well as creating a shareable component not dependent on the framework being used by the implementing part, is a quite exciting thing.

So what can we do?

Choose to be reachable by outer JavaScript or to entirely encapsulate the functionality.
Choose to be reachable by outer styling or to entirely encapsulate our styles.
Act on being attached, as well as detached, from a document. (See Life Cycle Callbacks)
Act on the attributes of the element being changed during runtime.

This means that we can choose whether we want to build an ecosystem by contributed functionality only, or by encapsulated visually designed components that matches our product’s graphical profile.

For this specific component we just wanted to deliver the functionality of a simple schedule visualizer and do not encapsulate any specified styling.

So let’s go to the functionality of the Up Next component as such and to compare a React component to what we can build in native JavaScript.

The feature set

The component as such solves a rather basic scenario which can be seen as a base plate functionality in any linear channel experience - Showing what’s currently playing and what’s Up Next.

This is implemented by pulling the schedule from which the linear experience is created out of the video files that already exists in the archive.

An example of such schedule json

[
  {
    "channelId": "eyevinn",
    "assetId": "urn:uuid:da80b21b-2e6e-42ae-82b8-3b1b5581b59a",
    "eventId": "735fecc8-cade-410d-993f-9860e4de9efe",
    "id": "urn:uuid:da80b21b-2e6e-42ae-82b8-3b1b5581b59a",
    "title": "TV Plus Joachim",
    "start_time": 1590226668810,
    "end_time": 1590226741810,
    "start": "2020-05-23T09:37:48.810Z",
    "end": "2020-05-23T09:39:01.810Z",
    "uri": "http://lambda.eyevinn.technology/stitch/master.m3u8?payload=eyJ1cmkiOiJodHRwczovL21haXR2LXZvZC5sYWIuZXlldmlubi50ZWNobm9sb2d5L3R2cGx1cy1hZC1qb2FjaGltLm1vdi9tYXN0ZXIubTN1OCIsImJyZWFrcyI6W3sicG9zIjowLCJkdXJhdGlvbiI6MTYwMDAsInVybCI6Imh0dHBzOi8vbWFpdHYtdm9kLmxhYi5leWV2aW5uLnRlY2hub2xvZ3kvYWRzL3NmZi0xNXMubXA0L21hc3Rlci5tM3U4In1dfQ==",
    "duration": 73
  },
  {
    "channelId": "eyevinn",
    "assetId": "urn:uuid:b8ff551a-6da3-485a-8a53-b11c5d28753f",
    "eventId": "c25f531a-3bf5-4645-b3f0-cbaf4c7f459d",
    "id": "urn:uuid:b8ff551a-6da3-485a-8a53-b11c5d28753f",
    "title": "TV Plus Johanna",
    "start_time": 1590226741810,
    "end_time": 1590226816810,
    "start": "2020-05-23T09:39:01.810Z",
    "end": "2020-05-23T09:40:16.810Z",
    "uri": "http://lambda.eyevinn.technology/stitch/master.m3u8?payload=eyJ1cmkiOiJodHRwczovL21haXR2LXZvZC5sYWIuZXlldmlubi50ZWNobm9sb2d5L3R2cGx1cy1hZC1qb2hhbm5hLm1vdi9tYXN0ZXIubTN1OCIsImJyZWFrcyI6W3sicG9zIjowLCJkdXJhdGlvbiI6MTYwMDAsInVybCI6Imh0dHBzOi8vbWFpdHYtdm9kLmxhYi5leWV2aW5uLnRlY2hub2xvZ3kvYWRzLzZjZDdkNzY4LWUyMTQtNGViYy05ZjE0LTdlZDg5NzEwMTE1ZS5tcDQvbWFzdGVyLm0zdTgifV19",
    "duration": 75
  }
]

React component vs Web Component

Both these components are built in quite a similar way as the feature set is quite similar, as mentioned earlier on in this post. So the things that really differs is the markup as such, the state handling and the rendering language implementation.

Life cycle handling

Life cycles in React and Web Components exposes the same possibilities, though with different terminology and minor limitations.

A simple table of the most common life cycle events can be shown as this

Event	React Component	Web Component
Added to DOM	componentDidMount	connectedCallback
Rendering	render	-
Attributes being changed	-	attributeChangedCallback
Removed from DOM	componentWillUnmount	disconnectedCallback

As seen the main structure is the same between the solutions, though the missing part doesn't mean that the functionality is missed as such.

React handles the changed attributes by simply changing the values on the props of the component, and re-render the component with those values in mind. In the web component you will have to trigger the render function yourself, both when added to DOM (the connectedCallback event) as well when you get the event for changed attribute values.

Rendering

The other major difference is how the rendering is handled. In React you render your inner html with a markup language called JSX. This gives us the ability to get great autocompletion in your code editor as well as evaluating expressions and values of properties to take visualisation decision based on those. For Web Components on the other hand, we will rather create the inner html elements in JavaScript and append them as childs or, as I prefer to do, write the html as such in a template literal. Even though this doesn't bring the excellent editor functionality as JSX might give us, we will have the same ability to handle expressions and property values in the rendition.

React Example

Web Component Example

Conclusion

To take the extra step and deliver a cross framework component is, as seen, quite a small job - bringing not only a value to the ecosystem that we want to build, but also letting the customer utilise our knowledge and development undependent on their framework choices.

The components can be downloaded through npm (React component and web component) and the source code can be found on our Github (React component and web component). Please feel free to contribute, either to the libraries and components that already exists or to the ecosystem around the Channel engine and Consuo by building open source functionality around them.

Act on storage events

Erik Hoffman — Tue, 25 Feb 2020 06:58:32 +0000

I'm a big fan of writing event driven applications both in intra app communication as well as cross app communication, like message queues, job queues or sockets.

One area which, when you look at the examples out there, seems to have been fallen behind when it comes to this is to act on data storage and the events there might be from the database engine.

The classic case in the greater database engines out there, like MongoDB, is to listen for updates on a specific table or document storage. But I want a more dynamic solution to solve more use cases.

Document Database Scenario

Quite a few years ago now, there came a new star in this market when it comes to storage for realtime applications, RethinkDB. It's a document database, or rather "json database" as they call it, which is built around the concept of listening to queries and get updates pushed to you rather then polling for new data.

Yes, that's correct - you get new data pushed not only on table level, but on each and every query, no matter how detailed it may be.

You can start by using it as an ordinary database, doing a query by their simple query language - ReQL. When you've got your query right, you'll just add .changes() to start listening for changes on that specific query.

For example if you think of a chat application where messages as posted into a database table, separated by their room id, room_id.

const ROOM_ID = "#myChatRoom";

r.table("messages").filter(
  r.row("room_id").eq(ROOM_ID)
).changes().run(conn, callback);

This would give us a callback, calling the defined callback method, each time a new message appears in the message table with our room id defined.

Going one step further - this do not only apply to new data being added to the query of your choice, but also a lot of other events, i.e. you can build whatever you want on whatever event that you want. What a world of possibilities!

Key value Database Scenario, the simpler one

In a lot of scenarios you might not need a document database, but you still want to act on data changes. It might be when things are set or get, but maybe even more likely on expiration an deletions.

Redis is one of the most well known key value databases out there. It is also quite well known that they have a PubSub solution, i.e. you can use Redis as an message queue. Though what is less known is that they do have an event subscriber which isn't related, that is used to listen for changes on your ordinary key storage. They call this Keyspace Notifications. It is quite a threshold to understand and get started with, so I will walk you through it.

Configuration

Step one in to configure your Redis instance to send these push events, as this is off by default due to performance optimization.

There are quite a few keys that can be set but in this example we will set KEA, i.e. all keyspace and keyevent events.

This can be done either through the cli $ redis-cli config set notify-keyspace-events KEA or by your application client that integrates with redis. Here is an example in JavaScript.

const redis = require("redis");
const redisSubscriber = redis.createClient();
redisSubscriber.config("SET", "notify-keyspace-events", "KEA");

When Redis is configured, you have to define the database you want to subscribe (a Redis has 16 databases, where the default one is 0) specific kind of event that you want to key notifications for, or if you want to get it for all of them (then subscribe to *).

const database = 0;
const event = "set";
redisSubscriber.subscribe(`__keyevent@${database}__:${event}`);

Event listener

When those are set up, you're ready to start listen to events.

First you connect to the subscription and then you start listening for messages.

redisSubscriber.on("subscribe", channel => {
  console.log(`subscribed to event of type ${channel}`);
  redisSubscriber.on("message", (channel, key) => {
    console.log(`Event type: ${channel}`);
    console.log(`KEY: ${key}`);

    // Here you can go and fetch data on that key 
    // or take whatever action that you might want to.
  });
});

Use cases

As seen for the document database there are endless possibilities in creating realtime application with these kinds of event - but for the key value storage my main usage of subscriptions has been to work with TTL and expiration of keys, taking actions on expiring cache etc.

The concept though is the same - why should you long poll for data from your database, when you wouldn't want to implement long polling between your clients and your server?

What use cases do you see for these kind of subscriptions?

Cache for performance, not for offloading

Erik Hoffman — Sun, 23 Feb 2020 12:40:05 +0000

Often when you talk about caching it's focused on offloading the origin by letting most of the request to hit a CDN or get some cached data delivered to avoid processing too often. Though in my mind there is more to it.

I recently had a project where the caching would to be implemented purely on performance focus, and not by any mean based on offloading needs.
The processing of each request was far too heavy to get the response times down to a acceptable level.

Of course with a classic CDN cache you will get the response times low and you will get an incredible performance. Until the cache times out. Each X minutes there will be a single request that will take the performance hit to generate new data to cache.

How should we solve this?

There are of course a few solutions out there to solve this, by keeping the cache up to date and hot without the need of an end user request to go through and hit the origin generating data.

One solution could be to have a separate worker, which generates data to the cache with a set interval, never having an expiration on the cache but rather updating it. This is, I would say, an optimal solution since you will update the cache without ever letting someone through neither taking the performane hit of generating the new data to often on the server.
Though one thing with this solution is that it can be quite architectural heavy for a small application, since you need to have an external worker as well as a CDN integration where you activaly can update your data etc.

My take

My take on solving this for smaller application is not as light on the origin but nearly as good as the solution above on performance, and you will never let the cache expire.

In my examples I'm using Fastify but it could rather be any route handler there is. I am also using node-cache for in memory cache, but that can as well be switched to any cache solution that you might want to use, like Redis or Memcached.

The main idea is to always respond with the cached data, but even then letting the application work even after responding the request, being able to fetch new data and put into cache - to be updated for the next request to come in.

const NodeCache = require("node-cache");
const storage = new NodeCache({
  stdTTL: 3600,
  checkperiod: 120,
  useClones: false
});

fastify.get("/data", async (req, res) => {
  // Using our unique url as key, i.e. query params etc will differ the key
  const key = req.req.originalUrl;
  // If already in cache, send the cached data as response
  const data = storage.get(key);
  if (data) {
    res
      .header("Cache-Control", "public, max-age=300")
      .code(200)
      .send(data);
  }
  // Please notice that we do not trigger a return, but let the execution go on
  // Here we would typically generate some data
  const newData = "Some new data";
  // Updating the cache
  storage.set(key, newData);
  // This response will only be sent the first time, since fastify have already sent a response and closed connection
  res
    .header("Cache-Control", "public, max-age=300")
    .code(200)
    .send(response);
});

So the first time, when no cache exists, it will go thorugh to the end setting new data into the cache, but also responding with that data. Each other time it will respond with the cached data, then keep execute and update the cache with the new data.

This way we will always get the performance of having data cached, though also having new data exposed all the time as we constantly updates the cache with new data.
The only way we offload is to have a cdn in front of this which will not be updated in this scenario.

Conclusion

You can either cache for performance on your server of for the performance of the end user. Far too often the focus is on the first one and not the latter.
My example is a great way to combine these by letting the cdn being the offload for your server, but when the requests gets through you still have an in memory cache in place for the performance enjoyment of your users.

How do you work with performance in your applications and APIs?

Were am I? A Streaming Service Experience

Erik Hoffman — Tue, 04 Feb 2020 20:36:23 +0000

As mentioned in the description this is yet another component in building a simple streaming service and its components, not having to buy everything on the market.

I have since before written about creating a simple recommendation engine

One of the basics that you would expect of all streaming services out there, though it isn't implemented everywhere just yet, would be a continue watching service to be able to continue at the position where you left watching the last time. Here we'll implement the simplest of such solution for anyone to implement.

What do we want to to?

A Continue Watching service can be very complex in showing reasons (new episode, left in the middle of an episode, new season etc) but we will start with an implementation built for the simplest needs - to keep watching where you left intra episode or movie.

For that to exist we would want the following implementations

Being able to post a position for a user on a specific asset.
Get a position for a user on a specific asset.
Get a list of the user's current saved positions, to show a Continue Watching carousel. Preferable in order.

Simplicity

Though you can of course implement this in the most complex ways of having data structure and metadata objects in infinity, stored in complex databases or graphs.

My ambition is rather to get both the architecture, as well as the application speed, as thin and light as possible.

I therefore choose to implement the storage in a simple key value store, in this case Redis.

Simplest implementation

When thinking about a continue watching service, it's essentially a bookmark in an asset for the specific user visiting the site, trying to keep on where he or she left. So which data do we need?

A user watches an asset, until a specific position.

So for this to be stored in a key value storage, what are the unique part of this? What should be the key? In my optinion the best solution is to create a "unique" identifier of the userId of the user consuming the video and the assetId for the asset watching, i.e. userId:assetId, storing the position as value.

So first iteration is to use the SET and GET methods in redis for simply setting the value and later fetching the value for that specific asset only.

const util = require('util');
const redisClient = redis.createClient(6379, "127.0.0.1");
// Using util.promisify from the native node library to avoid callback hell
const set = util.promisify(redisClient.set);
const get = util.promisify(redisClient.get);

// This is the unique identifier which we create out of user and asset
const KEY_ASSET = (userId, assetId) => `${userId}:${assetId}`;

const store = async (userId, assetId, position) => {
  if (!userId || !assetId || !position) return false;

  const success = await set(
    KEY_ASSET(userId, assetId),
    position
  );

  return success;
};

const fetch = async (userId, assetId) => {
  if (!userId || !assetId) return false;

  const position = await get(KEY_ASSET(userId, assetId));

  return position;
};

Next up - User experience

Though in most services you would want a list of which assets you have currently ongoing - a "continue watching carousel" - to not have to search and find the specific asset before knowing that you have a progress in it.

Propably the most scalable and performance efficient way of doing this would be to generate Redis lists or sets on input, rather than on data request. Though for this simple solution we choose to find all listings for the user and fetching them all.

For this we will use the Redis KEY command, where you can get all matching keys given a pattern which you send in - in this case this would be userId:*, fetching all keys for one single user. This does only provide us with a list of keys though, so for each key we will have to do the basic GET command to get their values.

const util = require('util');
const redisClient = redis.createClient(6379, "127.0.0.1");
const set = util.promisify(redisClient.set);
const get = util.promisify(redisClient.get);
// adding the keys function which we will use
const keys = util.promisify(redisClient.keys);

const KEY_ASSET = (userId, assetId) => `${userId}:${assetId}`;
// This is our wildcard pattern
const KEY_USER = userId => `*${userId}:*`;

const list = async userId => {
  if (!userId) return false;
  const userKeys = await keys(KEY_USER(userId));
  // If no matches we return an empty array
  if (!userKeys) return [];

  // when we fetched as many values as we have keys, we'll be happy
  const expectedLength = userKeys.length;
  let keepGoing = true;
  const continueWatchingList = [];
  while (keepGoing) {
    const key = userKeys.shift();
    const val = await get(key);
    // we generate an object for each, to be able to list out for the end user
    const item = { assetId: key.split(":")[1], position: val };
    continueWatchingList.push(item);
    if (continueWatchingList.length >= expectedLength) {
      keepGoing = false;
      return continueWatchingList;
    }
  }
};

But what about order?

Now we have progress saved, ability to fetch it by asset and even a list of such objects to show in a carousel. Though this list is rather unordered, not showing the most relevant one up front. What if we should order it by the newest saved progress at first, as well as expiration when you haven't continued in a set amount of time?

Let's try to implement it using Redis EXPIRE command to set a expiration date, and then the TTL command to get the remaining time until expiration. If we update this on each progress report, the one with the highest time remaingin should be the newest input? Right?

// we add the expire command to be able to set expiration
const expire = util.promisify(redisClient.expire);

// we set it to one year, so each stored value will be deleted after one year if no progress is being made
const ONE_YEAR = 1 * 60 * 60 * 24 * 365;

const store = async (userId, assetId, position) => {
  if (!userId || !assetId || !position) return false;

  const setSuccess = await set(
    KEY_ASSET(userId, assetId),
    position
  );

  // when stored, we set expiration
  const expirationSuccess = await expire(
    KEY_ASSET(userId, assetId),
    ONE_YEAR
  );

  return success && expirationSuccess;
};

// we add the ttl method to read out remaingin expiration
const ttl = util.promisify(redisClient.ttl);

const list = async userId => {
  if (!userId) return false;
  const userKeys = await keys(KEY_USER(userId));
  if (!userKeys) return [];

  const expectedLength = userKeys.length;
  let keepGoing = true;
  const continueWatchingList = [];
  while (keepGoing) {
    const key = userKeys.shift();
    const val = await get(key);
    // we'll fetch the expiration for each key
    const expiration = await ttl(key);
    // storing it in each object
    const item = { assetId: key.split(":")[1], position: val, expiration };
    continueWatchingList.push(item);
    if (continueWatchingList.length >= expectedLength) {
      keepGoing = false;
      // before returning the list, we sort it by remaingin time
      continueWatchingList.sort((a, b) => b.expiration - a.expiration);
      return continueWatchingList;
    }
  }
};

Conclusion

So now we have the ability to set, get and list assets in order for an end user to be exposed by his or hers progress in a streaming service.

For the entire project implemented as an API to clone, fork or be inspired by - please have a look over at our Github .
We love contributions.
https://github.com/Eyevinn/continue-watching-api

Personalize the Experience

Erik Hoffman — Fri, 13 Dec 2019 14:43:25 +0000

Originally posted on Medium https://medium.com/@eyevinntechnology/personalize-the-experience-37f991db2745

One of the major trends in video streaming services over the last few years has been to evolve the personalized experience of content being shown for each specific consumer. It can either be editorial recommendations for specified groups of users, or automatically generated lists of assets that you're supposed to like based on earlier consumption or behavior.

How hard can it be?

How hard can it be, you might ask yourself?
Well, ask yourself these questions

Do I like everything in a specific genre?
Do I like everything with a specific actor?
Do I like everything that my friends likes?

And to dive even deeper

Do I watch the same movies in the summer as during the holidays?
Do I watch the same movies when it's sunny outside as when it rains?
Do I watch the same kind of assets in the morning as in the evening?

It's hard, isn't it?

The expected ouput of a recommendation might, and probably will, vary depending on the context in which you are right now - and therefore we need to provide such recommendations.

Which solutions exists?

Meta data match

Looking at the recommendation solutions out there, the quality and complexity varies a lot. The most basic one which nearly all streaming services starts to implement is a basic meta data matching of the assets in the consumer's consumption history vs the rest of the catalog. I.e. if the consumer have watched an American Action movie with Brad Pitt, I will probably get a recommendation of other Action movies with Brad Pitt preferably.

But are we that simple as humans? Are the main actor that important, as we will choose the movie to watch depending on it?

Consumption match

Next step, following the meta data match recommendations, is often to match consumption history between users. The classic "Because you've watched X" carousel that a lot of streaming services shows.

From my point of view this is a far better approach, which will be even better over time as the dataset grows both when it comes to videos watched, but also consumers to compare against. If I have watched 10 movies that my friend has also watched, there's a big chance that we both will equally like a few more.

Though do we like everything that we've watched?

Ratings match

If we don't like everything we watch, how should we be able to compare your consumtion pattern? Maybe some kind of rating system which contains both positives, but maybe more important negatives as well?

Should we auto detect what we think is a negative? Is a bounce after just a few minutes a negative? Though it can be that you want to continue later on, and you might as well watch an entire movie which isn't that great but just time consumption?

Should we implement a ranking feature, being able to vote thumbs up or down depending on your experience of the movie? Or will the threshold be to high, as it requires action from the consumer.

My approach

Of course there is no holy grail here, as someone should've already implemented it by now. Though I think that you can implement a quite good solution based on aggregation of the ideas that we've been in touch with here.

What if we

Implement bounce within X minutes as a negative.
Removes this negative if the user continues to watch later on.
Implement a finished video as a positive.
Removes this positive if the rates it as negative.
Ability to rank thumbs up and down on the asset.

Finally we can as well add some filtering on the output of the recommendations - Should we really recommend typically christmas movies and series during the rest of the year?

We might as well add some bonus points if the metadata is matching my preferences. I.e. this will only change the order of the recommendations already there - not generate new ones.

Implementation

As the positives and negatives is the same, backend wise, even if triggered by different events - the implementation for this will be quite simple.

We want to be able to input a few events - like, dislike, unlike and undislike - i.e. set a rating and remove it later on if needed.

As our filtering will be quite simple, when it comes to metadata, we will apply this on the output generated from our recommendation engine.

Example

We at Eyevinn have choosed to build a simple example project managing these inputs and generating recommendations out of it by using the Jaccard coefficient to determine the similarity between users and k-nearest neighbors to create recommendations out of the user set generated.

It is a node module which you can install through npm and implement as the engine in your recommendation api as example as shown below implemented in Express routes.

Inputs

const eyeRecommender = require("@eyevinn/eye-recommender");
...

app.post("/vote/:userId/:action/:assetId", async (req, res) => {
  const userId = req.params.userId;
  const action = req.params.action;
  const assetId = req.params.assetId;

  switch(action) {
    case "like":
      await eyeRecommender.input.like(userId, assetId);
      break;
    case "dislike":
      await eyeRecommender.input.dislike(userId, assetId);
      break;
    case "unlike":
      // This one removes a set like
      // Typically if you've triggered a like on video end, but the user vote a dislike later on
      await eyeRecommender.input.unlike(userId, assetId);
      break;
    case "undislike":
      // This one removes a set dislike
      // Typically if you've triggered a dislike on bounce, but the user keeps watching or vote like.
      await eyeRecommender.input.undislike(userId, assetId);
      break;                  
  };
  res.set("Cache-Control", "public, no-cache");
  res.sendStatus(200);
});

Get recommendations and statistics

const eyeRecommender = require("@eyevinn/eye-recommender");
...

app.get("/recommendations/:userId", async (req, res) => {
  const userId = req.params.userId;
  // Get an array of recommendations in order from the most recommended to the least
  const recommendations = await eyeRecommender.statistics.recommendationsForUser(userId);

  // Here you would apply some filtering if you want to
  // Fetch assets by the asset id's, filter the metadata etc.

  res.send(recommendations);
});

Contributions

As we've built this purely on the aspect of evaluating the options there are, as well as looking into how to evolve the current solutions out there in a cheap scalable way - we've put this open source for you to use and contribute.

We would love engagement, opinions and contributions with code as well as ideas.

https://github.com/Eyevinn/eye-recommender

Generate tag suggestions from a text

Erik Hoffman — Thu, 17 Oct 2019 06:47:50 +0000

In most of the scenarios when you should publish a text or any other media, you need to set tags for visibility and for it to be found in search and browsing. To set this tags may not always be the simplest - not only by the editor itself, but in many scenarios the text should be published and tagged by another person than the one who was writing it.

What if we could help out with some suggestions for tags?

Theory

If we assume that the valuable words, which we want to tag the text with, are among the most used words in the text, except from prepositions and such smaller in between words.

Let's filter out the most used words from the text!

Get started

Let's say that we have a text - in this case I'll use my latest blog post Light and Safe with git hooks and partial testing and will refer to it in the example below as TEXT_INPUT.

First we want to get all words out of it, one by one, instead of being in sentences. Let's split on the RegExp \s metacharacter which will find all whitespaces in the text if so a space, new line, tab or any other whitespace there is.

const wordsArray = splitByWords(TEXT_INPUT);

function splitByWords(text) {
  return text.split(/\s+/);
}

The result will look something like this

[ 'Why?',
  'To',
  'write',
  'tests',
  'for',
  'your',
  'code,',
  'as',
  'well',
  'as',
  'following',
  'a',
  'code',
  'design',
  'pattern,',
  'is',
  'crucial',
  'parts',
  'of',
  'building',
  'a',
  'scalable',
  'and',
  'stable',
  'code',
  'base',
  'and',
  'deliver',
  'on',
  'the',
  'continuous',
  'integration,',
  'deployment',
  ... 500 more items ]

I.e. just a list of words, as promised.

Now let's count how many times each words exists in the text by iterating the array, adding each word as an object key with the number of presences as its value.

// As input we have the array created in the earlier code block
const wordsMap = createWordMap(wordsArray);

function createWordMap(wordsArray) {
  // This object will store the result during, and after, the iteration
  const wordsMap = {};
  // Let's iterate the array, sending in each word into the anonymous function
  wordsArray.forEach(function(key) {
    // If the word is already in the storing object, we'll add up on its presence number.
    // Else we just add it with its first presence, #1
    if (wordsMap.hasOwnProperty(key)) {
      wordsMap[key]++;
    } else {
      wordsMap[key] = 1;
    }
  });
  return wordsMap;
}

Now we have a giant object with all words, where they all have a count of presences. Something like this

{ 
  'Why?': 1,
  To: 2,
  write: 1,
  tests: 4,
  for: 6,
  your: 4,
  'code,': 1,
  as: 7,
  well: 2,
  following: 1,
  a: 11,
  code: 9,
  design: 1,
  'pattern,': 1,
  is: 8,
  crucial: 1,
  ...and more
}

Better but you still need to find the ones with most presence. let's start by filtering out smaller words, which quite often is prepositions and such - filtering in the same method as before.

// As input we have the array created in the earlier code block
const wordsMap = createWordMap(wordsArray);

function createWordMap(wordsArray) {
  const wordsMap = {};
  wordsArray.forEach(function(key) {
    // Let's start with handling different appearences of the same word, by normalizing them - removing commas, capitalizing etc
    key = key
      .trim()
      .toLowerCase()
      .replace(".", "")
      .replace(",", "")
      .replace("!", "");
    // Then filter by length to remove the short, often to be, prepositions
    if (key.length <= 5) return;
    // Then keep on as before
    if (wordsMap.hasOwnProperty(key)) {
      wordsMap[key]++;
    } else {
      wordsMap[key] = 1;
    }
  });
  return wordsMap;
}

The result out of this is a better list, like this

{
  safest: 1,
  implement: 1,
  should: 4,
  before: 1,
  commit: 5,
  broken: 2,
  integrated: 1,
  origin: 1,
  process: 1,
  struggling: 1,
  looking: 2,
  documentation: 1,
  fortunately: 1,
  community: 1,
  around: 1,
  javascript: 1,
  packages: 1,
  ...and more
}

Now let's sort them to have the most popular on top

// The unsorted list as input, wordsMap
const sortedWordsArray = sortByCount(wordsMap);

function sortByCount(wordsMap) {
  // This array will store our list as we'll now create an array of sorted objects
  var finalWordsArray = [];
  // Iterate all the keys in the word list object sent in, to map each key:alue row to an object in itself, to add to our array
  finalWordsArray = Object.keys(wordsMap).map(function(key) {
    return {
      name: key, // the word itself
      total: wordsMap[key] // the value
    };
  });

  // Now let's sort the array so the object with most appearances get in top
  finalWordsArray.sort(function(a, b) {
    return b.total - a.total;
  });

  return finalWordsArray;
}

The result will be something like this

[ 
  { name: 'lint-staged', total: 6 },
  { name: 'commit', total: 5 },
  { name: 'eslint', total: 5 },
  { name: '"hooks":', total: 4 },
  { name: '"pre-commit":', total: 4 },
  { name: '"husky":', total: 4 },
  { name: 'should', total: 4 },
  { name: 'install', total: 4 },
  { name: 'entire', total: 3 },
  { name: 'packagejson', total: 3 },
  ...and more
]

A lot more relevant!

What can we improve?

When filtering the array of words, it would of course be greatly improved if we could get some list word words to ignore, rather than doing the assumption that all short words should be removed. I still haven't found a reliable source for this though.

We could possibly using some kind of NLP, Natural language processing to find out, though that would lock ut down to use this only at english texts and not being language agnostic. It would also build on the complexity quite a lot.

This is part of the reason I would say that we generate suggestions rather than generating tags

We can't guarantee that the most used words are relevant
You might want to have broader tags, even if the generated ones are relevant (I added continuous integration and deployment for example) for the text content as such.

So what's the use case?

A text is quite readable and as the publisher it might be easy and relevant to read the text instead. But an applicable scenario might be to analyze a subtitle file for a video or transcription of a podcast - and out of that generate tags to make the media more searchable without watching or listening through the entire asset, making notes of tags to be set.

What can you see as a use case?
What do you think can be improved?

Light and Safe with git hooks and partial testing

Erik Hoffman — Wed, 16 Oct 2019 12:37:26 +0000

Why?

To write tests for your code, as well as following a code design pattern, is crucial parts of building a scalable and stable code base and deliver on the continuous integration, deployment and delivery promises that we all wants to reach.

The safest way to implement this should be before each commit or push, to avoid broken code to be integrated into the origin code base.

This is a process which may, at a first look, be quite struggling looking at the git hooks documentation

How?

Fortunately, with the great community around JavaScript and NPM, there are some packages that will greatly simplify the workflow for you to integrate this into your project.

Except from the testing and linting, which in this example is being made with Jest and ESLint, we will go through Husky for git hooks and lint-staged for partial affect.

Hook up Git events

As mentioned we're using Husky to simplify the git hooks implementation. First install it by running npm install --save-dev husky.

In your package.json set up the event on which you want to act - in our case the pre-commit event - by adding

  "husky": {
    "hooks": {
      "pre-commit": ""
    }
  }

Now let's find out what to do with this hook.

We probably want to run our tests and linting to be sure that we don't commit any broken code into the repository.

Accordning to our weapons of choice we'll run jest && eslint.

  "husky": {
    "hooks": {
      "pre-commit": "jest && eslint"
    }
  }

The problem now is that each time we do any minor change, the pre-commit hook will run our entire test suite and validate the code standards through our entire code base. Not that effective as it in a big project may take up to a minute, if not more.

Partial affect

Lucky us - both Jest and ESLint, with a little help from lint-staged, has functionality to only run the code in context of our commit.

First, install lint-staged by running npm install --save-dev lint-staged.

Then, in the package.json file once again, we should define what actions that should be triggered towards which parts of the code base.

  "lint-staged": {
    "src/**/*.js": "eslint && jest"
  }

To run this instead of ESLint and Jest directly on the hook, let's change our git hook

  "husky": {
    "hooks": {
      "pre-commit": "lint-staged"
    }
  }

Lint-staged will by itself send the staged files into ESLint, though for Jest it isn't the files themselves that should be sent into it and therefore no viable action is taken and it still test the entire suite.

Well, Jest have a few options parameters that may be suitable for our situation and in this scenario we will use --findRelatedTests which takes a list of files as argument, which is exactly what lint-staged is giving it.

So calling Jest --findRelatedTests will run the specific tests related to the files which we are committing in that actual commit.

This gives us the end result looking like this in our package.json

  "husky": {
    "hooks": {
      "pre-commit": "lint-staged"
    }
  },
  "lint-staged": {
    "src/**/*.js": "eslint && jest --findRelatedTests"
  }

And it will lint only the files being committed, as well as running the tests related to those files. This will be just a minor suite being run each time we commit and is therefore not such burden as a big heavy suite might be, affecting your productivity.

Happy days!