Forem: Rishit Bansal

Launching Dyte’s Live streaming SDK on Product Hunt

Rishit Bansal — Tue, 18 Jul 2023 08:51:43 +0000

What is Dyte?

For those of you who are hearing about Dyte for the first time, here’s some quick context -

We started building Dyte back in 2020 in the middle of the pandemic. Having started initially as yet another video calling platform we soon transitioned to offering video and voice SDKs after receiving several requests for the same. And it has been quite a journey ever since 🚀

We’re super excited to announce the launch of “Dyte’s live streaming SDK”, newest addition to the suite of solutions we provide 🎉

Dyte’s Live streaming SDK:

Dyte's Live-streaming SDK is designed to enhance your audio and video capabilities by providing a comprehensive range of features. As a versatile SDK provider, expanding our offerings to include live-streaming was a natural progression. Live-streaming caters to a variety of use-cases, particularly for companies already leveraging our video services. We recognized this and made it a priority to expand our offerings to meet the demands of our valued users. Needless to say, many of them have already successfully incorporated our live-streaming SDK, benefiting from its exceptional performance. After months of meticulous testing and optimization, we are thrilled to announce that our live-streaming SDK is out of beta and available to the public for seamless integration and utilisation!

Let's delve into some of the features that our Live-streaming SDK offers:

Key Features:

Low latency streaming: Experience lightning-fast streams with latency as low as 2 seconds, ensuring your audience enjoys real-time content from anywhere.
Custom layouts: Craft a visually captivating video feed with Dyte's UI Kits, allowing you to create a tailored and engaging user experience for your audience.
Live analytics: Gain comprehensive insights into your live stream's health, user activity, and user behaviour, empowering you with valuable data for optimization.
Adaptive bitrate: Enjoy smooth and uninterrupted live streams, regardless of network conditions, without requiring any manual intervention.
Built-in interactivity: Foster engagement and collaboration with features like whiteboard, chat, reactions, screenshare, polls, and quizzes, making your live streams interactive and captivating.
Seamless host addition: Enable viewers to effortlessly join the livestream and participate in games or activities with the streamer, enhancing the overall experience.
Scalability: With Dyte's robust infrastructure, effortlessly scale your streams to support over 10,000 viewers while maintaining exceptional video quality.
Platform agnostic: Stream live content from any device and browser, ensuring compatibility across a wide range of video codecs. Get started without a hitch!

We are live on Product Hunt today! Check us out right now: https://www.producthunt.com/posts/live-streaming-sdk-by-dyte

Thank you and I look forward to your comments and feedback! 🙇

Just Launched Dyte on Product Hunt

Rishit Bansal — Mon, 30 Jan 2023 09:28:42 +0000

What is Dyte?

Dyte is a live video calling SDK that lets you embed completely customizable and branded live video conferencing right in your product within minutes.

The Problem:

Back in 2020, while being stuck in the pandemic, we all had to shift to virtual meetings. Not only was it tiring to keep juggling between tabs and links; but for a business like ed-tech, it also created unnecessary friction for the user. Once a user clicks on the meeting link, they leave the platform with no visibility of their actions and behaviours.
Most of the video SDKs that already existed in the market either had a tightly coupled UI that provided close to no customizability or were so complex to implement that it was as good as building it from scratch.

The Solution:

This is when we decided to build Dyte. We wanted to create something that could be integrated with 0 hassle while also being completely customizable to fit into your product.

Dyte’s Key Features:

Customise your meetings end-to-end: From your meeting UI to the levels of user access you give your users, you can customise everything. Pick and choose every element of your call UI from our design library called “UI Kits” or build your own UI from scratch with the video infra being powered by our “Core SDKs”.
Ease of Integration: Dyte is so easy to implement that we have had customers go live in a day. Our in-depth documentation will help you navigate Dyte without any hiccups no matter which framework you use.
Plug-ins: Our out-of-the-box plugins such as whiteboard, polls, chat, code editor etc. help you create interactive and collaborative spaces that keep your team engaged!
Analytics: Our developer portal brings everything you need in one place. With GUI based configs and controls it also provides you with a granular view of all your meeting and user data.
More than video calls: With Dyte you can seamlessly switch to webinars or breakout rooms whenever required. You can also record and store your video meetings in a single place for easier access.
Support: We provide 24*7 support either through dedicated slack channels or through our community so you never find yourself stuck.

We are live on Product Hunt today! Check us out right now:

Thank you and I look forward to your comments and feedback! 🙇

https://dyte.io/ph

WebRTC 102: Understanding libWebrtc

Rishit Bansal — Mon, 28 Nov 2022 07:07:12 +0000

Jump to the "libWebRTC" section if you are in a hurry and not interested in a general overview of the WebRTC space.
If you've missed Part 1, here it is: WebRTC 102: Demystifying ICE | Dyte

Introduction

Anyone who has even slightly dwelled into the world of real-time video-audio communication over the internet would have surely heard about WebRTC: The standard for ultra-low latency, real-time media communications over the Internet! You may have also tried out building a simple P2P application using RTCPeerConnection, or tinkered around with high-level libraries such as Mediasoup, Pion, Jitsi to name a few. But something is missing...... what really powers this new family of protocols? How does it work? If you're someone who is interested in getting into the nitty-gritty of things to do with WebRTC, this series is for you!

Why Should You Care About WebRTC?

As previously stated, WebRTC is the protocol that powers all of the magic behind the scenes. Understanding WebRTC will help you understand how to optimize for a wider range of environments, enable you to make low-level modifications to cater to your particular case, and so on.
So without further ado, let's get started!

A Small Refresher to the Current State of WebRTC

To understand why libWebRTC exists and why it is so important, we should start by looking at its grassroots in 2011 when Google first announced a shiny new open-source project for web browsers.

The project has since moved to a brand new website webrtc.org, and the source can now be found at https://webrtc.googlesource.com/src/. This repository contains the native WebRTC library used by most modern browsers (Chrome, Firefox, Opera, etc.).

Of course, browser vendors rarely use the EXACT same library and versions due to compatibility reasons, and differing release cycles. For example, here is Firefox's clone of libWebRTC on Github with their own modifications to the project.

The Actual WebRTC Standard

libWebRTC is amazing, and everything, but the real mission to maintain interoperability between multiple platforms, browsers, etc., is realized by two projects:

IETF: IETF is an organization that maintains a collection of documents with various standards for real-time communication. These documents do not cover the actual WebRTC API, but rather, the different codecs, and protocols that are used by WebRTC clients to communicate with others. For example, ICE, STUN, and RTP.
W3: W3 maintains the spec for the official WebRTC API which you commonly see in browsers (RTCPeerConnection). All browsers are expected to comply with this specification to allow JavaScript clients to be cross-compatible. In actual practice, however, you would find that Chrome (or "Chromium") is by far the closest implementation of this spec and other projects currently implement a subset of the standard.

When we refer to WebRTC as a whole, we actually usually refer to these two specifications and not just chrome's WebRTC implementation.

Now that we have an idea of the WebRTC space let's dive deeper into libWebRTC!

libWebRTC

LibWebRTC is a C++/C native implementation of the WebRTC API which is compatible with Windows, MacOS, and Linux. On the mobile side of things, it also provides Java and Objective-C bindings for Android and iOS respectively.

Source for image: BlogGeek.me

Yes, the project is open source, but Google and other browsers (Firefox, Opera) are the primary contributors and driving force behind the codebase since inception. Despite the fact that the most popular use case for libWebRTC has been web browsers, there is nothing stopping you from using it as a native library to communicate with other WebRTC peers.

Should you be using libWebRTC?

Unless you are someone particularly keen on dealing with C/C++ and the million other pain points that come with it...you might find it easier to use some other open-source projects implementing the WebRTC specs which we cover below.

The "Mediasoup" project provides a high level JavaScript/TypeScript interface to the WebRTC APIs. The core logic of this project is implemented in C++/Rust. Consider taking a look at the project if you want an easy-to-use library instead of the low-level libWebRTC APIs.
A notable project to mention is the Pion/webrtc project which has a Golang implementation of the WebRTC API. Of course, we should mention the rust port WebRTC.rs. Let’s keep all the rustaceans happy too!

If "ease of use" is not particularly important, and you are looking for cross-platform support, libWebRTC is definitely the safest bet to make native WebRTC clients that are compatible with browser clients from Chrome, Firefox, and other browsers.

Getting started with libWebRTC

Now that we have talked about the WebRTC space, lets go over how you can start working on the libWebRTC codebase.

Cloning the repo

You can clone the source directly using git clone https://webrtc.googlesource.com/src, but it is recommended that you install Google's Depot Tools which includes a set of tools for working with the project.

Start by cloning depot tools:

git clone <https://chromium.googlesource.com/chromium/tools/depot_tools.git>

The repository's root folder has executable tools that we'll use later, so it can be convenient to add it to your PATH variable:

export PATH=/path/to/depot_tools:$PATH

Next, we can get the libWebRTC source by running:

mkdir webrtc-checkout
cd webrtc-checkout
fetch --nohooks webrtc
gclient sync

Now, the src folder should have the entire source code from libWebRTC's main branch!

Building the source

Chromium, libwebRTC, and other Google projects use the ninja build system.

But before we can use ninja to build the project, we must use another tool, gn.

gn uses the configuration options in the BUILD.gn file to generate ninja build files for multiple operating systems, executables, shared libraries, etc.

The webrtc.GNI file contains some boolean flags to control which components of the project are built, which you can mess with if you wish to do so. For now, we will just stick to the defaults.

To generate ninja's build files, run:

gn gen out/Default

This will by default generate files for a DEBUG build that has extra data that we will use later while debugging an example on the repo.

Finally, to start the build using ninja, run:

ninja -C out/Default

LibWebrtc’s Threading Model

The first step to working on any codebase is to locate the entry point of the code. Since libWebRTC is a library, it's hard to narrow it down to a single entry point. But the majority of codebases will start by using the CreatePeerConnectionFactory function in the codebase which lets you easily create a WebRTC PeerConnection. The first three parameters of this function are essential to understand the entire webRTC codebase. These three thread objects are instances of the rtc::Thread class, and are used globally in the entire codebase to allocate different tasks on separate threads to help avoid processing tasks from bottlenecking network calls and external APIs:

network_thread: This thread is responsible for writing the actual media packets flowing through your peer connection, and performing minimal processing tasks.
worker_thread: The most resource-intensive of the three, this thread deals with processing the individual video/audio streams received/sent to peers.
signaling_thread: This thread is where all the external API functions of the peer_connection class run. All external callbacks also run on this thread, therefore it is essential for users of the library to not block inside callbacks, since this will block the whole signaling thread.

Throughout the codebase, libWebRTC uses the RTC_DCHECK_RUN_ON macro to assert that function calls are running on the correct threads. This is also super helpful for someone reading the codebase to know which thread each function runs on.

LibWebRTC's peer_connection Example

The best way to get your hands dirty with libWebRTC is by using the peer_connection example available in the src/examples/peerconnection directory of the project. If you followed the steps previously covered in the “Building the source” section, the example should already have been built according to the default rules in the webrtc.gni file.

Note: As of the date of writing this article, we are using the commit (bdf25670973611ae0c2562a43922e6120edab070). Some of the information covered in this section might get outdated in the future.

The architecture of the example

The example outlines how to set up a simple peer connection to send a video stream from one client to another client and display it on the receiving end on top of a GTK window:

As we can see in the above diagram, there are two components in this example: the client and the server. In this blog post, we will concentrate on the client code as the server logic is irrelevant to LibWebRTC itself and is just a simple socket server that routes traffic between the clients. The architecture of the client application is as follows:

Running the Code

In the rest of this guide, we'll assume that you are using a Linux machine. To run the example, first, we need to start the server process. We can run the binary we previously built by running:

./src/out/Default/peerconnection_server

This should start up the socket server on port 8888:

Now, spin up two more terminal windows and run the client application on each of them. It will be useful to attach these windows to gdb . This will help us debug errors when we make changes to the codebase.

gdb ./src/out/Default/peerconnection_client
r

If the clients start up correctly, you should see two GTK windows popup.

Click on Connect on both of these windows, and now you should see the list of peers connected on both windows.

Now, click on the peer on any one of the windows. This should set up a WebRTC connection that streams video from that window to the other window! But at the time of writing this article, this seems to fail with the following error: 😞

Let’s try debugging why this error happens.

Debugging/Fixing the client binary crash #1 (Race condition in thread management) :

We can run bt on our GDB window to view the call stack when the application crashed and see that the following functions were called:

PhysicalSocketServer::Wait() is invoked in the main thread of the client, which blocks until we get a socket message.
On clicking the peer on the interface, a socket message is received which triggers OnMessageFromPeer.
This triggers the creation of a new Peer connection, which seems to call PhysicalSocketServer::Wait() recursively again…

On debugging the code, we found that this happens because of a caveat in the Threading Model we covered before. The thread where we invoke the CreatePeerConnectionFactory is actually the main thread of our client application. The constructor for the factory has a check which makes sure that all function calls run on the signaling thread. But, since we invoked the constructor from a different thread, it tried to queue a call on the signaling thread and blocked the main thread until the function call was completed on the former.

This causes Wait() to be called on our main application’s thread recursively, and the rtc::Thread class doesn’t allow Wait() to be called twice in this manner:

We can prevent this issue by simply queuing the function call on the signaling thread instead of synchronously calling it on the main thread of our application:

Here is a patch on github gist with the fixes, you can use this to apply the fixes on your local machine. Now, when we rebuild the files and run the code, the video stream shows up correctly!

This works for a small duration, but after a while, it appears that the video stream gets stuck on the client application on the receiving end, and now it crashes with a different segmentation fault error:

Let's look into why this happens.

Debugging/Fixing the client binary crash #2 (Buffer overflow in Video Renderer):

On viewing the call stack, we noticed that the crash occurs in the Redraw() function in the main_wnd.cc file:

On analyzing OnRedraw(), we found that the function is responsible for transferring the video of the peer from the remote_renderer to the draw_buffer. In the process, the code seems to be scaling the width of the source video by 2x, by duplicating adjacent pixels on the same row.

Since the memcpy is leading to a segfault after a duration, it is possible that the size of the source video is increasing to a larger size after a while, and since we initialize the draw_buffer just once, we ended up overflowing its bounds as it still has a smaller size. We can confirm this by adding some log lines in the code to log the width and height of the remote_renderer’s video :

RTC_LOG(LS_INFO) << "Video size: " << size;

We can fix this bug by re-initializing the draw buffer each time the size of the remote_renderer’s video changes, as outlined in this patch:

Now the video stream should render correctly for an indefinite period of time!

Conclusion

Congratulation, you reached the end of this article! You should now have a basic understanding of libWebRTC, and how to work with the project. If you are interested in contributing, you can head over to webrtc’s bug tracker and try your hands at squashing some bugs 😉

Try Dyte if you don't want to deal with the hassle of managing your own peer-to-peer connections!

If you haven't heard of Dyte yet, go to https://dyte.io to learn how our SDKs and libraries revolutionize the live video and voice-calling experience. Don't just take our word for it; try it for yourself! Dyte offers free 10,000 minutes every month to get you started quickly.If you have any questions or simply want to chat with us, please contact us through support or visit our developer community forum. Looking forward to it!

Why we Chose Flagsmith for Feature Flagging

Rishit Bansal — Wed, 28 Sep 2022 05:13:06 +0000

We at Dyte are super excited about the fact that we're growing at a super-fast pace, and so is our software delivery cycle. Almost every sprint, we introduce new features. However, the ever-increasing number of customers with different requirements warrants the need for an effective way to manage the feature delivery process as well. All of the features are not intended for all of our customers. The features that are not intended for all, are shipped behind the flags (indicators/controllers). These flags basically control the recipient of any such new features. In the software industry, the flags are commonly referred to as feature flags.

Before we dive into why Dyte chose Flagsmith for feature flagging, let's make sure we're all on the same page about feature flags.

The What's and Why's of Feature Flag

First thing first -:) Let's understand what Feature Flags are! A feature flag is a feature management solution that allows users to change the software's functionality without deploying new code. You can hide the code or behavior of software using feature flags without shipping new versions of the software.

In the simplest terms, consider the feature flag as a powerful "if" statement:

if (DyteFlagsmithFlags.hasFeature("newamazingfeature")) {
    renderNewAmazingFeature();
}

Feature flags go by different names (to list a few), in the software industry:

Feature Toggles
Feature Switch
Release Toggle
Feature Controls

Feature flags allow developers to conditionally turn on or off specific sections of their code (or code paths).

Now, come to why's:

There are a few ways to look at when you should use feature flags. Some of the most common use cases are:

Show or hide a button to access a new feature
Test a UI change
Try a new call to action to see how it performs
Let users try an experimental use case

To understand it better, imagine you're working on a feature that doesn’t provide support for all browsers yet or cannot be deployed in all geographic locations. For example, the new functionality is bound to work only on Chrome, and the standard support is not there yet for other browsers. Maybe the functionality doesn’t pass compliance of a few specific countries. In such a situation, you can use feature flags to manage the features easily.

A feature flag is a method of controlling the delivery of a specific feature (for example, user engagement metrics or facial gesture detection) to a subset of clients, countries, and browsers rather than rolling out all features to all clients.

You can easily create and manage feature flags with an advanced and user-friendly flag management platform, such as Flagsmith. Your changes can be applied in real-time, eliminating the need to deploy and ship your solutions repeatedly.

This is especially useful for SDK solutions like Dyte, where the release cycle is too agile. We can now instantly enable/disable a feature without requiring our clients to spare their developers countless hours in deployment.

Evaluation Phase

At Dyte, we evaluated multiple possible options and solutions before reaching the final decision!

We could have used a third party such as Flagsmith or LaunchDarkly.
Build an in-house solution.

After extensive discussion and evaluation, we decided to go with a third-party solution that is expert in this area. There were several reasons for this decision, but the most important one for us was not spending our developers' time and energy on something that was not our company's primary goal.

The next step was to figure out who:) This required a lot more research!

Evaluation Table

We wanted a solution that was cost-effective while still meeting our requirements.

There are many common and expected features of a feature flag solution, such as segmentation, flag types (boolean, string, JSON), A/B testing, and percentage rollouts, but we dug even deeper to find a solution that provided the most customization in terms of performance and functionality while remaining simple to use and integrate, just like Dyte!

So we researched, compared, and came up with a comparison table.

Symbols:
~ denotes approximately

— denotes “Information not readily available”

Function / Parameter	Flagsmith	LaunchDarkly	Optimizely	Split.io	DevCycle
SDK size (Vanilla JS)	~ 23 KB	~ 40 KB	~ 101 KB	~ 108 KB	~ 47 KB
SDK size (React)	~ 39 KB	~ 64 KB	~ 123 KB	~ 119 KB	~ 59 KB
SDK Support (Go, Java, Swift)	Available	Available	Available	Available	Available
On-premise hosting	YES	-	-	-	-
Load testing availability	YES	-	-	-	-
Website navigation index on the scale of 1-5. 5 being the most painful	1	2	4	3	1
Docs navigation, index on the scale of 1-5. 5 being extremely painful	2	3	3	2	1
Tracking the flags which were enabled for a session/call	YES (it was possible by having Integration of Webhook/ New Relic)	YES	YES (Needed events or integration)	YES	NO (We needed to add our own solution)
Dashboard to view audit logs	YES	YES	YES	YES	NO
Ways to find/define flag dependencies	-	YES	-	YES	NO
API Call response within xx ms	We could control it by having our custom-hosted solution. 100ms was the SLA expectation of flagsmith. https://docs.flagsmith.com/advanced-use/edge-api	200 ms	https://launchdarkly.com/how-it-works/	45ms to 400ms	130ms to 400ms https://status.split.io/
SLAs for issues	99% uptime or 10% credit https://flagsmith.com/service-level-agreement/	https://launchdarkly.com/policies/enterprise/premium-sla/ (Response in 1 day to 1 week or Pay more, 98% uptime or 10% credit)	No clear SLA.	99.9% uptime or moneyback if an issue arises for 2 months consequently https://www.split.io/legal/terms-of-service/	Response in 4 hours to 2 days based on the plan. 99.9% uptime. https://devcycle.com/pricing
What I liked?	API was easy to use and verbose. Functions such as hasFeature were aptly named. We could create our own infra as well as contribute. It is Open Source.	Documentation was better and rich. Promised the closest solution to what we needed.	Drag and drop for the layman. Advance event tracking to get insights in Optmizely’s in-built metrics section.	Better metrics and alert policies. A lot of features were really advanced but feel like overkill.	Extremely easy docs to follow.
What I didn’t like?	Docs were not updated or were updated in advance at various places.	Information was not readily available on the website. The free account had various limitations so couldn’t play around. SDK functions were named poorly.	Website navigation was not as intended. Features were called decisions, looks weird to read the code.	Weird naming conventions. The entire system was named believing that it will only serve A/B testing.	Still in the nascent stage. No auditing. No history.

*Evaluation as per March 2022

The Deciding Factors

For us, the most crucial points apart from the functionality were SLAs (can’t let the ship sink with others), SDK size (we are an SDK ourselves, bundle size matters, cannot afford more KBs), ease of use (API structure, Doc navigation), and the price.

Why Flagsmith?

We found Flagsmith to be the best solution available for our requirements. It had all the features, the ability to self-host, and more reasonable SLAs. Best of all, a very easy-to-follow API docs (https://docs.flagsmith.com/clients/rest). And just a single API call can do it all; no need to open a Postman collection and hit your head against the wall for days 😊

The cherry on top is that it is open source. If someday we get stuck on a requirement, with the help of a league of extraordinary folks, we can always contribute to their open-source codebase.

Given these considerations, we concluded Flagsmith to be the richest platform with a significantly lower fee.

Thanks to Flagsmith, we could avoid burning a hole in our pockets. Other service providers would have charged ten times the price for comparable features.

We highly recommend Flagsmith for all of your feature flag needs. Happy feature flagging with Flagsmith!

If you have any questions or simply want to chat with us, please contact us through support or visit our developer community forum. Looking forward to it!

Meeting Recordings at Dyte

Rishit Bansal — Fri, 25 Mar 2022 09:47:44 +0000

Authors: Kushagra Vaish, Rishit Bansal

Do you remember what people said on the last video call you attended? The average person attends 3-4 calls every day, making it difficult to remember everything that was discussed in the calls. At Dyte, we always have a single person taking notes for every call (generally the person that joins last 🧙‍♂️). But even this leads to frequently missing out on a few action items every call. Cue cloud recordings! Cloud recordings capture everything on your video call to a file that can be viewed anytime in the future. This article will cover the different approaches we covered while engineering this feature at Dyte and the hurdles we had to overcome in the process.

How to build it?

We knew we had to build recordings for our video calling company (duh) but we could have accomplished this in numerous ways. Firstly, we started by asking ourselves the following questions:

What do we want to record? Just the webcam and audio stream? Or the chat in the meeting as well? What about other interactive elements (polls, plugins, file uploads, etc.) in the meeting?
Do we want to record on the client-side or server-side?
Do we want to store the recording locally or server-side?

Let's consider these questions one by one. To answer the first question, we decided that we want to record any interactive element present on the screen. Also, we were sure that we wanted to store all the recordings on the server side since everyone in a meeting should be able to download the recording post-meeting. But, we didn't know how to answer the second question as all the approaches had trade-offs.

We came up with 3 major approaches to answer the second question:

Recording a browser tab: We could run a recording logic on the user’s browser that would record their screen and either store it locally or send it to the cloud. All browsers already ship with MediaRecorder APIs allowing you to record and store any media stream you generate. The flow looks something like this:
Recording a browser tab via MediaRecorder APIs
Record the streams received from the server: Most video call solutions send their streams to a server that processes them and sends them to all users who want to consume them. We can simply add another layer that sends these videos to a processing pipeline built using GStreamer or FFmpeg, which takes all the raw streams, stitches them together, and then stores them. In layman’s terms, think of this as taking everyone’s webcam videos from a call, then placing the videos side-by-side, and finally exporting to a single video.
Recording the streams received from the server
Good old screen capture: A simple approach is to join the meeting using a virtual “Bot” user and record the entire screen of this Bot using tools like FFmpeg or OBS. This approach preserves the interactive elements on a webpage as we now see the meeting just like how any other user in the meeting would perceive it. However, we would need to incur the hardware cost of running FFmpeg/OBS for every recorded meeting.
Recording the entire screen using tools like FFmpeg or OBS

The MVP

Our first approach prototype used the MediaRecorder API

We first thought of allowing users to record locally (using their own browser’s MediaRecorder API) but quickly realized that we can’t depend on the user’s machine to remain stable during lengthy recordings. What if a user leaves a meeting earlier than other users or runs into network issues and rejoins a meeting mid-way? To overcome these issues, we structured our recording MVP like this:

We ran a Chrome browser on the cloud that joined the call as a hidden participant.
This Chrome browser had an extension that activated as soon as a call was joined.
The hidden participant then starts to stream its tab. This stream is fed to the MediaRecorder API which then stores the recording in browser storage.
When the meeting ends, the recording is transferred to our S3 bucket.

We shipped the MVP within 2 weeks of planning that we had to do recording (oh, we do miss those days) and on the first day, it broke the entire system! Why? Well, there were a few things we failed to account for:

The entire recording was stored in memory, and as the calls become larger, the available memory decreases. We were running Chrome. This led to our cloud VMs running out of memory frequently.
Even when it had enough memory, the Chrome browser was unstable and crashed randomly.
Sometimes our recorder was unable to join the meeting.
Certain players couldn’t play the recording. Most importantly, it couldn't be played in QuickTime Player (Mac’s default player) and Windows Media Player (Windows default player).
... and a lot more.

Our MVP

On a side note, we were in the YC W21 batch during the time and recently came to know about a concept called PRD or Product Requirements Document (stop rolling your eyes OK, we had no clue about anything). Getting into the details of how we struggled with that will take another blog post, but we essentially realized this: we were just building a cool showcase by linking the maximum amount of systems possible, most of which weren’t even necessary.

Thinking along the same lines, we decided to start by writing a detailed PRD outlining the revamped architecture for the system. We clearly defined the goals with the rewrite, and how we could satisfy these requirements in a reliable manner.

The big refactor

This time, we decided to architecture a solution around the third approach, i.e., screen capture, described previously in this article. We built our own in-house docker image packaged with selenium, Xrandr, FFMPEG, and our application code. This container would join the meeting and then record the browser’s screen using FFMPEG. A lot of our time went into fine-tuning FFMPEG parameters to ensure the correct trade-off between quality and output size of the video file. Since FFMPEG continuously writes to the file system, we did not encounter the memory issue faced in the MediaRecorder approach. Moreover, FFMPEG provided a plethora of output encoding formats, allowing us to export videos to both H.264 and VP9 codecs. The vast majority of video players can easily play these.

FFMPEG gives a range of CLI options to control I/O formats and other parameters like bitrate (Source: https://ffmpeg.org/ffmpeg.html)

To ensure that the browser joined our meetings reliably, we added retry logic that would look for failures in joining the meeting and try to reload the page again when such errors happened. Moreover, we added logging in every component of the system, which helped us easily debug issues on live-call. A crucial step was monitoring these logs and setting up automated alerts on slack whenever we saw an increased frequency of “error” log lines.

An example of an alert on our slack channel from our recording service

Another issue we encountered was the long durations to spawn these docker containers on Amazon’s Elastic Container Service. ECS did not have any well-defined SLAs on how long it would take to spin up our containers. Our tests revealed that this startup time could range anywhere from 10 seconds to 5 minutes under stress and high load. We were later able to optimize this startup time to less than a second by using a neat trick, something we will cover in a future blog post.

AWS ECS

What we learned

LOG EVERYTHING!! This is probably the most understated thing that we have learned. Don’t worry about performance; use a proper logging library. Don’t worry about cluttering; use proper log levels and make the logging follow a proper structure. If you don’t have data on how to debug the problem, it’s really hard to find the issue. We ended up building a structured logging framework on top of winston which helped us identify edge cases with FFmpeg and our recording logic, and this was then shipped across all our microservices. Also, we ended up building alerts on top of these logs using Grafana, which allowed us to respond to issues even before customers reported them.
Try to reduce your iteration loop. We realized that testing a recording change locally was taking a lot of time and we didn’t have a proper reproducible test for testing our recording. For the redesign, we started with first writing a simple reproducible test and collecting baseline performance for that test. We built a simple script that allows us to deploy local change directly on the testing environment. That allowed us to shift from doing 1-2 iterations of recording per day to at least 5 iterations per day.
On the other end, once you are ready with the change, test everything. There was a random bug we saw where for some meetings, we saw the recording was never stopping. We saw then that these were meeting with a host, which when ends meeting for all participants causes recorder (who is also a hidden participant) to get stuck in an invalid state. You can’t test for everything but we ended up creating a feature checklist as part of our SDK and then each new feature is tested against the combinations of existing features.
Understand the cloud infra that you are building on top. Remember that random Chrome crash that we talked about? Turns out it was an issue on ECS. We had to go deep in the documentation for how ECS works and debug that.

Wrapping Up

In this post, we described our engineering journey in developing and perfecting meeting recordings on Dyte. We covered the different approaches we took, and what we learned in the process. Hope this helped you not only learn something new but also understand the importance of a well-defined developer workflow!

Do check out our recording APIs by triggering a recording using our REST API. Also, stay tuned for a future blog post on how we fine-tuned our infra for lightning-fast recording start times!

TypeORM Tips (Part 2: Use where() with care)

Rishit Bansal — Sat, 22 Jan 2022 19:15:01 +0000

Hey everyone! This is the second post in my series on tips while developing with TypeORM, a popular typescript ORM library. In this post I will remark on a common pitfall when using the where() method in the library.

Use `where()` with care

TypeORM's QueryBuilder provides a where() method to add SQL WHERE clauses in a your queries which allows you to specify a condition to control the records which your query processes. Here's a code snippet illustrating how you can use this method:

const query = await this.userRepository.createQueryBuilder()
             .select()
             .where("user.id=:userId", {userId: "123"})
             .getOne();

This method follows a similar pattern to several other methods of QueryBuilder which let you successively chain methods. Each method call returns back another QueryBuilder instance. Here is an example of a chain which performs multiple LEFT JOINS, filters by a where condition and finally limits the query result to just 10 rows:

const query = await this.userRepository.createQueryBuilder('user')
            .leftJoinAndSelect('user.posts', 'posts')
            .leftJoinAndSelect('user.comments', 'comments')
            .where('user.age > :minAge', { minAge: 18 })
            .limit(10)
            .getMany();

Neat! Now lets say I want to add another filter on this query to ensure that the user age is also under 65 years old. Naturally, if I were to follow the chain pattern offered by the library I might do the following:

const query = await this.userRepository.createQueryBuilder('user')
            .leftJoinAndSelect('user.posts', 'posts')
            .leftJoinAndSelect('user.comments', 'comments')
            .where('user.age > :minAge', { minAge: 18 })
            .where('user.age < :maxAge', { maxAge: 65 })
            .limit(10)
            .getMany();

TypeORM successfully executes the above and doesn't give any compile-time/runtime warnings at all. But this piece of code will not filter out records correctly!

What's the problem?

Adding multiple where() clauses to a query doesn't make sure all of them are satisfied. Rather, TypeORM only picks the last where() in the chain and uses that as the sole filter for the query. In other words, successive where() clauses just override previous clauses instead of adding new conditions. Thus the above code snippet will just return users whose age is less than 65 (i.e, The condition user > 18 won't be enforced!).

This is vague as the library doesn't complain with this usage and can sometimes blindside developers. If a developer didn't test the above code on corner-cases, he/she might unknowingly deploy this on production and may discover the edge case only much later when the bug is reported.

How do you fix this?

The correct usage is to use andWhere or orWhere depending on if you want to concatenate multiple conditions using AND or OR. For example, the above code snippet can be correct to:

const query = await this.userRepository.createQueryBuilder('user')
            .leftJoinAndSelect('user.posts', 'posts')
            .leftJoinAndSelect('user.comments', 'comments')
            .where('user.age > :minAge', { minAge: 18 })
            .andWhere('user.age < :maxAge', { maxAge: 65 })
            .limit(10)
            .getMany();

You can also use the Brackets function to create more complicated queries. Say I wanted to check if the user falls in either of two age ranges (18<age<35 OR 50<age<65), I could do the following:

const query = await this.userRepository.createQueryBuilder('user')
            .leftJoinAndSelect('user.posts', 'posts')
            .leftJoinAndSelect('user.comments', 'comments')
            .where(new Brackets(qb => 
                qb.where('user.age > :minAge', { minAge: 18 })
                  .andWhere('user.age < :maxAge', { maxAge: 35 }))
            .orWhere(new Brackets(qb => 
                qb.where('user.age > :minAge', { minAge: 50 })
                  .andWhere('user.age < :maxAge', { maxAge: 65 }))
            .limit(10)
            .getMany();

Note that here it was completely safe to use multiple where() clauses as the other usages actually operate on a seperate QueryBuilder instance and not the parent one. The basic rule to follow is to avoid multiple where() method calls on the same QueryBuilder instance.

TypeORM Tips (Part 1: Don't use save())

Rishit Bansal — Sat, 22 Jan 2022 13:09:43 +0000

Introduction to the series

As of January 21st 2022 (source), TypeORM is the 3rd most popular Javascript ORM library and certainly the most popular if we're talking about Typescript.

I have been extensively working with this library for the past two years, using it to serve millions of database queries every single day. In this series, I will list a few tips and pitfalls I learned while working with the project, which helped me catch bugs in production and optimize API calls. I'll try to keep each post short, informative and straightforward so you can quickly use these optimizations in your code.

In each post, we will go over:

What is the problem?
Why is it wrong?
How can you fix it?

So let's get started!

save() vs insert(), update()

Repositories have the .save() method, which has a dual function:

It inserts the entity if it doesn't already exist.
If the entity exists, it attempts to update the existing one.

Let's observe two example usages of the method:

Here is a code snippet taken from an API endpoint which registers a new user:

const user = this.userRepository.create({
    name: "Rishit",
    password: "test123",
});
await this.userRepository.save(user);

And here is a snippet from another endpoint which updates the name of an existing user:

const user = this.userRepository.findOne({
    name: "Rishit",
});
user.name = "John";
await this.userRepository.save(user);

Pretty handy, right? Well, not so much as this comes at the cost of a performance penalty. The implementation of save() executes 2 queries instead of a single one:

First, it uses a SELECT query to search for an existing entity.
If step 1 returns a record, it uses UPDATE to update the record. Otherwise, it uses INSERT to insert a new record.

Why is it bad?

These queries need two round trips to the database, meaning that you have to suffer the network latency due to each of the trips.
There are specific contexts in your code where you know for a fact that you are certainly inserting / updating and do not require to use save()'s dual functionality.
The SELECT query generated by TypeORM includes a subquery, which is highly inefficient for tables with millions of rows.

How do you fix this?

Look at the context you use your query in, and that should usually let you decide whether you meant to do an insert() or an update(). For example, you can refactor the above two snippets to:

registering a new user:

const user = this.userRepository.create({
    name: "Rishit",
    password: "test123",
});
await this.userRepository.insert(user);

updating an existing user:

await this.userRepository.update({
    name: "Rishit",
},{
    name: "John",
});

And just like that, you reduced the database load due to these queries by half: a 2x improvement!. The usage of save() might seem pretty obvious. Still, it's a prevalent practice to use .save() everywhere in code due to the documentation of TypeORM itself recommending it as the primary mode to update entities.

That said, save() does indeed pose a valid use case for code that actually requires the functionality to insert OR update depending on if the entity exists or not. However, I am sure that most use cases don't need this feature and intend to either insert or update a record, not both.

How MoleculerJS Powers Dyte!

Rishit Bansal — Mon, 15 Nov 2021 06:52:33 +0000

Microservices enable organizations to be more agile and build highly scalable applications. We, at Dyte, take advantage of MoleculerJS to connect our different microservices and APIs.

In this blog post, we will be talking about how we structure all of our services around Moleculer and use it to collect critical insights and metrics to debug and optimize our infrastructure to serve millions of minutes of #livevideo calls every month.

We would love to hear from you especially about what frameworks and tools you use to manage your microservices.

Read the blog here!

Reverse Engineering Keyboard Driver: Part 2 (Decompiling .NET applications)

Rishit Bansal — Tue, 05 Oct 2021 10:19:22 +0000

Introduction

This post is a part of my series to reverse engineer the keyboard driver/Omen Light Studio application on my HP Omen Laptop and re-implement its functionality on Linux. In this post, I will be covering how to decompile .NET services/DLLs

Locating the Service

I have been using the Light Studio Application for a while, and one of its best features is the ability to set up dynamic "wave" lighting on your keyboard. In this setting, the lighting on the keyboard keeps changing dynamically to show a sort of wave animation. An intriguing behavior I have observed with this feature is that the animation works just once you have booted into Windows. During the boot process in Ubuntu, the lighting is set to the last set color from windows and remains static. This means that the animation is most likely set by a background service on windows which starts on boot and is not a feature of the keyboard on the hardware level.

So, I began by opening a task manager and looking for any relevant background services that are running. I found two of particular interest:

Omen Gaming Hub

Light Studio Helper

The second service looks more interesting, so I right-clicked on the service and selected "Open File Location", and found that it was located at C:\Program Files\HP\LightStudioHelper

We can see several files, but one which caught my eye was Newtonsoft.Json.dll. I instantly recognized it as a C# library (https://www.newtonsoft.com/json), as I have worked with it in the past. This is important, as this means that the application was likely to be written in .Net in C#.

Decompiling the Executable and DLL Files

Next, I looked for tools to decompile .NET applications. The top result was a free tool by Jetbrains called DotPeek. I began by opening the folder on DotPeek and it was able to decompile all the DLLs. The two results of most importance to us are the ones for LightStudioHelper and OmenFourZoneLighting:

The LightStudioHelper binary is what runs the background service. We start by looking at the Main() method in this class:

 private static int Main(string[] args)
    {
      string productVersion = Process.GetCurrentProcess().MainModule.FileVersionInfo.ProductVersion;
      Logger.logger.Info("");
      Logger.logger.Info("----- Log Start, version: " + productVersion);
      if (Program.ProcessCommands(args))
      {
        Logger.logger.Info(string.Format("----- program exits, returnCode = {0}", (object) Program._returnCode));
        return Program._returnCode;
      }
      if (Program.IsAnotherInstanceRunning("OLS_HELPER"))
      {
        Logger.logger.Info("----- program exits");
        return Program._returnCode;
      }
      Program.Cleanup();
      Program.CreateTimer();
      Program.CreateAndRunThread();
      Logger.logger.Info("----- program exits");
      return Program._returnCode;
    }

This method first gets the version of the service, processes command line arguments, and then checks if another process of the OLS_HELPER service is running (not sure why yet) and then runs the Cleanup() method:

private static void Cleanup()
    {
      Logger.logger.Info("Cleanup()");
      TaskScheduler taskScheduler = new TaskScheduler("LightStudioHelperTemp");
      taskScheduler.Stop();
      taskScheduler.Delete();
      DirectoryInfo directoryInfo = new DirectoryInfo(Program.InstallDirTemp);
      if (!directoryInfo.Exists)
        return;
      directoryInfo.Delete(true);
    }

The cleanup() method creates a TaskScheduler class, which is another user-defined class in the source code of the service.
The implementation for Stop() and Delete() can be seen in the source, but it's irrelevant as it just deals with killing any already running process of the LightStudioHelper. We concentrate on the CreateAndRunThread() method which is run next:

 private static void CreateAndRunThread()
    {
      Logger.logger.Info("CreateAndRunThread()");
      Thread thread = new Thread(new ParameterizedThreadStart(Program.ThreadLightingUpdate));
      thread.IsBackground = true;
      thread.Start();
      if (thread == null || !thread.IsAlive)
        return;
      thread.Join();
    }

This method creates a new Thead which executes the ThreadLightingUpdate method in the background:

private static void ThreadLightingUpdate(object state)
    {
      Color[] colorArray = new Color[4];
      Logger.logger.Info("enter ThreadLightingUpdate()");
      while (Program._isRunning)
      {
        FourZoneLightingData zoneLightingData = LightStudioStorage<FourZoneLightingData>.ReadData();
        if (zoneLightingData != null && zoneLightingData.FourZoneColors != null)
        {
          bool flag = false;
          for (int index = 0; index < 4; ++index)
          {
            if (!flag && colorArray[index] != zoneLightingData.FourZoneColors[index])
              flag = true;
            colorArray[index] = zoneLightingData.FourZoneColors[index];
          }
          if (flag && FourZoneLighting.IsTurnOn())
          {
            Thread.Sleep(5);
            FourZoneLighting.SetZoneColors(zoneLightingData.FourZoneColors);
          }
        }
        Thread.Sleep(33);
      }
      Logger.logger.Info("leave ThreadLightingUpdate()");
    }

This is the main loop of the thread(), and on looking at it, we already got a lot of clues on how the background service works:

It maintains a 4 element array of Color, which is encouraging, as I know my keyboard has 4 configurable lighting zones, this might refer to the color of each zone!
The program has a while loop that seems to read colors from some sort of storage (LightStudioStorage<FourZoneLightingData>.ReadData()), and then stores the color data in the 4 element color array. It maintains a flag variable to check if any of the regions has a different color. Finally, if the flag variable is set, and FourZoneLighting.IsTurnOn() is true (presumably checking if the keyboard lights are turned on), it calls FourZoneLighting.SetZoneColors to set the colors.

I went in a little side adventure in checking out the LightStudioStorage and where it stores data, and found that it is a MemoryMappedFile:

namespace CommonLib.SharedMemory
{
  public sealed class LightStudioStorage<T>
  {
    private static MemoryMappedFileHelper _mmfHelper = new MemoryMappedFileHelper(typeof (T).Name);

    private LightStudioStorage()
    {
    }

    public static void WriteData(T data)
    {
      if (LightStudioStorage<T>._mmfHelper == null)
        return;
      LightStudioStorage<T>._mmfHelper.WriteData<T>(data);
    }

    public static T ReadData()
    {
      T obj = default (T);
      if (LightStudioStorage<T>._mmfHelper != null)
        obj = LightStudioStorage<T>._mmfHelper.ReadData<T>();
      return obj;
    }
  }
}

This likely refers to some shared memory logic, and another process might be writing to this memory-mapped file and calculating colors based on an algorithm. For now, I stopped here, but this area might be interesting to look at in the future as well.

Moving on, I searched for the implementation of FourZoneLighting.SetZoneColors, and found it was implemented in OmenFourZoneLighting.dll:

public static bool SetZoneColors(Color[] zoneColors)
    {
      if (zoneColors.Length != 4)
        return false;
      byte[] returnData = (byte[]) null;
      int num = FourZoneLighting.Execute(131081, 2, 0, (byte[]) null, out returnData);
      Thread.Sleep(5);
      if (num == 0 && returnData != null)
      {
        byte[] inputData = returnData;
        returnData = (byte[]) null;
        for (int index = 0; index < 4; ++index)
        {
          inputData[25 + index * 3] = zoneColors[index].R;
          inputData[25 + index * 3 + 1] = zoneColors[index].G;
          inputData[25 + index * 3 + 2] = zoneColors[index].B;
        }
        num = FourZoneLighting.Execute(131081, 3, inputData.Length, inputData, out returnData);
      }
      return num == 0;
    }

This file basically seems too transfer the colors to another data structure, inputData, and then passes them to Execute():

 private static int Execute(
      int command,
      int commandType,
      int inputDataSize,
      byte[] inputData,
      out byte[] returnData)
    {
      returnData = new byte[0];
      try
      {
        ManagementObject managementObject1 = new ManagementObject("root\\wmi", "hpqBIntM.InstanceName='ACPI\\PNP0C14\\0_0'", (ObjectGetOptions) null);
        ManagementObject managementObject2 = (ManagementObject) new ManagementClass("root\\wmi:hpqBDataIn");
        ManagementBaseObject methodParameters = managementObject1.GetMethodParameters("hpqBIOSInt128");
        ManagementBaseObject managementBaseObject1 = (ManagementBaseObject) new ManagementClass("root\\wmi:hpqBDataOut128");
        managementObject2["Sign"] = (object) FourZoneLighting.Sign;
        managementObject2["Command"] = (object) command;
        managementObject2["CommandType"] = (object) commandType;
        managementObject2["Size"] = (object) inputDataSize;
        managementObject2["hpqBData"] = (object) inputData;
        methodParameters["InData"] = (object) managementObject2;
        InvokeMethodOptions invokeMethodOptions = new InvokeMethodOptions();
        invokeMethodOptions.Timeout = TimeSpan.MaxValue;
        InvokeMethodOptions options = invokeMethodOptions;
        ManagementBaseObject managementBaseObject2 = managementObject1.InvokeMethod("hpqBIOSInt128", methodParameters, options)["OutData"] as ManagementBaseObject;
        returnData = managementBaseObject2["Data"] as byte[];
        return Convert.ToInt32(managementBaseObject2["rwReturnCode"]);
      }
      catch (Exception ex)
      {
        Console.WriteLine("OMEN Four zone lighting - WmiCommand.Execute occurs exception: " + ex?.ToString());
        return -1;
      }
    }

This method seems to be doing the actual interaction with the hardware of the keyboard. I did a bit of research about ManagementObject and found that it's a class used to interact with WMI (Windows Management Instrumentation). WMI, specifically WMIACPI allows you to interact with the Bios and hardware devices, but more on this on the next blog post, for now, let us just treat this function as a black box which does some magic to set the colors of the keyboard.

Since now we have enough information on how the service works, I tried to implement everything in the OmenFourZoneLighting.dll file in my command line C# program for windows.

Rewriting the WMI Code in a C# program

I started by setting up a .NET console application on Rider and added the System.Drawing, and System.Management DLLs as assembly references from my system.

I copied most of the Code from the dotPeek decompiled result, and fixed some variable references, and wrote a CMD application which is available on this Github repository:

thebongy / omen-cli

omen-cli

A CLI to customize your keyboard backlight 🚀

Explore the docs »

View Demo
·
Report Bug
·
Request Feature

🎯 Table of Contents

📖 About The Project

omen-cli is a lightweight CLI tools built in C# to customize keyboard backlights on HP Omen laptops similar to how Omen Light studio does.

Built With

✈️ Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

.NET Framework 4.8
Nuget.exe CLI
MSBuild.exe CLI

Installation

Clone the repo

git clone https://github.com/thebongy/omen-cli.git

Install dependencies

nuget install .\CLI\packages.config -OutputDirectory packages

Run the following command from the root dir to build the project

MSBuild.exe

🔧 Usage

To view all the options available, use the --help command:

The set4 command is used to set 4 colors to…

View on GitHub

Conclusion

In this post, we saw how to decompile a C# application, and then implemented is using .NET Framework. In the next post, I will research more into ACPI and WMI drivers for Linux, to get a better idea of how to implement this functionality on Linux.

Reversing Keyboard Driver: Part 1 (Introduction)

Rishit Bansal — Sat, 21 Aug 2021 20:00:05 +0000

Tldr; (What is the series about)

This post is an introductory post to a series that will describe my journey to analyze and learn how the keyboard backlight driver on my laptop works, and then re-implement it on my own. I wanted a place to dump my thoughts and different things I try, and learn in the process.
This post is just an intro to the series, if you're looking for the actual action, head over to the next post in the series!

Why?

I recently upgraded to a new laptop (HP Omen 15-en0037AX). The laptop has a four-zone RGB LED backlight on the keyboard. Like most laptops that ship with such a keyboard, the manufacturer shipped a preinstalled app (Omen Light Studio) on windows to customize the various lighting modes on the keyboard.

HP Omen Light Studio

Unfortunately, this app is only available for Windows, which is a problem, as I am primarily a Linux user.

Since I have always been interested in cybersecurity and have taken part in multiple CTF (Capture the Flag) events, I have a basic understanding of how to Reverse Engineer C/C++ binaries using tools like Ghidra but have never reversed an actual real-life application. In this series I want to (hopefully) achieve two things:

Reverse Engineer the Omen Light Studio App, and figure out how it interfaces with the hardware on the laptop to control the keyboard's lighting.
Re implementing the functionality on Linux (maybe in C/C++/Python/NodeJS) and possibly make a similar desktop app.

Disclaimer

This series is going to be me documenting all my approaches (right or wrong), and also explaining various concepts I learned in the process. Currently, I have no idea where this will lead to or what tools/languages I need to know/learn, but all that begins from the next post. See you there!