Forem: Ahmed Ossama

Mongodb text index vs regex for text searching

Ahmed Ossama — Sat, 27 Aug 2022 14:13:04 +0000

In this post I'll discuss the difference between two methods for text searching in mongodb collection and compare their complexities, pros and cons.

A case study

Consider having posts collection where each object consists of title and content:

"_id": ObjectId(""),
"title": "PostA",
"content": "This is the content for the first post."
},
{
"_id": ObjectId(""),
"title": "PostB",
"content": "This is a different content for the second post."
}

Our objective here is to search for some text and return the matching documents.

Using REGEX

In case we want to search for title it will be easy to use normal find filter

db.posts.find({title: "PostA"})

if we know exactly the title or to use regex if we know a part of it

db.posts.find({title: 'pattern', $options: '<options>'})

But if we want to search in the content field, using this

db.posts.find({content: "first"})

will return nothing as it will search for an exact matching.
So, we can use regex here

db.posts.find({content: {$regex: /first/}})

this query will return the PostA document as its content contains the word first.
But this will do a full collection scan of O(n) and it will have a poor performance on larger datasets.

Using Text Index

Text Indexes : It converts the text into array of single words and it remove all the stop words (is, a, an, etc)
Let's create a text index on our content field

db.posts.createIndex({content: "text"})

and don't forget to specify "text" to remove unneeded words and store keywords.
to search for a word

db.posts.find({$text: {$search: "first"}})

This will return PostA.

Why we didn't search inside our content in the above query? 🤔
Since mongo treat this index as an array of words in order if you want to add another field for this text index, for example we can add both title and content to the index and it will treat them under one text only.

Example

db.posts.createIndex({title: "text", content: "text"})

Note: we can't add another text index while there is already another one so these next lines are illegal and we should add them at once,

 db.posts.createIndex({content: "text"})
 db.posts.createIndex({title: "text"})

Ok, now we have a combined index on both title and content so if we search with any keyword whether in title or content it will return the correct matching document.
This approach will be very efficient in terms of complexity as it uses indexScan ( O(log(n)) ) also in usability instead of searching in a specific field this will search in a combined multiple fields which is more practical.

Exclude words using text index

Let's try to search a post with content contain the 'post' keyword

db.posts.find({$text: {$search: "post"}})

This query will return both PostA and PostB docs, but we can return only PostB if we exclude the 'first' keyword :

db.posts.find({$text: {$search: "post -first"}})

This will exclude the docs with content having the word 'first'.

Conclusion

Finally we saw that using text index is faster, easier and preferable to be used and support keyword exclusion, but we have some other cases when we want to search for a substrings or partial word matches like the word Post in PostB in this case we have to use regex.

Adaptive bitrate streaming

Ahmed Ossama — Tue, 15 Feb 2022 15:33:19 +0000

Introduction

In this post I will try to explain how video streaming happen, for example when you watch a YouTube video or any online streaming, we will know the protocols behind this and focusing on the Adaptive bitrate over HTTP.
ALERT! The article is a little bit academic so it's ok if you miss some concepts, Happy reading ')

What was before?

Just having a single stream sent out, if you are lucky you will stream it, otherwise you will suffer from repetitive pauses or to watch with the lowest quality.
Another approach is to download the whole video in the main memory in order to guarantee watching, but what if you don’t have enough memory, however it's data consuming since most of us don't watch a whole video, most people nowadays watch only 20% of the video as the content growth is very high.
So we conclude that we have more than one type of video streaming.

Types of http video streaming

The Internet uses HTTP streaming over TCP most commonly for video streaming services.
(Youtube, Netflix, etc).

Regular HTTP Streaming: It completely downloads the video to the client server.
Progressive download: Progressive downloading of the video at a fixed quality.
Adaptive streaming over HTTP: Combination of Adaptive video quality control and progressive Downloading.

Adaptive streaming

Adaptive bitrate streaming is a technique used in streaming multimedia over computer networks. While in the past most video or audio streaming technologies utilized streaming protocols such as RTP with RTSP, today's adaptive streaming technologies are almost exclusively based on HTTP and designed to work efficiently over large distributed HTTP networks such as the Internet. It works by detecting a user's bandwidth and CPU capacity in real time and adjusting the quality of the media stream accordingly. It requires the use of an encoder which can encode a single source media (video or audio) at multiple bit rates. The player client switches between streaming the different encodings depending on available resources. "The result: very little buffering, fast start time and a good experience for both high-end and low-end connections."

More specifically, adaptive bitrate streaming is a method of video streaming over HTTP where the source content is encoded at multiple bit rates. Each of the different bit rate streams are segmented into small multi-second parts. The segment size can vary depending on the particular implementation, but they are typically between two and ten seconds. First, the client downloads a manifest file that describes the available stream segments and their respective bit rates. During stream start-up, the client usually requests the segments from the lowest bit rate stream. If the client finds that the network throughput is greater than the bit rate of the downloaded segment, then it will request a higher bit rate segment. Later, if the client finds that the network throughput has deteriorated, it will request a lower bit rate segment.

Transcoding: Converting from one or more codecs, bitrates, or resolutions to others.

The bitrate changes with time as the resources change in order to adapt with it.

So, what about the client side or actually how the player handles all of this..

Player Switching algorithm depends on:

Network throughput
Screen size (Playing with larger bitrates in a small screens will not make sense)
Rendering speed(fps) in games for example.
Buffer size
Previous ABR switches

Popular Implementations

MPEG-DASH
Adobe HTTP Dynamic Streaming
Apple HTTP Live Streaming
Microsoft Smooth Streaming
QuavStreams Adaptive Streaming over HTTP
Uplynk
Self learning clients

Youtube MPEG dash

MPEG-DASH is the only adaptive bit-rate HTTP-based streaming solution that is an international standard.

Here simulating youtube mpeg dash operations where changes occur in user resources may lead to an adaptive change from a higher bitrate to a lower one whether in the video or the audio but it’s most likely in the video since it leads to a bigger change than the audio.

Trick mode : is when we seek back (rewind) the video it will reset and go back to the default resolution.

Relation between the bitrate and resolution

Bitrate is the amount of data encoded for a unit of time, and for streaming is usually referenced in megabits per second (Mbps) for video, and in kilobits per second (kbps) for audio. From a streaming perspective, a higher video bitrate means a higher quality video that requires more bandwidth. So why doesn’t everyone just upload at the highest bitrate possible? Well, not every viewer can download at the highest bitrate possible.

Some Cons

HTTP based adaptive bitrate technologies are more operationally complex than conventional streaming technologies. Here are few thoughts that outline some of those complexities:

File Size: In HTTP based adaptive bit rate streaming, content is segmented into smaller blocks or objects. Most CDNs (Content Delivery Networks) struggle with large VOD files, especially while handling block sizes in storage. Decreasing block sizes may result in inefficient storage, affecting performance and causing a poor viewing experience.

Network Overload: HTTP based adaptive bit rate video streams cause more network traffic than MMS or RTSP streams. This is because HTTP based adaptive bit rate video streaming technology is very aggressive and fills up pipes whenever it can whereas regular streaming mechanisms have certain caps and limits.

Questionable Quality of Service: While viewers expect a quality viewing experience, there’s no guarantee to the quality of service delivered to viewers as QoE technologies sometimes do not meet up to the requirements. A quality video viewing experience is one where there is no buffer lag, but here there is no guarantee of delivering a true HD video.

The “Thundering Herd'' Problem: The thundering herd problem occurs because caching servers are primarily designed to cache web sites. When the request for an object is initiated, the cache tries to pull the video from the file’s origin and share it with the requester. When multiple requests are initiated simultaneously, the video server is bombarded with requests and this puts a strain on system resources. This is exactly what HTTP adaptive bitrate streaming does and can be inefficient in this way.

What's exactly Websockets

Ahmed Ossama — Wed, 17 Nov 2021 06:30:17 +0000

In this article we will talk about websockets, it is a computer communication protocol, providing full connections over a single TCP connection.
Before we dive deeper into how it works, let’s discuss briefly on how HTTP works to know why we have web sockets!

How HTTP works? (http:// , https://)

Consider having a client and a server, the following flow will show us how the communication occur :

The client opens a TCP connection to communicate with the server.
The client sends a request to the server for example a GET request for an index.html file
The server responds to that request with a certain status code such as:404 not found or 201 ok. And this flow continues until the client closes the connection. We can conclude here that this protocol is unidirectional as we can’t receive a response without sending a request and vice versa.

Now we knew how http works, but in many situations this protocol will overhead, imagine we are having an application like facebook, while you are browsing there’s notifications maybe pop up, guess :
Is it good to be fetched using HTTP?
Here you as a client don’t know that the resource is modified because another user modified it or the service itself modified it, so in this case using HTTP is not the best thing, HTTP will be preferred when a resource that changes hourly or changes only after it knows that a related resource is modified, so that we need something STATEFUL.

How does websocket work? (ws:// , wss://)

Having the same client and server :

The client opens a TCP connection.
A websocket handshake request occurs, this is a normal http request with upgrade header that asks the server whether to establish a websocket connection or not.
If it’s ok:
- The client now is able to send and receive information to and from the server without waiting, then this is a bidirectional protocol.
- If anyone closes the connection, the other will not be able to send information and the connection is terminated from both sides.

When to use websockets?

There are some scenarios where websockets are preferred over HTTP such as Fast Reaction Time, Ongoing Updates.
It’s used in:

Chat Applications
*Consider a chat application that allows multiple users to chat in real-time. If WebSockets are used, each user can both send and receive messages in real-time. Anyone can send you a message without requesting to receive it!
Real time Applications
- Consider a notification system in a feed application as we talked earlier.
Gaming Applications
- Consider having a multiplayer game running in the browser which its ui is automatically updated upon the current state.

Pros of using websockets

Full duplex
- No polling required and we don’t have to ask for a response repeatedly.
Moderate overhead in establishing connection, and minimal overhead in sending/receiving messages.
Secure. ## Cons of using websockets
Since it’s stateful it’s difficult to be horizontally scaled.

Implementing a websocket

We will use python and js to write a very simple websocket to establish connection between a created server and the browser client side.
First of all, implement the server in python language that recieves a messages and instantly reply to it.
Don't forget to "pip install websockets"
Create a folder and place this script file in it.

import asyncio

import websockets

# create handler for each connection

async def handler(websocket, path):

    data = await websocket.recv()

    reply = f"Client:{data}\nServer:I am fine, thanks!"

    await websocket.send(reply)



start_server = websockets.serve(handler, "localhost", 7000)



asyncio.get_event_loop().run_until_complete(start_server)

asyncio.get_event_loop().run_forever()

Second, implement the client side using html and js

<!DOCTYPE html>

<html lang="en">

<head>

    <meta charset="UTF-8">

    <meta http-equiv="X-UA-Compatible" content="IE=edge">

    <meta name="viewport" content="width=device-width, initial-scale=1.0">

    <title>Client</title>

</head>

<body>

    <button onclick="contactServer">How are you?</button>

</body>

<script>

    const socket = new WebSocket('ws://localhost:7000');

socket.addEventListener('open', function (event) {

    socket.send('How are you?');

});



socket.addEventListener('message', function (event) {

    console.log(event.data);

});

const contactServer = () => {

    socket.send("Initialize");

}

</script>

</html>

After running the file and opening the HTML page, the results as follow

Conclusion

Finally, each technology has its pros and cons and using websockets or http depends on the nature of your application and the requirements needed.
If you only want to receive an update every hour, you will want to go with HTTP. If you want to receive updates every second, a WebSocket might be a better option, because establishing an HTTP connection takes a lot of time.

How browsers handle authentication?

Ahmed Ossama — Sat, 25 Sep 2021 14:33:55 +0000

Today, I’ll talk about authentication and how your browser handles such a process, First of all, authentication is the act of validating that users are who they claim to be. This is the first step in any security process. There are different ways to authenticate a user, the most popular methods are using SESSION IDs and JWT each of them has its pros and cons, In this article, I’ll discuss a brief about them. so let’s dive into the article.

Imagine the login process that you are doing every day when you enter your username/mail and your password to enter a website...

How the browser communicates with the server

The browser sends a POST request to the server containing the username and password
The server checks whether this user exists or not then compare the saved hashed password to that in the request if everything is fine:
The server sends a success response 201 ok to the browser indicating that the user is found then browser sets a cookie that contains…….

HTTP COOKIES

The trivial old way is setting a cookie with key, value pairs containing data that refer to the logged user, the key could be something like 'username' and the value would be the username.

{
 "username" : "ahmedossama"
}

so, every time the browser communicates with the server it checks the cookie to allow the user to do stuff behind the login, but by this method, it will be easy to deceive the server and act as another user so we need a unique identifier and this gets us into the session ids!

Session IDs

Instead of returning raw info about the user, the server will respond with only a random id that only it can understand and then the browser set the cookie to be something like that

{
"sessionID" : "A123"
}

whenever the user makes a request, the server has a 3rd party translate this id into understandable information, then it sends a 200 ok if the request is from an authenticated user.
But a problem may occur when having multiple servers running in parallel.

As we saw, passing user's info by value is non-secure so, we used a token(session ID) to pass user info by reference(need to go to the server each time), this leads us to a great question, what about passing a token by value 🤔 ??

JWT (JSON web tokens)

The server response will be data about the user and a unique signature from the server to guarantee that data.
The jwt token consists of three parts which are the Header, payload, and signature.

Header

{ 
  "typ": "JWT",    
  "alg": "HS256" //This could be HMAC, SHA256, RSA, HS256 or RS256
}

Payload

It contains session data about the user (claims), claims are customizable but take care don't put sensetive or large data in them.

{
  "sub": "1234567890", //subject
  "name": "Ahmed",
  "iat": 1516239022 , //issued at
  "ex" : 3000 ,  //expiry
  "admin" : true
}

Signature

Signature is calculated by encoding the header and payload using Base64url Encoding and concatenating them with a period separator. Which is then given to the cryptographic algorithm.

HMACSHA256(
  base64UrlEncode(header) + "." +
  base64UrlEncode(payload),
  secret)

The final jwt token will be something like that

You can play with just here : JWT
Also, there's lots of drawbacks to using jwt i.e XSS, CSRF attacks and others can make security issues, there's no perfect method ')

Forem: Ahmed Ossama

Mongodb text index vs regex for text searching

A case study

Using REGEX

Using Text Index

Exclude words using text index

Conclusion

Adaptive bitrate streaming

Introduction

What was before?

Types of http video streaming

Adaptive streaming

Player Switching algorithm depends on:

Popular Implementations

Youtube MPEG dash

Relation between the bitrate and resolution

Some Cons

What's exactly Websockets

How HTTP works? (http:// , https://)

How does websocket work? (ws:// , wss://)

When to use websockets?

Pros of using websockets

Implementing a websocket

Conclusion

How browsers handle authentication?

How the browser communicates with the server

The server sends a success response 201 ok to the browser indicating that the user is found then browser sets a cookie that contains…….

HTTP COOKIES

Session IDs

JWT (JSON web tokens)

Header

Payload

Signature