Forem: Sharfin Jahan Sakib

Exploring emoji encodings

Sharfin Jahan Sakib — Wed, 22 Jan 2025 19:15:46 +0000

Have you ever tried to covert a string with emojis to bytes?

text = 'Hi 🙋🏽‍♂️'

print(bytes(text, 'utf-8'))

b'Hi \xf0\x9f\x99\x8b\xf0\x9f\x8f\xbd\xe2\x80\x8d\xe2\x99\x82\xef\xb8\x8f'

And wonder TH are the \x thingies and how do they even translates back to emojis?

Diving deep, you will find it takes five unicode characters to represent this single emoji.

emoji = '🙋🏽‍♂️'
print("Length: ", len(emoji))
for c in emoji:
    code = ord(c)
    print("{}\t{}\t{}".format(code, hex(code), c))

Length:  5
128587  0x1f64b 🙋
127997  0x1f3fd 🏽
8205    0x200d  ‍
9794    0x2642  ♂
65039   0xfe0f  ️

First we have the base emoji: person raising hand 🙋 128587 or 0x1f64b in hexadecimal. Often represented as U+1F64B

Then we have the skin tone modifier: light skin tone 127997

If you add these two, you will get a gender neutral person with this skin tone raising hand.

print(chr(128587) + chr(127997))

🙋🏽

The second last one represents a male symbol, and the last one is called Variation Selector-16 (VS16), which indicates the previous character should be displayed as an emoji, rather than a symbol.

There are many characters can be displayed either as symbol or emoji.

heart_symbol= '♥'
print(heart_symbol)
print(heart_symbol + chr(65039))

♥
♥️

Note: if the system or the renderer doesn't support that emoji, it might be rendered as a symbol. In that case, copy the output and paste it into a different text editor or browser to see the difference.

So, the first two and lat two represents two separate emojis. The middle character is called a Zero Width Joiner (ZWJ), which is used to combine two or more emojis.

woman = '👩'
laptop = '💻'
print(woman + chr(8205) + laptop)

👩‍💻

Of course, the system or font has to support these combinations, otherwise it will be rendered just separate.

man = '👨'
toilet = '🚽'

print(man + chr(8205) + toilet)

👨‍🚽

Now let's see how each unicode characters in general can be represented as bytes in the utf-8 encoding.

Let's consider the character character U+1F64B.

First we have to determine how many bytes will be required to store this, according to the range of the character. We could store this number itself with just three bytes, but then there would be no way to decode it back to one single character. So this pattern is used instead: (why this?)

11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Now lets put the bits of the number 0x1f64b in this pattern. Lets first convert it to binary.


print(bin(0x1F64B))

0b11111011001001011

The pattern has 21 bits, so let's add 0s at start to match

000011111011001001011
000 011111 011001 001011

putting in the pattern: 11110000 10011111 10011001 10001011

Now, what are the hexadecimal of these bytes?

print(list(map(hex, [0b11110000, 0b10011111, 0b10011001, 0b10001011])))

['0xf0', '0x9f', '0x99', '0x8b']

Notice that these matches with the first 4 thingies we wanted to understand. Now you know what they mean.

Find the notebook here: https://github.com/sjsakib/emoji-encoding/
Leave a star if you enjoyed.

Making your own NestJS on top of express under 180 lines

Sharfin Jahan Sakib — Thu, 03 Aug 2023 19:56:04 +0000

It can be a little overwhelming when using NestJS for the first time. Let's try to understand how some of it's components work.

Before you start

I am assuming that you are familiar with NestJS and the features it provides. Most of the features that feels like black magic is achieved with decorators and experimental metadata API. Make sure you have basic understanding of these.

I've set up this repository with necessary config to run the working snippets I am about to share. You can put them in the playground.ts file and run them with the command npm run playground. I highly recommend running, changing and playing around with the snippets.

Defining routes

The key here is that, decorators can attach metadata to classes and methods, and those metadata can be accessed later on runtime. Let's dive right in.

import 'reflect-metadata';

const PATH_KEY = Symbol('path');
const HTTP_METHOD_KEY = Symbol('method');

/**
 * Post is a decorator factory that takes a path and returns a decorator.
 * That decorator attaches the path and the HTTP method post to the class method it is applied to.
 **/
function Post(path: string) {
  return function (target: any, key: string) {
    Reflect.defineMetadata(PATH_KEY, path, target, key);
    Reflect.defineMetadata(HTTP_METHOD_KEY, 'post', target, key);
  };
}
/* 👆 these are codes that the framework might provide */

/* So user can write something like this */
class AuthRoute {
  @Post('/login')
  async login() {
    return 'login success';
  }
}

/* Then the framework can use the class to create the actual routes */
function createApp(ControllerCls: any) {
  // first get all the properties of that class
  const properties = Object.getOwnPropertyNames(ControllerCls.prototype);
  properties
    .filter(
      (
        method // keep the ones that as HTTP method metadata
      ) => Reflect.hasOwnMetadata(HTTP_METHOD_KEY, ControllerCls.prototype, method)
    )
    .forEach(method => {
      const path = Reflect.getMetadata(PATH_KEY, ControllerCls.prototype, method);
      const httpMethod = Reflect.getMetadata(
        HTTP_METHOD_KEY,
        ControllerCls.prototype,
        method
      );
      console.log(`Mapping: ${httpMethod.toUpperCase()} ${path}`);
      // now that we have access to the method name and path at runtime,
      // these could be attached to an express app
    });
}

createApp(AuthRoute);

Note that, using symbol is not mandatory here, could just use plain strings for the keys.

Dependency injection

The basic idea of dependency injection is that instead of instantiating the dependencies of a class in the constructor, you pass the dependencies to the constructor. The mystical thing that NestJS does here is, you can define the dependencies in the constructor with shorthand express like constructor(private service: Service) and NestJS will do the instantiating and pass it down to the constructor for you. Let's see how something like that is possible.

Again metadata API comes into play. The parameters of a constructors is available with the design:paramtypes key in the metadata. The catch is that, the class has to be decorated with at least one decorator. Otherwise typescript will not record the parameter data while transpiling to runnable javascript. This is where the @Injectable() decorator comes into play. You might have noticed that NestJS will have you decorate the controllers with the Controller() decorator even when no controller prefix is required. This is because all classes need to be decorated with at least one decorator in order for them to be instantiated with correct params.

import 'reflect-metadata';

function Injectable() {
  return function (target: any) {}; // it doesn't have to do anything
}

class UserRepository {
  async findUser() {
    return 'user exists';
  }
}

@Injectable()
class AuthService {
  constructor(private readonly userRepo: UserRepository) {}
  login() {
    return this.userRepo.findUser();
  }
}

function instantiate(ProviderCls: any) {
  const params = Reflect.getMetadata('design:paramtypes', ProviderCls).map(
    DependencyCls => new DependencyCls()
  );

  const provider = new ProviderCls(...params);

  provider.login().then(console.log);
}

instantiate(AuthService);

Note that, it didn't need to be a decorator factory, since it is not taking params. We could define it like function Injectable(target: any) {} then it could be used like @Injectable, without the braces. Just making it look like the other decorators.

What about passing data down to the route handlers?

With a similar technique, parameters can be decorated to indicate what kind of data is needed in that parameter, and then the framework can pass down the appropriate data, taking form the underlying platform.

import 'reflect-metadata';

const HTTP_METHOD_KEY = Symbol('method');
const PATH_KEY = Symbol('path');
const PARAMS_META_KEY = Symbol('paramsMeta');

// just like the first snippet
function Get(path: string) {
  return function (target: any, key: string) {
    Reflect.defineMetadata(PATH_KEY, path, target, key);
    Reflect.defineMetadata(HTTP_METHOD_KEY, 'get', target, key);
  };
}

// decorator to indicate that data is required from route parameter
export function Param(key: string) {
  return function (target: any, methodName: string, index: number) {
    const paramsMeta = Reflect.getMetadata(PARAMS_META_KEY, target, methodName) ?? {};
    paramsMeta[index] = { key, type: 'route_param' };
    Reflect.defineMetadata(PARAMS_META_KEY, paramsMeta, target, methodName);
  };
}

class AuthRoute {
  @Get('/profile/:id')
  async profile(@Param('id') id: string) {
    return `user: ${id}`;
  }
}

function createApp(ControllerCls: any) {
  Object.getOwnPropertyNames(ControllerCls.prototype)
    .filter(method =>
      Reflect.hasOwnMetadata(HTTP_METHOD_KEY, ControllerCls.prototype, method)
    )
    .forEach(method => {
      const PARAM_DATA = { id: '123' }; // could get from req.params

      const paramsMeta =
        Reflect.getMetadata(PARAMS_META_KEY, ControllerCls.prototype, method) ?? {};

      const paramsToPass = Reflect.getMetadata(
        'design:paramtypes',
        ControllerCls.prototype,
        method
      ).map((_, index) => {
        const { key, type } = paramsMeta[index];
        if (type === 'route_param') return PARAM_DATA[key];
        return null;
      });
      ControllerCls.prototype[method](...paramsToPass).then(console.log);
    });
}

createApp(AuthRoute);

Putting it all together

I've tried to keep the snippets as short and easy to understand as possible. You can see the lib.ts where I've put them together with actual express. And in the index.ts there is a working web app using this 'framework'. Their you have it, a nestjs-like framework on top of express under 180 lines of code. This is how it would look like in action:

import {
  Body,
  Controller,
  Get,
  Injectable,
  Module,
  Param,
  Post,
  createApp,
} from "./lib";

class LoginDto {
  username: string;
  password: string;
}

@Injectable()
class UserRepository {
  async findOne(id: string) {
    return { userId: id };
  }
}

@Injectable()
class AuthService {
  constructor(private readonly userRepository: UserRepository) {}

  async login({ username }: LoginDto) {
    return `login successful for ${username}`;
  }

  async findUser(id: string) {
    return this.userRepository.findOne(id);
  }
}

@Controller('auth')
class AuthController {
  constructor(private readonly authService: AuthService) {}

  @Post('/login')
  login(@Body() loginData: LoginDto) {
    console.log({ loginData });
    return this.authService.login(loginData);
  }

  @Get('/profile/:id')
  async profile(@Param('id') id: string) {
    const user = await this.authService.findUser(id);
    return `user: ${user.userId}`;
  }
}

@Module({
  controllers: [AuthController],
  providers: [AuthService],
})
class AppModule {}

const app = createApp(AppModule);

app.listen(3001, () => {
  console.log('listening on port 3001');
});

It doesn't handle a lot of things of course, but it is a good starting point to understand how a framework like NestJS works under the hood.

Thanks for reading. Leave a star if you liked it.

A quick tip for frontend developers

Sharfin Jahan Sakib — Fri, 26 Mar 2021 05:39:37 +0000

When we click on any part of the chrome dev tools, the main page loses focus. It is a problem in some cases, for example when inspecting dropdowns and stuff.
To prevent this, press Shift + Ctrl + P, type Emulate a focused page, then press Enter. Now the main page will not lose focus while interacting with the dev tools.