Forem: Furkan Kalkan

Sorting Multilanguage Text Properly on OpenSearch

Furkan Kalkan — Fri, 15 Nov 2024 07:11:36 +0000

If you have multi-language or non-English content and use OpenSearch, default sort method will not sort the content alphabetically. Default sort method use Unicode values of characters in order to sort texts alphabetically, it works in English but fails in most non-English languages. OpenSearch documentation did not address this problem. In Elasticsearch documentation there is plugin named analysis-icu mentioned to solve this issue ¹. This plugin supported by OpenSearch too ². There is not much information about OpenSearch specific version of this plugin but usage is same as Elasticsearch one:

Install the plugin on each node(s):
```
/usr/share/opensearch/bin/opensearch-plugin install analysis-icu --batch
```
You can use init containers method to make this if you use Kubernetes. Don't forget the mount the plugin directory /usr/share/opensearch/plugins/ on both container.
After installation of plugin, restart your nodes.
Add sort subfield to your fields ³ :

{
  "mappings": {
    "properties": {
      "title": {   
        "type": "text",
        "fields": {
          "sort": {  
            "type": "icu_collation_keyword",
            "index": false
          }
        }
      }
    }
  }
}

You can add language and country parameters after type if your content in single language. Also you can add numeric: true parameter to sort numbers in text in correct order.

Since our subfield used only for sorting, use "index": false to turn off indexing of field.

How I manage multiple kubernetes clusters w/ kubectx & kubens

Furkan Kalkan — Tue, 01 Oct 2024 15:27:37 +0000

Kubernetes is a powerful platform for automating the deployment and scaling of applications. It has become the industry standard for managing server clusters. Managing a single Kubernetes cluster is pretty straightforward. You put your cluster configuration in a directory (usually ~/.kube) and start using kubectl. But if you have to manage multiple kubernetes clusters for reason it can be quite mess to track them.

I used to switch between clusters by renaming config directories or modifying the KUBECONFIG environment variable, but I realized that it's not sustainable.

I recently saw kubectx project on Github and I decide to use it.

Step 1: Merging kubernetes configs

I merged my kubernetes configs like below and put it on ~/.kube directory.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <certificate authority data>
    server: https://<cluster1-ip>:6443
  name: my-cluster-1
- cluster:
    certificate-authority-data: <certificate authority data>
    server: https://<cluster2-ip>:6443
  name: my-cluster-2
contexts:
- context:
    cluster: my-cluster-1
    namespace: <default-namespace-name>
    user: my-user-1
  name: cluster-one
- context:
    cluster: my-cluster-2
    namespace: <default-namespace-name>
    user: my-user-2
  name: cluster-two
current-context: cluster-one
kind: Config
preferences: {}
users:
- name: my-user-1
  user:
    <add auth config here>
- name: my-user-2
  user:
    <add auth config here>

Step 2: Installing kubectx & kubens

Installing kubectx & kubens tools are easy.

You can install with sudo apt install kubectx in Debian/Ubuntu or brew install kubectx in MacOS. More install methods described on README file. I also installed fzf package for easy navigation using arrow keys.

It is possible to switch between clusters using kubectx and between namespaces using kubens commands. Once namespace selected, you don't have to use -n option to define namespace kubectl, all bare kubectl commands applied on selected namespace.

Step 3: Setting the powerline to tracking clusters and namespaces

I love powerline over traditional PS1 (aka command prompt). It's beautiful and highly customizable.

You can refer the powerline documentation for install and configure powerline itself.

I use powerline-k8s plugin to see currently active kubernetes cluster and namespace in prompt.

You can install it using pip:
pip install powerline-k8s

After install I add this lines to bottom of groups section ~/.config/powerline/colorschemes/default.json

...
        "k8s":           { "fg": "solarized:blue", "bg": "solarized:base2", "attrs": [] },
        "k8s_namespace": { "fg": "solarized:red",  "bg": "solarized:base2", "attrs": [] },
        "k8s_context":   { "fg": "solarized:blue", "bg": "solarized:base2", "attrs": [] },
        "k8s:divider":   { "fg": "gray4",          "bg": "solarized:base2", "attrs": [] }
...

Final result:

You may want to use https://github.com/jonmosco/kube-ps1 for more lightweight alternative.

Why you should use environment managers to manage multiple versions?

Furkan Kalkan — Mon, 30 Sep 2024 23:03:58 +0000

Different projects has different needs. Different versions of libraries, different versions of SDKs, even different versions of compilers/interpreters... But installing and managing multiple versions of them is huge pain.

How should install that:

Compiling from source ??

Official repositories (you're lucky) ??

Adding some questionable (!) PPAs ??

Homebrew, nuget, ... ??

Congratulations, your computer has turned into the Amazon rainforest over the years, complete with its own delicate ecosystem.

Compiling from source code

It's quite popular method for installing software in Unix and Linux. But properly removing or upgrading source-compiled software is usually hard. There is a high chance that after a long time you will not remember where and how you installed it. Another issue is handling symlinks and dependencies conflicts.

So, don't do it yourself :)

Using official repositories

Using own repositories of distro or official repositories/packages is better solution. It handles removing, upgrading, symlinks and dependencies conflicts for you. Sometimes they permit installing different versions of compilers/interpreters simultaneously.

Unfortunately, my experiences was not shiny with this method. Removing packages in Debian package system may gone bad in some cases, especially when they're part of meta-packages (don't know about RPM, I'm not use it).

Another issue is managing the versions. In modern Linux distros, lot of application (including some of system applications) written in Python. They use system Python by default. Installing multiple versions of Python may break them. Installing multiple versions of NodeJS, Golang or Java may break your installed applications, too.

Using PPAs or 3rd party package managers

They have similar advantages and disadvantages with using official repositories. In additions to them using 3rd party PPAs or package managers like Homebrew or Nuget may risky due supply-chain attacks. You don't simply trust random guy's PPA blindly, promoted in blog post or forum post. Don't put risk you and your company.

If your project needed very specific version, you may not find any repository or package for that. In this situation, you should package it or compile it yourself.

Using environment managers

We actively use environment managers pyenv, goenv and nvm for managing multiple versions of Python, Golang and NodeJS at Mantis. These provide us with an easily manageable, robust development environments.

Using environment managers has several advantages over above methods:

Less chance of breaking system
Easy removal of unneeded version
Versions usable in both system-wide or project wide
Can install, remove and manage all versions in one place
Predictable installation path

Caveats of using environment managers:

Versions installed with environment managers still interference with system versions and break applications
You may need install required dependencies manually before installing versions
In rare conditions environment managers can broken and solving this may need long debugging sessions or re-installing environment manager itself and all installed versions

Automate everyday boring tasks with scripting

Furkan Kalkan — Tue, 11 Oct 2022 12:32:10 +0000

The average day of a programmer is often filled with boring tasks such as bulk operations. Today I share my tips/tricks about how I automate this kind of tasks easily with help of REPLs.

Usecase 1: I have long list of record ids and want only show/update them in a database table.

with open('id_list.txt') as f:
    id_list = f.read().split('\n')
    print(f"select * from my_table where id in {tuple(id_list)}")

tuple data type represent in Python as comma seperated values between parantheses that perfectly fit my need of where in query here.

Usecase 2: I have to migrate one table to another in PostgreSQL.

In this case I have to migrate the old database tables to new ones. Column names may different, some columns may removed or remapped and new table may have new columns that contains aggregated data from other columns/tables.

I created views/materialized views using old tables that map new ones. And write little sql script that return insert queries from output of materialized view as:

select format('insert into table_v2 (field_a, field_b, field_c, ...) values (%L, %L, %L,...);', field_a, field_b, field_c,..)
from table_v2_mw

%L format string handles quotation and null values for us here.

This method is super useful for applying one shot complex changes and report them. You can attach exported list of insert/update queries to Jira.

Usecase 3: I hate spreadsheets.

Excel/CSV files are industry standard if you work with data. Sometimes I need to manipulate them and spreadsheet softwares usually sucks, even in simple tasks. Instead of fighting with bloated nested IFs and non-sense functions, I prefer convert them in CSV format and manipulate them in Python. Also, Python can handle really big CSV files with help of Pandas.

with open("some_file_revised.csv", "w+") as csv_out:
    with open("some_file.csv") as csv_in:
        csv_reader = csv.DictReader(csv_in)
        csv_writer = csv.DictWriter(csv_out, fieldnames=csv_reader.fieldnames)
        csv_writer.writeheader()
        for row in csv_reader:
            # Manipulate data here
            row["foo"] = row["foo"][4:]
            csv_writer.write(row)

Quick Hack: Converting MathML to LaTeX

Furkan Kalkan — Tue, 02 Jul 2019 21:45:45 +0000

Recently, I need to convert some MathML codes in article metadata from SCOAP3 to LaTex format. Most of institutional repositories escapes XML entities, so MathML doesn't render correctly. I tried the Wiris' API but it's very slow and give errors in most of long formulas.
Finally, I found Yaroshevich's XSL Schema that works without problem.

Example Python code:

import lxml.etree as ET
import re

def to_latex(text):

    """ Remove TeX codes in text"""
    text = re.sub(r"(\$\$.*?\$\$)", " ", text) 

    """ Find MathML codes and replace it with its LaTeX representations."""
    mml_codes = re.findall(r"(<math.*?<\/math>)", text)
    for mml_code in mml_codes:
        mml_ns = mml_code.replace('<math>', '<math xmlns="http://www.w3.org/1998/Math/MathML">') #Required.
        mml_dom = ET.fromstring(mml_ns)
        xslt = ET.parse("mmltex/mmltex.xsl")
        transform = ET.XSLT(xslt)
        mmldom = transform(mml_dom)
        latex_code = str(mml_dom)
        text = text.replace(mml_code, latex_code)
    return text

Fully automated metadata objects with Python 3.7's brand new dataclass library.

Furkan Kalkan — Sun, 23 Sep 2018 19:49:45 +0000

Dataclass is a brand new data structure which featured in Python 3.7. Recently @btaskaya write about it a great article. If you hadn't read yet, you can read on here.

Dataclass has promising features to create reusable, self-verified and automated metadata objects. Before that, I used to use dict format to create metadata objects but copying and pasting the same object all the time is boring and conflict with against DRY (Don't Repeat Yourself) rule.

It was like this:

Metadata = {}
Metadata["id"] = id
Metadata["url"] = url

if something:
    Metadata["some_field"] = some_data

Metadata["media"] = {}
Metadata["media"]["id"] = media_id 
...

I can use NamedTuple or something instead of dict but they have some limitations and I really didn't have enough time to implement something fancier in the early days of the project. When I refactor code I realize that dataclass is more functional for my needs.

In this article, I will show you how to implement fully automated metadata objects with dataclasses step by step.

Part 1: Implement metadata fields that don't need calculation

There is no problem at all in this step. It's just standard implementation.

from dataclasses import *


@dataclass
class Metadata:
    title: str
    url: str
    created_at: str = None    # Fields may have default value

Part 2: Add some fields that need calculation and let's calculate it automatically.

This fields will get values only after calculations. In our case, post_id should equal the random number plus url.

import random
from dataclasses import *
@dataclass
class Metadata:
    # Normal fields
    title: str
    url: str
    created_at: str = None
    # Calculated fields
    post_id: str = field(init=False)
    def __post_init__(self):
        random_number = random.randint(100000, 999999)
        self.post_id = f"{random_number}_{self.url}"

__post_init__ function will calculate our field post_id after initiliasion.

Let's call it:

>>> Metadata(
...  title="Some Article",
...  url="https://example.com/article.html",
...  created_at="2018-09-23"
... )
Metadata(title='Some Article', url='https://example.com/article.html', created_at='2018-09-23', post_id='696953_https://example.com/article.html')

Gotcha!

Part 3: Make our hands dirtier; add `__post_init__` only pseudo fields

We may want to build autonomous complex structures. For instance, if one field annotated, dataclass can build the whole substructure for us. In our case, we use additional fields author_names and author_ids to construct authors field as list of dict. If author information not provided for the article, the value of authors field should be None.

import random
from dataclasses import *


@dataclass
class Metadata:
    # Normal fields
    title: str
    url: str
    created_at: str = None
    authors: list = None
    # Calculated fields
    post_id: str = field(init=False)
    # Non-nullable Pseudo fields
    author_names: InitVar[list]
    author_ids: InitVar[list]

    def __post_init__(self, author_names, author_ids): # You have to pass pseudo fields as the parameter.
        random_number = random.randint(100000, 999999)
        self.post_id = f"{random_number}_{self.url}"
        self.authors = []
        for i in range(0, len(author_names)):
            self.authors.append({"id": author_ids[i], "name": author_names[i]})

Let's call it:

>>> Metadata(
...  title="Some Article",
...  url="https://example.com/article.html",
...  created_at="2018-09-23"
... )

TypeError: non-default argument 'author' follows default argument.

It didn't work:(

Important Note: You have to group default and non-default fields.

Try again:

import random, json
from dataclasses import *


@dataclass
class Metadata:
    # Non-nullable Pseudo fields
    author_names: InitVar[list]
    author_ids: InitVar[list]
    # Normal fields
    title: str
    url: str
    created_at: str = None
    authors: list = None
    # Calculated fields
    post_id: str = field(init=False)

    def __post_init__(self, author_names, author_ids): # You have to pass pseudo fields as the parameter.
        random_number = random.randint(100000, 999999)
        self.post_id = f"{random_number}_{self.url}"
        self.authors = []
        for i in range(0, len(author_names)):
            self.authors.append({"id": author_ids[i], "name": author_names[i]})

    def to_json(self):
        json.dumps(asdict(self))

Let's call it again:

>>> Metadata(
... title="Some Article",
... url="https://example.com/article.html",
... created_at="2018-09-23",
... author_names=["Furkan Kalkan", "John Doe"],
... author_ids=["1", "2"]
... )
Metadata(title='Some Article', url='https://example.com/article.html', created_at='2018-09-23', authors=[{'id': '1', 'name': 'Furkan Kalkan'}, {'id': '2', 'name': 'John Doe'}], post_id='692728_https://example.com/article.html')

Yeah!

But wait... Where the author_names and author_ids are gone?

Note: Pseudo fields that InitVar instance, only used in __post_init__() as parameters, they are not a part of object.
>> Metadata.author_names
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'Metadata' has no attribute 'author_names'

Part 4: We don't need to define `author_names`.

We can make pseudo fields as optional, too.

import random, json
from dataclasses import *


@dataclass
class Metadata:
    # Non-nullable Pseudo fields
    author_ids: InitVar[list]
    # Normal fields
    title: str
    url: str
    created_at: str = None
    authors: list = None
    # Nullable Pseudo fields
    author_names: InitVar[list] = field(default=None)
    # Calculated fields
    post_id: str = field(init=False)

    def __post_init__(self, author_names, author_ids): # You have to pass pseudo fields as the parameter.
        random_number = random.randint(100000, 999999)
        self.post_id = f"{random_number}_{self.url}"
        if author_names:
            self.authors = []
            for i in range(0, len(author_names)):
                self.authors.append({"id": author_ids[i], "name": author_names[i]})

    def to_json(self):
        json.dumps(asdict(self))

Call it:

>>> Metadata(
... title="Some Article",
... url="https://example.com/article.html",
... created_at="2018-09-23",
... author_ids=["1", "2"]
... )
Metadata(title='Some Article', url='https://example.com/article.html', created_at='2018-09-23',authors=None,post_id='692728_https://example.com/article.html')

Part 5: We need JSON.

Python objects are good but we need to dump it as JSON to POST it to web services, MQs, etc. Dataclass library has builtin function asdict() which can dump our object to dict.

Let's write the wrapper for our object.

import random, json
from dataclasses import *


@dataclass
class Metadata:
    # Non-nullable Pseudo fields
    author_names: InitVar[list]
    author_ids: InitVar[list]
    # Normal fields
    title: str
    url: str
    created_at: str = None
    authors: list = None
    # Calculated fields
    post_id: str = field(init=False)

    def __post_init__(self, author_names, author_ids): # You have to pass pseudo fields as the parameter.
        random_number = random.randint(100000, 999999)
        self.post_id = f"{random_number}_{self.url}"
        if author_names:
            self.authors = []
            for i in range(0, len(author_names)):
                self.authors.append({"id": author_ids[i], "name": author_names[i]})

    def to_json(self):
        return json.dumps(asdict(self))

Check it:

>>> m = Metadata(
... title="Some Article",
... url="https://example.com/article.html",
... created_at="2018-09-23",
... author_names=["Furkan Kalkan", "John Doe"],
... author_ids=["1", "2"]
... )
>>> m.to_json()
{"title": "Some Article", "url": "https://example.com/article.html", "created_at": "2018-09-23", "authors": [{"id": "1", "name": "Furkan Kalkan"}, {"id": "2", "name": "John Doe"}], "post_id": "466969_https://example.com/article.html"}

Part 6: Remove unnecessary fields from json.

We want to remove None valued fields from json except the url field. It's possible with a little bit of change:


def to_json(self):    
    metadata = asdict(self)
    for key in list(metadata):
        if key != "url" and metadata[key] == None:
                del metadata[key]
    return json.dumps(metadata)