Forem: Vladislav Filinkov

Recruiting is broken. Let’s fix it!

Vladislav Filinkov — Mon, 22 May 2023 08:51:18 +0000

Introduction:

Job hunting can feel like a wild roller coaster ride, full of peaks, valleys, and unexpected turns.

Resume Format Frustrations: Hours spent polishing your resume can seem pointless if it doesn't fit a specific format. The ATS (Applicant Tracking System) has no room for non-conformity, just like many recruiters. It feels like wrestling with mismatched puzzle pieces.
Endless Applications: Job applications can feel like an eternal marathon with no end in sight. It seems you're scattering applications like confetti, but responses are as elusive as a lucky clover. The process seems to prioritize quantity over quality, leaving you questioning if your painstakingly crafted cover letters ever get read.
The Ever-changing Resume: Adapting your resume for each job application can feel like you're continuously dressing up your qualifications to fit the mold of the day, often spending more time on this than on actual work-related tasks.
The Feedback Void: After application and interview, you're often left waiting in silence. Feedback is a rarity, making the job hunt feel like casting calls into the abyss, with no echoes to guide you.

Navigating job hunting can be arduous, but I hope this ~~humorous~~ look at its challenges brings you a smile. It's essential to step back sometimes, have a laugh, and remember that this roller coaster ride is a shared experience. Happy job hunting!

What if we can solve these problems?

Imagine a job-hunting utopia requiring minimum effort. Although we can't address all problems but we can surely tackle a few.

Create a resume whatever you want. Describe everything you think is important, feel free about the format and size of description.
Do not need an application anymore - just upload a resume and wait! Wait until our app finds the best fit for you.
Now you will know exactly what positions your resume is relevant to, and in which open positions you are in what "place" compared to other candidates. The level of openness can be radically different.

The Solution:

The solution revolves around using contextual embeddings and vector similarities.

Embeddings:

Embeddings are mappings of discrete variables to a vector of continuous numbers. They are low-dimensional, learned continuous vector representations of discrete variables within a neural network. Their uses include finding nearest neighbors in the embedding space, serving as input to a machine learning model, and for visualization of concepts and relations.
Contextual embeddings assign each word a representation based on its context to solve the issue of contextual meaning.

In a nutshell, our solution involves these steps:

We generate embeddings for both CVs and job descriptions.
We calculate the Euclidian distance between the CV and Jobs (many-to-many)
This distance indicates the match level between a CV and a job description.
We identify the top-X CVs for each job and top-X jobs for each CV.
These lists represent the best-fitting candidates for each job and the most suitable jobs for each CV.

Dataset example:

resumes = [
    {
        "id": "1",
        "name": "John Doe",
        "description": "Software engineer with 5+ years of experience in Python and JavaScript."
    },
    {
        "id": "2",
        "name": "Jane Doe",
        "description": "Digital marketing specialist with a focus on SEO and content creation."
    },
…

jobs = [
    {
        "id": "1",
        "name": "Software Engineer",
        "description": "Seeking a Software Engineer with 5+ years of experience in Python and JavaScript."
    },
    {
        "id": "2",
        "name": "Digital Marketing Specialist",
        "description": "Looking for a Digital marketing specialist with a focus on SEO and content creation."
    },
…

High-level concept:

Embedding calculation
In the next segment, I'll share a code example that illustrates how you can generate embeddings using two leading models: Ada from OpenAI and BERT from Hugging Face. I won't be going into detail about choosing the ideal model for your specific needs. Nonetheless, from my investigations, it's clear that in terms of accuracy, OpenAI's model significantly outshines BERT, as well as several other models. Here, we measure accuracy using the chat CV, where we expect the model to precisely identify relevant job ads based on a distance metric.

The Python code snippet provided below outlines how to compute embeddings using the OpenAI Ada model and the Hugging Face BERT model:

import openai
from transformers import AutoTokenizer, AutoModel
import torch

openai.api_key = '<OPENAI_API_KEY>'

#  OpenAI API embeddings
def openai_embedding(text, model="text-embedding-ada-002"):
   text = text.replace("\n", " ")
   return openai.Embedding.create(input = [text], model=model)['data'][0]['embedding']

# Bert Transformer embeddings
def bert_embedding(text):
    model_name = "bert-base-uncased"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModel.from_pretrained(model_name)

    # Tokenize and encode the text
    input_ids = tokenizer.encode(text, return_tensors="pt", max_length=512, truncation=True)

    # Get the embeddings
    with torch.no_grad():
        output = model(input_ids)
        hidden_states = output.last_hidden_state

    # Calculate the mean of the token embeddings to get the sentence embedding
    sentence_embedding = torch.mean(hidden_states, dim=1).squeeze()
    return sentence_embedding


# Load the CV to variable
cv = Path('cv.txt').read_text()
cv = cv.replace('\n', '')

# Create embeddings for the CV
cv_embed_openai = openai_embedding(cv)
cv_embed_openai = bert_embedding(cv)

Euclidian distance
Fantastic, we're now ready to move on to the next step: calculating the Euclidean distance between two vectors. However, keep in mind that a singular distance value doesn't provide much insight. For instance, a distance of 12432 - is it a significant or minor distance? This is where the power of relativity comes in. We should calculate the distances between numerous CVs - say 10, 100, or 1000 - and select the top 10 or top 50 CVs that are closest to our job description. The more comparisons we make, the more precise our decision will be regarding the relevance of a CV to the open position.

Now, let's take a look at a basic outline of our architecture

The first step is to establish collections (tables) for the CVs and job descriptions. We will employ MilvusDB, an open-source vector database that is user-friendly and easy to configure. All you need to do is grab the pre-configured Docker compose, and you're good to go! I've decided to omit the part about saving the matching results to a relational database like PostgreSQL, as it's not essential for demonstrating our solution.

Here is an example of how to set this up in Python:

def collection_create(collection_name, dimension):
    if utility.has_collection(collection_name):
        utility.drop_collection(collection_name)

    fields = [
        FieldSchema(name='id', dtype=DataType.INT64, descrition='Ids', is_primary=True, auto_id=True),
        FieldSchema(name='name', dtype=DataType.VARCHAR, descrition='name', max_length=200),
        FieldSchema(name='description', dtype=DataType.VARCHAR, description='description', max_length=20000),
        FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, description='CV embedding', dim=dimension)
    ]
    schema = CollectionSchema(fields=fields, description='Datasets descrption')
    collection = Collection(name=collection_name, schema=schema)

    index_params = {
        'index_type': 'IVF_FLAT',
        'metric_type': 'L2',
        'params': {'nlist': 1024}
    }
    collection.create_index(field_name="embedding", index_params=index_params)
    print(f"Collection '{collection_name}' has been created.")

def openai_embedding(text, model="text-embedding-ada-002"):
   return openai.Embedding.create(input = text, model=model)['data'][0]['embedding']

def data_insert(collection_name, data):
    collection = Collection(name=collection_name)
    for row in data:
        ins=[[row['name']], [row['description']], [openai_embedding(row['description'])]]
        collection.insert(ins)
        time.sleep(1)  # Free OpenAI account limited to RPM
    print(f"Inserted {len(data)} objects to the {collection_name} collection.")

And the last part is execution.
Here are a few important considerations to keep in mind:

The process of searching for relevant open positions for a CV and vice versa is technically challenging and represents a classical high-load problem. However, in this discussion, we're not delving into that aspect.
For our demonstration, we're selecting the top three most relevant jobs. However, in practice, you can choose any number that suits your needs. The more CVs and job descriptions you have, the more relevant the results will be, giving you flexibility in adjusting this variable.
It's crucial to understand that our approach to relevant search is based on the entire description, rather than just tags, keywords, or other simplified elements. This approach offers the freedom to create a solution that is highly productive and adaptable to your specific needs.

if __name__ == "__main__":
    # Connection to the Milvus DB
    connections.connect(host='localhost', port='19530')
    print("Connected to Milvus DB...")

    # Create collections for resumes and jobs
    collection_create('resumes', 1536)  # The dimension of your embeddings should match the output of the model used. Here it is assumed to be 768.
    collection_create('jobs', 1536)

    # Insert the data into the respective collections
    data_insert('resumes', resumes)
    data_insert('jobs', jobs)

    # Demonstrate a search
    cv = resumes[0]['description']  # Let's say we're searching for jobs that match the first resume
    name = resumes[0]['name']
    print(f"Let's find related jobs for {name}")
    print(f"His CV is {cv}")
    print(f"Search results:")
    print(search('jobs', cv, 3))  # The 3 most relevant jobs will be printed

Connected to Milvus DB...
Collection 'resumes' has been created.
Collection 'jobs' has been created.
Inserted 10 rows to the resumes collection...
Inserted 10 rows to the jobs collection...
Let's find related jobs for John Doe
His CV is Software engineer with 5+ years of experience in Python and JavaScript.
Search results:
[{'name': 'Software Engineer', 'description': 'Seeking a Software Engineer with 5+ years of experience in Python and JavaScript.', 'distance': 0.07440714538097382}, {'name': 'Content Writer', 'description': 'Seeking a Content Writer specializing in tech and software development topics.', 'distance': 0.3115929365158081}, {'name': 'IT Consultant', 'description': 'Looking for an experienced IT consultant with a background in network security.', 'distance': 0.3668628931045532}]

Link to the python source code

Additional Considerations:

Recruiters typically search for candidates with "relevant experience," which is a significant aspect of their decision-making process. To cater to this, we could match individual sections of a CV, each representing different periods of a person's work history, instead of the entire CV. This approach would allow us to identify candidates whose most recent job aligns perfectly with what the recruiter is looking for, even if their overall CV isn't as relevant. Consequently, this would significantly boost such candidates' chances of being considered for the position.

In conclusion, envisioning this as an open-source project could indeed be a game-changer, aiming to solve the longstanding issues in recruitment in a more effective manner. By eliminating opaque mechanisms and introducing transparent and open algorithms and results, we can revolutionize the recruitment process.

Thank you for joining us on this exploration! I hope you found this article insightful. Enjoy the rest of your day!

Team Topologies - A New Way To Organize Your Teams

Vladislav Filinkov — Wed, 17 Aug 2022 14:19:44 +0000

From One To Many Teams

Have you ever organized productive work of a team? Then you probably started by asking yourself a set of questions like “How many people do you need?”, “What roles would they have?” and “What organizational structure should it have?”. Perhaps, you might think that a team can organize themselves the way they want because they are smart and independent professionals. Maybe, you would like use some framework to be aligned to corporate approach or be more transparent to management.

But what if your organisation has several teams, more that 3 or 5 or, maybe, 8, do you need to use use some framework to allow these teams work together? What if each team has their own internal framework? Teams need to interact with each other. Sometimes they need do cross planning, sometimes they need some kind of coordination to be sure that everyone is not doing the same feature, or features from different teams should be integrated.

To be sure that every team works on their own problem and teams openly collaborate with each other and at the same time are aligned to the company’s goals. You have to find the way of organisation these team interactions.

The Problem

I like the concept of of self organization of a team. Each company’s goal is pretty individual with their individual solution and the way to achieve this goal is unique too. It is impossible to find one-solution-for-all-teams approach it impossible in the higher level - teams interaction organisational framework.

What to do in this situation?

Of course you can find some framework which exists already and try to use it in your organisation. But there are no guarantees that any of these frameworks will fit your organisation or culture, especially if you are a startup where you don’t have much time to focus on the process rather the technology or product or your customers.

Team Topologies

My experience said will be better presented with some kind of lego-like framework, where you have prepared building blocks but you can build whatever you want. Build your own framework for your company specifically!

But what kinds of building blocks should they be?

I think it pretty simple and clear, we have to have Types of Teams and Types of their Interaction. Types of teams help understand what this team doing in general, internal team rules and processes are not important because you also have Interaction Type, like an API for a micro-service, when you don’t have to know how exactly this service works internally, but instead you just have to know what API uses as input and as an output - same for the team.

This approach is called Team Topologies and this is something that can can help to adapt your working process for your company, department and whatever you want.

Team Topologies is a clear, easy-to-follow approach to modern software delivery with an emphasis on optimizing team interactions for flow.

There are four fundamental types of teams - Team Topologies and three core team interaction modes combine with awareness of Conway’s Law, team cognitive load, and responsive organization evolution to define a no-nonsense, team-friendly, humanistic approach to building and running software systems.

Devised by experienced IT consultants Matthew Skelton and Manuel Pais, the Team Topologies approach is informed by the well-known DevOps Team Topologies patterns (also authored and curated by Matthew and Manuel). Matthew and Manuel have worked with many organizations around the world to help them shape their teams for modern software delivery, and Team Topologies is the result of this experience.

The combination of different types of teams and their Interaction modes

I’m not trying to explain the whole methodology, you can do this using materials form the authors I just want to explain why, from my perspective, this a very interesting approach. This approach of organizing work is so natural (and well structured) that I like it very much.

And last thing which I find to be extremely interesting is that you don not even need to design the teams and the interaction they should have. What you need to integrate these approach into your company is just let your teams choose the type and interaction mode that they would like to you and you will be super close to self-organisation approach, just because you already have the necessary blocks.

Thank for reading this article till the end, I hope you’ll enjoy these methodology and it will help your teams and your company to focus not on processes but on building products and make the world a better place!!