Forem: Artlist

React Testing Simplified

Dan Jackson — Mon, 03 Oct 2022 12:47:46 +0000

Javascript testing has a tendency to become unnecessarily complex - there's nearly always multiple solutions to the same problem.

This article aims to cover our chosen route at Artlist, the thinking behind these choices, and some simple code examples to get started with React Testing Library and Jest.

Why test a web frontend?

Confidence

When creating and maintaining increasingly complex web apps, we need the confidence that our app isn't going to fall over at a moment's notice. We need assurance that future changes will not break our components.

Depending on an app's user base; preventing bugs can save a company vast amounts of money in the long-run.

Design

Writing tests forces us to think about how a feature will be used. Each valid and invalid path through a feature can be described using tests. This also means that requirements can be easily proven.

Documentation

As a developer working on a new part of a shared codebase, how do you know what each file/component does?

If written correctly, tests provide a specification of a feature. This is why tests are often saved with a .spec extension.

test('clicking save button submits the form' , () => {});

test('clicking close button closes modal' , () => {});

Jest

As Jest is the default test runner for React Testing Library, we will use this for code examples.

Jest allows us to:

describe our test suite
Create each test case
Define what we expect each test case to do

describe('complex calculations', () => {
  test('1 + 1 = 2', () => {
    expect(1 + 1).toEqual(2);
  });
});

Mocking

Jest also provides utilities to mock functions and modules. Mocks replace functionality with a function that Jest can use to check:

If a mock function was called
How many times it was called
Which parameters it was called with

const mockFn = jest.fn();
mockFn();
expect(mockFn).toHaveBeenCalled();

Mocks can also be useful in components that contain external functionality that we don't want to use in our tests. If you do need to retain functionality use jest.spyOn - this can call the original function but return a mock function.

React Testing Library

React Testing library is a set of helpers that can be used in conjunction with a test runner such as Jest to render React components and make assertions on the resulting DOM elements.

The main functions are:

render: Renders a react component
getBy*/findBy*/queryBy*: A collection of matcher functions to find elements in the DOM
userEvent: Emulate user events such as click or type

React Testing library is best suited to creating Unit and Integration tests; isolated sections of our app that can be tested individually without the influence of components higher up the tree.

Unit Test

Unit tests are best performed on pure components - components that will always give the same output given the same props. These kind of tests are usually performed on basic visual components that are used to form larger structures and sections in our app.

const Button = ({text}) => (
  <button>{children}</button>
);

import { render, screen } from '@testing-library/react';

test('button text is displayed correctly',  () => {
  render(<Button>Click Me</Button>);
  expect(screen.getByText("Click Me")).toBeInTheDocument();
});

Integration Test

An integration test proves that multiple components work correctly together. In the context of React, this usually consists of rendering a compound component, and interacting with it in some way.

import { render, screen, userEvent } from '@testing-library/react';

import handleSubmit from './handleSubmit';

jest.mock('./handle-submit');

test('can submit form with first and last name',  () => {
  render(<Form/>);

  userEvent.type(screen.getByLabelText('firstname'), 'Dan');
  userEvent.type(screen.getByLabelText('lastname'), 'Jackson');
  userEvent.click(screen.getByText('submit'));

  expect(handleSubmit).toHaveBeenCalledWith('Dan Jackson');
});

In this example, handleSubmit is an imaginary function that would be used inside our Form component. By calling jest.mock('./handle-submit'), we are replacing the actual implementation with a mock function. Without a mock, we cannot test in this way.

We think it's important to create components in a composable way; making sure that responsibility is split into child components and not aggregated in one place. You should not need to render an entire page just to test one form control.

End-to-End?

E2E testing involves testing that entire pages function correctly when interacted with. This kind of testing is outside the realms of React Testing Library, and here at Artlist we have Automation Engineers working on adding the E2E framework Playwright to our infrastructure.

Whilst E2E testing inherently provides the most overall coverage (and therefore code confidence), it is still important to test your application at various different levels. Kent C. Dodds has an Excellent Article explaining how utilising a variety of test types can be beneficial.

Dynamic Content

Components using animations or transitions can be difficult to test due to the the varying DOM output at any given time. For example, you may have an animated notification that appears after a delay, or an exit animation that doesn't remove an element from the DOM until that animation has finished.

Take Artlist's download button for example:

React Testing Library has various async utilities for this exact purpose, such as waitFor.

import { render, screen, userEvent, waitFor } from '@testing-library/react';

test('clicking download shows success notification',  () => {
  render(<Form/>);

  userEvent.click(screen.getByLabelText('download song'));
  await waitFor(() => {
    expect(getByText('Download Successful')).toBeInTheDocument()
  });
});

waitFor tells jest to wait until the assertions inside the callback are fulfilled, or the test timeout has been reached.

Why not use Enzyme?

Enzyme is another popular React testing utility - but it operates in a fundamentally different way. With Enzyme, we are given the ability to test the internals of our components, such as props, state, and lifecycle methods.

React Testing Library on the other hand, gives no access to implementation details; instead providing ways to render components and interact with them. This means a steeper learning-curve, but tests that are closer to real-world user interactions.

Conclusion

Here at Artlist we are always on the lookout for ways to make our developer's lives easier; and utilising a tried-and-true React testing framework to provide code confidence is one of the ways we are working towards that goal.

Next Steps

Check out the official React Testing Library docs for in depth explanations, a full API reference, and more advanced topics such as configuration options.

Happy testing!

How to migrate data when refactoring a monolith to microservices (with zero downtime)

Meitar Bruner — Mon, 12 Sep 2022 11:14:41 +0000

Microservices architecture is a well-known subject that is covered a lot in the tech industry.

Every person who ever tried breaking a monolith, probably found there aren't a lot of articles that speak about one of the greatest pains when moving from monolith to microservices - Data migrations.

After making some mistakes with migrations, and the fact that we decided to do it with zero downtime, I'll be glad to share some strategies, tools, and best practices we've learned for this kind of migration. Let's dive in.

First, what are data migrations?

According to Wikipedia: "Data migration is the process of selecting, preparing, extracting, and transforming data and permanently transferring it from one computer storage system to another."

Let's understand what are the risks when migrating data:

Having corrupted data (data that violates your new service expectations).
Losing data.
Creating data duplications.
The new data can contain mistakes.

The risks when switching services with the aim of zero downtime:

Data might return differently than expected.
The new service might not handle the requested rates.
New data that was created and updated won't be available anymore in the old system.

In this article, we will cover the practices to lower these risks. It is important to mention that you can probably migrate your data without all the practices I am going to cover, but keep in mind that having all of them will lower your risk to a minimum.

A. Preparations

Understand the data you are going to transfer

Learning the old entities you are going to transfer, the relationships between them, and how they will be represented in the new database is the first step. Also, it's important to understand the size (# of records & bytes) you are going to transfer.

Knowing the nature of the data will save you a lot of time when creating your new API and your migration process.

In case you designed or built your new API, you probably already did it.

If not, pay attention to the relations and the properties of the objects: do you need all the properties/columns? Will one object from the old database be represented by 2 objects in your new database? Or vice versa, a joined object from 2 SQL tables can become one object in the new database.

There are many possibilities so I won't try to cover them all. The main point is, that you should understand the data you are about to transfer and its scale, so you'll be able to create your new service properly and understand how each piece of data will be mapped to your new database.

Create the new API

An important approach I'd recommend embracing is to never access your new database without the API layer.

This is very important because accessing the database without the server layer and then using the same data from it can cause some mutations and problems you want to avoid (risk #1).

For example, your new API may have a restriction for some string property to be camelCase only, which the old system did not enforce. In that case, if you create a migration without going through the API layer, the new database will be corrupted with data that violates your new API rules.

In that case, unless you don't have your API ready, it is better to not insert new data into it.

Create your API first, better covered with DTO validations and integration/e2e tests, then move to the migration phase.

Try to be incremental as possible (1 feature at a time).

As we mentioned, the migration process comes with a risk, so it is very important to try and do it in small steps rather than migrating everything at once.

Think if you can migrate only some part of the data from the old database to the new one, so at least some endpoints from the old monolith could be replaced at the beginning.

You can lower the risk even more by starting with the feature which has the least traffic.

In Artlist for example, we had a case when we needed to migrate all data related to the user's favorite songs, artists, and albums, including folders users created for themselves.

In that case, we decided to do it step-by-step for each object and chose to start with favorite artists because this feature has the smallest traffic and was less used by users.

Consider using third-party tools

At this point, you already know the old structure and the new expected structure of your data. Plus, you already created the API which is connected to the new database, and decided about what objects you would like to migrate first to your new database.

Before you dive into implementing big guns (developing huge code implementations), this phase is a reminder that for all the next phases, it is always good to search for some known, tested third-party tools, packages, or services that already implemented what you are about to do.

Because every problem is different and many people have different stacks and cloud services, I'm avoiding pointing to a specific solution. Just remember a lot of these problems are already solved by many developers, and they might have thought about issues we haven't.

B. Get to work

Synced create/update/delete operations

Now we are starting for real. The first thing we would like to do is start writing to the new database and have it always synced with the old one. Remember we are aiming for zero downtime, so at the moment we will switch to the new service and database, it should have all the old data migrated to it.

The best way to do it is to call the new API from the old system whenever a mutation operation is happening (create, update or delete). As we mentioned above, we would like to go through the API and not update the database directly, to prevent corrupted data from being transferred.

Another important thing is from where in the old system you are calling to your new API. Having the calls as close as you can to the old database (like shown in the graphs above), will lower the risk of losing the call due to some error.

Keep in mind that now your new API will start handling requests, so it is better to verify that it can handle the rates of the old service and database.

Another great layer you can add, that will lower the risk even more, is sending the operation to some queueing component with an acknowledgment mechanism. That way you will never lose the call (risk #2) unless the new API processes it, plus you will be able to throttle calls if needed.

Create a job for history data

Good, we now have our new database populated with every new data piece inserted by real users! Also, updates and deletes are synced with the old database.

Now it's time to migrate all the data that was inserted before the operations sync.

We will need a job that can fetch data from the old database and then create an API call to insert it into the new service. Once you have this component, you can start and run it to migrate all the past data.

*It is much safer to read from a replica and not directly from the production database.

Depending on the size of your data, running the job can be time-consuming and take minutes, hours, or even days in extreme cases. Also, if you already implemented the queuing mechanism, it might be worth using it with this job as well, just create new operations to the queue you already have.

Create a mechanism to prevent duplications

It is also important to create a mechanism that prevents inserting duplications in your new service.

Let's say that your migration setup is broken, and your job creates 2 POST calls to the new API, for every 1 object from the old database (risk 3#).

A way to prevent the new API from creating duplicates is by checking if the unique identifier of the object is already located in the database, and then only if it isn't, insert it.

If this object should be able to have duplication (1 to many, many to many), you can save a property with the old ID of the object in the new database object. Doing it will let you do a quick check if this object already exists.

C. Some more protection

Verification tests

You now have your new database populated and synced with all past and present data.

Before you do the final switch from the old monolith to your new service, consider doing some verification tests. Verification tests can be created in many ways, but there are 2 that can be very useful:

Tests that validate that the databases are synced (prevent risks #1-4).
Test that validates that the end user will get the same data when you switch to the new service.

The first ones can be checked by a job, which will go over all data objects from the old database, and check if the same data exists and returns as expected from the new service.

The second one can be validated statistically with real traffic. For every GET call that fetches data from the old monolith, you can make a call to the new service, and then before/after returning the data to the consumer, send the received objects to some component that will compare them and store if the result is expected or not.

Create a backup mechanism

As a developer, you know anything might happen. That's why it is worth considering a backup mechanism.

What it means is that everything we just did from the old monolith to the new API, we can also create in the opposite direction (sync calls from the new database to the old system, history sync, etc.).

That way, in case the worst happens, we will have a mechanism to roll back to the old known system.

D. Switch and monitor

Connect the consumer to the new service

We are one step before the final switch, if you did the comparison tests, you already have the GET calls in your consumer. If not, this is the time to implement these calls to the new service along with switching the mutation calls (create/update/delete) to use the new service.

A thing worth considering in this phase, is a gradual rollout mechanism, to test what will happen to users if they'll be switched to the new service.

Rollout 100% & monitoring

That's it, we can now switch all calls to the new service. In case you have some feature flags related to a rollout, you can now get rid of them.

A new service with maybe new technology, can raise some problems we might not even think about, that's why it is important to keep and monitor the new service and the consumers using it even more during that phase.

Summary

We covered the risks, and the practices meant to lower the risks, of migrating from one service to a new service and database, while doing it with zero downtime.

Migrations can be scary, but using the practices mentioned in this article, will lower the potential risks and issues.

Even though I learned all these practices, there is a disclaimer that we didn't use all of them. Implementing them all comes with the cost of resources like time, which sometimes you do not have.

In that case, you should analyze your situation, and think which practices you can afford, and which would be less valuable.

Rings and Onion in Your Next Node Application

Raphael Ben Hamo — Sun, 07 Aug 2022 13:40:36 +0000

Sometimes we need to create a new service or refactor the existing one. Most of the time, we spend a couple of hours initializing the application - Node application setup, creating files, folders, etc. The way we structure the folders’ skeleton can be something similar to other services in our repos or based on a service we have done in the past.

This moment is highly important! If we stop and take the time to plan before starting this process, it will be worth it in the long run.

In this article, we will do it by creating the folders’ skeleton using the Clean Architecture approach.

Why software architecture is so important
About Clean Architecture
Node API Service with Clean Architecture
Folders skeleton
Summary

Why Software Architecture Is so Important

In a large project, we want it to be easy to maintain, stable (not quick & dirty), and open to new features as quickly as possible.

To achieve that, we need to separate our files and folders into components with different responsibilities that can change independently without affecting other components.

Clean Architecture

In short, Clean Architecture is a system architecture guideline proposed by Robert C. Martin (Uncle Bob).
You can read about it here and here and here.

The main concept can be shown by Robert Martin’s illustration:

Each circle represents a different area in our software. According to the dependency rule, the outer layer can depend on inner layers but not on any layer outer from it, which means that as a general rule, the deeper the layer, the less it is prone to changes.

Since this is a little bit abstract, let us demonstrate what it looks like in Node API Service.

Node API Service with Clean Architecture

We will start creating a Node service that has an API and Database.
In Clean Architecture, we will have these layers:

Domain layer - the abstract layer, responsible for all of our business data, our business functions and entities, but abstracted - interfaces and abstract classes.
Application layer - the logic layer, each file here is responsible for a flow or use case in our project.
Adapter layer - the API presentation layer, which includes routes, controllers, etc.
Infrastructure layer - database configurations, implementations of the entities, providers, etc.

Folders skeleton

The layers break down into these folders: domain, use cases, API, and infrastructure.

In the Artlist world, it can be implemented as a service for managing artists and songs.

First, let us see what it looks like for each folder:
domain - contains files such as enums, interfaces, models, etc.
In Artlist, it would be the place for artist and song properties and abstract classes of the logic for each flow.

usecases - contains files of the project logic, flow, and use cases.
Each file can present a separate flow, like "download song use-case" or all song use-cases.

api - contains files such as controllers, interceptors, guards, etc.
For Artlist, this is the API for the client side to get the artist and song data.
Here we can call from the controller 'a to use case 'a.x' and use case 'a.y'. If we do so, we make a dependency from the adapter layer to the application layer. That is ok because the dependency is from the outer layer to a deeper layer.

infrastructure - contains database configurations - table entity implementations, external providers, DAL repositories, etc.
Here we can implement the abstract classes declared in the domain layer - also, the database entities and ORM.

Now we have a basic folder structure for Node API service using Clean Architecture.

You can save it as a template repo and start other services from it.

Summary

In this article, we learned the basics of Clean Architecture. First, we translated it to the Node world, then demonstrated how to build a project skeleton for Node API service using Clean Architecture and finally showed how it can be implemented in the Artlist world.

Thanks for reading!

ML Configuration Management

Sephi Berry — Thu, 28 Jul 2022 10:40:00 +0000

This post follows our blog describing our ML Ops manifest. In this post we will dive into the our configuration management within our ML projects

Background

Working in a real-world business environment, requires moving codes between research/development, test and production environments, which are crucial for development velocity. While doing so, it is important to allow for a common language & standards between various AI & Development teams, for frictionless deployment of codes. Additionally configuration management assist in:

ML work with many parameters,
hyper params etc. ,
we want to separate config from code (12 factor app)

These days, it comes without a surprise that there are several OpenSource (OS) configuration frameworks that can be utilized for this. After reviewing several options (including Hydra), we decided on dynaconf, since it fulfilled our requirements of being:

Python based
Simple
Easily configurable and extendable
Allow for overriding and cascading

Development Environment

Artlist runs in multiple cloud environments, however currently most of the ML workloads run on GCP. Following GCP best practices, we have set up different projects for each environment, thus allowing for strict isolation between them, in addition to enabling billing segmentation. This separation needs to be easily propagated into the configuration, for seamless code execution.

In this post we review our configuration in relation to

Basic implementation
Advance Templating
Simple Overriding
Project vs. Module settings
Updating Configurations

Now let’s see how dynaconf can help out with this.

Basic Implementation

We decided to work with configuration settings that are stored in external toml files, which are easily readable and are becoming one of the de-facto standards in python.

A code snippet from our basic configuration file is as follows:

[default]
PROJECT_ID = "@format artlist-{this.current_env}"
BASE_NAME = “my_feature_name”
BASE_PIPELINE_NAME_GCP = "@jinja {this.BASE_NAME | replace('_', '-')}"
BUCKET_NAME = "@format {this.BASE_NAME}--{this.current_env}"

[dev]
SERVICE_ACCOUNT="service-account1@artlist-dev.iam.gserviceaccount.com"

[tst]
SERVICE_ACCOUNT="service-account2@artlist-tst.iam.gserviceaccount.com"

[prd]
SERVICE_ACCOUNT="service-account3@artlist-prd.iam.gserviceaccount.com"

Now let's break it down.

Whenever dynaconf runs - it runs in a specific environment. The default environment is called DEVELOPMENT. However, since we wanted to move easily between the environments (and GCP resources), we changed the naming convention of the environments (to a 3 letter acronym = dev, tst, prd), so we can readily reference the relevant GCP project while specifying the environment.

Using the env_swithcher, we can indicate to dynaconf which configuration to load and what GCP project to access with the following line:

PROJECT_ID = "@format artlist-{this.current_env}"

Using the @format as a prefix to the string, we can parse the parameter that is within the curly brackets. For example, if the current environment is set to ‘dev’ the PROJECT_ID variable will be artlist-dev, thus accessing only the resources from the dev project, whereas if the environment is set to ‘prd’ the PROJECT_ID will be artlist-prd.

Accessing the rest of the relevant variables is based on the various sections in the toml file.

For example, referencing the Production Service Account (SA) will be by accessing the SERVICE_ACCOUNT variable which is under the [prd] section

Advance Templating

Dynaconf includes the ability to work with Jinja templating - this can be useful for manipulating strings. GCP has a quark that requires naming containers within the GCP registry so as not to have ‘_’ (underscore) as separators, but rather ‘-’ (hyphen). And since we wanted to sync our registry and the artifacts coming out of Vertex AI pipelines (that are stored within buckets / Cloud Storage), we were able to keep the python naming convention of ‘_’ , while converting the strings to the GCP convention when required.

Using the jinja’s text replace method we can easily alter the text as necessary:

BASE_PIPELINE_NAME_GCP = "@jinja {this.BASE_NAME | replace('_', '-')}"

Simple Overriding

Another useful feature of dynaconf is that you can easily override the configuration using local settings. This is very convenient since local settings for development doesn’t need to be checked into source control, while the general settings should be synced to the entire team.

All that is required to differentiate between the settings is to add the .local suffix to the file name, see example below:

General Settings - settings.toml
Local Settings - settings.local.toml

Whenever dynaconf identifies the suffix .local.toml it will overwrite the variables configuration that exists in the settings.toml with the loaded settings.local.toml file.

An example for overwriting with local credentials

Project vs. Module settings

Our ML framework is KubeFlow (hosted by GCP VertexAI pipelines), which requires various configurations: some at the component level (which are reused independently in various pipelines), while others are at the pipeline/project/cross-component level. To load both settings, we can use another feature of dynaconf which can define a specific file name template that will automatically be loaded by dynaconf. Here is our implementation practice:

Any configurations that are at the project level will be written in the project settings - settings_prj.toml (see dynaconf settings_files configuration)
Any configurations that are at the component level will be written in the component settings - settings.toml .

Updating Configurations

Sometimes there is a need to update the configuration during runtime, this can be challenging since the entire configuration is loaded immediately when the library is called. To do so, we can use a decorator to update the configuration. Assuming the cfg is the configuration settings, we can write the following decorator:

@input_to_config(cfg)
def input_to_config(config, sequence_override=True):
   """[decorator] override config parameters with function inputs
   Args:
       config: Dynaconf configuration / settings to be updated
       wrapped_func ([function]): [the function to capture it's input and push to the config]
       sequence_override: configures the option for overriding keys or merging the values
   """

   def decorator(wrapped_func):
       @wraps(wrapped_func)
       def inner(*args, **kwargs):
           _override_config(kwargs, config, sequence_override=sequence_override)
           return wrapped_func(*args, **kwargs)

       return inner

   return decorator

Summary

In this blog post, we have laid out our configuration implementation using dynacof library. We saw how we

Used the basic setup of dynaconf configuration
Synced our GCP project with dynaconf environments
Worked with the advanced dynaconf settings

In our next posts, we will extend our description of the various elements that have been incorporated into our ML project workflow, while developing our internal base library, which include standardization of:

Logging
Accessing our secrets (using GCP Secret Manager)
Conduct our experiment tracking (using ClearML )

# Image copyright

The banner image was co-created using DALLE2

Lessons learned on the road to MLOps

Amit Bendor — Thu, 07 Jul 2022 09:49:45 +0000

The start

Hello and welcome to our series of posts on MLOps at Artlist!

The goal of this series is to share our MLOps journey (which is still undergoing) and how we applied our vision and perspective to a real-world production infrastructure.

In the very early days of our department, when I (Amit), just joined the company and was the first Data science employee, I had the rare opportunity to stop and think with myself about how to "build things right" this time.

After managing a few data science teams and projects, and seeing failures and success stories I felt like I had a quite clear vision of the “values” which should guide us as we build our infrastructure and practices.

It took me about 2 days of just distilling and spilling out into a notion page all of my thoughts and we were ready to go.

At that time the buzzword “MLOps” was rising in popularity and we felt like it has many similar notions to ours - although some of them are vague in terms of implementation.

Core values

A battle of ultimate goals

Before starting with values we need to understand our main goals. they will be our "guiding star" whenever we have a decision to make.

If I had to choose 1 superior goal it would be “business impact”. Quite a high level but eventually we want to bring value to our users and to impact the company’s bottom line.

Also, we can measure it for every decision we're taking - for example what features to focus on next? which implementation to choose for our pipeline tool? we can answer all of those with an estimation of the business impact.

In order to get to this goal - we can say we’d like to be:

Independent - can bring value without relying on the development teams or data engineers (ideally)
Focus on DS - in order to bring our unique value to the company, we need to do mostly data science/research work and not general engineering tasks.

As you can figure out these two goals are conflicting with each other - and this is exactly the fine line each team needs to define. We defined our own guidelines as you’ll see below.

The modern Data Science team

One of our first decisions was to build the team the “modern way”. And by that I mean we decided we’re not going for the traditional way algorithm teams used to provide some dirty research code - and letting a software engineer decipher and convert it into a running production code.

We decide to rather strive towards a “Full Cycle Data Science” paradigm, where we take ownership over every activity related to our core data science activity.

What does it means exactly? What are the bounds?

We call it the “Up to the API” approach- all activities from research, to automation using pipelines and holding APIs to externalize our predictions.

What not?

Any operational backend/frontend work
Data engineering ETLs

Turning “values” into infrastructure

Ok, so now let's move on to the most important subject - values.
You might be asking - so the rest of the articles just going to be a bunch of clichés? I promise not. let’s see how we took every “value” and created actionable decisions & guidelines we use every day.

Enabling creativity

What does it means?

Inspired by our company, we believe that researchers should “push the boundaries” - and that our MLOps infrastructure should enable it.

We truly believe we should enable the Maximum flexibility in the research side of our work - because this is our core value in the company.

Decisions

Research with notebooks/scripts

👉 As long as it supports fast iterations - use Jupyter notebooks, scripts, or a combination of both

CYOF (Choose your own framework)

👉 Prefer Tensorflow over PyTorch - not a problem. Use the framework/tools that make the most sense to you

Simplicity

What does it means?

We already talking about conflicts so here is another one.

We don’t want to activate our human decision-making process for operations that are outside of our core contribution.

Therefore, we decided to give minimal flexibility on the “engineering side” of our work.

Decisions

One implementation ^TM

👉 minimal set of tools and just one implementation for 3 variants of our main components: pipeline, API, Python package

Example: for API use only FastAPI

Write once

How is it being reflected?

Write code templates (cookiecutters in Python) - to provide a fully functional environment - full of nifty automations
- Use it when starting every project
- Look below the links section for a few examples
- Deep dive post is coming up soon
Standard operations - a library we developed covering “must-haves” of any script or notebook. (production or research)
- Establish a good standard
- Includes: Logging, configuration, and tracking modules
- Deep dive post is coming up soon
Everything is containerized
- A standard way to deploy our code in different services without changing anything (!)

Transparency (of tools & infra)

What does it means?

This one is connected to the previous value. (simplicity)

We want things to “just work” without touching them when possible.

We rather sacrifice flexibility and cost

Decisions

No DevOps

👉 We’ll prefer serverless, self-managed, cloud-native solutions. Even in the cost of flexibility or cost of operation

👉 For example we chose vertex pipelines as our main pipelines tool and GCP native monitoring for those features.

Fewest interactions - we chose libraries and implementations that don’t require the least lines of code to run

Example:

from aistdops.logging import logger logger.info()

over

logger = Logger(), logger.info()

How is it being reflected?

Cloud - prefer managed services, serverless
Choice of transparent libraries - for example, ClearML for experiment tracking as it logs most things *implicitly *

Robustness + Well engineered

What does it means?

So frankly, we really don’t like waking up in the middle of the night from a pager duty call.

Therefore, we’re doing all we can to provide testable, robust solutions

Decisions

Invest in testing code, data validation, and model monitoring
Choose production-ready tools
Track everything we can
1. To build a lineage map, reproducibility
Reduce production risks as possible
1. Prefer batch jobs over online APIs
2. Always have a fallback

How is it being reflected?

Data validation - with great expectations
Tracking experiments with ClearML
Embrace best practices in core components - configuration management, logging, dataset and model management

What's next?

In the next articles in this series, we'll go deeper into our core implementations of those values into our infrastructure.

About the writer

Amit is the head of data science at Artlist. He’s an active contributor to the Association of software architecture and Cloud security alliance organizations.

If you google his name, you’ll see he is talking about technology at every given chance.

Co-hosting the award-winning podcast “Osim Tochna” in Israel, recording videocasts, and speaking at conferences and meetups.

When he is not running into walls with the VR headset, he is spreading the world of AI for developers with cloudaiworld.com

Building a team's culture

Asaf David — Wed, 11 May 2022 11:37:20 +0000

Twenty years ago, I was a platoon commander in a classified unit in the army (that is no longer classified). Back then, making decisions and applying authority was straightforward. Officers were to make most decisions, pushing them down the command chain through non-officer commanders all the way to the soldiers. Decision enforcement in that ecosystem was called "a command."

There was no place for discussion once a decision (command) was made, and the expectation was that it would get fulfilled right away, or else…
This thick layer of "simple" soldiers who are untrained in the military decision-making process and unaware of much of the intel requires a top-down and highly hierarchical management approach. Add to that combat conditions where soldiers put their life at risk, and the need for such a hierarchy with high discipline only intensifies.

Fast forward to a year ago. As I approached building my R&D team at Artlist - my first position as a team leader in the tech industry - I was trying to define my approach to managing and leading such a team. Obviously, there are many differences between leading an infantry team and a team of software developers, so you can already understand that much of the conclusion would be very different from what I described above.

In this article, I'll try to lay out some of the dilemmas I had and insights I gained regarding decision-making and authority as a team leader in R&D.

Before I dive in, I think some context is needed about the company I work for, my position, and my personal history.

In 2016, Artlist was founded as a bootstrap to enable video creators to find high-quality music for their videos with a revolutionary licensing subscription. Since then, it has grown tremendously, acquiring two companies and reaching 300 employees at the beginning of 2022. It has also changed its goal to support the whole content creator economy, not only video creators.

As for my position, my team is in charge of two of the company's web applications, artlist.io and artgrid.io, and while writing this article, we are shifting into a group-of-teams structure, so my title is changing from team lead to group manager :-)

Now here's an abridged version of my personal history to give you the whole picture. Besides being in the army, before becoming a software developer, I had a pretty enjoyable career in the food & beverages industry, mainly in management positions such as managing a bar, leading the on-trade market in one of the biggest wine & spirits importers in my country while managing 12 employees and a few million dollars of sales per year, running my own restaurant (and losing all my money), and finally establishing and leading a new online wine & spirits e-commerce site.

Now that we've set the context, let's discuss the key factors that have impacted my approach to management. I'll also star* the insights I have derived from these factors and will try to construct from them my holistic culture and attitude as a manager.

Team = Leverage

The basic idea of a team is leverage. Yes, I can write code, but I'm limited by my own velocity, knowledge and experience. Leading a team enables you to accomplish more, thanks to an increased workforce and, more importantly, the increased shared knowledge and expertise. As such, the team's ability is not only defined by its members' velocity but also by the type of knowledge each member might hold.

There are a few key factors to maximizing your team's leverage.

Diversity*

When you have a diverse team, each member's added value holds great significance. Conversely, when all team members are the same, the added value of a single team member is lost.

Of course, there are many other benefits of having a diverse team, but since this is not the main topic of this article, I'll leave it aside and perhaps address it more in-depth in the future. I'll also leave aside the option of growing your team's size, as it is more of an external solution.

Velocity

Another vital factor in optimizing your team's leverage is increasing each team member's velocity, and I find that two main actions increase an individual's production capability:

A personal development plan* focusing on areas that simultaneously benefit the employee and the company. Such a plan should be defined and owned by the team member with the full support of their manager, and I will take the story of one of my team members to illustrate this point. Parish is a great front-end developer and has worked at Artlist for several years. However, despite his experience, Parish lacked ownership skills, resulting in a tendency to push things to production with insufficient quality, thus introducing new bugs that required him to go back to his developments repeatedly. When one of our QA engineers told me that he needs to run some QA on the site every time Parish deploys to production, the severity of the situation became clear to me. Once I had Parish understand the problem and its impact on the company, we raised some suggestions for improving the quality of his work and avoiding future issues. Long story short, a month and a half later, the feedback I got from external stakeholders was how satisfied they were with Parish's work, and he had reduced his QA cycles to almost zero. While this example might seem simplistic, I find it to be the general case in most PDPs, both for technical and soft skills.
Frequent candid feedback* - This is quite a big topic on which many books have been written, so I'll try to avoid going deep into this rabbit hole and simplify it by stating the following: When we receive frequent, ongoing candid feedback, we can alter our behavior and actions in a way that helps us grow at a faster pace. And the compound interest of this growth holds tremendous value to us and our surroundings. I'll just add a few tips for increasing feedback loops in your team:

a. Start with the why - Show them the benefits they can reap from it.

b. Do it gradually and through a controlled process until you are 100% sure you can expand it.

c. Teach the team how to give feedback and how to receive it.

d. Practice a lot with the team. For example, when someone gave another team member some feedback, I asked someone else to give feedback on how the feedback was given. Then I asked for feedback on the feedback's feedback and so on. It was excellent practice for providing feedback.

Agility

While Artlist has just grown to more than 300 employees, it still has a dominant start-up approach, focusing on innovation and our ability to react to changes and decisions as fast as possible. This point is crucial in defining the culture of my team as it's a key factor in the success or failure of any start-up. To emphasize this point, we can look at the last year from my team's point of view. Product-wise, we started by taking ownership of three websites - artlist.io, artgrid.io and the Artlist jobs site. But since then, we ended up rebuilding Artlist's primary site (artlist.io), are now working on a significant product change, and are starting to build a new product from scratch.
But the product side of things is only one part of the story. Agility is required from an HR perspective. In fast-paced start-ups, the need to adjust the organizational structure is an ongoing process. For example, only a year ago, I had zero team members, while today, my team consists of 16 members and is about to break into several groups and add more team/group members.
So what does this all mean to our forming culture? First and foremost, team members must be flexible and adjust quickly to any new initiative or structure. To enable this, they need to have a good understanding of the high-level plan and goals. They also have to be connected to the "why" at all times and understand the reasons for each task and what we aim to achieve.
I found that there are two things I can do to support this behavior - high transparency* and high ownership*. While the former is relatively straightforward and means that team members are familiar with the decisions as well as the decision-making process, the latter is very significant and not always obvious. Perhaps explaining how I give tasks to a team member will shed some light here. Here's the basic rule: Provide enough information to enable your team member to make their own decisions on how to complete the task. Then, understanding the big picture, why this task is needed and how it helps us, they will make wiser decisions and take ownership of the mission along the way.

Creativity

Start-ups are highly reliant on continuous innovation, which is synonymous with creativity. But how does one create an environment that increases a team's creativity or, at least, its creative results?
These questions have already been answered in the highly recommended book 'No Rules Rules' by Erin Meyer and Reed Hastings, in which they describe how much of this can be achieved by reducing the rules* you apply to your employees, increasing their responsibility, ownership, decision-making, and problem-solving. This isn't an easy solution, and I'll address it later on in this article.
Increasing each member's part in the whole team's decision-making, transparency, and putting more power and influence in their hands* will also increase the creativity and inspire them to question and criticize anything we do.

Knowledgeable workers

Unlike most of my prior management experience - leading soldiers, sales representatives, bartenders, and so on - which didn't require any formal education or high analytical skills of my subordinates, software developers tend to be pretty damn smart. Some have completed a demanding degree in CS, while others are more autodidacts, but they all have high knowledge and skills in software development. Such workers require a specific management approach to get the most out of them. And by "getting the most out of them," I mean not only from the company's point of view but also the workers' perspective, as they tend to focus on self-growth and learning. This brings me back to several points I've already discussed before in this article, such as having a personal development plan* and ownership* . It also means that the team members can have a highly active part in the team's decision-making process*. Such decisions can vary from mission-oriented to prioritization to the culture itself.

Recap

So let's take a quick look at the key points we have gathered so far regarding team building and management:

Diversity
Personal development plan
Frequent candid feedback
Transparency
Ownership
Rules Reduction
Putting more power and influence in the individual hands
Make them an active part of the team decision-making process

Trying to connect the dots, I think the first thing that jumped into my mind was that many of these points are focused on empowering each member. Some even overlap, and others were considered for management only in my low-tech experience.
That made me rethink what it meant to lead such a team. How much should the leading approach be egalitarian vs. hierarchical? If my team consists of such intelligent people, how much of the decision-making should I make?
Here are my conclusions:

The team as a whole is smarter than any single member, including me.
This doesn't mean that I, as team leader, won't be making decisions. But it does mean that in many cases, my role should be enabling the team to make decisions in a healthy manner, rather than making them on my own.
Such an approach can result in better decisions and empower the team simultaneously.
Much of my focus should be on establishing good communication within the team.
This point, which is far from easy to achieve, serves many aspects of a team.
It helps the team reach better decisions.
It creates better alignment, which reduces problems and time waste, thus increasing velocity.
It enables frequent candid feedback between team members, which helps each of them improve and grow, thus increasing the team's leverage.

Looking at these two conclusions places the manager in a new position. If back in the day, I would have thought that the leader's place was on top of the pyramid, making the decisions and controlling everything; nowadays, I believe the leader's best spot is actually between the team members, forming the "glue" that connects all the members and guides them in the right direction.

Aftermath

A year forward, looking at my team, I have to say that we are on the right track.
Our team is highly diverse and comprises 15 members from Israel, the UK, Russia, Belarus, Bangladesh and one newcomer from Ukraine (celebrating a year in Israel now).
Many of the team's decisions are made by its members, from prioritization through processes to culture. Evidence for that was my recent three-and-a-half week-long vacation, during which the team performed amazingly well as if I was not needed at all (note to self: take long vacations more often).
Open, candid feedback between all members is taking place, much is still to be done in this area, but we are heading in the right direction.

There are still many challenges ahead of us in implementing this culture, and most of this thought process is taken with the team, and soon probably with my new team leads. It seems that each of the factors of this approach requires more analysis as you grow with it. For example, diversity requires more thought about adjusting each member's culture to the team culture.
But I am sure that with the current state of mind combined with the super talent of each team member, we as a team will achieve all that and more.

Forem: Artlist

React Testing Simplified

Why test a web frontend?

Confidence

Design

Documentation

Jest

Mocking

React Testing Library

Unit Test

Integration Test

End-to-End?

Dynamic Content

Why not use Enzyme?

Conclusion

Next Steps

How to migrate data when refactoring a monolith to microservices (with zero downtime)

First, what are data migrations?

A. Preparations

B. Get to work

C. Some more protection

D. Switch and monitor

Summary

Rings and Onion in Your Next Node Application

Table of contents

Why Software Architecture Is so Important

Clean Architecture

Node API Service with Clean Architecture

Folders skeleton

Summary

ML Configuration Management

Background

Development Environment

Basic Implementation

Advance Templating

Simple Overriding

Project vs. Module settings

Updating Configurations

Summary

# Image copyright

Lessons learned on the road to MLOps

The start

Core values

A battle of ultimate goals

The modern Data Science team

Turning “values” into infrastructure

Enabling creativity

What does it means?

Decisions

Simplicity

What does it means?

Decisions

How is it being reflected?

Transparency (of tools & infra)

What does it means?

Decisions

How is it being reflected?

Robustness + Well engineered

What does it means?

Decisions

How is it being reflected?

What's next?

About the writer

Building a team's culture

Team = Leverage

Diversity*

Velocity

Agility

Creativity

Knowledgeable workers

Recap

Aftermath