Forem: jzfrank

HFDP(13) - Patterns in the real world (a review and a summary)

jzfrank — Mon, 10 Jun 2024 08:06:56 +0000

What is a pattern? Simply put, a pattern is a solution to a problem in a context. Note the context should be a recurring situation. Patterns are a collection of wisdom to solve problems.

Review

Let's take a brief review of what we have learnt so far:

Decorator: wraps an object to provide new behavior
State: encapsulates state-based behaviors and uses delegation to switch between behaviors.
Iterator: provides a way to traverse a collection of objects without exposing its implementation.
Facade: simplifies the interface of a set of classes.
Strategy: encapsulates interchangeable behaviors and uses delegation to decide which one to use.
Proxy: wraps an object to control access to it.
Factory method: allows a client to create families of objects without specifying their concrete classes.
Adapter: wraps an object and provides a different interface to it.
Observer: allows objects to be notifies when state changes.
Template method: Subclasses decide how to implement steps in an algorithm.
Composite: Clients treat collections of objects and individual objects uniformly.
Singleton: ensures one and only one object is created.
Abstract factory: allows a client to create families of objects without specifying their concrete classes.
Command: encapsulates a request as an object.

There are more patterns for sure, but they are probably the most popular ones.

Category of Patterns

There are many ways to classify patterns. Understanding the categories help you memorize them and communicate them better.

One way is to partition patterns into creational, behavioral, structural pattern.

Creational Pattern: involves object instantiation. Provides a way to decouple a client from the object it needs to instantiate.
Behavioral Pattern: is concerned with how classes and objects interact and distribute responsibilities.
Structural Pattern: lets you compose classes or objects into larger structures.

Another way is to classify them into class patterns and object patterns.

Class Patterns describe how relationships between classes are defined via inheritance. Relationships in class patterns are established at compile time.
Object Patterns describe relationships between objects and are primarily defined by composition. Relationships in object patterns are typically created at runtime and are more dynamic and flexible.

Remarks

Do not use patterns just to use patterns. Always prefer simplicity (KISS: keep it simple). If a problem could be solved without using patterns, prefer easier solution. Only use patterns whenever you see benefits (e.g. You see a practical change might happen, which could be solved by using patterns).
Refactoring is usually patterns time. e.g. Many conditional statements may hint for State pattern.
Patterns are a common vocabulary that accelerate communication among developers.

Anti-patterns

Anti-patterns tell you a bad solution to a problem. Why it could be tempting. And what should be the right direction.

See anti-patterns.

HFDP(12) - Compound Pattern

jzfrank — Wed, 29 May 2024 16:21:30 +0000

Compound Pattern is to combine two or more patterns to solve a general problem. One of the most famous compound pattern is the MVC (model view controller) pattern.

MVC involves Model, View, and Controller. When user touch a button, it will invoke controller to do certain things. The controller then invoke the model to modify its state. Finally, the model will notify the view that something changes. The view will be rendered accordingly.

Learning MVC from top down is a bit involving. However, learning it from bottom up is very simple. MVC is nothing but a combination of patterns.

The controller and the view are using the Strategy pattern. The view does not care the concrete behavior, it just invokes the controller. A concrete controller is just an implementation of an interface and could be easily replaced by another implementation.

The model and the view are using the Observer pattern. The model is the Observable and the view is the Observer. The model could notify the subscribed view the changes of its status. The observer view will update the view accordingly.

The view itself is using the Composite pattern. Consider the relationship between a window and a button. They are both visual elements but have an inclusive relation. A view could be a standalone or could contain other views.

Sometimes we want to reuse a controller. Often we need to use Adapter pattern to adapt the interface.

MVC pattern is so useful that it gets adapted and used in many popular frameworks. In the web world, a thin client approach is where the model, most of view, and the controller all reside in the server, while the browser renders the view. Single Page Application approach is all the model, view and controller reside in the browser. Most frameworks lie in between the two extremes.

HFDP(11) - Proxy Pattern

jzfrank — Tue, 28 May 2024 09:37:39 +0000

Proxy manages and controls access. Proxy is a stand in for a real object. Proxy pattern has a lot of variations.

For example, if we want to call a method of an object living remotely in another JVM, we need to make use of the Remote Proxy. In java, we could achieve this by using rmi technology. Basically, the client calls a stub that implements the same Remote interface as the real subject. The stub then forwards the request to the skeleton living in the JVM, which then calls the real subject and forwards back returned value. Through the time, the client thinks it is accessing the real subject, which it is not doing this directly.

Another well-known example is the Virtual Proxy. It is used for wrapping an expensive object like an image fetched from a network. Let's say you want to display an image on screen. Instead of fetching the image and manages the waiting yourself, you could delegate the task to a virtual proxy. It will fire a thread to fetch the image, and renders waiting texts while waiting. In this manner, you reduced the coupling in code.

We may also need to manage access, which brings the Protection Proxy. It wraps an object and decides if the client could access certain methods of it. In Java it is implemented via reflection. e.g.

Person getOwnerProxy(Person, person) {
  return (Person) Proxy.newProxyInstance(
    person.getClass().getClassLoader(),
    person.getClass().getInterfaces(),
    new OwnerIncovationHandler(person));
}

where OwnerIncovationHandler implements InvocationHandler. Its method signature is Object invoke(Object proxy, Method method, Object[] args).

We've seen a number of patterns that serve as wrapper. They look similar but serves different purposes:

Decorator Pattern: wrap an object add behaviors
Facade Pattern: wrap possible many objects and simplify interfaces
Adapter Pattern: wraps an object and implements another interface
Proxy Pattern: wraps an object and controls access.

Proxy pattern has many variations:

Firewall Proxy (control access to network recourses) -> e.g. corporate firewall system.
Smart Reference Proxy (provide additional actions when a subject is referenced e.g. count)
Caching Proxy (temp store expensive resources) -> web server proxies and content management and publishing systems.
Synchronization Proxy (provides safe access to a subject from multiple threads)
Complexity Hiding Proxy (pretty much like Facade)
Copy-on-Write Proxy (lazy copying, variant of Virtual Proxy)

Set up a local mysql server on Mac with prisma

jzfrank — Sat, 21 Oct 2023 14:00:22 +0000

mysql is one of the most popular dialects of SQL. And prisma is an ORM tool that helps developer interact with database safer and better.

In this post, we will discuss 1. how to set up a local mysql server 2. how to use it with prisma

Set up a local mysql server

Download mysql

Go to mysql website and choose downloads

Choose community version

Choose community server

download the version matching your Operation system

Click No thanks, just start my downloads

Install mysql

Create a new Database

Finish install guide.

Now, open terminal and type

mysql -u root

hit enter, you should see:

Now, create a database with commands CREATE DATABASES, here, we use myFirstDB

Check if database is created successfully: SHOW DATABASES

Now that we have successfully created the database, it's time to connect to it with prisma

Connect to DB with prisma

I assume you have already installed prisma in your project. If not, go to setup-prisma and set it up.

Now, what remains is to connect to our local database.
mysql connection string should be of format:
mysql://USER:PASSWORD@HOST:PORT/DATABASE

3306 is the default port. In our case, we don't have a password, we are using root, connect to db myFirstDB. So the connection string is simply mysql://root@localhost:3306/myFirstDB

In schema, prisma, for datasource, use

datasource db {
  provider = "mysql"
  url      = env("DATABASE_URL")
}

In .env file, add

DATABASE_URL="mysql://root@localhost:3306/myFirstDB"

Now you are good to go! To check if it actually works, open the database with sql db visualization tools like Beekeeper Studio:

click import from url and

connect

Now, you see no table, which is expected.

To test, we add a dummy schema to schema.prisma

model User {
  id   String @id @default(cuid())
  name String
}

run

npx prisma migrate dev --name init

in command line, this should generate db schema for you.

Check Beekeeper studio

We have successfully initialized the local database with prisma!

That's all for today, happy hacking!

Set up data science development: from zero to hero

jzfrank — Thu, 22 Jun 2023 12:03:19 +0000

When it comes to data science, python with jupyter-lab seems to be the go-to choice.

However, when we start a new project, it always costs us half an hour or two to configure the right environment. Even just launching jupyter-lab and install our kernel to it.

In this tutorial, I give a quick step-by-step guidance on setting up a new data science development environment.

First, make sure anaconda or miniconda is installed.

Then, create a new python environment using

$ conda create -n myenv python=3.9

Now you create a new environment called myenv with python version 3.9.

Next, activate the environment

$ conda deactivate

$ conda activate myenv

You should see something like this in terminal:

(myenv) $

Next, install jupyter-lab

(myenv) $ conda install -c conda-forge jupyterlab

As the last step, install current kernel to jupyterlab

(myenv) $ python -m ipykernel install --name myenv --display-name "myenv-display-name"

Now, you can run

(myenv) $ jupyter-lab

A new window of jupyter-lab will pop up.
OK, you are all set! Let's do real data science! ;)

How to Set up a Local mysql Docker and View it using TablePlus

jzfrank — Wed, 21 Jun 2023 18:50:50 +0000

Create an empty folder. Open the folder.

Create two files: docker-compose.yml and docker.env.

In docker-compose.yml file:

version: "3.7"
services:
  db:
    image: mysql:8.0.33
    env_file: docker.env
    ports:
      - "3306:3306"

Then in docker.env file:

MYSQL_ROOT_PASSWORD=pass
MYSQL_DATABASE=mydb
DB_NAME=mydb
DB_HOST=db

Make sure docker is opened. Run

docker compose up --build

Now your mysql instance is running with connection string mysql://root:pass@127.0.0.1:3306/mydb.

To view the database, open TablePlus, create a new connection with above connection string.

Big Data for Engineers - Chapter 2: Lessons Learned from the past

jzfrank — Sun, 05 Mar 2023 10:22:44 +0000

Data independence

The fundamental socle on which modern data management is based (Edgar Codd 1950).

Data independence means the independence of the logical view on the data and physical storage.

A database management system stack is four-layer:

A logical query language
A logical model
A physical compute layer that processes the query
A physical storage layer where the data is physically stored

In the past two decades, the physical layer has changed: clusters with thousands of machines rather than a single machine. However, the look and feel of querying data remain the same to the end user.

Formal prerequisites

Sets and relations

set, inclusion: are trivial concepts
Cartesian product: AxB
relation: a relation R on a family of sets is a subset of their cartesian product.
partial function: A partial function p between two sets A and B is a relation is a relation that does not associate any element of A to more than an element of B
function: a partial function that is "injective"

Sets commonly used:

natural integers, relative integers, decimals, rational numbers, real numbers, a set of all values

Rational database management systems (RDBMS)

main concepts

RDBMS is based on a tabular data format. Its first class citizens:

table: a collection of records
attribute: a property that records can have
row: a record in a collection. Also called: record, entity, document, item, business object
primary key: attribute(s) tha uniquely identify a record

formalism behind relational model

Tables can be defined formally (as purely mathematical objects)

Relational integrity

A collection T fulfills relational integrity of all records have identical support.

Domain integrity

The values associated with each attribute should be restricted to a domain.

Atomic integrity

values are only atomic values: the collections does not contain any nested collections or sets or lists or anything has a structure of its own.

If these constraints are relaxed, we enter the world of NoSQL databases:

Relational Algebra

Selection

Projection

Grouping

Renaming

Changes the name of a column

Extended projections

Compute values and assign results to a new column.

Cartesian products

Joins
Join is a "filtered Cartesian proect" in which we only combine directly related tuples and omit all other non-matching pairs.

Normal Forms

Anomalies might occur in poorly designed databases. To avoid anomalies, best practice is to follow normal forms.

First normal form: atomic integrity

Second normal form: each column in a record contains information on the entire record. A table should be only about one type of thing. Each non key field must be about the same thing as the primary key.

Third normal form: forbids functional dependency on anything else than the primary key

SQL language

SQL is a declarative language. SQL is also a functional language (to some extent).

Example

SELECT c.century AS cent,
       COUNT(c.name) AS num_captains,
       SUM(s.id) AS ships
FROM captains c, ships s
WHERE c.id = s.captain
GROUP BY century
HAVING COUNT(c.name) >= 3

Internally:

Languages

Transactions

ACID:

Atomicity: either an update is applied completely, or not at all
Consistency: Properties of database stays the same
Isolation: the system "feels like" the user is the only one using the system, however in fact thousands are using concurrently.
Durability: all data written to database is durably stored and will not be lost (e.g. electricity shortage or a disk crash)

Scaling up and out

If we have a lot of rows: object storage, column storage, distributed file systems, massive parallel processing.

If we have a lot of columns: column storage.

If we have a lot of nesting: syntax, data models, document stores, querying.

So, if you do not fully understand normal forms, 3 integrities, and ACID. It is OK for this course, we will very soon break these conventions and enter the Big Data world.

HFDP(10) - State Pattern

jzfrank — Mon, 27 Feb 2023 21:40:28 +0000

Suppose you are to maintain a gumballMachine, which takes quarters, gives gumball after turning the crank. How would you implement it? The gumball machine certainly behaves differently when in different states (no quarter, inserted one quarter, sold, sold out etc.). Well, we can certainly use if else clauses. However, if new functionalities might be added (for example, randomly generate a winner that gets two gumballs), we can already foresee the code will be messy to maintain.

A natural solution to problem like this, where the object behaves differently under different context, could be the State Pattern.

Formal Definition

The State Pattern allows an object to alter its behavior when its internal state changes. The object will appear to change its class.

We say "appear" because the object cannot change its class after it is initialized. However, it does appears to change its class, as it will behave differently under different context. This is because its state has changed. The way we implement that is to maintain state inside the object, and switch it whenever needed.

Our Context object will have an internal member that maintains State. When request() is called, we are calling state.handle() under the hood. The magic is the State interface. We will have concrete implementation of this interface, and implement handle() differently under different states.

As a side note, we may see similarity between State Pattern and Strategy Pattern. Both of them delegate execution to another composition class (remember our Duck app?). However, the State Pattern is meant for different behaviors of an object under different states. The strategy pattern, in my humber opinion, is like saving us from implementing tons of classes that differ only by a set of behaviors (Don't get me wrong, all of them have the behaviors. They just behave differently).

Big Data for Engineers - Chapter 1: Introduction

jzfrank — Fri, 24 Feb 2023 16:00:17 +0000

Magnitude is an interesting concept. We have a good feeling about measuring real world objects:

Our height -> meter
Driving Cars -> kilometers
Size of Earth -> megameters
Distance of Earth to Sun -> gigameters
Distance of Jupyter to Sun -> terameters
Entire Solar System -> petameters ......

However, we seem to fail to appreciate the magnitude of data: we take 1GB, 1TB for granted, without thinking what a large number that is.
In terms of bytes, approximately:

kB = 1,000 B
MB = 1,000,000 B
GB = 1,000,000,000 B
TB = 1,000,000,000,000 B
PB = 1,000,000,000,000,000 B
EB = 1,000,000,000,000,000,000 B
ZB = 1,000,000,000,000,000,000,000 B
YB = 1,000,000,000,000,000,000,000,000 B

kilobyte, megabyte, gigabyte, terabyte, petabyte, exabyte, zettabyte, yottabyte. Isn't that impressive?

To understand the importance of learning big data, we discuss 3 V's.

3 Vs of big data

Volume

The volume of data stored is increasing exponentially. In 2021, it's around 100ZB.

Variety

Data exists in various forms:

trees (JSON, XML)
unstructured (text, pictures, audio, video)
cubes
graphs (neo4j, Oracle PGX)

Velocity

How fast can we process data? Three factors affect the velocity: capacity, thoughput, latency.

From 1956 to 2021, capacity has increased by a factor of 200,000,000,000, throughput by 10,000, latency (decreases) by 150. So the throughput and latency cannot catch up with the capacity.

However, if we do parallel or batch processing, we may reduce that gap.

HFDP(9) - Iterator Pattern and Composite Pattern

jzfrank — Tue, 21 Feb 2023 09:36:27 +0000

In this post we discuss two patterns: the Iterator pattern and the Composite Pattern.

Iterator Pattern

Intro

Suppose you are merging two restaurants. Each has their own menus and implemented differently. Say one uses ArrayList, another uses Array. How would you iterate over the menus? We may of course do a type check and treat them differently. However, this will expose the internal representation and seems not so generic. Do we have a better approach? The Iterator Pattern comes to our rescue.

Instead of exposing the internal representation, we may define a method called createIterator() that returns an Iterator. In such manner, we could use the returned Iterator interface hasNext(), next().

Formal Definition

The Iterator Pattern provides a way to access the elements of an aggregate object sequentially without exposing its underlying representation.

That is, the client should deal with two interfaces: the Aggregate and the Iterator. And the Aggregate should have a method called createIterator which returns an Iterator.

Composite Pattern

Intro

Suppose now you are merging more restaurants, whose menus have submenus. Now we have recursive structure here. Instead of treating menu and menuItem separately, we may use Composite Pattern to treat them transparently/uniformly. We may define an interface MenuComponent and let Menu and MenuItem implement that.

Formal Definition

The Composite Pattern allows you to compose objects into tree structures to represent part-whole hierarchies. Composite lets clients treat individual objects and compositions of objects uniformly.

What's more, since our Composite often contain many subcomponents, we may implement a createIterator() method. This is the connection between Iterator Pattern and Composite Pattern.

HFDP(8) - Template Method Pattern

jzfrank — Tue, 14 Feb 2023 14:22:56 +0000

Intro

Many of us are living on caffein beverages. Some of us drink coffee, some of us drink tea. Check out the steps needed to make a coffee:

Boil some water
Brew Coffee in boiling water
Pour coffee in cup
Add sugar and milk

and steps to make a tea:

Boil some water
Brew tea in boiling water
Pour tea in cup
Add lemon

We can already see there exists an underlying similar algorithm for the two processes, namely:

Boil water
Brew
Pour in cup
Add condiments

Of course, we may add different condiments to tea and coffee. That should be left for the subclass to implement.

So yes, this is it! Abstract the steps into a template method (encapsulating algorithms), then leaving specific step(s) in the algorithm to the subclasses. This is the intuition for our Template Method Pattern

Definition

The Template Method Pattern defines the skeleton of an algorithm in a method, deferring some steps to subclasses. Template Method let's subclasses redefine certain steps of an algorithm without changing the algorithm's structure.

When we implement this, we usually declare the template method as final, and to-be-implemented methods as abstract. We also define one or many hook method, which are optional for the subclasses to implement.

Design Principle

The Hollywood Principle

Don't call us, we'll call you.

This principle states that one should avoid the circular dependencies between up and low level components. -> Just let the high-level component controls the low-level components, but not the converse.

Examples

Template Method pattern is particularly useful in frameworks.

Java Arrays Sorting

Say we have an array of self-defined classes Duck, to use Sorting provided by Java, we need to make Duck implement Comparable interface and implement compareTo method.

Swing JFrame

JFrame's update algorithm calls paint(), But by default paint() does nothing. It is a hook and can(should) be overwritten.

HFDP(7) - Adapter and Facade Pattern

jzfrank — Sat, 21 Jan 2023 15:10:03 +0000

In this post we will discuss two pattens: Adapter Pattern and Facade Pattern. Why we put them in the same post? Because their purpose is kinda similar: both "converting" something unsuitable/complex into something suitable/simple.

Adapter Pattern

Imagine you are on travel. You travel many countries: U.S., U.K., Switzerland, China etc. You realized that different areas are using different sockets. What do you do? You use adapters.

This is exactly the idea of Adapter Pattern. Say you were using the java's interface Enumeration. However, as java gets updated in new versions, it has a new interface Iterator. You want to use Iterator object in your code, however, instead of changing everything, you want to keep old code as much as possible. What do you do? You write an adapter to adapt Iterator to Enumeration.

An example of implementing:

public class IteratorEnumeration implements Enumeration { 
    Iterator iterator;

    public IteratorEnumeration(Iterator iterator) { 
        this.iterator = iterator;
    }

    public boolean hasMoreElements() { 
        return iterator.hasNext();
    }

    public Object nextElement() { 
        return iterator.next();
    }
}

Formal Definition

The Adapter Pattern converts the interface of a class into another interface the clients expect. Adapter lets classes work together that couldn’t otherwise because of incompatible interfaces.

In fact, there are two kinds of adapter patterns: object adapter and class adapter. While the object adapter pattern achieves this by passing object, the class adapter pattern achieves this by multiple inheritance.

Facade Pattern

Imagine you are setting your personal home movie theater. You bought the equipments, you settled the locations, you connected the internet, you bought a popcorn machine, and you even bought Netflix. OK, everything seems cool. Let's sit down and enjoy our well-earned leisure of watching movie. Now what do you need to do? You need to 1) turn on the projector 2) close lights 3) turn on popcorn machine 4) open netflix and search for target movie 5) set volume to medium (say 15) etc. ... Hey, why it is so complicated? What's worse, when we want to quit, we need to undo everything. Wouldn't it be nice if we have a remote controller that turns on everything once we hit "on" mode, turns off everything once we hit "off" mode. This will need the facade pattern, which tries to provide a simplified interface for a large and complex subsystem. The facade pattern provides intuitive interfaces and reduces coupling with the subsystems.

Formal Definition

The Facade Pattern provides a unified interface to a set of interfaces in a subsystem. Facade defines a higher- level interface that makes the subsystem easier to use.

Design Principle

Principle of Least Knowledge - talk only to your immediate friends.

Facade Pattern is a good illustration of this principle. No matter how complex the subsystem is, if the client code only talks to the simplified interface (the facade), things will become easier and coupling will be reduced.

Using this principle, we should not call methods on returned objects. For example,

Now it comes to the end of this post. We have discussed adapter pattern and facade pattern. Further, we introduced principle of least knowledge in light of facade pattern. Adapter Pattern: When we have two similar interfaces, we use Adapter Pattern if we want to use one pattern as another pattern. Facade Pattern: When we have a overly complex subsystem while we want simplified interface, we use Facade Pattern.

See you next post!