Forem: chamodperera

Replacing For Loops with Vectorization in Python

chamodperera — Tue, 31 Jan 2023 03:11:15 +0000

Loops are an integral part of programming languages, as they allow for the repetition of a block of code a specific number of times or until a certain condition is met. However, loops can be slow and resource-intensive, especially when the operations inside the loop are repeated many times. This can be a disadvantage when working with large datasets, as loops can limit performance.

One way to improve the performance of these types of operations is through a technique called Vectorization. With this approach, operations can be performed on entire arrays or datasets at once, rather than looping through each element individually. This can be much more efficient, particularly when working with large datasets or when the operations are repeated many times. Vectorization is a useful tool for tasks such as data analysis, machine learning, and financial modeling. In this article, let's see how this technique can enhance the performance of these types of operations.

What is Vectorization

Vectorization is a technique that allows for faster computation by storing and manipulating data in an array or vector format rather than as individual units. The fundamental structure for storing and manipulating data in a vectorized manner is the N-dimensional array, or ND array, which is a data structure that represents a multi-dimensional grid of values.

Vectorized operations enable element-wise operations to be performed on entire ND arrays using a single function call, rather than iterating over the array and applying the operation to each element individually. When working with ND arrays that have more than two dimensions, vectorized operations can still be used by processing each element in the array one at a time. For example, with a 2D array, you can iterate over the rows and process one row at a time.

for  i  in range(len(A)):
  B[i] = A[i] * 2

This loop multiplies each element in the array A by 2 and stores the result in the corresponding element of B.

# Vectorized implementation
B = A * 2

This vectorized operation performs the same operation as the loop, but the operation is performed on the entire array at once.

Getting Started with NumPy

NumPy is the most used library for working with vectorized operations in Python. It provides N-dimensional arrays, a wide variety of vectorized operations, and an easy-to-use API. NumPy also provides functions for performing vectorized operations on ND arrays, including mathematical operations, statistical operations, and operations related to sorting, searching, and indexing. These features make NumPy well-suited for working with vectorized operations on multi-dimensional arrays.

Installation :

pip install numpy

import numpy as np  #import package

Create an ND array

a = np.array([[1,2,3],[4,5,6]]) #2-Dimentional array

In the above example, you've created a 2D array. The dimensions of the array are represented by sets of square brackets, []. The dimensions of a ND array are called axes. The number of axes is called the rank. You can check the no of dimensions, and shape ( A tuple having the no of elements in each dimension ) by calling the following methods.

a.ndim

This returns the no of dimensions

>> 2

a.shape

This returns a tuple containing the no of elements in each dimension

>> (2,3)

Why Vectorization

When you perform a vectorized operation on an ND array using NumPy, the operation is applied to each element in the array simultaneously, rather than iterating over the array and performing the operation on each element individually. This can be much faster than using a loop to perform the same operation, particularly when working with large datasets or when the operation is computationally expensive. Numpy vectorized operations are implemented using low-level, highly optimized code that can take advantage of the specialized architecture of modern processors, resulting in significant performance improvements NumPy. Learn more. Let’s see an example,

b = np.random.rand(10000,1)

Here b is a two-dimensional array of shape (10000,1). random.rand() is a numpy method that assigns a random value (between 0,1) to every element in the array.

Now Let's get the sum of all the values in


 using *For loops* and *Numpy*.

![](https://i.imgur.com/H72n7eg.png)

Calculate sum using For loops


```python
sum = 0
for i in range(b.shape[0]):
    sum += i

Vectorized Implementation

sum = np.sum(b)

Performance comparision,

import timeit   #timeit is a package used for tracking elapsed time for both operations

Measure the execution time of the for loop implementation

for_loop_time = timeit.timeit(
    # Define the for loop code as a string
    stmt="""
sum = 0
for i in range(b.shape[0]):
    sum += i
    """,
    # Define the variables used in the code as a string
    setup="import numpy as np\n" +
          "b = np.random.rand(10000,1)",
    # Execute the code 1000 times
    number=1000
)
print("For loop elapsed time: {:.5f} seconds".format(for_loop_time))

>> For loop elapsed time: 0.61476 seconds

Measure the execution time of the vectorized implementation

vectorized_time = timeit.timeit(
    # Define the vectorized code as a string
    stmt="sumt = np.sum(b)",
    # Define the variables used in the code as a string
    setup="import numpy as np\n" +
          "b = np.random.rand(10000,1)",
    # Execute the code 1000 times
    number=1000
)
print("vectorized elapsed time: {:.5f} seconds".format(vectorized_time))

>> vectorized elapsed time: 0.01074 seconds

The vectorized implementation is nearly 50 times faster than the non-vectorized version. This performance difference becomes more significant for larger calculations.

From now on, we'll get into some important methods of NumPy that will be essential for your project if you use vectorization. By understanding how these operations work, you can make more informed decisions about how to structure your code and choose the most efficient approaches for your specific use case. In some cases, working with multi-dimensional arrays can be challenging. The reshape() function in NumPy is a useful tool for converting an array to a different shape. It allows you to change the dimensions of an array by specifying the new shape as an argument. Remember you can always use this to convert your multidimensional array into a 2D or 1D array.

b = np.random.rand(10,5,8) #3 dimensional array
a = np.reshape(b,(400)) #convert to 1D array

a.shape

>> (400,)

Note that here, the total number of elements in an array should match the new shape when reshaping the array. For example, think that the original dimensions of the array are d1, d2, d3,..., dN and the new reshaped dimensions are r1, r2,..,rM. The condition that must be satisfied in order to reshape the array is d1 * d2 * d3 * ... * dN = r1 * r2 * r3 * ... * rM.

Operations on Vectors

Now let's look at some common vectorized operations using NumPy and compare the performance of for loop implementations to vectorized implementations.

Element-wise operations

Element-wise operations are performed on each element of an array individually. Some examples of element-wise operations include addition, subtraction, multiplication, and division.

Addition

To add arrays using a for loop, we can iterate over their elements and add them element-wise. For example, consider the following 2D arrays.:

Define two 2D arrays

a = np.array([[1, 2], [3, 4]])
b = np.array([[5, 6], [7, 8]])

Add the arrays using a nested for loop

result = np.empty(a.shape) # Initialize an empty result array

for i in range(a.shape[0]):
    for j in range(a.shape[1]):
        result[i, j] = a[i, j] + b[i, j]
print(result)

>> [[ 6  8] [10 12]]

To add two vectors using vectorization, we can simply use the + operator.

Add the arrays using vectorization

result = a + b
print(result)

>> [[ 6  8] [10 12]]

To compare the performance of these two approaches, we can use the %timeitmagic command in IPython or Jupyter notebooks, or the timeit module in a script.

Measure the execution time of the for loop implementation

for_loop_time = timeit.timeit(
    # Define the for loop code as a string
    stmt="""
for i in range(a.shape[0]):
    for j in range(a.shape[1]):
        result[i, j] = a[i, j] + b[i, j]
    """,
    # Define the variables used in the code as a string
    setup="import numpy as np\n" +
        "a = np.array([[1, 2], [3, 4]])\n" +
        "b = np.array([[5, 6], [7, 8]])\n" +
        "result = np.empty(a.shape)",
    # Execute the code 1000 times
    number=1000
)
print("For loop elapsed time: {:.5f} seconds".format(for_loop_time))

For loop elapsed time: 0.00324 seconds

Measure the execution time of the vectorized implementation

vectorized_time = timeit.timeit(
    # Define the vectorized code as a string
    stmt="result = a + b",
    # Define the variables used in the code as a string
    setup="import numpy as np\n" +
        "a = np.array([[1, 2], [3, 4]])\n" +
        "b = np.array([[5, 6], [7, 8]])",
    # Execute the code 1000 times
    number=1000
)
print("Vectorized elapsed time: {:.5f} seconds".format(vectorized_time))

Vectorized elapsed time: 0.00051 seconds

Similarly, subtraction, multiplication, and division can be performed using the same method.

Broadcasting is a powerful feature in element-wise operations that allows operations to be performed between arrays of different shapes. It allows the smaller array to be "broadcast" or resized to match the shape of the larger array so that the element-wise operation can be performed.

Multiplying by a scalar

To multiply the array a by a scalar using a for loop, we can iterate over its components and multiply them by the scalar. For example,

Define a 2D array

  a = np.array([[1, 2], [3, 4]])

Define a scalar

  s = 2

Multiply the array by the scalar using a nested for loop

  result = np.empty(a.shape) # Initialize an empty result array

  for i in range(a.shape[0]):
      for j in range(a.shape[1]):
          result[i, j] = a[i, j] * s
  print(result)

  >> [[2 4] [6 8]]

To multiply a vector by a scalar using vectorization, we can simply use the * operator. Here, the scalar value s is broadcast to a temporary array of the same shape as a, and the multiplication is performed element-wise. This allows us to easily multiply a vector by a scalar value without having to write a loop.

Multiply the array by the scalar using vectorization

  result = a * s
  print(result)

  >> [[2 4] [6 8]]

Performance comparition,

Measure the execution time of the for loop implementation

  for_loop_time = timeit.timeit(
      # Define the for loop code as a string
      stmt="""
  for i in range(a.shape[0]):
      for j in range(a.shape[1]):
          result[i, j] = a[i, j] * s
      """,
      # Define the variables used in the code as a string
      setup="import numpy as np\n" +
            "a = np.array([[1, 2], [3, 4]])\n" +
            "s = 2\n" +
            "result = np.empty(a.shape)",
      # Execute the code 1000 times
      number=1000
  )
  print("For loop elapsed time: {:.5f} seconds".format(for_loop_time))

For loop elapsed time: 0.00320 seconds

Measure the execution time of the vectorized implementation

  vectorized_time = timeit.timeit(
      # Define the vectorized code as a string
      stmt="result = a * s",
      # Define the variables used in the code as a string
      setup="import numpy as np\n" +
            "a = np.array([[1, 2], [3, 4]])\n" +
            "s = 2",
      # Execute the code 1000 times
      number=1000
  )
  print("Vectorized elapsed time: {:.5f} seconds".format(vectorized_time))

  Vectorized elapsed time: 0.00106 seconds

Dot product

The dot product, also known as the inner product or scalar product, is a binary operation that takes two arrays and returns a scalar. It is originally defined as the sum of the element-wise products of the two 1D arrays. However, when applied to multidimensional arrays, the output can be different than a scalar. It may return an array of higher dimensions, depending on the implementation.

Vector dot product

The vector dot product takes two 1D arrays (vectors) and returns a scalar. For example, given two vectors a and b with n elements each, the vector dot product is calculated as a[0]*b[0] + a[1]*b[1] + ... + a[n-1]*b[n-1].

To calculate the vector dot product of two 1D arrays (vectors) using a for loop, we can iterate over the elements of each array and multiply them element-wise, then sum the results.

Define two vectors

a = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
b = np.array([10, 9, 8, 7, 6, 5, 4, 3, 2, 1])

dot_product = 0

for i in range(len(a)):
    dot_product += a[i] * b[i]
print(dot_product)

>> 385

To calculate the vector dot product using vectorization, use the np.dot() function.

dot_product = np.dot(a, b)
print(dot_product)

>> 385

Performance comparision,

Measure the execution time of the for loop implementation

for_loop_time = timeit.timeit(
    # Define the for loop code as a string
    stmt="""
dot_product = 0
for i in range(a.shape[0]):
    dot_product += a[i] * b[i]
    """,
    # Define the variables used in the code as a string
    setup="import numpy as np\n" +
          "a = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])\n" +
          "b = np.array([10, 9, 8, 7, 6, 5, 4, 3, 2, 1])",
    # Execute the code 1000 times
    number=1000
)
print("For loop elapsed time: {:.5f} seconds".format(for_loop_time))

For loop elapsed time: 0.00366 seconds

Measure the execution time of the vectorized implementation

vectorized_time = timeit.timeit(
    # Define the vectorized code as a string
    stmt="dot_product = np.dot(a, b)",
    # Define the variables used in the code as a string
    setup="import numpy as np\n" +
          "a = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])\n" +
          "b = np.array([10, 9, 8, 7, 6, 5, 4, 3, 2, 1])",
    # Execute the code 1000 times
    number=1000
)
print("vectorized elapsed time: {:.5f} seconds".format(vectorized_time))

vectorized elapsed time: 0.00193 seconds

Matrix multipication

The matrix multipication is the dot product applied to two 2D arrays(matrices) which returns another 2D array. It is calculated by taking the dot product of each row of the first matrix with each column of the second matrix. For example, given two matrices A with m rows and n columns, and B with p rows and q columns, the matrix multipication is poerformed as follows:

result[i, j] = A[i, 0]*B[0, j] + A[i, 1]*B[1, j] + ... + A[i, n-1]*B[n-1, j]

This results in a 2D array with m rows and q columns.
For example, given the following matrices:

A = [[1, 2, 3], [4, 5, 6]]  # A has 2 rows and 3 columns
B = [[7, 8], [9, 10], [11, 12]]  # B has 3 rows and 2 columns

result = [[58 64], [139 154]]  # Result has 2 rows and 2 columns

To calculate the dot product of two matrices using a for loop, we can iterate over their elements and multiply them element-wise, then sum the products. For example:

Define two 2D arrays

a = np.array([[1, 2], [3, 4]])
b = np.array([[5, 6], [7, 8]])

result = np.zeros((a.shape[0], b.shape[1]))

for i in range(a.shape[0]):
    for j in range(b.shape[1]):
        for k in range(a.shape[1]):
            result[i, j] += a[i, k] * b[k, j]
print(result)

>> [[19 22] [43 50]]

To calculate the dot product of two matrices using the numpy library, we can use the np.matmul() function.

result = np.matmul(a, b)
print(result)

>> [[19 22] [43 50]]

Performace comparision,

Measure the execution time of the for loop implementation

for_loop_time = timeit.timeit(
    # Define the for loop code as a string
    stmt="""
for i in range(a.shape[0]):
    for j in range(b.shape[1]):
        for k in range(a.shape[1]):
            result[i, j] += a[i, k] * b[k, j]
    """,
    # Define the variables used in the code as a string
    setup="import numpy as np\n" +
          "a = np.array([[1, 2], [3, 4]])\n" +
          "b = np.array([[5, 6], [7, 8]])\n" +
          "result = np.zeros((a.shape[0], b.shape[1]))",
    # Execute the code 1000 times
    number=1000
)
print("For loop elapsed time: {:.5f} seconds".format(for_loop_time))

For loop elapsed time: 0.00811 seconds

Measure the execution time of the vectorized implementation

vectorized_time = timeit.timeit(
    # Define the vectorized code as a string
    stmt="result = np.matmul(a, b)",
    # Define the variables used in the code as a string
    setup="import numpy as np\n" +
          "a = np.array([[1, 2], [3, 4]])\n" +
          "b = np.array([[5, 6], [7, 8]])",
    # Execute the code 1000 times
    number=1000
)
print("Vectorized elapsed time: {:.5f} seconds".format(vectorized_time))

Vectorized elapsed time: 0.00131 seconds

However, both np.dot() and np.matmul() functions can be used to perform a matrix-matrix dot product, but they may return different results when applied to 1D arrays. When applied to 1D arrays, np.dot() performs the vector dot product, while np.matmul() performs element-wise multiplication. For example,

Define two 1D arrays

u = np.array([1, 2, 3])
v = np.array([4, 5, 6])

Calculate the dot product using np.dot()

result = np.dot(u, v)
print(result)

>> 32

Calculate the dot product using np.matmul()

result = np.matmul(u, v)
print(result)

>> [4 10 18]

Using np.matmul() is the recomended way to perform matrix multipication.

Advanced techniques for manipulating elements in NumPy arrays

NumPy provides a set of built-in methods for working with multidimensional arrays that are similar in functionality to those provided by Python for working with lists. Some examples of these methods include np.sort(), np.append(), etc.

`concatenate()`

The concatenate() function allows you to combine multiple arrays into a single array.Here is an example of concatenating two 3D arrays using a for loop,

Define two 3D arrays

array_1 = [[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]
array_2 = [[[13, 14, 15], [16, 17, 18]], [[19, 20, 21], [22, 23, 24]]]

Iterate over the elements in each array and append them to the combined list

combined_array = [] # Initialize an empty list to store the combined arrays

for array in array_1:
    combined_array.append(array)
for array in array_2:
    combined_array.append(array)

# Convert the list to a numpy array
combined_array = np.array(combined_array)
print(combined_array)

>> [[[ 1  2  3]
  [ 4  5  6]]

 [[ 7  8  9]
  [10 11 12]]

 [[13 14 15]
  [16 17 18]]

 [[19 20 21]
  [22 23 24]]]

The default behavior of the concatenate() function is to concatenate the arrays along the first axis (axis 0). This means that the arrays are joined together one after the other along the first axis, resulting in a new array with the same number of dimensions as the original arrays but with an increased size along the first axis. You can change the axis along which the arrays are concatenated by providing the axis parameter.

Use the concatenate() function to combine the two arrays

combined_array = np.concatenate((array_1, array_2))
print(combined_array)

>> [[[ 1  2  3]
  [ 4  5  6]]

 [[ 7  8  9]
  [10 11 12]]

 [[13 14 15]
  [16 17 18]]

 [[19 20 21]
  [22 23 24]]]

On the other hand hstack() and vstack() functions can also be used to combine arrays horizontally or vertically, respectively. The hstack() function is used to stack arrays in sequence horizontally (along the second axis), whereas the vstack() function is used to stack arrays in sequence vertically (first axis).

Using hstack to stack the arrays horizontally,

result_hstack = np.hstack((array_1, array_2))
print(result_hstack)

>> [[[ 1  2  3] [ 4  5  6] [13 14 15] [16 17 18]]
[[ 7  8  9] [10 11 12] [19 20 21] [22 23 24]]]

Using vstack to stack the arrays vertically,

result_vstack = np.vstack((array_1, array_2))
print(result_vstack)

>> [[[ 1  2  3] [ 4  5  6]]
[[ 7  8  9] [10 11 12]]
[[13 14 15] [16 17 18]]
[[19 20 21] [22 23 24]]]

`sort()`

The sort() function sorts the elements in a NumPy array in ascending order. Here is an example of sorting an array using a for loop,

Define a 3D array

data = [[[3, 1, 4, 3], [2, 1, 4, 3]], [[4, 5, 6, 4], [5, 6, 7, 5]]]

Iterate over the elements in the array and add them to the list

sorted_elements = [] # Initialize an empty list to store the sorted elements

for array in data:
    for subarray in array:
        for element in subarray:
            sorted_elements.append(element)

# Sort the list in ascending order
sorted_elements.sort()
print(sorted_elements)

>> [1, 1, 1, 2, 3, 3, 3, 4, 4, 4, 5, 5, 6, 6, 7]

The same example using the sort() function,

sorted_elements = np.sort(data)
print(sorted_elements)

>> [1, 1, 1, 2, 3, 3, 3, 4, 4, 4, 5, 5, 6, 6, 7]

`where()`

The where() function in NumPy allows to return elements from one of two possible arrays depending on whether a given condition is met. The elements where the condition is true will come from the first array, while the elements where the condition is false will come from the second array.Here is an example,

Define arrays

x = np.array([1, 2, 3, 4, 5])
y = np.array([10, 20, 30, 40, 50])

condition = x > 2

The condition variable is a boolean array that indicates which elements of x are greater than 2.

Iterate over the elements of the input arrays and the condition array, and use an if statement to decide which element to add to the output array.

result = []
for i in range(len(x)):
    if condition[i]:
        result.append(x[i])
    else:
        result.append(y[i])
result = np.array(result)
print(result)

>>> [10 20 3 4 5]

The np.where() function uses the condition to select elements from x where the condition is True and elements from y where the condition is False.

result = np.where(condition, x, y)
print(result)

>>> [10 20 3 4 5]

Working with a dataset

Working with a dataset can often involve calculating statistical measures such as the mean, standard deviation, and variance. This can be a tedious process if you have to do it manually using for loops, especially for large datasets. Fortunately, NumPy provides vectorized functions that can perform these calculations efficiently on an entire array of data at once.

In the other hand Python's built-in math module also provides functions for these calculations like the mean, standard deviation, and variance. The real advantage of using NumPy for statistical calculations comes with the axis argument, which allows you to calculate statistical measures along a specific dimension of a multidimensional array. This cannot be done with Python's built-in math module. Let's see some examples,

Define a 2D array

data = np.array([[1, 2, 3], [4, 5, 6]])

Median

The median of a data set is the middle value when the data is arranged in ascending or descending order. If the data set has an odd number of values, the median is the middle value. If the data set has an even number of values, the median is the average of the two middle values. Learn more

Calculate the median of each row using a for loop

medians = np.empty(data.shape[0])
for i in range(data.shape[0]):
    medians[i] = np.median(data[i, :])
print(medians)

>> [2. 5.]

To calculate the median using Numpy library, you can use the np.median()function.

medians = np.median(data, axis=1)
print(medians)

>> [2. 5.]

In these examples, data is a 2D NumPy array and the axis=1 argument tells the function to compute the median for each row. If you want to compute the median for each column you can change the axis argument to axis=0.

Mean

The mean (average) of a data set is found by adding all numbers in the data set and then dividing by the number of values in the set. Learn more

Calculate the mean of each column using a for loop

means = np.empty(data.shape[1])
for i in range(data.shape[1]):
    means[i] = np.mean(data[:, i])
print(means)

>> [2.5 3.5 4.5]

To calculate the mean using Numpy library, you can use the np.mean() function.

means = np.mean(data, axis=0)
print(means)

>> [2.5 3.5 4.5]

Standard Deviation

The standard deviation is a measure of how spread out the data is from the mean. It is the square root of the average of the squared deviations from the mean. std = sqrt(mean(x)), where x = abs(a - a.mean())**2. A low standard deviation indicates that the data is tightly grouped around the mean, while a high standard deviation indicates that the data is more spread out. Learn more

# Calculate the standard deviation of each column using a for loop
stds = np.empty(data.shape[1])
for i in range(data.shape[1]):
    stds[i] = np.std(data[:, i])
print(stds)

>> [1.5 1.5 1.5]

To calculate the mean using Numpy library, you can use the np.std() function.

# Calculate the standard deviation of each column using np.std() with the axis argument
stds = np.std(data, axis=0)
print(stds)

>> [1.5 1.5 1.5]

Can For loops be completely replaced?

Now you know how to use vectorized operations and Numpy functions to optimize the performance and readability of your code. However, in some cases, the sequential nature of for loops may still be necessary. For example, when iteratively updating values in a function, where the output at time t depends on the output at time t-1, a for loop may be the only option. This is because vectorized operations are typically performed in parallel, so the output at t-1 cannot be used to calculate the output at t in a vectorized manner.

def update_values(prev):
       new = prev + np.log(prev + 2)
       return new

x = 0
y = 0
iterations = 10

for i in range(iterations):
       x += update_values(x)
       y += x + 10

print(x)
print(y)

>> 1150.00069088284
>> 2367.2066419361945

Conclusion

Considering vectorization other than explicit For Loops where necessary can make your next project more robust since it is an accelerated way to carry out calculations. But as above there can be times when using Loops is the only option. So choose wisely by analyzing the problem you are working on. If it can be solved in a parallel nature, moving to vectorization is the best option. Here I've shown you some important methods of NumPy to implement vectorization efficiently. You can also find the documentation here . Make sure to check that out to find some more important details.

Test-Driven Development with Python

chamodperera — Sat, 30 Jul 2022 06:15:00 +0000

Test-driven development (TDD) is an established technique for delivering better software more rapidly and sustainably over time. It is a standard practice in software engineering.

In this blog, I'll be explaining about test-driven development (TDD) and a how-to guide to TDD with python. In case you didn't even hear about it, also tag along.

What is Test-Driven Development & Why?

Test-driven development (TDD) is a software development process relying on software requirements being converted to test cases before software is fully developed, and tracking all software development by repeatedly testing the software against all test cases. In short, It is when you write tests before you write the code that's being tested.

It has number of benefits

Improved design of the system
Better code quality and flexibility
Good documentation as a byproduct
Lower development costs and increased developer productivity
Enhanced developer satisfaction
Easier to maintain

The Process

Write tests
Get the tests to pass
Optimize the design (Refactor)
Repeat the process

Now let's look at how do TDD in python

Prerequisites

There are several widely used libraries in python to write tests. In this guide, I am using a library called pytest. So make sure to install it first.

pip install -U pytest

Let's consider a small function to check whether a password is valid or invalid

Writing Tests

Let's say a valid password meets the following requirements.

length must be greater than 8
should contain a uppercase letter [A-Z]
should contain a lowercase letter [a-z]
should contain a digit [0-9]

So the first step is to write the tests for the function. I'll start by declaring an empty function.

validator.py

def pw_validator(password):
    pass

Now you can start writing tests in the same file or a separate file. The second option improves the code readability. If you choose to store the tests in a separate file, it should start with test_ to make it recognizable by pytest

test_validator.py

from validator import pw_validator

A test is basically a function that uses assert() method to check our validator returns the correct output. This test function also should start with test_.

Below you can see how I implemented some test functions with several valid & invalid passwords with the expected output.

def test_case_1():
    assert pw_validator("8c4XTH&Z4a5z1Cxo") == True

def test_case_2():
    assert pw_validator("m6oj4l*6r#s$") == False #doesn't contain any upper case letter

def test_case_3():
    assert pw_validator("DU$8$256Q*W@V6!KSED@H") == False #doesn't contain any lower case letter

def test_case_4():
    assert pw_validator("DO!OPhXnqCjBR&J") == False #doesn't contain any digits

def test_case_5():
    assert pw_validator("9s@X85") == False #length is lower than 8

Now you can simply type pytest in the console to run the tests.
Of course, the tests will fail first as we didn't implement the validator function yet.

Here each failed test is represented by a "F". Passes tests will be represented by a "."

When writing test functions you can include several assert methods in a single function. But the pytest understands each test by functions. So the best practice is to include assert methods in separate functions.

Implement the function to pass tests

After writing tests, we can start working on the validator function. You can check each required condition with an if else statement as below.

def pw_validator(password):

    if len(password) >= 8:

        if any(char.isdigit() for char in password):

            if any(char.isupper() for char in password):

                if any(char.islower() for char in password):
                    return True
                else:
                    return False

            else:
                return False

        else:
            return False

    else:
        return False

Now you can see all the tests have passed after running pytest.

You might definitely not come up with the correct implementation at first. For example, say you forgot to add functionality to check whether the password length is greater than 8 characters. Then one of the tests will return as failed.

def pw_validator(password):

        if any(char.isdigit() for char in password):

            if any(char.isupper() for char in password):

                if any(char.islower() for char in password):
                    return True
                else:
                    return False

            else:
                return False

        else:
            return False

So your job is to improve the function until it passes all the tests.

Refactor

In this step, you'll make your code more readable, concise, and clean. In our case, you might notice the validator function seems a bit untidy with the if else tree. We can refactor it as below to improve quality.

def pw_validator(password):
    return True if len(password) >= 8 and any(char.isdigit() for char in password) and any(char.isupper() for char in password) and any(char.islower() for char in password) else False

Now you can repeat the process by applying it to another function.

Conclusion

At this time, you might have a clear understanding of what TDD is and how to use it in your project. TDD is widely used in data science & machine learning as it involves lots of testing.

Tutorial - How to build your own LinkedIn Profile Scraper in 2022

chamodperera — Sun, 17 Jul 2022 12:32:27 +0000

LinkedIn is the world's largest professional network on the internet. You can use LinkedIn to find the right job or internship, connect and strengthen professional relationships, and learn the skills you need to succeed in your career. A complete LinkedIn profile can help you connect with opportunities by showcasing your unique professional story through experience, skills, and education.

In this tutorial, Let's look at how to implement a web scraper to gather job details and company profiles from a posted jobs list on Linkedin and save them in a .JSON file using Python.

This tutorial is a complete beginner guide to web scraping using python.

What is Web Scraping?

Web scraping refers to the extraction of data from a website. This information is collected and then exported into a format that is more useful for the user. Be it a spreadsheet or an API. In most cases, automated tools are preferred when scraping web data as they can be less costly and work at a faster rate.

Getting started

In order to complete this task, we need these widely used python libraries in web scraping.

The first one, Selenium is used to navigate web pages and interact with them. The other one is widely used to scrape data from web pages.

So let's install them.

pip install selenium

pip install beautifulsoup4

Installing the web driver

To work with Selenium you need to install the web driver for your browser. WebDriver is an open source tool for automated testing of web apps across many browsers. It provides capabilities for navigating web pages, user input, JavaScript execution, and more. If you are using chrome, you can download the driver by this link.

It's important to check your browser version before downloading the driver.

Task explanation

On LinkedIn, a user account is not compulsory to search for jobs. We can simply navigate to linkedin/jobs and search for any job vacancies available in your area.

So that our task is to automatically search for jobs on Linkedin and save the job list and company profiles as.JSON.

Navigating to webpage

As mentioned earlier, we use Selenium for navigation purposes. Let's import it into our program.

from selenium import webdriver

Then we need to establish our web driver as a driver object.

driver = webdriver.Chrome(location)

Replace location with your web driver location. Also see other supported browsers.

Now we can use the get() method to locate the website by URL.

driver.get("https://www.linkedin.com/jobs") #URL

If you run the program now, you can see that it spins up a new browser and navigates to the URL. You'll also notice that it has a address bar saying it is being controlled by automated test software.

Interacting with the page

The next step is to search for jobs. First, let's save the job title and location that we want to search for in separate strings.

job_title = 'software engineer' 
job_location = 'sri lanka'

A web page consists of HTML elements. In order to interact with it, we need to find the elements we need to act on and then find the selector or locator information for those elements of interest. The easiest way is to Inspect pages using developer tools. Place the cursor anywhere on the webpage, right-click to open a pop-up menu, then select the Inspect option. In the Elements window, move the cursor over the DOM structure of the page until it reaches the desired element. From there, we can find the HTML tag, the defined attribute, and the attribute values.

Next, we need to pass this information to the selenium web driver to simulate user actions on elements. Selenium provides various find_element methods to find elements based on their attribute/value criteria or selector value that we supply in our script. For that, the By class needs to import from Selenium.

from selenium.webdriver.common.by import By

These are the various ways the attributes are used to locate elements on a page.

find_element(By.ID, "id")
find_element(By.NAME, "name")
find_element(By.XPATH, "xpath")
find_element(By.LINK_TEXT, "link text")
find_element(By.PARTIAL_LINK_TEXT, "partial link text")
find_element(By.TAG_NAME, "tag name")
find_element(By.CLASS_NAME, "class name")
find_element(By.CSS_SELECTOR, "css selector")

In our case I am using XPATH method to locate the input tags of the title and location.

XPATH is the language used for locating nodes in an XML document. As HTML can be an implementation of XML (XHTML), Selenium users can leverage this powerful language to target elements in their web applications.

You can copy the XPATH by right clicking on the element.

Now save them in separate variables.

search_title = driver.find_element(By.XPATH, '//*[@id="JOBS"]/section[1]/input')
search_location = driver.find_element(By.XPATH, '//*[@id="JOBS"]/section[2]/input')

Now to pass the string values to the inputs, we can use send_keys() method.

search_title.send_keys(job_title)
search_location.clear()
search_location.send_keys(job_location,Keys.ENTER)

The location input is sometimes auto-filled by default with a location based on the IP address. clear() method is used clear any default values in the input.

Keys.ENTER argument is used to send ENTER key after passing the input values. Before that Keys should be imported.

from selenium. webdriver. common. keys import Keys

It is important to stop the program for some time as the search results should be properly loaded before the next steps. For this, we can use the inbuilt time library.

import time

time.sleep(3) #sleeps for 3 seconds

Finally we can get the UL which contains the job list by calling the By.CLASS_NAME method.

jobs_list = driver.find_element(By.CLASS_NAME,'jobs-search__results-list')

Scrape data from the web page

Now we can use BeautifulSoup to scrape necessary data from the job list. Let's import is first.

from bs4 import BeautifulSoup

As an initial step, we need to direct the job_list to BeautifulSoup.

soup = BeautifulSoup(jobs_list.get_attribute('outerHTML'), 'html.parser')

Similar to Selenium we can retrieve all li items in the UL by the tag name into a list. See the Bs4 documentation for more information.

jobs = soup('li')

Now let's make a list of the information we need to extract from every job item.

job title
location
link to the job details
link to the company profile

We use a for loop to iterate through each item in the jobs list and retrieve information.

data=[]
for job in jobs:
    item ={}
    item["job_title"] = job.find("h3",class_="base-search-card__title").text.strip(" \n")
    item["company"] = job.find("h4",class_="base-search-card__subtitle").text.strip(" \n")
    item["location"] = job.find("span",class_="job-search-card__location").text.strip(" \n")

    job_details = job.find("a",class_="base-card__full-link")
    item["job_details"] = job_details["href"].split('?', 1)[0]

    company_profile = job.find("a",class_="hidden-nested-link")
    item["company_profile"] = company_profile.attrs["href"].split('?', 1)[0]
    data.append(item)

In the above code, I declared an empty data array to store the information. And In every iteration in the for loop, I used find methods to locate the elements which we need to get data from.

text method is used to get the innerHTML values of the HTML tags and the attrs[] method is used to get the attribute values of an element. split() & split() methods are used for basic text formatting.

In every iteration, all the information is saved in a separate JSON object.

Save data in a JSON file

Now that all the retreived data is passed to the data array, we can use the built in json library to save them in a new json file.

import json

with open("jobs.json", "w") as writeJSON:
   json.dump(data, writeJSON, indent=4)

The final output will look like this.

Conclusion

The final program

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium. webdriver. common. keys import Keys
from bs4 import BeautifulSoup
import json
import time


job_title = 'software engineer' #replace job title
job_location = 'sri lanka' #replace location


driver = webdriver.Chrome('webdriver/chrome/chromedriver') #replace the webdriver location

driver.get("https://www.linkedin.com/jobs")


search_title = driver.find_element(By.XPATH, '//*[@id="JOBS"]/section[1]/input')
search_location = driver.find_element(By.XPATH, '//*[@id="JOBS"]/section[2]/input')
search_title.send_keys(job_title)
search_location.clear()
search_location.send_keys(job_location,Keys.ENTER)
time.sleep(3)

jobs_list = driver.find_element(By.CLASS_NAME,'jobs-search__results-list')
soup = BeautifulSoup(jobs_list.get_attribute('outerHTML'), 'html.parser')

jobs = soup('li')

data=[]
for job in jobs:
    item ={}
    item["job_title"] = job.find("h3",class_="base-search-card__title").text.strip(" \n")
    item["company"] = job.find("h4",class_="base-search-card__subtitle").text.strip(" \n")
    item["location"] = job.find("span",class_="job-search-card__location").text.strip(" \n")

    job_details = job.find("a",class_="base-card__full-link")
    item["job_details"] = job_details["href"].split('?', 1)[0]

    company_profile = job.find("a",class_="hidden-nested-link")
    item["company_profile"] = company_profile.attrs["href"].split('?', 1)[0]
    data.append(item)

with open("jobs.json", "w") as writeJSON:
   json.dump(data, writeJSON, indent=4)

driver.quit()

With this program, you can easily scrape job details and company profile URLs on Linkedin.

You can also download the program via my Github repository.

Thank You

Getting started with Arduino ( build a knight rider LED simulation)

chamodperera — Tue, 24 May 2022 08:05:58 +0000

Hi everyone, today I thought of making a complete beginner guide to the people who are new to Arduino. In this post I am going to give a detailed description of making a knight rider LED simulation circuit (beginner project) from scratch. For this project I am using an Arduino UNO board.

Well, Arduino is an open-source electronics platform based on easy-to-use hardware and software. Arduino boards are able to read inputs - light on a sensor, a finger on a button, or a Twitter message - and turn it into an output - activating a motor, turning on an LED, publishing something online. You can tell your board what to do by sending a set of instructions to the microcontroller on the board. To do so you use the Arduino programming language (based on Wiring), and the Arduino Software (IDE), based on Processing. Learn more about Arduino and boards visit arduino.cc

As I mentioned earlier I am using an Arduino UNO board for this project. You can buy it online or you can simply simulate the whole project by visiting tinkercad.

Here is an detailed image of the board

And you need to install the Arduino IDE software to start programing. Download and install it as well.

So now let's get started with the project

Connecting the components together

For this project we need following components

Arduino UNO x1
LED x6
wires
project board x1

connect the components together as shown as below

Writing the program

Now that the components are wired together, connect your board to the pc and open Arduino IDE. First make sure that the correct port is selected from the tools.

The Arduino program contains two main parts: setup () and loop (). The name of the functions implies their purpose and activity: setup () sets up the Arduino hardware, such as specifying which I/O lines is planned to use, and whether they are inputs or outputs. The loop () function is repeated endlessly when the Arduino runs.

void setup() {
  // put your setup code here, to run once:

}

void loop() {
  // put your main code here, to run repeatedly:

}

So at first we need to declare our output pins. As the 6 LEDs are connected to the 2-7 digital pins, Let's make them as output using pinMode(). This part goes to the setup().

void setup() {
  pinMode(2,OUTPUT);
  pinMode(3,OUTPUT);
  pinMode(4,OUTPUT);
  pinMode(5,OUTPUT);
  pinMode(6,OUTPUT);
  pinMode(7,OUTPUT);
}

Note :- each line should end with a ";"

To simply this part we can use a For loop as below

{
  for (int i = 2; i <= 7; i++) {
    pinMode(i , OUTPUT); 
 }

Now let's think about the output we need. The LEDs should blink like the Knight Rider Car light. Left to right, right to left, on repeat. To accomplish that we need to blink each LED one after other as a line. If we consider 1st LED in the row, at the beginning it needed to be turned on and after few milliseconds needed to be turned off. here's how we do it

digitalWrite(2, HIGH);
delay(100); //interval in milliseconds
digitalWrite(2, LOW);

digitalWrite() method is used to set the pin value to HIGH or LOW, which means toggles the LED. delay() method is used to set the inteval.

Now if we use a For loop to do the same thing to every single LED one after other, we can get the expected output. So the loop() part looks like this.

void loop()
{
  for (int i = 2; i <= 7; i++) {
    digitalWrite(i, HIGH);
    delay(100); 
    digitalWrite(i, LOW); 
  }

   for (int i = 7; i >= 2; i--) {
    digitalWrite(i, HIGH);
    delay(100); 
    digitalWrite(i, LOW); 
  }
}

Try to better understand the above code considering the expected output.

Now you can upload the program and see the output.

Check out the simulation of this on tinkercad.

Thank you. Comment your questions below.

Use firebase in your React App

chamodperera — Sun, 22 May 2022 11:03:15 +0000

Hi everyone, this is my first post. Feel free to comment any mistakes below.

So recently I've been creating my personal-portfolio site with reactjs. It has few sections where I need to update in the future ex:- contacts, projects...etc. And I needed to update those sections without directly changing the code. That's where I used firebase as a solution for this case.

Firebase is a platform developed by Google for creating mobile and web applications. It was originally an independent company founded in 2011. In 2014, Google acquired the platform and it is now their flagship offering for app development.

Then let's get started. I am using firebase V9 for this.

Step 1 (setting up a new react project)

First steps first! setting up a react project. If you are new to react check out the React documentation to get started. React is a free and open-source front-end JavaScript library for building user interfaces based on UI components. You can create a new react app by running the following code in your console.

npx create-react-app my-app

Step 2 (Create to new firebase project)

You can create a new firebase project by visiting the official firebase site. Create an account and "Go to console" to set up a new project.
I already have few projects, you can set up a new project by by clicking on "Add project".

Step 3 (Installing firebase)

To install firebase to your app, go to the project folder and run,

npm install firebase

Step 4 (Adding firebase project to your app)

Now that the firebase has been installed, Let's add your firebase project to your app.

Go to the root of your app and create a new file,

firebase-config.js

Get the firebase project configuration. You can find your configuration by visiting project settings in the firebase console.
Now save your configuration in the firebase-config.js. Here I had pushed my project to a git hub repo. So for the further security, I have saved my configuration in a .env file. Of course you can save your configuration directly, but if you need to host your project some where, using the .env is the better solution. Learn more about using environmental variables in react.

Step 5 (Reading data from firestore)

So that now you've added firebase to your project,you can use firebase features in your project. Check out the firebase documentation.

For addition I'll tell you how to read data from firestore database.

Go to your firebase project and set up Firestore Database. In my case I've created some contact information in the firestore database.
import firebase in your component

import {db} from '../../firebase-config'
import {collection, getDocs} from "firebase/firestore"

Now you can read data by creating a function. Use UseEffect to wrap it in your react component. Let's read my data in the "contacts" collection.

const [contacts,setContacts] = useState([])

    useEffect(() => {
        const contactsCollection = collection(db,'contacts')
        const getContacts = async () => {
            const snapshot = await getDocs(contactsCollection);
            setContacts(snapshot.docs.map((doc) => ({...doc.data(),id:doc.id})));
        }

        getContacts();


      }, []);

Now you can render your data in the template

Comment any questions below. Thank you.