Forem: Dev Khadka

Everything you need to know about git rebase

Dev Khadka — Sun, 15 Oct 2023 16:44:29 +0000

Everything you need to know about git rebase

How rebase works

Lets say we have commits c1, c2 and c3 on branch b1
When we do git rebase <ref> with b1 as current branch,
- git will checkout <ref> where <ref> can be commit hash, branch name, tag anything that points to a git commit
- Then git will replay (apply) each commits c1, c2 and c3 one at a time. This will be same as you started working by checking out <ref> and redoing all your changes.
- If there are conflicts when applying the commits, it will stop and allow you to resolve the conflict and continue using git rebase --continue
- Finally git will reset current b1 to points to new HEAD
We can use interactive rebase to pick or drop commits, reorder them, combine them and more. We'll see examples below.

When can we use rebase

When you have feature branch which may takes few days before you can create PR to merge. In this case you can rebase your branch against the main branch daily and get new codes and minimize conflicts.
Rebasing before merging your branch to main branch makes history cleaner
When working locally you can do micro commits and before creating PR you can rebase and reorganize your commits into atomic changes.
In some cases you may need to maintain temporary branch for testing features from multiple team members, lets say UAT branch. you can use interactive rebase to bring multiple changes from another branch to UAT branch.

What are the risks of using rebase and how to minimize them

The major risk of using rebase is it modifies the history and we need to do force push to upstream. So rule of thumb is don't use rebase in a shared branch. Some times there could be situation we want to use rebase when 2 people working on same feature. Lets say Backend and Frontend engineers working on same feature, In such case we can minimize the risk by using --force-with-lease when pushing to upstream. This prevents you from replacing other peoples work but for --force-with-lease to work as expected you need to keep few things in mind.

How force-with-lease works

Git tracks upstream branches with remote branches, for example if you fetch branch branch1 from remote named origin there will be a remote branch named origin/branch1
Now lets say we have some changes on branch1 locally and also rebased changing history. At the same time your colleague have pushed some changes on upstream.
If you push with --force you will replace the changes from your colleague.
If you push with --force-with-lease it will only replace the upstream history if origin/branch1 has the same commit hash as the HEAD on upstream. So if there is some change on upstream after you last fetched it will fail so that we can fetch the new changes and include them in you changes before pushing again.

Following these steps will reduce risk of replacing others changes

Never do git fetch or git pull with out specifying a branch name. Doing so will fetch all the remote branches without you not knowing. So when doing --force-with-lease git will check newly fetched origin/branch1 with upstream head and replace the new changes.
Instead use git fetch origin branch1 or git pull origin branch1 --rebase. git pull actually does two things git fetch and then merge, you can use rebase instead by giving --rebase option
Requlary update your local branch with upstream changes fetching and rebasing.

Common practices to use git rebase

Setup repo for practice

Summary
We'll clone the example repo and make two copies of the folder, one will be used as our local repo and other as upstream repo

It is recommended that you open these two folders on separate window of your IDE to avoid confusion.

git clone https://github.com/devbkhadka/git-rebase-blog.git to get the sample repo
make a copy of repo cp -r git-rebase-blog git-rebase-blog-upstream

Add git-rebase-blog-upstream as origin url for git-rebase-blog

cd git-rebase-blog
git remote set-url origin $(readlink -f '../')/git-rebase-blog-upstream

Get remote changes to local

Lets do few changes on local repo
git switch branch1 checkout branch1 and switch to branch1
add this text at the end of playground.txt

  - Nepali flag is the only triangular flag in the world.

add changes and commit

  git add .
  git commit -m "Added fact about flag"

Similarly add the following two lines, one in each commit

  - Momo: is the most popular fast food. Momo: is made of flour and filled with chicken, meat or vegetable fillings.
  - About 80% of the Nepalese population is Hindu.

Now open a new terminal and cd into git-rebase-blog-upstream
switch to branch1 git switch branch1
Add following line to playground.txt

  - Nepal is rich in diverse culture and language. Nepal has over 80 ethnic group and over 120 languages.

add changes and commit

  git add .
  git commit -m "Added a fact in upstream"

go back to terminal with git-rebase-blog
git fetch origin branch1 to get upstream changes
git rebase origin/branch1
There will be one conflict, resolve that and do git rebase --continue

Git force update example

Switch to terminal with git-rebase-blog-upstream
add a line to playground.txt and commit

  this text will be overwritten with --force-with-lease

git switch main switch to main so that we can push changes from downstream to branch1.
Now go back to terminal with git-rebase-blog
Do git fetch to fetch from upstream
Now do git push origin branch1 --force-with-lease
Switch to terminal with git-rebase-blog-upstream and do git log --oneline branch1 to see the commits. You will see last commit on upstream repo to be replaced. This happened because we fetched all the branches from the upstream but didn't get the changes to branch1.
switch to branch1 git switch branch1 in git-rebase-blog-upstream and add another line to playground.txt and commit

  this text will not be overwritten with --force-with-lease

git switch main switch to main so that we can push changes from downstream to branch1.
Now go back to git-rebase-blog and do git push origin branch1 --force-with-lease, this should fail

Interactive git rebase

Lets reset the upstream to main branch, go to terminal with git-rebase-blog-upstream and switch to git switch branch1 and run git reset --hard origin/main
Now come back to terminal with git-rebase-blog and do git fetch origin branch1
Now lets do interactive rebase git rebase -i origin/branch1, you will see list of commits in your branch1 as below

  pick 4905fc5 Added fact about ethnicity
  pick 93ffd54 Added fact about flag
  pick 0516467 Added fact about momo
  pick 8b19d6a Added fact about population

We can now re-organize the commits like below

  pick 4905fc5 Added fact about ethnicity
  # reordered as commit
  # squash will combine this commit with previous
  # commit and let you change commit message
  squash 8b19d6a Added fact about population
  # this will stop the rebase after this commit and you can
  # do changes and commit with --amend option to edit it
  edit 93ffd54 Added fact about flag
  squash 0516467 Added fact about momo

Save and exit the rebase edit file, there will be some conflict because of changing order resolve them and do git rebase --continue until it is rebased successfully
check the final commit history using git log --oneline

TDD in React using Jest — beginner tutorial

Dev Khadka — Mon, 13 Apr 2020 14:10:25 +0000

Overview

In this tutorial we’ll get started with using Jest library to test react application. This tutorial will cover following topics

Setup react project which includes jest library
How to write test using jest
Some common jest matchers
Concept of mocking and how to do it using jest
UI testing of react using react testing library
Finally I will also add reference where you can get in depth knowledge

For grasping above topics we’ll create a demo application which lists restaurants which can be filtered by distance from a center location. We’ll use TDD approach to build this application and give you simple exercise along the way to play with.

Prerequisite

You need to

be familiar with javascript
have some understanding of react like (JSX, Function based components, few hooks like useState, useEffect, useMemo). I will try to explain them as we use it

The codes are in Github with separate branch for each step. You can directly jump into any branch and play around.

If you are familiar with setting up react app you can directly skip to “List Restaurants”

Setup New React Project

You need nodejs installed before you can continue

Create a new folder named “jest-tutorial” and cd into that folder

cd /path/to/jest-tutorial

Run “create-react-app” command

# if create-react-app doesn't exists npx will install it and run
npx create-react-app .

Now you can run your app in browser. You should see a spinning react native logo in browser

npm start

press “ctrl+c” to stop the server in terminal

Lets Check Some Important Files

package.json — below is a part of the package json file. It lists project dependencies and commands that you can run

"dependencies": {
"@testing-library/jest-dom": "^4.2.4",
"@testing-library/react": "^9.5.0",
"@testing-library/user-event": "^7.2.1",
"react": "^16.13.1",
"react-dom": "^16.13.1",
"react-scripts": "3.4.1"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},

index.js — It is entry point for the app, it mounts the “App” component to element with id “root” in “public/index.html” file

ReactDOM.render(
<React.StrictMode>
   <App />
</React.StrictMode>,
document.getElementById('root')
);

App.js — It is the root component for our application. We can think of a react application as a tree where “App” component is root and it and its descendants can have one or more components as branches.

import './App.css';
function App() {
return (
<div className="App">
    ...
</div>
);
}
export default App;

Some Explanations

It imports “./App.css” as a global css file
“App” function returns JSX which is HTML like syntax in Javascript (What is JSX?)
It exports “App” component to be used in other files

Basic Layout

Replace content of “App.css” file
replace whole content of App.css file with css in following gist. This css includes basic styling for our demo application.

.App {
  display: flex;
  flex-direction: column;
  height: 100vh;
  color: white;
  overflow: hidden;
}

.App-header {
  background-color: #282c34;
  display: flex;
  flex-direction: column;
  align-items: center;
  justify-content: center;
  font-size: calc(10px + 2vmin);
  color: white;
  border-bottom: 1px solid rgb(143, 143, 143, 30);
}

.App-content {
  padding-top: 16px;
  background-color: #40444d;
  display: flex;
  flex-direction: column;
  flex-grow: 1;
  overflow: scroll;
}

.App-content form{
  margin: 16px 10% 16px 10%
}

.App-content input {
  box-sizing: border-box;
  width: 100%;
  height: 32px;
  font-size: 20px;
  padding: 4px;
}


.App-content ul {
  box-sizing: border-box;
  margin: 16px 10% 16px 10%;
  padding: 0px;
}

.App-content li {
  box-sizing: border-box;
  width: 100%;
  font-size: 20px;
  padding: 16px;
  list-style-type: none;
  background-color:  #282c34;
  border-bottom: 1px solid rgb(143, 143, 143, 30);
}

.App-link {
  color: #61dafb;
}

Replace the JSX in “App.js” replace all JSX content ( and its contents) with following

<div className="App">
    <header className="App-header">
        <h2>Welcome to Jest Tutorial</h2>
    </header>
    <div className="App-content">
    </div>
</div>

List Restaurants

Lets start by listing restaurants in UI. For that we need list of restaurants, which we may need to fetch from an api and then display it in UI. It sounds bit complex, if we try to implement all the functionality at once it will be complex to implement and hard to debug.

In TDD we build application in small increments. We first plan the next incremental feature to be implemented, then we write test code to validate that the implementation works as expected, with some given preconditions and inputs. Then only we write code for the feature

App Component

Start here by checking-out “1-skeleton” branch

Implementation Steps

We’ll implement the “List Restaurants” feature in following steps

Instead of directly showing list in “App” component we’ll create “Restaurants” component which will be included in “App” component. This will separate the responsibility and make it more testable.
“Restaurants” component will take list of restaurants as input and display it

Test Cases for App Component

Now lets write test cases for above steps.

App Component
    - Should call "fetchRestaurants" function to get restaurants
    - Should render "Restaurants" component with result from "fetchRestaurants"

Lets write the first unit test, for that lets create a “tests” folder in “src” and move “src/App.test.js” in it. It is common practice to put tests under “tests” folder.

Now replace content of “App.test.js” with following code

import React from 'react';
import { render } from '@testing-library/react';
import App from '../App';
describe("App Component", ()=>{
    it('Should call "fetchRestaurants" function to get restaurants', ()=>{
        fail("not implemented")
    })
})

Some explanation

“npm test” runs the jest command, which will look for js files inside tests or *.test.js or *.specs.js files and runs tests inside it one at a time in not particular order
“describe” is function provided by jest which will be available without import when running test with jest. It is used to group similar tests.
“it” is also function available in test environment it represents a single test case. Here we intentionally wrote test to fail.

Command to Run Test

npm test

it should show result ‘Failed: “not implemented”’ in the console

Using Mock for Testing

If you notice, the test above depends on a function called “fetchRestaurants”. Do we have to implement the function first? No, here is why
If we try to implement another functionality while working on one it will complicate things, which is against TDD principals
If we use real “fetchRestaurants” in test then when “fetchRestaurants” fails in future, testing depending on it will also fail. It will make pin-pointing the problem harder

So what is the solution for it?

Solution is to make a fake “fetchRestaurants” function which will return the value we need for testing, this is called mocking.

Lets see it in action

import React from 'react';
import { render } from '@testing-library/react';
import App from '../App';
import Restaurants from '../Restaurants'
import {fetchRestaurants} from '../utils'
import * as fixtures from '../fixtures'
import { act } from 'react-dom/test-utils';

// First mock whole '../Restaurants' and '../utils'
// By default it will mock all the functions in module to return undefined
jest.mock('../Restaurants')
jest.mock('../utils')

// Provide fake return values for the functions
Restaurants.mockReturnValue(null)
// we want fetchRestaurants to return promise that resolves to fixtures.dummyRestaurants
fetchRestaurants.mockResolvedValue(fixtures.dummyRestaurants)

describe("App Component", ()=>{

  // function passed to before each is called before running each test
  // It is used to setup pre-condition for each test
  beforeEach(()=>{
    // mockClear clears call history of the mock function
    Restaurants.mockClear()
    fetchRestaurants.mockClear()
  })

  it('Should call "fetchRestaurants" function to get restaurants', async ()=>{
    await act(async () => {
      render(<App />)
    })
    expect(fetchRestaurants).toBeCalled()
  })

  it('Should render "Restaurants" component with result from "fetchRestaurants"', async ()=>{
    await act(async () => {
      render(<App />)
    })
    expect(Restaurants.mock.calls[1][0]).toEqual({list: fixtures.dummyRestaurants})
  })
})

Some Explanations

“jest.mock(modulepath)” will modifies the original model by hooking into the import functionality. This is called monkey patching. Any other modules imported in this test file will also see the modified module.
So when “App” component see “Restaurants” component in its JSX it will use mock “Restaurants” instead of real one. This gives us chance to monitor how it is being used, like what property being passed.
“render” function renders the components in a virtual DOM implemented by “jest-dom” so that the test can be run without a browser
We need to wrap render inside “async act(async ()=>{})” because we are updating state in useEffect function which will update state and trigger UI update
“expect” function gives us access to variety of matcher that can be used to check if certain condition is satisfied in test.

Steps to Make the Tests Pass

At this point your test will fail, to make the test to pass you have to do following changes step by step which will take your test little further in each change

Create file “src/Restaurants.js ” and add code below

export default function Restaurants() {
}

create file “src/utils.js” and add code below

export function fetchRestaurants() {
}

create file “src/fixtures.js” and add code below

export const dummyRestaurants = "Dummy Restaurants"
Add code below before “return” in App.js. Don’t forget to import “fetchRestaurants” in the file
useEffect(()=>{
    fetchRestaurants()
})

change App function in App.js to look like below. Don’t forget to import “Restaurants”

import React, { useEffect, useState } from 'react';
import './App.css';
import { fetchRestaurants } from './utils';
import Restaurants from './Restaurants';

function App() {
  const [restaurants, setRestaurants] = useState(null)
  useEffect(()=>{
    fetchRestaurants()
      .then(setRestaurants)
      .catch(()=>console.log("error in fetching"))
  }, [])

  return (
    <Restaurants list={restaurants}/>
  );
}

export default App;

Some Explanations

callback of “useEffect” is called before each render of App component if values in second parameter changed. Values in second parameter must be a prop or state, empty array means it will run for 1st time only. We are calling “fetchRestaurants” before each render and calling “setRestaurants” function with value resolved by promise to update restaurants. This will re-render Restaurants component by updating list prop
You tests should pass now. Now lets move on to testing “Restaurant Component”

Exercise: Add a test case to do snapshot test of the component. You can get some detail here

Hint: Object returned by render function will have “baseElement” property. you can call “expect(baseElement).toMatchSnapshot()” which will create snapshot of html rendered for first time and test “baseElement” against the saved snapshot from next time. It will prevent accidental change in UI.

Exercise: Handle error case of fetchRestaurants.

Hint: Resolve object with structure {data: …} for success and {error: …} for error and check condition App component to show or hide error message element

Restaurants Component

Start here by checking-out “2-App-Component” branch

Implementation Steps for Restaurants Component

Restaurants component will receive restaurant list as “list” prop and render it by looping through each restaurant
It will take distance in a input field and filter the restaurants within the distance. To implement this feature we need a function to calculate distance, which is not implemented yet, so for doing the test we need to mock it.

Test Cases for Restaurants Component

Restaurants Component
    - should render restaurants passed to it
    - should be able to filter restaurants by distance from the center

The test cases should look like shown below

import React from 'react'
import {render, fireEvent} from '@testing-library/react'
import Restaurants from '../Restaurants'
import * as fixtures from '../fixtures'
import {calculateDistance} from '../utils'

jest.mock('../utils')
describe("Restaurants Component", ()=>{
    it("should render restaurants passed to it", ()=>{
        // render function returns a handle 
        const {getAllByText} = render(<Restaurants list={fixtures.dummyRestaurants}/>)
        // get elements matching regex
        expect(getAllByText(/Restaurant\d/).length).toBe(5)
    })

    it("should be able to filter restaurants by distance from center", ()=>{
        const {queryAllByText, getByTestId} = render(<Restaurants list={fixtures.dummyRestaurants}/>)

        // following block set five different return value for five calls to calculateDistance
        calculateDistance
            .mockReturnValueOnce(30)
            .mockReturnValueOnce(110)
            .mockReturnValueOnce(80)
            .mockReturnValueOnce(60)
            .mockReturnValueOnce(300)

        const inpDistance = getByTestId('inpDistance')
        // fire change event on inpDistance to set distance
        fireEvent.change(inpDistance, {target:{value: 100}})

        expect(queryAllByText(/Restaurant\d/).length).toBe(3)
    })
})

Some Explanation

In short, we interact with rendered DOM using handle returned by “render” function. We can also fire different event on DOM element by using “fireEvent” object. Like we used “change” event to trigger filter and check that list is filtered . More details are on comments in code.

Steps to Make Test Pass

Enter code below to “Restaurants.js” file for layout

import React from 'react'
export default function Restaurants({list}) {
   return <div className="App">
        <header className="App-header">
            <h2>Restaurants</h2>
        </header>
        <div className="App-content">
        </div>
    </div>
}

Create “distance” state by adding following line above “return” const [distance, setDistance] = useState(null)
Add the code block below before “return” line in “Restaurants” function. It will create a memorized value “filteredList” which is changed when either “list” or “distance” state changes

const filteredList = useMemo(()=> {
    return filterWithinDistance(list, distance)
}, [list, distance])

To render “filteredList” insert code below inside “App-content” div in JSX. This should make first test pass

{
    filteredList && filteredList.map((restaurant, i)=>
        <li key={restaurant.id}>{restaurant.name}</li>
    )
}

In “utils.js” add following function

export function calculateDistance(location){
}

Add “filterWithinDistance” function below the “Restaurants” function at the bottom of page. Don’t forget to import “calculateDistance” from “utils”

function filterWithinDistance(restaurants, distance) {
    return distance?
        restaurants.filter(restaurant=> calculateDistance(restaurant.location) <= distance):
        restaurants
}

Now add the following “form” in JSX above “ul” element

<form onSubmit={(e)=>e.preventDefault()}>
    <input onChange={(e)=> setDistance(e.target.value*1)}
        data-testid="inpDistance"
        placeholder="Enter distance in meters"/>
</form>

Now all of your tests should pass.

Excercise: Handle the case when restaurant list is empty or null by showing a message in UI

Hint: In test, render “Restaurant” component with list property “null” and “[]” then verify that you can find element containing the message text. In “Restaurant” component, conditionally show message or list based on “list” prop

Excercise: Show calculated distance in each restaurant list item.

Hint: modify “filterWithinDistance” to return restaurants with calculated distance and show it in UI. In test verify that mocked distance is show in the rendered UI

Implement “fetchRestaurants”

Start here by checking-out “3-Restaurants-Component” branch

Test Cases for fetchRestaurants

fetchRestaurants
    - should call fetch api with correct parameters
    - should return response on fetch success
    - should return empty array on fetch error

The test codes should look like

import {fetchRestaurants, RESTAURANTS_URL} from '../utils'
import * as fixtures from '../fixtures'


jest.spyOn(global, 'fetch')

describe('fetchRestaurants', ()=>{
    beforeEach(()=>{
        global.fetch.mockClear()
        global.fetch.mockResolvedValue({text: ()=>JSON.stringify(fixtures.dummyRestaurants)})
    })
    it('should call fetch api with correct parameters', ()=>{
        fetchRestaurants()
        expect(global.fetch).toBeCalledWith(RESTAURANTS_URL)
    })

    it("should return response on fetch success", async ()=>{
        const restaurants = await fetchRestaurants()
        expect(restaurants).toEqual(fixtures.dummyRestaurants)
    })

    it("should return null on fetch error", async ()=>{
        global.fetch.mockRejectedValue("some error occured")
        const restaurants = await fetchRestaurants()
        expect(restaurants).toEqual([])
    })
})

Some Explanations

‘fetch’ is a global variable so we used “jest.spyOn” function to mock
‘fetch’ property of “global” object. “global” object is equal to “window” object in browser.
“mockResolvedValue” sets mimic value resolved by fetch by passing object with text function.
“mockRejectedValue” mimics the error case in fetch

Steps to Make the Test Pass

Add “RESTAURANTS_URL” constant in “utils.js” file

export const RESTAURANTS_URL = "https://gist.githubusercontent.com/devbkhadka/39301d886bb01bca84832bac48f52cd3/raw/f7372da48797cf839a7b13e4a7697b3a64e50e34/restaurants.json"

fetchDistance function should look like below

export async function fetchRestaurants() {
    try{
        const resp = await fetch(RESTAURANTS_URL)
        const respStr = await resp.text()
        return JSON.parse(respStr)
    }
    catch(e) {
        console.log(e)
        return []
    }
}

Some Explanations

We are getting the restaurants list for git raw url which returns text response. So we are using “text” property of “resp”.
We are parsing response string to javascript object

Implement Calculate Distance

Start here by checking-out “4-fetch-restaurants” branch

Test Cases for calculateDistance

calculateDistance
    - should return distance in meters from center to a location given in degree

Test code for calculateDistance should look like below. Add it at the bottom of utils.test.js file

describe('calculateDistance', ()=>{
it('should return distance in meters from center to a location given in degree', ()=>{
    const testLocationPairs = [
        [ 40.76404704,-73.98364954],
        [ 26.212754, 84.961525],
        [27.699363, 85.325500],
        [ -11.166805, 38.408597],
    ]
    const expectedDistances = [12109725, 168479, 1181, 6647488]
    const calculatedDistances = testLocationPairs.map((location)=>{
        return calculateDistance(location)
    })
    // Test calculated values with in 1km range of expected value
    expect(calculatedDistances.map(d=>Math.floor(d/100)))
        .toEqual(expectedDistances.map(d=>Math.floor(d/100)))
    })
})

Steps to Make the Test Pass

Add constants below at top of utils.js file

export const CENTER_LOCATION = [27.690870, 85.332701]
const EARTH_RADIUS_KM = 63710
const PI_RADIAN_IN_DEGREE = 180
Add following code for calculating distance
export function calculateDistance(location){
    const [x1, y1] = convertCoordinateFromDegreeToRadian(location)
    const [x2, y2] = convertCoordinateFromDegreeToRadian(CENTER_LOCATION)
    const term1 = Math.sin((x2-x1)/2)**2
    const term2 = Math.sin((y2-y1)/2)**2 * Math.cos(x1) * Math.cos(x2)
    const distance = 2*EARTH_RADIUS_KM*Math.asin(Math.sqrt(term1+term2))
    return distance * 100
}
function convertCoordinateFromDegreeToRadian(point) {
    const [x, y] = point
    return [x*Math.PI/PI_RADIAN_IN_DEGREE, y*Math.PI/PI_RADIAN_IN_DEGREE]
}

We are using haversine distance formula to calculate distance in earth’s surface

Excercise: Validate the location parameter, Latitudes range from 0 to 90. Longitudes range from 0 to 180

Hint: verify that passing invalid value throws error using “expect(function).toThrow()”

Your tests should pass now. You can check in browser if it works or not by running “npm start”

I will appreciate any feedback, question and criticism. Your small encouragement means a lot, please don’t forget to clap like.

References

Learning Numpy by Playing: Indexing, Broadcasting, Vectorization and More

Dev Khadka — Sun, 10 Nov 2019 07:22:09 +0000

Introduction

This blog is meant to be playground for beginners to learn about numpy by trying real codes. I have kept texts content as little as possible and code examples as much as possible.

This is also meant to be quick future reference guide of numpy features you already learned. The output of each cell has details describing results.

Prerequisite

Basic programming knowledge
Some familiarity with python (loops, arrays etc.)

What Will We Cover

Basics

Creating Array
Understanding Structure of Numpy Array (dimension, shape and strides)
Data Types and Casting
Indexing Methods
Array Operations

Advance

Broadcasting
Vectorization
Ufunc and Numba

More for Exploration

What is Numpy?

Numpy is fundamental computing library for python. It supports N-Dimensional array and provides simple and efficient array operations.

NumPy is library of algorithms written in the C language which stores data in a contiguous block of memory, independent of other built-in Python objects and can operate on this memory without any type checking or other python overheads. NumPy arrays also use much less memory than built-in Python sequences.

Why Use Numpy?

Python was not initially designed for numerical computation. As python is interpreted language it is inherently slower than compiled languages like C. So numpy fills that gap here, following are few advantages of using numpy

It provides efficient multidimensional array operations, both memory and computation wise
It provides fast mathematical operations on entire array without need to use loop
It also provides scientific operations related to linear algebra, statistics, Fourier transform and more
It provides tool for interoperability with c and c++

How to Play With Numpy?

I would recommend two ways to play around with Numpy

kaggle or Google Colab: you can jump right into coding without needing any setup
Jupyter Notebook: you need to install jupyter notebook and then install numpy library using pip (numpy may be already installed if you have anaconda or miniconda)

Following This Tutorial

You can try this tutorial on kaggle or google colab by forking or making copy of following notebooks respectively

Fork this kaggle notebook
or
Make copy of this colab notebook

Some Tips On Jupyter Notebook

To see auto completion suggestions press tab
To see parameters of a function press shift + tab after typing function name and '('. eg. type np.asarray( then press shift + tab
To view doc string use '?' like np.asarray? then press shift + enter

Enough of reading!!
Lets get our hand dirty

Creating Array

From python list

import numpy as np
print(np.array([1,2,3,4]))

print('\n', 'array of 16 bit integers')
print(np.array([1,2,3,4], dtype=np.int16))

print('\n', '2 dimensional array')
print(np.array([[1,2,3], [4,5,6]]))

    [1 2 3 4]

     array of 16 bit integers
    [1 2 3 4]

     2 dimensional array
    [[1 2 3]
     [4 5 6]]

Numpy's Methods

print('Numpy array from range')
print(np.arange(3,8))

print('\n', '2D 3X3 array of zeros')
print(np.zeros((3,3)))

print('\n', '2D 2X3 array of ones')
print(np.ones((2,3)))

print('\n', 'Triangular array with ones at and below diagonal')
print(np.tri(3, 4))

print('\n', 'Index matrix with ones at diagonal')
print(np.eye(3))

print('\n', '20 equally spaced values between 1 and 5')
print(np.linspace(1, 5, 20))

    Numpy array from range
    [3 4 5 6 7]

     2D 3X3 array of zeros
    [[0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]]

     2D 2X3 array of ones
    [[1. 1. 1.]
     [1. 1. 1.]]

     Triangular array with ones at and below diagonal
    [[1. 0. 0. 0.]
     [1. 1. 0. 0.]
     [1. 1. 1. 0.]]

     Index matrix with ones at diagonal
    [[1. 0. 0.]
     [0. 1. 0.]
     [0. 0. 1.]]

     20 equally spaced values between 1 and 5
    [1.         1.21052632 1.42105263 1.63157895 1.84210526 2.05263158
     2.26315789 2.47368421 2.68421053 2.89473684 3.10526316 3.31578947
     3.52631579 3.73684211 3.94736842 4.15789474 4.36842105 4.57894737
     4.78947368 5.        ]

Using `np.random`

print('3X2 array of uniformly distributed number between 0 and 1')
print(np.random.rand(3,2))

print('\n', 'Normally distributed random numbers with mean=0 and std=1')
print(np.random.randn(3,3))

print('\n', 'Randomly choose integers from a range (>=5, <11)')
print(np.random.randint(5, 11, size=(2,2)))

print('\n', "Randomly selects a permutation from array")
print(np.random.permutation([2,3,4,5,6]))

print('\n', "This is equivalent to rolling dice 10 times and counting \
occurance of getting each side")
print(np.random.multinomial(10, [1/6]*6))

    3X2 array of uniformly distributed number between 0 and 1
    [[0.32564739 0.97679242]
     [0.26588925 0.89020385]
     [0.18024366 0.90879681]]

     Normally distributed random numbers with mean=0 and std=1
    [[-1.18661884 -0.43561077  1.21316858]
     [-1.47847545  0.69296328  1.01348937]
     [-0.03562709 -1.90675623  0.44639003]]

     Randomly choose integers from a range (>=5, <11)
    [[8 7]
     [8 9]]

     Randomly selects a permutation from array
    [6 4 3 2 5]

     This is equivalent to rolling dice 10 times and counting occurance of getting each side
    [3 2 2 2 0 1]

Understanding Structure of Numpy Array (dimension, shape and strides)

import numpy as np
arr = np.array([[1,2,3], [2,3,1], [3,3,3]])

print('Number of array dimensions')
print(arr.ndim)

print('\nShape of array is tuple giving size of each dimension')
print(arr.shape)

print('\nstrides gives byte steps to be moved in memory to get to next \
index in each dimension')
print(arr.strides)

print('\nByte size of each item')
print(arr.itemsize)

    Number of array dimensions
    2

    Shape of array is tuple giving size of each dimension
    (3, 3)

    strides gives byte steps to be moved in memory to get to next index in each dimension
    (24, 8)

    Byte size of each item
    8

More on Strides

print('Slice indexing is done by changing strides, as in examples below')

print('Strides of original array')
print(arr.strides)

print('\n', 'Slice with step of 2 is done by multiplying stride(byte step size) by 2 in that dimension')
print(arr[::2].strides)

print('\n', 'Reverse index will negate the stride')
print(arr[::-1].strides)

print('\n', 'Transpose will swap the stride of the dimensions')
print(arr.T.strides)

    Slice indexing is done by changing strides, as in examples below
    Strides of original array
    (24, 8)

     Slice with step of 2 is done by multiplying stride(byte step size) by 2 in that dimension
    (48, 8)

     Reverse index will negate the stride
    (-24, 8)

     Transpose will swap the stride of the dimensions
    (8, 24)

Some Stride Tricks: Inner product by changing strides

It is very rare that you may want to use these tricks but it helps us understand how indexing in numpy works

as_strided function returns a view to an array with different strides and shape

from numpy.lib.stride_tricks import as_strided

arr1 = np.arange(5)
print('arr1: ', arr1)

arr2 = np.arange(3)
print('arr2: ', arr2)

print('\n', 'Adding a dimension with stride 0 allows us to repeat array in that dimension without making copy')

print('\n', 'Making stride 0 for rows repeats rows.')
print('As step size is zero to move to next row it will give same row repeatedly')
r_arr1 = as_strided(arr1, strides=(0,arr1.itemsize), shape=(len(arr2),len(arr1)))
print(r_arr1)

print('\n', 'Making stride 0 for columns repeats columns.')
r_arr2 = as_strided(arr1, strides=(arr2.itemsize, 0), shape=(len(arr2),len(arr1)))
print(r_arr2, '\n')

print('Inner product: product of every value of arr1 to every value of arr2')
print(r_arr1 * r_arr2)

    arr1:  [0 1 2 3 4]
    arr2:  [0 1 2]

     Adding a dimension with stride 0 allows us to repeat array in that dimension without making copy

     Making stride 0 for rows repeats rows.
    As step size is zero to move to next row it will give same row repeatedly
    [[0 1 2 3 4]
     [0 1 2 3 4]
     [0 1 2 3 4]]

     Making stride 0 for columns repeats columns.
    [[0 0 0 0 0]
     [1 1 1 1 1]
     [2 2 2 2 2]] 

    Inner product: product of every value of arr1 to every value of arr2
    [[0 0 0 0 0]
     [0 1 2 3 4]
     [0 2 4 6 8]]

Using Broadcast

print('Above example is equivalent to using broadcast to do inner product')
print(arr1[np.newaxis, :] * arr2[:, np.newaxis])

print('arr1[np.newaxis, :].strides => ', arr1[np.newaxis, :].strides)
print('arr2[:, np.newaxis].strides => ', arr2[:, np.newaxis].strides)

    Above example is equivalent to using broadcast to do inner product
    [[0 0 0 0 0]
     [0 1 2 3 4]
     [0 2 4 6 8]]
    arr1[np.newaxis, :].strides =>  (0, 8)
    arr2[:, np.newaxis].strides =>  (8, 0)

Data Types and Casting

Notes

Numpy array can store items of only one data type
np_array.dtype attribute will give dtype of the array
following table shows some common datatypes with their string names


Numpy Attribute                                 | String Name                | Description
------------------------------------------------------------------------------------------------------
np.int8, np.int16, np.int32, np.int64           | '<i1', '<i2', '<i4', '<i8' | signed int
np.uint8, np.uint16, np.uint32, np.uint64       | '<u1', '<u2', '<u4', '<u8' | unsigned int
np.float16, np.float32, np.float64, np.float128 | '<f2', '<f4', '<f8', '<f16'| floats
np.string_                                      |'S1', 'S10', 'S255'         | string of bytes (ascii)
np.str                                          |'U1', 'U255'                | string of unicode characters
np.datetime64                                   |'M8'                        | date time
np.Object                                       |'O'                         | python object
np.bool                                         |'?'                         | boolean

Break down of string name '<u8': here '<' means little-endian byte order, 'u' means unsigned int and '8' means 8 bytes. Other options for byte order are '>' big endian and '=' system default
All of the array initialization functions discussed above takes 'dtype' parameter to set datatype of the array eg: np.random.randint(5, 11, size=(2,2), dtype=np.int8)

Casting

import numpy as np
arr = np.arange(5, dtype='<f4')
print('arr: ', arr)

print('\n', 'Cast to integer using astype function which will make copy of the array')
display(arr.astype(np.int8))

print('\n', 'By default casting is unsafe which will ignore the overflow. e.g. `2e10` is converted to 0')
arr[3] = 2e10
print(arr.astype('<i1'))

print('\n', 'Casting from string to float')
sarr = np.array("1 2 3 4 5.0".split())
print(sarr)
print(sarr.astype('<f4'))

print('\n', 'Use casting="safe" for doing safe casting, which will raise error if overflow')
# print(arr.astype('<i1', casting='safe'))

    arr:  [0. 1. 2. 3. 4.]

     Cast to integer using astype function which will make copy of the array



    array([0, 1, 2, 3, 4], dtype=int8)



     By default casting is unsafe which will ignore the overflow. e.g. 2e10 is converted to 0
    [0 1 2 0 4]

     Casting from string to float
    ['1' '2' '3' '4' '5.0']
    [1. 2. 3. 4. 5.]

     Use casting="safe" for doing safe casting, which will raise error if overflow

Reshaping

ndarray can be reshaped to any shape as long as total number of element are same

arr = np.arange(20)
print('arr: ', arr)

print('\n', 'reshape 1D arr of length 20 to shape (4,5)')
print(arr.reshape(4,5))

print('\n', 'One item of shape tuple can be -1 in which case the item will be calculated by numpy')
print('For total size to be 20 missing value must be 5')
print(arr.reshape(2,2,-1))

    arr:  [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]

     reshape 1D arr of length 20 to shape (4,5)
    [[ 0  1  2  3  4]
     [ 5  6  7  8  9]
     [10 11 12 13 14]
     [15 16 17 18 19]]

     One item of shape tuple can be -1 in which case the item will be calculated by numpy
    For total size to be 20 missing value must be 5
    [[[ 0  1  2  3  4]
      [ 5  6  7  8  9]]

     [[10 11 12 13 14]
      [15 16 17 18 19]]]

Array View With Different dtype

arr.view() method gives new view for same data with new dtype. Creating view with different dtype is not same as casting. eg. if we have ndarray of np.float32 ('<f4') creating view with dtype byte ('<i8') will read 4 bytes float data as individual bytes

arr = np.arange(5, dtype='<i2')
print('arr: ', arr)

print('\n', 'View with dtype "<i1" for array of dtype "<i2" will breakdown items to bytes')
print(arr.view('<i1'))

print('\n', 'Changing little-endian to big-endian will change value as they use different byte order')
print(arr.view('>i2'))

print('\n', 'Following will give individual bytes in memory of each items')
arr = np.arange(5, dtype='<f2')
print(arr)
print(arr.view('<i1'))

    arr:  [0 1 2 3 4]

     View with dtype "<i1" for array of dtype "<i2" will breakdown items to bytes
    [0 0 1 0 2 0 3 0 4 0]

     Changing little-endian to big-endian will change value as they use different byte order
    [   0  256  512  768 1024]

     Following will give individual bytes in memory of each items
    [0. 1. 2. 3. 4.]
    [ 0  0  0 60  0 64  0 66  0 68]

Indexing Methods

Integer and Slice Indexing

This method of indexing is similar to indexing used in python list
Slicing always create view to the array ie. does not copy the array

import numpy as np
arr = np.arange(20)
print("arr: ", arr)

print('\n', 'Get item at index 4(5th item) of the array')
print(arr[4])

print('\n', 'Assign 0 to index 4 of array')
arr[4] = 0
print(arr)

print('\n', 'Get items in the index range 4 to 10 not including 10')
print(arr[4:10])

print('\n', 'Set 1 to alternate items starting at index 4 to 10 ')
arr[4:10:2] = 1
print(arr)

    arr:  [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]

     Get item at index 4(5th item) of the array
    4

     Assign 0 to index 4 of array
    [ 0  1  2  3  0  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]

     Get items in the index range 4 to 10 not including 10
    [0 5 6 7 8 9]

     Set 1 to alternate items starting at index 4 to 10 
    [ 0  1  2  3  1  5  1  7  1  9 10 11 12 13 14 15 16 17 18 19]

Slice Indexing in 2D Array

For multidimensional array slice index can be separated using comma

arr = np.arange(20).reshape(4,5)

print('arr:\n', arr)

print('\n', 'Set 0 to first 3 rows and and last 2 columns')
arr[:3, -2:] = 1
print(arr)

    arr:
     [[ 0  1  2  3  4]
     [ 5  6  7  8  9]
     [10 11 12 13 14]
     [15 16 17 18 19]]

     Set 0 to first 3 rows and and last 2 columns
    [[ 0  1  2  1  1]
     [ 5  6  7  1  1]
     [10 11 12  1  1]
     [15 16 17 18 19]]

Boolean Indexing

Boolean array of same shape as original array (or broadcastable to the shape) can be used as index. Which will select the items where index value is true
Boolean array can also be used to filter array with certain conditions
Boolean indexing will return copy instead of view to the array

arr = np.arange(6).reshape(2,3)
print('arr:\n', arr)

print('\n', 'Following index will gives last two items of 1st row and 1st element of 2nd row')
indx = np.array([[False, True, True], [True, False,False]])
arr[indx]

print('\n', 'Boolean index to filter values greater than 3 from arr')
filter_indx = arr>3
print('Filter Index:\n', filter_indx)

print('\n', 'Set 3 to values greater than 3 in arr')
arr[filter_indx] = 3
print(arr)

    arr:
     [[0 1 2]
     [3 4 5]]

     Following index will gives last two items of 1st row and 1st element of 2nd row

     Boolean index to filter values greater than 3 from arr
    Filter Index:
     [[False False False]
     [False  True  True]]

     Set 3 to values greater than 3 in arr
    [[0 1 2]
     [3 3 3]]

Fancy Indexing

Fancy Indexing means using array of index(integer) as index to get all items at once
Fancy Indexing will also return copy instead of view to the array

import numpy as np

arr = np.arange(10)
print('arr:\n', arr)

print('\n', 'Get items at indexes 3,5 and 7 at once')
print(arr[[3,5,7]])

print('\n', 'Sorting arr based on another array "values"')
np.random.seed(5)
values = np.random.rand(10)
print('values:\n', values)
print('\n', 'np.argsort instead of returning sorted values will return array of indexes which will sort the array')
indexes = np.argsort(values) 
print('indexes:\n', indexes)
print('Sorted array:\n', arr[indxes])

print('\n', 'You can also use fancy indexing to get same item multiple times')
print(arr[[0,1,1,2,2,2,3,3,3,3]])

    arr:
     [0 1 2 3 4 5 6 7 8 9]

     Get items at indexes 3,5 and 7 at once
    [3 5 7]

     Sorting arr based on another array "values"
    values:
     [0.22199317 0.87073231 0.20671916 0.91861091 0.48841119 0.61174386
     0.76590786 0.51841799 0.2968005  0.18772123]

     np.argsort instead of returning sorted values will return array of indexes which will sort the array
    indexes:
     [9 2 0 8 4 7 5 6 1 3]



    ---------------------------------------------------------------------------

    NameError                                 Traceback (most recent call last)

    <ipython-input-14-30ac3b9b19e2> in <module>()
         14 indexes = np.argsort(values)
         15 print('indexes:\n', indexes)
    ---> 16 print('Sorted array:\n', arr[indxes])
         17 
         18 print('\n', 'You can also use fancy indexing to get same item multiple times')


    NameError: name 'indxes' is not defined

Tuple Indexing

Multi-dimensional array can be indexed using tuple of integer array of equal length, where each array in tuple will index corresponding dimension
If number of index-array in tuple is less than the dimension of array being indexed they will be used to index lower dimension (i.e. dimension starting from 0 to length of tuple)

arr2 = np.arange(15).reshape(5,3)
print('arr2:\n', arr2)

print('\n', 'Will give items at index (4,0) and (1,2)')
indx = ([4,1],[0,2])
print(arr2[indx])

print('\n', 'Tuple of length one will return rows')
indx = ([4,1],)
print(arr2[indx])

    arr2:
     [[ 0  1  2]
     [ 3  4  5]
     [ 6  7  8]
     [ 9 10 11]
     [12 13 14]]

     Will give items at index (4,0) and (1,2)
    [12  5]

     Tuple of length one will return rows
    [[12 13 14]
     [ 3  4  5]]

Assignment With Advance Indexing

Advance Indexing(i.e. Boolean, Fancy and Tuple) returns copy instead of view to the indexed array. But direct assignment using those index will change the original array, this feature is for convenience. But if we chain the indexing it may behaves in a way which seems to be somewhat unexpected

import numpy as np

arr = np.arange(10)
print('arr: ', arr)

print('\n', 'Direct assignment will change the original array')
arr[[3,5,7]] = -1
print(arr)

print('\n', 'When we chain the indexing it will not work')
arr[[3,5,7]][0] = -2
print(arr)

print('\n', 'But chaining index will work with slicing indexing')
arr[3:8:2][0] = -2
print(arr)

    arr:  [0 1 2 3 4 5 6 7 8 9]

     Direct assignment will change the original array
    [ 0  1  2 -1  4 -1  6 -1  8  9]

     When we chain the indexing it will not work
    [ 0  1  2 -1  4 -1  6 -1  8  9]

     But chaining index will work with slicing indexing
    [ 0  1  2 -2  4 -1  6 -1  8  9]

Mixed Indexing

In multi dimensional array we can use different indexing method (slicing, boolean and fancy) for each dimension at same time
For mixture of boolean and fancy index to work, number of True's in boolean index must be equal to length of fancy index

arr = np.arange(64).reshape(4,4,4)
print('arr: ', arr)
print('\n', 'Following mixed indexing will select 1st and 3rd item in 0th dimension')
print('and item at index 0 and 2 at 1st dimension and item at index >=2')
print(arr[[True, False, True, False], [0,2], 2:])

    arr:  [[[ 0  1  2  3]
      [ 4  5  6  7]
      [ 8  9 10 11]
      [12 13 14 15]]

     [[16 17 18 19]
      [20 21 22 23]
      [24 25 26 27]
      [28 29 30 31]]

     [[32 33 34 35]
      [36 37 38 39]
      [40 41 42 43]
      [44 45 46 47]]

     [[48 49 50 51]
      [52 53 54 55]
      [56 57 58 59]
      [60 61 62 63]]]

     Following mixed indexing will select 1st and 3rd item in 0th dimension
    and item at index 0 and 2 at 1st dimension and item at index >=2
    [[ 2  3]
     [42 43]]

Array Operations

Simple Array Operations

Numpy provides simple syntax to perform mathematical and logical operations between array of compatible shapes. Here compatible shape means, shape of one of the array can be expanded to match shape of other using broadcast rule, which we'll discuss below
For this section we'll only see two cases
- Arrays has same shape in which case operation will be element wise
- One of operand is scalar in which case operation will be done between scalar and each element of array
These operations between arrays are called vectorization and are way faster than same operation using loop.
Vectorization are faster because it is implemented in C and don't have overheads like type checking etc.

import numpy as np

print('Evaluate expression (x1*x2 - 3*x1 + 30) for x1 and x2 in range 0 to 10')
x1 = np.linspace(0,10,20)
x2 = np.linspace(0, 10, 20)
z = x1*x2 - 3*x1 + 30
print(z)

print('\n', 'Spatial distance between corresponding points in two array')
p1 = np.random.rand(20,2)
p2 = np.random.rand(20,2)

'''np.sum will add values along given axis (dimension). If shape of array is (3,4,5)')
then axis 0,1 and 2 corresponds to dimension with length 3, 4 and 5 respectively'''
d = np.sum((p1-p2)**2, axis=1)
print(d)

print('\n', 'Element wise comparison, ">=" will give boolean array with True where element')
print('of p2 is greater than or equal to p1')
r = p2>=p1
print(r)

print('\n', 'Element wise logical operation, "&" will give True where point of p2 is ahead')
print('in both x and y direction from corresponding point in p1')
print(r[:,0] & r[:,1])

    Evaluate expression (x1*x2 - 3*x1 + 30) for x1 and x2 in range 0 to 10
    [ 30.          28.69806094  27.9501385   27.75623269  28.11634349
      29.03047091  30.49861496  32.52077562  35.09695291  38.22714681
      41.91135734  46.14958449  50.94182825  56.28808864  62.18836565
      68.64265928  75.65096953  83.2132964   91.32963989 100.        ]

     Spatial distance between corresponding points in two array
    [0.54052263 0.17505988 0.59108818 0.41593393 0.03548522 0.29946201
     0.84649163 0.24975051 0.90016153 0.54062043 0.00097261 0.39826495
     0.64710327 0.40655563 0.00531519 0.94567232 0.33333277 0.01713418
     0.53797027 0.48080742]

     Element wise comparison, ">=" will give boolean array with True where element
    of p2 is greater than or equal to p1
    [[ True False]
     [False False]
     [False False]
     [ True  True]
     [ True False]
     [ True  True]
     [ True  True]
     [ True False]
     [ True False]
     [False  True]
     [ True False]
     [False False]
     [ True False]
     [ True False]
     [False False]
     [ True False]
     [False False]
     [ True False]
     [False  True]
     [False False]]

     Element wise logical operation, "&" will give True where point of p2 is ahead
    in both x and y direction from corresponding point in p1
    [False False False  True False  True  True False False False False False
     False False False False False False False False]

Functions for Array Operations

Numpy also has function version of above operations like np.add, np.substract, np.divide, np.greater_equal, np.logical_and and more
Array operations we see in section above using operators like +, * are operator overloaded version of function operation
Function version of the operation will give us extra parameters to customize, one of commonly used parameter is out. It is None by default, which will create a new array for result.
If We pass an array with shape and dtype matching to expected result to out parameter result will be filled to the passed array. It will be efficient memory wise if we are doing multiple operations

import numpy as np

print('Evaluate expression (x1*x2 - 3*x1 + 30) using functions')
x1 = np.linspace(0,10,20)
x2 = np.linspace(0, 10, 20)

'''Create empty output array with expected shape'''
z = np.empty_like(x1)

'''Code is not very clean as using operator but it will perform very well memory wise'''
np.multiply(x1, x2, out=z)
np.subtract(z, 3*x1, out=z)
np.add(z, 30, out=z)
print(z)

    Evaluate expression (x1*x2 - 3*x1 + 30) using functions
    [ 30.          28.69806094  27.9501385   27.75623269  28.11634349
      29.03047091  30.49861496  32.52077562  35.09695291  38.22714681
      41.91135734  46.14958449  50.94182825  56.28808864  62.18836565
      68.64265928  75.65096953  83.2132964   91.32963989 100.        ]

Broadcasting

Rules for broadcasting

When doing array operations between two array's whose shape doesn't match exactly then few simple steps are taken which changes shapes to match each other if they are compatible.

Check Array Dimensions: If dimensions doesn't match added to left of array with smaller dimension
Match Shape On Each Dimension: If shape in any dimension doesn't match and shape is 1 for one of the array then repeat it to match shape of other array in that dimension
Raise Error if Dimension and Shape Not Matched: If dimension and shape don't match till this step then raise error

Let's Visualize the Broadcasting Rule With Custom Implementation

lets do our custom implementation to visualize in code how the broadcast rule works

import numpy as np

arr1 = np.arange(20).reshape(10,2)
arr2 = np.random.rand(2)
arr3 = arr2.copy()

print('arr1.shape: ', arr1.shape)
print('arr2.shape: ', arr2.shape)

# Step 1: Check Array Dimensions
print('\n', 'arr1 has dimension 2 and arr2 has dimension 1, so add 1 dimension to\
left side of arr2')
# np.newaxis is convenient way of adding new dimension
arr2 = arr2[np.newaxis, :]
print('arr1.shape: ', arr1.shape)
print('arr2.shape: ', arr2.shape)

# Step 2: Match Shape On Each Dimension
print('\n', 'Now in axis=0 arr1 has 10 items and arr2 has one item, so repeat it 10\
times to match arr2')
arr2 = np.repeat(arr2, 10, axis=0)
print('arr1.shape: ', arr1.shape)
print('arr2.shape: ', arr2.shape)

print('\n', 'Now both array has same dimension and shape, we can multiply them')
print('arr1*arr2:\n', arr1*arr2)

print('\n', 'Lets see if broadcasting also produce same result')
print('arr1*arr3:\n', arr1*arr3)

    arr1.shape:  (10, 2)
    arr2.shape:  (2,)

     arr1 has dimension 2 and arr2 has dimension 1, so add 1 dimension toleft side of arr2
    arr1.shape:  (10, 2)
    arr2.shape:  (1, 2)

     Now in axis=0 arr1 has 10 items and arr2 has one item, so repeat it 10times to match arr2
    arr1.shape:  (10, 2)
    arr2.shape:  (10, 2)

     Now both array has same dimension and shape, we can multiply them
    arr1*arr2:
     [[ 0.          0.11111075]
     [ 1.71941377  0.33333225]
     [ 3.43882755  0.55555375]
     [ 5.15824132  0.77777525]
     [ 6.8776551   0.99999675]
     [ 8.59706887  1.22221824]
     [10.31648264  1.44443974]
     [12.03589642  1.66666124]
     [13.75531019  1.88888274]
     [15.47472397  2.11110424]]

     Lets see if broadcasting also produce same result
    arr1*arr3:
     [[ 0.          0.11111075]
     [ 1.71941377  0.33333225]
     [ 3.43882755  0.55555375]
     [ 5.15824132  0.77777525]
     [ 6.8776551   0.99999675]
     [ 8.59706887  1.22221824]
     [10.31648264  1.44443974]
     [12.03589642  1.66666124]
     [13.75531019  1.88888274]
     [15.47472397  2.11110424]]

Let's Try Few Examples With Shapes

you can try it by creating arrays of given shape and doing some operation between them

Before Broadcast        |Step 1                      | Step 2 and 3  
Shapes of arr1 and arr2 |                            | Shapes of result 
-------------------------------------------------------------------------
(3, 1, 5); (4, 1)       | (3, 1, 5); (1, 4, 1)       | (3, 4, 5)        
(10,); (1, 10)          | (10, 1); (1, 10)           | (10, 10)         
(2, 2, 2); (2, 3)       | (2, 2, 2); (1, 2, 3)       | Not Broadcastable
(2, 2, 2, 1); (2, 3)    | (2, 2, 2, 1); (1, 1, 2, 3) | (2, 2, 2, 2, 3)

Some Usage of Broadcast

Evaluate Linear Equation Using Broadcast

print("Let's evaluate equation c1*x1 + c2*x2 + c3*x3 for 100 points at once")
points = np.random.rand(100,3)
coefficients = np.array([5, -2, 11])
results = np.sum(points*coefficients, axis=1)
print('results first 10:\n', results[:10])
print('results.shape: ', results.shape)

    Let's evaluate equation c1*x1 + c2*x2 + c3*x3 for 100 points at once
    results first 10:
     [ 6.35385279  0.85639146 12.87683079  5.99433896  4.50873972 10.44691041
      3.87407211  6.62954602 11.00386582 10.09247866]
    results.shape:  (100,)

Find Common Elements Between Arrays

np.random.seed(5)
## Get 20 random value from 0 to 99
arr1 = np.random.choice(50, 20, replace=False)
arr2 = np.random.choice(50, 15, replace=False)
print("arr1: ", arr1)
print("arr2: ", arr2)
print('\n', 'arr1 and arr2 are 1D arrays of length 20, 15 respectively.')
print('To make them broadcastable Change shape of arr2 to (15, 1)')
arr2 = arr2.reshape(15, 1)
print('\n', 'Then both arrays will be broadcasted to (15, 20) matrix with all possible pairs')
comparison = (arr1 == arr2)
print('\n', 'comparison.shape: ', comparison.shape)
print('\n', 'Elements of arr1 also in arr2: ', arr1[comparison.any(axis=0)])

    arr1:  [42 29  6 19 28 17  2 43  3 21 31  4 32  0 23  5 48 34 37 26]
    arr2:  [40 37 41 48  4 20 10 18 34 28 19 32 17 22 23]

     arr1 and arr2 are 1D arrays of length 20, 15 respectively.
    To make them broadcastable Change shape of arr2 to (15, 1)

     Then both arrays will be broadcasted to (15, 20) matrix with all possible pairs

     comparison.shape:  (15, 20)

     Elements of arr1 also in arr2:  [19 28 17  4 32 23 48 34 37]

Find k-nearest Neighbors

import numpy as np

np.random.seed(5)

points = np.random.rand(20, 2)
print('To calculate distance between every pair of points make copy of points ')
print('with shape (20, 1, 2) which will broadcast both array to shape (20, 20, 2)', '\n')
cp_points = points.reshape(20, 1, 2)

## calculate x2-x1, y2-y1
diff = (cp_points - points)
print('diff.shape: ', diff.shape)

## calculate (x2-x1)**2 + (y2-y1)**
distance_matrix = np.sum(diff**2, axis=2)
print('distance_matrix.shape: ', distance_matrix.shape, '\n')

## sort by distance along axis 1 and take top 4, one of which is the point itself
top_3 = np.argsort(distance_matrix, axis=1)[:,:4]
print("Get the points with it's 3 nearest neighbors")
points[top_3]

    To calculate distance between every pair of points make copy of points 
    with shape (20, 1, 2) which will broadcast both array to shape (20, 20, 2) 

    diff.shape:  (20, 20, 2)
    distance_matrix.shape:  (20, 20) 

    Get the points with it's 3 nearest neighbors





    array([[[0.22199317, 0.87073231],
            [0.20671916, 0.91861091],
            [0.16561286, 0.96393053],
            [0.08074127, 0.7384403 ]],

           [[0.20671916, 0.91861091],
            [0.22199317, 0.87073231],
            [0.16561286, 0.96393053],
            [0.08074127, 0.7384403 ]],

           [[0.48841119, 0.61174386],
            [0.62878791, 0.57983781],
            [0.69984361, 0.77951459],
            [0.76590786, 0.51841799]],

           [[0.76590786, 0.51841799],
            [0.62878791, 0.57983781],
            [0.69984361, 0.77951459],
            [0.87993703, 0.27408646]],

           [[0.2968005 , 0.18772123],
            [0.32756395, 0.1441643 ],
            [0.28468588, 0.25358821],
            [0.44130922, 0.15830987]],

           [[0.08074127, 0.7384403 ],
            [0.02293309, 0.57766286],
            [0.22199317, 0.87073231],
            [0.20671916, 0.91861091]],

           [[0.44130922, 0.15830987],
            [0.32756395, 0.1441643 ],
            [0.41423502, 0.29607993],
            [0.2968005 , 0.18772123]],

           [[0.87993703, 0.27408646],
            [0.96022672, 0.18841466],
            [0.76590786, 0.51841799],
            [0.5999292 , 0.26581912]],

           [[0.41423502, 0.29607993],
            [0.28468588, 0.25358821],
            [0.44130922, 0.15830987],
            [0.2968005 , 0.18772123]],

           [[0.62878791, 0.57983781],
            [0.48841119, 0.61174386],
            [0.76590786, 0.51841799],
            [0.69984361, 0.77951459]],

           [[0.5999292 , 0.26581912],
            [0.41423502, 0.29607993],
            [0.44130922, 0.15830987],
            [0.87993703, 0.27408646]],

           [[0.28468588, 0.25358821],
            [0.2968005 , 0.18772123],
            [0.32756395, 0.1441643 ],
            [0.41423502, 0.29607993]],

           [[0.32756395, 0.1441643 ],
            [0.2968005 , 0.18772123],
            [0.44130922, 0.15830987],
            [0.28468588, 0.25358821]],

           [[0.16561286, 0.96393053],
            [0.20671916, 0.91861091],
            [0.22199317, 0.87073231],
            [0.08074127, 0.7384403 ]],

           [[0.96022672, 0.18841466],
            [0.87993703, 0.27408646],
            [0.5999292 , 0.26581912],
            [0.76590786, 0.51841799]],

           [[0.02430656, 0.20455555],
            [0.28468588, 0.25358821],
            [0.2968005 , 0.18772123],
            [0.32756395, 0.1441643 ]],

           [[0.69984361, 0.77951459],
            [0.62878791, 0.57983781],
            [0.63979518, 0.9856244 ],
            [0.76590786, 0.51841799]],

           [[0.02293309, 0.57766286],
            [0.00164217, 0.51547261],
            [0.08074127, 0.7384403 ],
            [0.22199317, 0.87073231]],

           [[0.00164217, 0.51547261],
            [0.02293309, 0.57766286],
            [0.08074127, 0.7384403 ],
            [0.02430656, 0.20455555]],

           [[0.63979518, 0.9856244 ],
            [0.69984361, 0.77951459],
            [0.48841119, 0.61174386],
            [0.62878791, 0.57983781]]])

Vectorization

In numpy vectorization means performing optimized operations on sequence of same type of data.
In addition to having clean structure of code vectorized operations are also very performant because codes are compiled an avoids overhead of python, like type checking, memory management etc.
The examples we see on Broadcast section above are also good example of vectorization

Vectorization Vs Loop

Let's say we have a polynomial equation of degree 10 of single variable like a1*x + a2*x^2 + a3*x^3 ... + a10*x^10. Let's try evaluating the equation for large number of x using python only and numpy vectorization see how they compare

import numpy as np
np.random.seed(32)
X = np.random.rand(10000)
coefficients = np.random.randn(10)*20 + 50

def evaluate_polynomial_loop():
  result_loop = np.empty_like(X)
  for i in range(X.shape[0]):
    exp_part = 1
    total = 0
    for j in range(coefficients.shape[0]):
      exp_part *= X[i]
      total+=coefficients[j]*exp_part
    result_loop[i] = total
  return result_loop


def evaluate_polynomial_vect():
  ## repeates x's in 10 columns
  exponents = X[:, np.newaxis] + np.zeros((1, coefficients.shape[0]))
  exponents.cumprod(axis=1, out=exponents)
  result_vect = np.sum(exponents * coefficients, axis=1)
  return result_vect

print('Verify that both gives same result')
print('Loop:\n', evaluate_polynomial_loop()[:10])
print('Vectorization:\n', evaluate_polynomial_vect()[:10])

    Verify that both gives same result
    Loop:
     [222.57782534  30.62439847  59.69953776 373.52687079 123.89007218
     179.70369976   6.49315699 321.685257    73.14575517  69.71437596]
    Vectorization:
     [222.57782534  30.62439847  59.69953776 373.52687079 123.89007218
     179.70369976   6.49315699 321.685257    73.14575517  69.71437596]

Compare

For fair comparison I used numpy array in both whose indexing is much faster than python list. By comparison we see the vectorization is about 80 times faster

%timeit evaluate_polynomial_loop()

    114 ms ± 5.02 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit evaluate_polynomial_vect()

    1.19 ms ± 79 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Ufunc and Numba

Ufunc: Also called Universal Functions are vectorized wrapper for a function. Ufunc can operate on ndarray and support broadcasting and type casting. np.add, np.multiply etc are examples of Ufunc which are implemented in C. We can create custom Ufunc using np.frompyfunc or using numba.

Numba: Numba is just in time compiler which generate optimized machine code from pure python array and numerical functions. You can use numba.jit decorator on a function which makes the function to be compiled on its 1st run. You can use numba.vectorize decorator to convert python function to Ufunc.

Let's compare different implementations of adding two big arrays as follow

Create Big Arrays

arr1 = np.arange(1000000, dtype='int64')
arr2 = np.arange(1000000, dtype='int64')

Implementation Using Python Loop

def add_arr(arr1, arr2):
  assert len(arr1)==len(arr2), "array must have same length"
  result = np.empty_like(arr1)
  for i in range(len(arr1)):
    result[i] = arr1[i] + arr2[i]
  return result

%timeit _ = add_arr(arr1, arr2)

    596 ms ± 24.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Creating Ufunc Using `np.frompyfunc`

def add(a, b):
  return a+b

vect_add =  np.frompyfunc(add,2,1)

%timeit _ = vect_add(arr1, arr2)

    195 ms ± 12.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Using Numba JIT

'nopython=True' means convert all to machine level code, if can't convert raise error

import numba
@numba.jit(nopython=True)
def add_arr_jit(arr1, arr2):
  assert len(arr1)==len(arr2), "array must have same length"
  result = np.empty_like(arr1)
  for i in range(len(arr1)):
    result[i] = arr1[i] + arr2[i]
  return result

_ = add_arr_jit(arr1, arr2)
%timeit _ = add_arr_jit(arr1, arr2)

    3.66 ms ± 467 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Creating Ufunc Using `numba.vectorize`

'numba.vectorize' takes signature of function being converted as parameter. 'int64(int64,int64)' means takes 2 'int64' parameters and returns 'int64'

import numba

@numba.vectorize(['int64(int64,int64)'], nopython=True)
def vect_add(a, b):
  return a+b

%timeit _ = vect_add(arr1, arr2)

    2.93 ms ± 569 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Conclusion solution using numba.jit and numba.vectorize are performing much better. You can also check how numpy vectorization compares with these

More for Exploration

Some Useful Functions

np.where: element wise if .. then .. Else
np.select: select values from multiple arrays based on multiple conditions
np.concatenate, np.vstack, np.r_, np.hstack, np.c_: join multiple ndarray row wise, column wise or in a given axis
np.ravel, np.flatten: converts multidimensional array to 1D array
np.roll: do circular shift of the array along given axis

Set Operations

*np.unique(x): * gives array of unique elements on the array
*Intersect1d(x, y): * gives 1D array of elements common to both arrays
*Union1d(x, y): * gives 1D array of unique elements from both arrays
*In1d(x, y): * check if each element of x is also present on y and returns array of length equal to x with boolean values
*Setdiff1d(x, y): * gives elements of x not in y
*Setxor1d(x, y): * give elements that is either in x or y but not in both

Save and Load ndarray From Disk

np.save("filename.npy", x): save single numpy array to disk
np.load("filename.npy"): load single numpy array from disk
*np.savez("filename.npz", key1=arr1, key2=arr2): * saves multiple arrays with given key
np.savetxt("filename.npy", x): save single numpy array to disk as delimited text file
np.loadtxt("filename.npy"): load single numpy array from text file

Memory Mapping

To work with large numpy array that doesn't fit in RAM you can use numpy.memmap function to map the array to file in disk. It will transparently loads only segments of array needed for current operations.

np.memmap(filename, dtype, mode, shape): create a memory mapped array to a given file
mmap.flush(): flush all in memory changes to disk

Thank you

Heartily thanks for reading through the blog. Hope it was helpful for getting you up and running in numpy. Any comment, suggestion and constructive criticism are welcome. If you like the content please don’t forget to like 💕

Linear Regression Using Gradient Descent for Beginners - Intuition, Math and Code

Dev Khadka — Tue, 13 Aug 2019 08:23:20 +0000

Linear Regression Using Gradient Descent for Beginners- Intuition, Math and Code

In this blog we'll first try to understand Linear Regression and Gradient Descent intuitively then we'll see math behind the algorithm and then do a basic implementation of it in python.
Structure of this blog is as follow

We'll setup a hypothetical problem for using linear regression.
We'll gradually develop intuition of each step of the algorithm. We'll answer why we use that step and what it does.
Then we'll look at math behind each step and represent it in and mathematical expression
We'll implement the algorithm using the mathematical expressions we derived using python.

Prerequisite

Even if you don't have any pre-knowledge you can get some intuition about how linear regression using gradient descent works but it is better if you know

Some concept of derivative from calculus
How matrix multiplication works
Basic python programming knowledge

Problem Setup

Let's assume a hypothetical experiment in which students of agriculture science collected some data from different farms/green houses through out many years. Structure of the collected data is as follow

What we want to do is predict harvests of different farms/green houses given Average Temperature and Average Nitrite In Soil. For this we have made some assumptions based on initial analysis of the data
Relation of Temperature and Average Nitrite In Soil is somewhat linear with yield

For simplicity we also assume that other factors like water in soil, light etc. either don't affect the yield or are kept fairly similar across all the farms

Lets represent average temperature (°C) as x1 average Nitrite in Soil(ppm) as x2 and harvest yield (kg/sq. m) as y
Then based on the above assumptions we can say that there exists a linear equation

Which will fairly describe relationship between our independent and dependent variables x1, x2 and y. This relationship is also called hypothesis equation. We'll also refer to this as a model.
where w0 is a bias term and w1 and w2 are weights for x1 and x2 respectively
Now our aim is to find value of w0, w1 and w2 which will best describes the relationship

Cost Function

Now let's find the way to measure how good the model is. One straight forward way is to measure how much error the model makes in predictions. Lets say

Where
Y is target values of known data in our example "yield"
P is corresponding predicted values and
err is errors made in each predictions

Here err is array of values, we need a single numerical value to compare which model is better.

We need to combine the err values to give a single value, straight sum or average will not work because positive and negative error will cancel each other.

So we take average of square of individual error. We are taking average because we don't want the metric to depend on number of items on training data. This value is called Mean Square Error(MSE), this is our cost function which we need to minimize.

Replacing pi in above equation with our hypothesis function we get

We added subscript i to represent i^(th) row of data. So, x(i,1) means 1st feature of i^(th) row of training data
Obviously another valid cost function will be average of absolute value of individual error also known as Mean Absolute Error(MAE) but we will use MSE for now due to its simplicity.

Finding Optimum Model (Bias and Weights)

Now we got our cost function as equation(2.2) above, lets look at it in detail

MSE=MSE(w0, w1, w2) is equation with three variables w0, w1, w2 as they are the unknown values. Whereas the x’s and y’s are constants from training data
We want to find w0, w1 and w2 which will give minimum value of MSE

How would we do it?

Side Note There is a straight forward way to do it using calculus which need finding partial derivatives of the function and equating them to ? zero. As at lowest point of function with continuous derivative, tangent will be equal to zero (i.e at this point tangent is changing from positive to negative or vice versa). Then find the value of variables by solving the equations.
But this method is not always practical due to following reasons

Process will be different for different type of equation

If cost function is very complex then it is very hard to find symbolic partial derivative or solve the partial derivative equations

Gradient Descent

There is a general algorithm to minimize any function (**function must be convex else it will only find local minima) known as gradient descent. We can summarize gradient descent algorithm as

Choose an arbitrary point lets say Wa, ie arbitrary values of w0, w1 and w2
Find small change in w0, w1 and w2 which will result in fastest decrease/descent of the MSE.
For finding the small change we’ll find rate of change of MSE in direction of w0, w1 and w2 separately, these are called partial derivatives. The combined vector of rate of change in each direction is called gradient. Then we multiply the gradient by a small number.

where η is a small step size and ∇MSE is gradient of MSE at point Wa. ∇MSE represents the direction in which MSE changes fastest at point Wa and its magnitude gives rate of change of cost in that direction.
Now find new point using following equation

which will have lower MSE value. Then replace Wa with Wb and repeat from step 2 for a specified number of times or until change on MSE in each step become very small

Thought Experiment

To understand the algorithm visually/intuitively, let’s do a thought experiment.
Lets say you are operating a nano robot in a temperature field (temperature varies in 3D space and is function of location) and your objective it to take the robot to lowest temperature as fast as possible else you risk damaging it.

You can ask the robot for temperature at its current position and move robot in any direction in 3D space, how will you guide the robot to lowest temperature?

Some points to note

Here temperature is analogous to cost and location of robot is analogous to point w0, w1, w2 your objective is to find the point where cost is lowest.
Lets say w0,w_1 and w2 represents left-right, top-bottom and front-back axis
It is not possible to plot MSE against w0, w1, w2 at once because it will be in 4D plane. So we need to visualize it by plotting MSE against w0, w1, w2 one at a time.
If we plot temperature against only w0 keeping w1 and w2 at some fixed value, we’ll see curve like below. Which is a quadratic curve which has convex shape.

You can follow the following steps

Lets say the robot starts at it current location W_a
To find rate of change of temperature in left-right direction

move robot to right little bit get temperature there and comeback to previous position.
Rate of change of temperature is change in temperature divided by the distance moved.
When the distance moved tends to zero it is called the partial derivative with respect to w0 because we kept w1 and w2 constant

Similarly you can find rate of change in temperature in top-bottom and front-back direction
Combining the rate of change in all 3 directions as vector gives the gradient of temperature in the field. Which is a vector and has direction in which temperature is changing fastest and its magnitude gives the rate of change. Lets represent it by ∇T
Now we need to move the robot in direction opposite of ∇T one step so that in every step we move to lower temperature. The step size should be small enough so that it doesn’t jump to other side of curve where temperature could be higher (see curve above). So new location of the robot will be

Now repeat the steps from 2 until either you reach a satisfactory temperature point or you run out of battery (i.e. steps)

Partial Derivative

If we take partial derivative of equation(1.1) w.r.t wj, where j is coefficient index

from equation(1.1) we know

then if we take partial derivative of pi w.r.t wj all other terms except wj is constant. so the result will be

Rearranging

Matrix Representation

Matrix operations like matrix multiplication are optimized and are more efficient than doing simple array calculations. Also matrix representation make the equations more readable. So we’ll convert above equations to matrix operations

Prediction using matrix multiplication

We can represent equation 1.1 which predicts value for one instance using matrix operation to predict for all instances at once as

Here:
P is prediction which is a column vector of length m
X is matrix with m rows and n columns, m is number of data points and n number of features plus one bias term which is always 1
W is column vector of length n

Gradient using matrix operations

In equation (4.1) we found partial derivative of MSE w.r.t w_j which is j th coefficient of regression model, which is j th component of gradient vector. Based on equation(4.1) Whole gradient can be represented using matrix operation like

Here:
X^T is transpose of matrix X. It will have n rows and m columns

Training step

Coding Time

In this part we’ll implement linear regression using basic python and numpy library. numpy library is the fundamental package for scientific computing in python. Its make computation on large array of data easy and efficient.
Numpy Introduction
Please don’t get overwhelmed by numpy, we’ll only use some basic functions of it. It will be explained on comment what each function does.
Installing Numpy If you have installed python using conda or miniconda it should be already installed. If its not installed in you system you can use pip to install it

pip install numpy

Then you can import numpy library in you jupyter notebook or python file as follow

import numpy as np

Numpy Array Operations Numpy provides shorthand notations to performs arithmetic operation on array, some examples are as follow

arr1+arr2 # will returns new array which is element wise sum of two arrays
2*arr1 # will multiply each element of arr1 by 2
arr1*arr2 # will return new array which is element wise multiplication of two array

Let's Generate Data

import numpy as np
# np.random.randn(100) will give array of 100 normally distributed random 
# numbers with mean 0 and std-dev 1
# 2*np.random.randn(100)+12 changes the normal distrubution to have mean 12 and std-dev 2
nitrate = 2*np.random.randn(100)+12
temperature = 4*np.random.rand(100) + 26
# np.c_ concatinates (joins) two array column wise
x_farm = np.c_[nitrate,temperature]
# This is imaginary equation describing relation between yeild, nitrate and temperature.
# Obiously this is not know in real world problem. We are using it to generate dummy data.
yeild_ideal = .1*nitrate + .08*temperature +.6
# adding some noise on the ideal equation. 
# The noise is normally distributed with 0 mean and std-dev 0.4
yeild = yeild_ideal + .4*np.random.randn(100)
print("few instances of generated data\n", x_farm[:5])
print("\nfew instances of generated targets\n", yeild[:5] )

Output

few instances of generated data
 [[12.15862374 26.41754744]
 [ 7.7100876  26.58160233]
 [12.39040209 27.5565463 ]
 [14.89781833 26.14161419]
 [12.89437241 29.96356333]]

few instances of generated targets
 [4.37946646 3.41541999 3.41400057 4.3290903  3.70089309]

Lets Define Some Functions

# Our equations above need bais term in x which should always be 1 
def add_bais_term(x):

    # np.ones(n) will give new array of length n whose all elements are 1 
    # np.c_ concatinates two array column wise
    return np.c_[np.ones(len(x)),x]
# Root mean square cost function
def rmse_cost_func(P,Y):
    ## model is array with bais and coffecients values
    return np.sqrt(np.mean((P-Y)**2))
# Finds gradient of cost function using eq(5.2) above
def gradient_of_cost(x,y,model):
    preds = predict(x,model)

    error_term = preds-y

    # np.matmul performs matrix multiplication
    # x.T is transpose of matrix x
    xt_dot_error_term = np.matmul(x.T,error_term)/len(x)
    return xt_dot_error_term
# Do prediction using eq(5.1) above
def predict(x,model):
    #np.matmul performs matrix multiplication
    return np.matmul(x,model)
def find_linear_regression_model(x,y):
    n_epochs = 10000
    neta = 0.001

    # Initialize all parameters(wj's) to zero
    model = np.zeros(len(x[0]))

    # do n_epochs iteration
    for _ in range(n_epochs):
        grad = gradient_of_cost(x,y,model)

        # move parameters closer to optimum solution in every step 
        next_model = model - neta*grad
        model = next_model
    return model

Train Model

x_farm_with_bais = add_bais_term(x_farm)
print("First 3 data with bias\n",x_farm_with_bais[:3])
model = find_linear_regression_model(x_farm_with_bais, yeild)
print("\nmodel(w0,w1,w2)\n", model)

Output

First 3 data with bias
 [[ 1.         12.15862374 26.41754744]
 [ 1.          7.7100876  26.58160233]
 [ 1.         12.39040209 27.5565463 ]]

model(w0,w1,w2)
 [0.03759733 0.09930154 0.10164468]

Predict

# predict yeild for 
yeild_predicted = predict(x_farm_with_bais, model)
yeild_predicted[:15]

Output

array([3.93017065, 3.50509946, 4.06895977, 4.17412975, 4.36366529,
       3.79386452, 3.74566146, 3.81563257, 4.19976404, 3.90262785,
       4.21451357, 4.05253678, 4.34120954, 4.06686309, 4.24635708])

Evaluate

cost = rmse_cost_func(yeild_predicted,yeild)
print(cost)

Output

0.3679644024562277

Hurray!! We have now implemented linear regression and used it to make predictions. We also measured RMSE, which is equivalent to mean of absolute error made when doing a prediction. The mean error is close to std-dev of random error added which mean the model is working quite good.

Disclamer

Above examples are only for illustration purpose to understand the working of linear regression from inside. Following are few things to be noted

In real world we will not know the relationship of dependent and independent variables in advance as we do now. Or even we don't know if relationship is linear, we can infer something about the relation using scatter plot or other data analysis.
We usually don't evaluate performance of a model using training data because a model can memorize the training data to get high performance in training data but may not work well with new unseen data. To avoid this - - we usually separate data into three sets, training, validation and testing. Validation set is used to validate model performance and used to improve model. Testing set is used only after final model is found to estimate performance of the model on unseen data.
This is a very basic implementation of linear regression to understand the core concepts. We hardly ever need to do our own implementation. We'll always use linear regression implementation from popular libraries which will be much more robust, efficient and provides customization parameters

Summary

Here is short summary of linear regression algorithm

Please find the jupyter notebook file for whole blog and code here

Forem: Dev Khadka

Everything you need to know about git rebase

Everything you need to know about git rebase

How rebase works

When can we use rebase

What are the risks of using rebase and how to minimize them

Common practices to use git rebase

TDD in React using Jest — beginner tutorial

Overview

Prerequisite

Setup New React Project

Lets Check Some Important Files

Some Explanations

Basic Layout

List Restaurants

App Component

Implementation Steps

Test Cases for App Component

Some explanation

Command to Run Test

Using Mock for Testing

Some Explanations

Steps to Make the Tests Pass

Some Explanations

Restaurants Component

Implementation Steps for Restaurants Component

Test Cases for Restaurants Component

Some Explanation

Steps to Make Test Pass

Implement “fetchRestaurants”

Test Cases for fetchRestaurants

Some Explanations

Steps to Make the Test Pass

Some Explanations

Implement Calculate Distance

Test Cases for calculateDistance

Steps to Make the Test Pass

References

Learning Numpy by Playing: Indexing, Broadcasting, Vectorization and More

Introduction

Prerequisite

What Will We Cover

What is Numpy?

Why Use Numpy?

How to Play With Numpy?

Following This Tutorial

Some Tips On Jupyter Notebook

Creating Array

From python list

Numpy's Methods

Using np.random

Understanding Structure of Numpy Array (dimension, shape and strides)

More on Strides

Some Stride Tricks: Inner product by changing strides

Using Broadcast

Data Types and Casting

Casting

Reshaping

Array View With Different dtype

Indexing Methods

Integer and Slice Indexing

Slice Indexing in 2D Array

Boolean Indexing

Fancy Indexing

Tuple Indexing

Assignment With Advance Indexing

Mixed Indexing

Array Operations

Simple Array Operations

Functions for Array Operations

Broadcasting

Rules for broadcasting

Let's Visualize the Broadcasting Rule With Custom Implementation

Let's Try Few Examples With Shapes

Some Usage of Broadcast

Evaluate Linear Equation Using Broadcast

Find Common Elements Between Arrays

Find k-nearest Neighbors

Vectorization

Vectorization Vs Loop

Using `np.random`

Creating Ufunc Using `np.frompyfunc`

Creating Ufunc Using `numba.vectorize`