Forem: Saish JagTap

Installing Hadoop 3.1.0 in Windows 10

Saish JagTap — Wed, 18 Jan 2023 16:01:36 +0000

Step 1: Install Java JDK 1.8 in system

After downloading java check your java version through this command on cmd.

Step 2: Download Hadoop version 3.1 and extract it to C:\ drive.
Hadoop version 3.1.0

Step 3: Setup System Environment Variables

To edit the system environment variable, go to environment variable in system properties.

Create 2 new user variables:

Variable Name: HADOOP_HOME Variable Value: The path of bin folder where extracted Hadoop.

Variable Name: JAVA_HOME Variable Value: The path of the bin folder in the Java directory.

To set Hadoop bin directory and Java bin directory path in system variable path, edit Path in the system variable.

Click on New and add the bin directory path of Hadoop and Java in it.

Step 4: Configuration
Edit some files located in the Hadoop directory of the etc folder.

Go to this link

Download the files and replace it at same location

add bin_new in hadoop directory.
add core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml in etc/hadoop directory.

Inside etc/hadoop directory edit hadoop-env.cmd

Replace %JAVA_HOME% with the path of the java folder where the jdk1.8 is installed and save it.

Rename bin folder to bin_old and rename bin_new to bin

Check whether hadoop is successfully installed by running this command hadoop version on cmd.

Hadoop is successfully installed in the system.

In above downloaded GitHub folder copy hadoop-yarn-server-timelineservice-3.1.0.jar file and paste in hadoop/share/hadoop/yarn folder.

Step 5: Format NameNode
Once the hadoop is installed, the NameNode is formatted. This is done to avoid deletion of all the data inside HDFS.

Run following command to format NameNode:
hdfs namenode -format

Step 6: To start run all the Apache Hadoop Distribution
Change the directory in cmd to sbin folder of hadoop directory using following command
cd C:\hadoop\hadoop-3.1.0\sbin

Start namenode and datanode using start-dfs.cmd command.

Two cmd windows will open for NameNode and DataNode

Start yarn using start-yarn.cmd command.

Two more windows will open, one for yarn resource manager and one for yarn node manager

Step 7: Verification
To access information about resource manager current jobs, successful and failed jobs, go to this link in browser- http://localhost:8088/cluster

Thank You for Reading

Styled Components <💅>

Saish JagTap — Sun, 16 Oct 2022 17:23:04 +0000

Styled Component is a library built for React and React Native.
It allows to use component level styles in application.
Styled Components are built on tagged template literals, which means, the CSS code is written in between backticks.

Installation

with npm

npm install styled-components

with yarn

yarn add styled-components

Import before using

import styled from styled-components;

Basic Styled Components

Create Button Component with the help of styled-components and can be reuse anywhere without worrying about its CSS.

Button.jsx

import styled from "styled-components";

const StyledButton = styled.button`
  border: 2px solid #4caf50;
  background-color: #4caf50;
  color: white;
  padding: 15px 20px;
  text-align: center;
  text-decoration: none;
  display: inline-block;
  font-size: 12px;
  cursor: pointer;
  transition: 0.5s all ease-out;
`;

export default StyledButton;

App.jsx

function App() {
  return (
    <div className="App">
      <StyledButton>Styled Button</StyledButton>
    </div>
  );
}

Styled Components with Props

Pass a function to a styled components template literal to adapt it based on its props.

Button.jsx

import styled from "styled-components";

const StyledButton = styled.button`
  border: 2px solid #4caf50;
  background-color: #4caf50;
  color: white;
  background-color: ${(props) =>
    props.variant === "outline" ? "#fff" : "#4caf50"};
  color: ${(props) => (props.variant === "outline" ? "#4caf50" : "#fff")};
  padding: 15px 20px;
  text-align: center;
  text-decoration: none;
  display: inline-block;
  font-size: 12px;
  cursor: pointer;
  transition: 0.5s all ease-out;
`;
export default StyledButton;

App.jsx

function App() {
  return (
    <div className="App">
      <StyledButton>Styled Button</StyledButton>
      <div>
        <br />
      </div>
      <StyledButton variant="outline">Styled Button</StyledButton>
    </div>
  );
}

Extending Styles

To easily make a new component that inherits the styling of another, just wrap it in the styled() constructor.

Button.js

import styled from "styled-components";

export const StyledButton = styled.button`
  border: 2px solid #4caf50;
  background-color: ${(props) =>
    props.variant === "outline" ? "#fff" : "#4caf50"};
  color: ${(props) => (props.variant === "outline" ? "#4caf50" : "#fff")};
  padding: 15px 20px;
  text-align: center;
  text-decoration: none;
  display: inline-block;
  font-size: 12px;
  cursor: pointer;
  transition: 0.5s all ease-out;
`;

export const FancyButton = styled(StyledButton)`
  background-image: linear-gradient(to right, #f6d365 0%, #fda085 100%);
  border: none;
`;

App.jsx

function App() {
  return (
    <div className="App">
      <StyledButton>Styled Button</StyledButton>
      <div>
        <br />
      </div>
      <StyledButton variant="outline">Styled Button</StyledButton>
      <div>
        <br />
      </div>
      <FancyButton as="a">Fancy Button</FancyButton>
      {/* as - is a polymorphic prop => pass anchor tag */}
    </div>
  );
}

Conclusion

Styled Components are a CSS-in-JS tool that bridges the gap between components and styling, offering numerous features to running in a functional and reusable way.
Documentation: https://styled-components.com/docs

Scrape E-Commerce Site using BeautifulSoup and store Data in csv file.

Saish JagTap — Sun, 21 Aug 2022 10:12:00 +0000

Scrape Flipkart website using beautifulsoup and store data in csv file.

IDE Used: PyCharm

First install all packages one by one

pip install bs4
pip install requests
pip install pandas

Import all modules

import requests
from bs4 import BeautifulSoup
import pandas as pd

requests to send HTTP request.
BeautifulSoup extract all content from web page (title, etc.)
pandas to make csv file.

Step 1: Loading web page with 'request'

request module allows to send HTTP requests. The HTTP request returns a Response object with all the response data (content, encoding, etc.)

url = "https://www.flipkart.com/search?q=laptop&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off"

req = requests.get(url)

Step 2: Extracting title with BeautifulSoup

BeautifulSoup provides lot of simple methods for navigating, searching, and modifying a DOM tree.

Extract the title

page = BeautifulSoup(req.content, "html.parser")
print(page.prettify())

title = page.head
print(title.text)

page.prittify() print all the content of page into a nicely format.

title.text print title of the page.

print without calling .text it will print full markup.

all_products = []

Add all extract data in this all_products list. Print this list at the end.

Step 3: Start scrapping

products = page.findAll("div", {"class": "_3pLy-c row"})

findAll is used for returning all the matches after scanning the entire document.

findAll(tag, attributes, recursive, text, limit, keywords)

In our case find all div tags with class name "_3pLy-c row"

.select() it returns a list of elements. The select() method to locate all elements of a particular CSS class. It find elements by Attr., ID, Class, etc.

.strip() method use to remove any extra newlines/whitespaces.

Step 4: Extract Name and Price of products

for product in products:
    lname = product.select("div > div._4rR01T")[0].text.strip()
    print(lname)
    price = product.select("div > div._30jeq3 ")[0].text.strip()
    print(lprice)

    all_products.append([lname, lprice])

print("Record Inserted Successfully...")

all_products.append append Name and Price in all_products list.

print(all_products)

Step 5: Add Data in CSV file.

col = ["Name", "Price"]

First create list for column name.

data = pd.DataFrame(all_products, columns=col)
print(data)

DataFrame It is a 2 dimensional data structure, like a 2 dimensional array, or a table with rows and columns.

data.to_csv('extract_data.csv', index=False)

to_csv used to export pandas DataFrame to csv file.

Code

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://www.flipkart.com/search?q=laptop&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off'
req = requests.get(url)

page = BeautifulSoup(req.content, "html.parser")
print(page.prettify())

title = page.head
print(title.text)

all_products = []
col = ['Name', 'Price']

products = page.findAll("div", {"class": "_3pLy-c row"})
print(products)

for product in products:
    lname = product.select("div > div._4rR01T")[0].text.strip()
    print(lname)
    price = product.select("div > div._30jeq3 ")[0].text.strip()
    print(lprice)

    all_products.append([lname, lprice])

print("Record Inserted Successfully...")

data = pd.DataFrame(all_products, columns=col)
print(data)
data.to_csv('laptop_products.csv', index=False)

GitHub

Thank You 😊