Forem: shubham mishra

Mastering Machine Learning :Why One-Hot Encode Data in Machine Learning?

shubham mishra — Sat, 18 Jan 2025 07:10:33 +0000

Understanding Categorical Data

Categorical data refers to information divided into specific groups or categories. For instance, when an organization collects biodata of its employees, the resulting data is categorized based on variables such as gender, state of residence, or department. This type of data is called categorical because it can be grouped by these shared attributes.
Examples of Categorical Data:

A “pet” variable with values: dog, cat.
A “color” variable with values: red, green, blue.
A “place” variable with values: first, second, third.

Categorical data must often be converted into numerical data to be utilized effectively in machine learning models. Two common methods to achieve this are:

Integer Encoding
One-Hot Encoding

What Is One-Hot Encoding?

One-hot encoding is a method of converting categorical variables into a numerical form that machine learning algorithms can process. It transforms each category value into a new binary column. Each binary column represents one category, with a value of 1 indicating the presence of the category and 0 indicating its absence.

Why Use One-Hot Encoding?

One-hot encoding is crucial because most machine learning algorithms cannot work with categorical data directly. Algorithms require numerical input to compute distances, probabilities, and patterns effectively. Here’s why one-hot encoding is often preferred:

** Prevents Misinterpretation:**

a. Unlike integer encoding, one-hot encoding prevents algorithms from interpreting category values as having a numerical hierarchy or relationship. For example, it avoids assuming that a “category 3” is greater than a “category 1.”
Ensures Data Compatibility: a.Many machine learning models like logistic regression, neural networks, and decision trees perform better with one-hot encoded data.
Widely Supported: a.Libraries such as scikit-learn (“sklearn”) provide robust support for implementing one-hot encoding efficiently.

How to Apply One-Hot Encoding?

In Python, one-hot encoding can be implemented using libraries like pandas or scikit-learn. Below is an example using pandas:
Python Code Example:
`
import pandas as pd

Sample dataset

data = {
'Bike': ['KTM', 'Ninza', 'Suzuki'],
'Price': [100, 200, 300]
}# Create a DataFrame
df = pd.DataFrame(data)# Apply one-hot encoding
df_encoded = pd.get_dummies(df, columns=['Bike'])print(df_encoded)`

Conclusion

One-hot encoding is a critical step in data preprocessing for machine learning. By converting categorical data into binary columns, it ensures algorithms can interpret and process the data correctly. Whether you’re working with simple datasets or advanced models, mastering one-hot encoding will significantly enhance your ability to work effectively with categorical data.

Would you like assistance with implementing one-hot encoding in your machine learning project? Let us know in the comments below!

https://www.orientalguru.co.in/myArticles/what-is-the-full-form-of-yarn
https://www.developerindian.com/articles/understanding-decision-trees-for-regression-step-by-step-explanation

https://www.developerindian.com/articles/outlier-detection-for-machine-learning-a-comprehensive-guide

What is Logical Planning

shubham mishra — Sun, 12 Feb 2023 19:44:32 +0000

Taking user code and converting it into a logical plan is the first phase of execution. The logical plan only converts the user’s set of expressions into the most optimized version.

It does this by converting the user code into an unresolved logical plan. This logical plan is unresolved because although your code might be valid, the tables or columns that it refers to may or may not exist. Spark uses a repository of all table, catalog and DataFrame information to resolve columns and tables in the analyzer. The analyzer may reject the unresolved logical plan if the required table or column name does not exist in the catalog. If the analyzer is able to resolve it, the result is passed through the Catalyst Optimizer.

The Packages can extend Catalyst to include their own rules for domain specific optimizations.

SQL query to find second highest salary using dens_rank function ?

shubham mishra — Sun, 12 Feb 2023 19:39:40 +0000

This is Question is most use full in Sql interview ,Below is Employee Table and we are going to write Sql Query using Dense rank function that is windowing function :

Name age salary department
shubham mishra 31 20000 computer
rahul kumar 20 30000 electric

select Name , age , salary   , 
dens_rank  over (partition by  department    order by  salary desc ) as rank 
from 
Employee 
where rank =2

This Solution is provided by Shubham mishra This article is contributed by Developer Indian team. Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above. Also folllow our instagram , linkedIn , Facebook , twiter account for more

What is Sql Constraints ?

shubham mishra — Sun, 12 Feb 2023 19:34:28 +0000

SQL Constraint are used to implement business rules.
Used to define rules to allow or restrict what values can be stored in columns.
The purpose of inducing constraints is to enforce integrity of database.
Type Of Constraints

NOT NULL
UNIQUE
PRIMARY KEY
FOREIGN KEY or Referential Key
CHECK

Below details description on Type of SQL Constraints
In table level constraints are declared after declaring all columns. Use table level when more than one column participates in constraint declaration.

NOT NULL:
NOT NULL constraint allows to specify that a column can not contain any NULL value.
NOT NULL can be used to CREATE and ALTER a table.

UNIQUE :
The UNIQUE constraint does not allow to insert a duplicate value in a column.
More than one UNIQUE column can be used in a table.

PRIMARY KEY:

A PRIMARY KEY constraint enforces the table to accept unique data for a specific column
and this constraint create a unique index for accessing the table faster.

FOREIGN KEY:

A FOREIGN KEY in L creates a link between two tables by one specific column of both table.
The specified column in one table must be a PRIMARY KEY
and referred by the column of another table known as FOREIGN KEY.

CHECK:

A CHECK constraint controls the values in the associated column.
The CHECK constraint determines whether the value is valid
or not from a logical expression.

This is Example primary key Constrains:

CREATE TABLE Student(roll_no varchar(7),name varchar(20) ,address
varchar(50) ,tot_marks int, branch varchar(6), PRIMARY KEY(roll_no));

*Conclusion *:
Here we Learn What is Sql Constarints ?

What is type of SQL Constraints?

It Is clear we are using constraints to restrict certain activity on data of Table.

If You have any doubt reach out to us

css interview questions and answers 2023 -dev.to

shubham mishra — Sun, 12 Feb 2023 19:27:13 +0000

What is Position related
What is Pseudo css?
What is the Box model in CSS?
What are the advantages of using CSS?
What are the limitations of CSS?
What is CSS and how does it work with HTML?
What are selectors and what are their different types?
What is CSS selector specificity and how does it work?
Describe z-index and how stacking context is formed.
Describe BFC (Block Formatting Context) and how it works.
Have you ever used a grid system, and if so, what do you prefer?
Have you used or implemented media queries or mobile specific layouts/CSS?
Explain how a browser determines what elements match a CSS selector.
Describe pseudo-elements and discuss what they are used for.
Explain your understanding of the box mode.
What does * { box-sizing: border-box; } do? What are its advantages?
What is the CSS display property and can you give a few examples of its use?
What's the difference between inline and inline-block?
What's the difference between the "nth-of-type()" and "nth-child()" selectors?
What's the difference between a relative, fixed, absolute and statically positioned element?
What existing CSS frameworks have you used locally, or in production? How would you change/improve them?
Have you used CSS Grid?

What is different type of Selector in CSS -dev.to

shubham mishra — Sun, 12 Feb 2023 19:22:49 +0000

What is different type of Selector in CSS

Basically Cascading style sheet css used to design a font and color etc on html element therefore we need to specify target html element .

So if you want to select or target HTML element ,following are selector :

Universal selector
Type selector
class selector
ID selector

** Introduction of Css and History of Css**
Cascading Style Sheets (CSS) is a simple mechanism for adding style
(e.g., fonts, colors, spacing) to Web documents

Althrogh HTML isvery essentialto generate adding
a web page bu their aresome important drawback which HTML
suffer

HTML has no style for layout
Using HTML we can’t add position of element
HTML does’t support for multiple unit like px ,em etc

Background image display using HTML repeate itself

To improve look and feel WC3 school launch a new language
called css in year 1996 .

The term css stand for Cascading Style Sheets
It is purly a design language
It only to handle look and feel of our page .

This means that still we have to use HTML
to process element but there presentation manage
by CSS

CSS Syntex :

Selector { property : Value }

“:+” single value

As I mention CSS are provide below type selector which are :

Universal selector : This selector are apply to all element of Html.
Type selector : This selector are apply to particular HTML tag.
Class selector :This selector are apply on same value of class attriute.
ID selector : This selector are apply on attribute of ID of Html element

Learn CSS - dev.to

shubham mishra — Sun, 12 Feb 2023 19:15:08 +0000

How to Write CSS

If you familiar with HTML syntax will notice that CSS syntax looks a different in nature.
CSS is apply on HTML tags.Learning css is much easier.
There is multiple method to apply CSS in HTML,Instead of listing page content, CSS lists the style rules that are assigned to HTML elements,
CSS is apply on entire HTML document, or even multiple HTML documents.
These rules are processed by the web browser loading the HTML file.So you can say our bowser (chrome, Edge ,Mozilla ) contain compiler for CSS.
CSS Comments
Similarly with HTML, you can write comments in CSS. Comments are ignored by the browser and are useful in providing context and meaning and functionality notes for your code.
To comment in CSS, write /*, then your comment text, then end with */.

Why is CSS Used For in development ?

The CSS is provide design , animation and position to web application as we know CSS would look as graphic property as well. To illustrate, our HTML Element on website.

What is a CSS Selector?
Here i describe full explaination of CSS Selector.i.e CSS Selector.

What is CSS style declaration?
This means that still we have to use HTML to process element but there presentation manage by CSS.

CSS Syntex :
Selector { property : Value }

What is a CSS Style property?
A CSS property is a characteristic (like color, animation , position) whose connect value defines one aspect of how the browser compile and display the element after changes.

There are CSS properties are follow that you can use to style a html element :

Float
Clear
Overflow
Text-transform
Z-index
Font-size
Background-color
Width
Height
Border
Padding
Margins
Font-family

Below is an example of a basic CSS that would print the words Hello World.

My CSS experiment

Hello World!

This is my first CSS example

Mandatory Tools you will need to install before work on Css or HTML and javascript.

you will need a Pentium 200-MHz computer with a minimum of 128 MB of RAM.
Linux 7.1 or Windows 95/98/2000/XP operating system.
chrome browser.
Microsoft Notepad or any other text editor or any tool like visual studio.

Scala class inherit or extend another case class not possible

shubham mishra — Fri, 20 Jan 2023 10:52:16 +0000

Scala case class inherit , #extend , #case class , #case class #inheritance #code #coding_question

It is not possible that we can not inherite another case class .

If you want to inherite then you can create Trait
Here we create Student Trait
case class Student(identifier: String) {}

case class SportTeam (salary: Long) extends student
Here we define a Student
trait Student{ def identifier: String }

Here we create class which is extends Student
case class SportTeam(identifier: String, salary: Long) extends Student

Case classs not able to extend of another case class because of following -

It led to wrong circumstances and unexpected behaviour.
It produce issue in multiple problems in the pattern matcher.
Case class most consistent with inheritance would violate the spec.
Scala doesn’t generate a copy method if one already exists in the class/traits the case class inherits.

Interview questions for sql queries 2023

shubham mishra — Sun, 15 Jan 2023 12:45:20 +0000

If you want to improve SQL skills, then install a SQL package like MySQL and start practicing with it. To get you started, we’ve outlined a few SQL query questions in this post.
The interview questions for sql queries we’ve filtered out of interviews held by top IT MNC like Flipkart and Amazon. So you’ll gain real-time experience by going through them.
Interview questions for sql queries is contain easy to hard variety of question and now a days windowing function is necessary in advance SQL queries

If you interested to grap a command on SQL ,we providing sql test questions and answers which is easy to help in crack interview exam and sql technical exam. Here we have example of emp and dept table in mysql . emp table in sql contain data of his employee . check out employee and department table queries in mysql

The MySQL schema for this database

DROP TABLE IF EXISTS dept;
DROP TABLE IF EXISTS salgrade;
DROP TABLE IF EXISTS emp;

CREATE TABLE salgrade(
grade int(4) not null primary key,
losal decimal(10,2),
hisal decimal(10,2));

CREATE TABLE dept(
deptno int(2) not null primary key,
dname varchar(50) not null,
location varchar(50) not null);

CREATE TABLE emp(
empno int(4) not null primary key,
ename varchar(50) not null,
job varchar(50) not null,
mgr int(4),
hiredate date,
sal decimal(10,2),
comm decimal(10,2),
deptno int(2) not null);

insert data in dept table

insert into dept values (20,'Research','Dallas');

insert into dept values (30,'Sales','Chicago');

insert into dept values (40,'Operations','Boston');

Insert data in emp table

insert into emp values (7369,'SMITH','CLERK',7902,'93/6/13',800,0.00,20);

insert into emp values (7499,'ALLEN','SALESMAN',7698,'98/8/15',1600,300,30);

insert into emp values (7521,'WARD','SALESMAN',7698,'96/3/26',1250,500,30);

insert into emp values (7566,'JONES','MANAGER',7839,'95/10/31',2975,null,20);

insert into emp values (7698,'BLAKE','MANAGER',7839,'92/6/11',2850,null,30);

insert into emp values (7782,'CLARK','MANAGER',7839,'93/5/14',2450,null,10);

insert into emp values (7788,'SCOTT','ANALYST',7566,'96/3/5',3000,null,20);

insert into emp values (7839,'KING','PRESIDENT',null,'90/6/9',5000,0,10);

insert into emp values (7844,'TURNER','SALESMAN',7698,'95/6/4',1500,0,30);

insert into emp values (7876,'ADAMS','CLERK',7788,'99/6/4',1100,null,20);

insert into emp values (7900,'JAMES','CLERK',7698,'00/6/23',950,null,30);

insert into emp values (7934,'MILLER','CLERK',7782,'00/1/21',1300,null,10);

insert into emp values (7902,'FORD','ANALYST',7566,'97/12/5',3000,null,20);

insert into emp values (7654,'MARTIN','SALESMAN',7698,'98/12/5',1250,1400,30);

Insert data into salgrade table

insert into salgrade values (1,700,1200);

insert into salgrade values (2,1201,1400);

insert into salgrade values (3,1401,2000);

insert into salgrade values (4,2001,3000);

insert into salgrade values (5,3001,99999);

Q.1 Write an SQL query to print the first three characters of ename from emp table
Select substring(ename ,1,3) from emp;

Q.2 Write an SQL query to print the ename from Worker table after removing white spaces from the right side.
Select LTRIM(ename) from emp;

Q.3 Write an SQL query to print the job from Worker table after removing white spaces from the left side
Select LTRIM(job ) from emp;

Q.4 Write an SQL query that fetches the unique values of job from emp table and prints its length.
Select distinct length(job ) from emp ;

Q.5 Write an SQL query to print all emp details from the emp table order by ename Ascending.
Select * from emp order by ename asc;

Q.6 Write an SQL query to print details for emp with the ename as “ADAMS” and “JAMES” from emp table.
Select * from emp where ename in ('Vipul','Satish');

Q.7 Write an Query print sum of salary based on department ?
selet sum(Sal) ,deptno from emp group by deptno having sum(sal) > 5000

Q.8 Write an Query print sum of Salary based on job and also print ename ,job

select ename ,job, salary from emp b
inner join
(select sum(Sal) as salary ,job from emp group by job ) a
on b.job = a.job

Q.9 Write an Query select depart name is not present in emp table

select b.dname , b.location form emp a
left outer join dept b
on b.deptno = a.deptno
where a.deptno is null

Q.10 Write an SQL query to print details of emp with Job name as "CLERK".
Select * from emp where job like 'CLERK%';

Q.11 Write an SQL query to print details of the emps whose ename contains ‘a’.
Select * from emp where ename like '%a%';

Q.12. Write an SQL query to print details of the empswhose ename ends with ‘a’.
Select * from emp where ename like '%a';

Q.13. Write an SQL query to print details of the emp whose ename ends with ‘h’ and contains six alphabets.
Select * from emp where ename like '_____h';

Q.14. Write an SQL query to print details of the emps whose SALARY lies between 100000 and 500000.
Select * from emp where sal between 100000 and 500000;

Q.15. Write an SQL query to print details of the emps who have joined in Feb’2014.
Select * from emp where year(hiredate) = 2014 and month(hiredate) = 2;

Q.16. Write an SQL query to fetch the count of employees working in the job ‘CLERK’.
SELECT COUNT(*) FROM emp WHERE DEPARTMENT = 'CLERK';

Q.17Write an SQL query to fetch the no. of emp for each Job in the descending order.
SELECT DEPARTMENT, count(WORKER_ID) No_Of_Workers
FROM worker
GROUP BY DEPARTMENT
ORDER BY No_Of_Workers DESC;

Q.18 Write an SQL query to fetch duplicate records having matching data in some fields of a table.
SELECT ename , COUNT()
FROM emp
GROUP BY ename
HAVING COUNT() > 1;

Q.19 Write an SQL query to clone a new table from another table.
SELECT * INTO empClone FROM Worker;

Q.20 The general way to clone a table without information is:
SELECT * INTO empClone FROM Worker WHERE 1 = 0;

Q.21 Write an SQL query to fetch intersecting records of two tables.
(SELECT * FROM emp)
INTERSECT
(SELECT * FROM EmpClone);

Q.22 Following MySQL query returns the current date:
SELECT CURDATE();

Q.21 Following MySQL query returns the current date and time:
SELECT NOW();

Q.23 Following SQL Server query returns the current date and time:
SELECT getdate();

Q.24 Following Oracle query returns the current date and time:
SELECT SYSDATE FROM DUAL;

Q.25. Write an SQL query to show the top n (say 10) records of a table.
SELECT * FROM Worker ORDER BY Salary DESC LIMIT 10;

Q.26 Following SQL Server query will return the top n records using the TOP command:
SELECT TOP 10 * FROM Worker ORDER BY Salary DESC;

Q.27 Following Oracle query will return the top n records with the help of ROWNUM:
SELECT * FROM (SELECT * FROM Worker ORDER BY Salary DESC)
WHERE ROWNUM <= 10;

Q.28 How can we avoid duplicating records in a query?
SELECT distinct * FROM Worker
OR
SELECT count(*),ename FROM Worker group by ename
The SELECT DISTINCT statement is used to return only distinct (different) values. Inside a table, a column often contains many duplicate values;
In Group by taking column name and provide us disctinct result based on Column value.

Q.29 Explain the difference between Rename and Alias.
RENAME TABLE emp TO emp_old, emp_new TO emp;
Or
An alias is created with the AS keyword.
SELECT ename AS EmployeeName FROM Emp

Q.30 What is schema?
In a SQL database, a schema is represent a list of logical structures of data. A database user owns the schema,It can be single or multiple . depend upon requirment . which has the same name as the database manager. As of SQL Server 2005, a schema is an individual entity (container of objects) distinct from the user who constructs the object.

Q.31 View contain Data?
No ,It is contain virtual represtation of data, with cutomize out put.
Answer: No, Views are virtual structures.

Q.32 Can a View be based on another View?
Answer: Yes, A View is based on another View.

Q.33 What is CTE?
Answer: A CTE or common table expression is an expression that contains a temporary result set which is defined in a SQL statement.

Q-34. Write an SQL query to print the name of employees having the highest salary in each department.
Ans.
The required query is:

SELECT t.dname ,
t.ename,
t.sal as salary,
from(SELECT max(sal) as
TotalSalary,dname from dept group by deptno ) as TempNew
Inner Join emp t on TempNew.deptno =t.deptno
and TempNew.TotalSalary=t.sal;

Q-35 Write an SQL query to fetch three max salaries from a table of employee.
Ans.
SELECT distinct Salary
from emp
a WHERE 3 >= (SELECT
count(distinct Salary) from emp b
WHERE a.Salary <= b.Salary)
order by a.Salary desc;

Q-36 Write an SQL query to fetch three min salaries from a table of employee.
Answer
SELECT distinct sal
from emp a
WHERE 3 >= (SELECT count(distinct Salary)
from worker b
WHERE a.Salary >= b.Salary)
order by a.Salary desc;

Q-37 Write an SQL query to fetch nth max salaries from a table of employee.
Answer
SELECT distinct Salary
from worker a
WHERE n >= (SELECT count(distinct Salary)
from worker b
WHERE a.Salary <= b.Salary)
order by a.Salary desc;

Q-38 Write an SQL query to fetch the names of employee who earn the highest salary in table :
Answer:
SELECT ename, sal
from emp
WHERE sal=(SELECT max(sal) from emp);

Q-39 Select the department name of the company which is assigned to the employee whose employee id is grater 103
Answer : select dname from empdept where deptno in (select dept from emp where empid>103)

Q-40 Select the name of the employee who is working under shubham.
Answer :select ename from emp where mgr =(select empid from employee where empname='shubham')

Q-41 Select the name of the employee who is department head of HR.
Answer : select ename from emp where empno =(select mgr from dept where dname='hr')

Q-42 select the details of all employee working in development department.
Answer : select * from emp where deptno in(select deptno from dept where dname='development')

Q-43 Write an SQL query to delete only emp table data.
Ans. We can use the TRUNCATE query to delete data from the SQL database table.
TRUNCATE TABLE emp;

Q-44 Write an SQL query for removing duplicates from a table without using a temporary table.

DELETE FROM StudentStipend
WHERE StudId IN (
SELECT StudId
FROM StudentStipend
GROUP BY Project, Stipend
HAVING COUNT(*) > 1));

Q-45 Write an SQL query for creating a new table with data and structure copied from another table.
Ans. We can perform the required operation using the SELECT INTO query.
SELECT * INTO newTable FROM StudentDetails;

Q-46 Write an SQL query for finding current date-time.
Ans. SQL queries for various Databases are as described below.
SQL Query In Oracle
SELECT SYSDATE FROM DUAL;

Q-47 SQL Query In SQL Server
SELECT getdate();

Q-48 SQL Query In MySQL
SELECT NOW();

Q-49 Using limit clause(MySQL)
SELECT ename FROM emp ORDER BY ename DESC LIMIT N-1,1;

Q-50 How to display Date in DD-MON-YYYY Employee table?
Answer: Select to_date (Hire_date,’DD-MON-YYYY’) Date_Format from Employee;

Q-51 Write SQL query for fetching top n records using LIMIT in emp table ?
Ans. SQL queries for fetching top n records using LIMIT for various Databases are as described below.
In MySQL in emp table
SELECT * FROM emp ORDER BY ename DESC LIMIT N;

In SQL server using the TOP command
SELECT TOP N * FROM emp ORDER BY ename DESC

Conclusion

Here in this Article We can see sql questions practice Details of list of basic sql practical questions . sql query questions is help you to start your learning in sql . We also provide you question of emp and dept table in mysql. sql queries interview questions

Hadoop HDFS Commands

shubham mishra — Sat, 14 Jan 2023 05:54:48 +0000

Hadoop HDFS Commands
Most of the commands in FS shell behave like corresponding Unix commands. Differences are described with each of the commands. Error information is sent to stderr and the output is sent to stdout.

We will start with some very basic help commands of hadoop and go into more detail as we go through this lesson.

HDFS Commands
fsck
HDFS Command to check the health of the Hadoop file system.
Command: hdfs fsck /

ls
HDFS Command to display the list of Files and Directories in HDFS.
Command: hdfs dfs –ls /

mkdir
HDFS Command to create the directory in HDFS.
Command: hdfs dfs –mkdir /new_deveoperIndian

touchz
HDFS Command to create a file in HDFS with file size 0 bytes.
command: hdfs dfs –touchz /directory/new_deveoperIndian

du
HDFS Command to check the file size.Command:
hdfs dfs –du –s /new_developerIndian/testfile

cat
HDFS Command that reads a file on HDFS and prints the content of that file to the console as output.
Command: hdfs dfs –cat /new_developerIndian/testfile

text
HDFS Command that takes a source file and outputs the file in text format.
Command: hdfs dfs –text /new_developerIndian/testfile

copyFromLocal
HDFS Command to copy the file from a Local file system to HDFS.
Command: hdfs dfs –copyFromLocal /home/edureka/test /new_developerIndian

copyToLocal
HDFS Command to copy the file from HDFS to Local File System.
Command: hdfs dfs –copyToLocal /new_developerIndian/test /home/developerIndian

put
HDFS Command to copy single source or multiple sources from local file system to the destination file system.
Command: hdfs dfs –put /home/edureka/test /user

get
HDFS Command to copy files from hdfs to the local file system.
Command: hdfs dfs –get /user/test /home/developerIndian

count
HDFS Command to count the number of directories, files, and bytes under the paths that match the specified file pattern.
Command: hdfs dfs –count /user

rm
HDFS Command to remove the file from HDFS.
Command: hdfs dfs –rm /new_developerIndian/test

rm -r
HDFS Command to remove the entire directory and all of its content from HDFS.
Command: hdfs dfs -rm -r /new_developerIndian

cp
HDFS Command to copy files from source to destination. This command allows multiple sources as well, in which case the destination must be a directory.
Command: hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2

mv
HDFS Command to move files from source to destination. This command allows multiple sources as well, in which case the destination needs to be a directory.
Command: hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2

expunge
HDFS Command that makes the trash empty.
Command: hdfs dfs -expunge

rmdir
HDFS Command to remove the directory.
Command: hdfs dfs –rmdir /user/hadoop

usage
HDFS Command that returns the help for an individual command.
Command: hdfs dfs -usage mkdir

help
HDFS Command that displays help for given command or all commands if none is specified.
Command: hdfs dfs -help

Uese of Hadoop HDFS

Hdfs is used for Very Large Files
HDF work with Streaming Data Access
It is support low cost hardware.

Type of data in hadoop

shubham mishra — Sat, 14 Jan 2023 05:17:00 +0000

First developed by Doug Cutting and Mike Cafarella in 2005, Licence umder forApache License 2.0
Apache Hadoopis an open source software framework for storage and large scale processing of data-sets on clusters of commodity hardware. The Hadoop Distributed File System (HDFS) is Hadoop's storage layer. Housed on multiple servers, data is divided into blocks based on file size. These blocks are then randomly distributed and stored across slave machines. HDFS in Hadoop Architecture divides large data into different blocks
Cloudera HadoopCloudera's open source platform, is the most popular distribution of Hadoop and related projects in the world .

** What is big data ?
**
Big Data is a collection of data that is huge in volume, yet growing exponentially with time. It is a data with so large size and complexity that none of traditional data management tools can store it or process it efficiently. Big data is also a data but with huge size. For this we can useApache Hadoopandcloudera hadoop

Type of data used in big data(Apache Hadoop / Spark)?

**Structured –
**that which can be stored in rows and columns like relational data sets

Unstructured – data that cannot be stored in rows and columns like video, images, etc.
Semi-structured – data in XML that can be read by machines and human
Unstructured
Structured –that which can be stored in rows and columns like relational data sets

**Semi-structured
**Unstructured – data that cannot be stored in rows and columns like video, images, etc.

Lights
Semi-structured – data in XML that can be read by machines and human

About Apache Hadoop

Hadoop is the most important framework for working with Big Data. The biggest strength of Hadoop is scalability. It can upgrade from working on a single node to thousands of nodes without any issue in a seamless manner.
Advantages of hadoop
Apache hadoopstores data in a distributed fashion, which allows data to be processed distributedly on a cluster of nodes

In short, we can say thatApache hadoopis an open-source framework. Hadoop is best known for its fault tolerance and high availability feature
Apache hadoopclusters are scalable.
TheApache hadoopframework is easy to use.
In HDFS, the fault tolerance signifies the robustness of the system in the event of failure. The HDFS is highly fault-tolerant that if any machine fails, the other machine containing the copy of that data automatically become active.

Hadoop hdfs

shubham mishra — Fri, 13 Jan 2023 20:03:12 +0000

The File System (FS) shell includes various shell-like commands that directly interact with the Hadoop Distributed File System (HDFS) as well as other file systems that Hadoop supports, such as Local FS, WebHDFS, S3 FS, and others. The FS shell is invoked by:

**Hadoop HDFS Commands
**Most of the commands in FS shell behave like corresponding Unix commands. Differences are described with each of the commands. Error information is sent to stderr and the output is sent to stdout.

If HDFS is being used, hdfs dfs is a synonym. Relative paths can be used. For HDFS, the current working directory is the HDFS home directory /user/ that often has to be created manually. The HDFS home directory can also be implicitly accessed, e.g., when using the HDFS trash folder, the .Trash directory in the home directory.

We will start with some very basic help commands and go into more detail as we go through this lesson.

fsck
ls
mkdir
touchz
du
cat
text
copyFromLocal
copyToLocal
put
get
count
rm
rm -r
cp
mv
expunge
rmdir
usage
help