Forem: Ashwin Telmore

How to Install Hadoop on Ubuntu 18.04 or 20.04

Ashwin Telmore — Sat, 19 Nov 2022 12:52:44 +0000

Prerequisites

Access to a terminal window/command line
Sudo or root privileges on local /remote machines

Use the following command to update your system before initiating a new installation:

sudo apt update

Type the following command in your terminal to install OpenJDK 8:

sudo apt install openjdk-8-jdk -y

Once the installation process is complete, verify the current Java version:

java -version; javac -version

Install OpenSSH on Ubuntu
Install the OpenSSH server and client using the following command:

sudo apt install openssh-server openssh-client -y

Create Hadoop User

Utilize the adduser command to create a new Hadoop user:

sudo adduser ashwin

enter the corresponding password:

su - ashwin

Generate an SSH key pair and define the location is is to be stored in:

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

Use the cat command to store the public key as authorized_keys in the ssh directory:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Set the permissions for your user with the chmod command:

chmod 0600 ~/.ssh/authorized_keys

hdoop user to SSH to localhost:

ssh localhost

Download and Install Hadoop on Ubuntu
Visit the official Apache Hadoop project page, and select the version of Hadoop you want to implement.

tar xzf hadoop-3.2.1.tar.gz

Single Node Hadoop Deployment (Pseudo-Distributed Mode)

Edit the .bashrc shell configuration file using a text editor of your choice (we will be using nano):

sudo nano .bashrc

Define the Hadoop environment variables by adding the following content to the end of the file:

#Hadoop Related Options
export HADOOP_HOME=/home/ashwin/hadoop-3.2.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS"-Djava.library.path=$HADOOP_HOME/lib/nativ"

It is vital to apply the changes to the current running environment by using the following command:

source ~/.bashrc

Use the previously created $HADOOP_HOME variable to access the hadoop-env.sh file:

sudo nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh

. If you have installed the same version as presented in the first part of this tutorial, add the following line:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Use the provided path to find the OpenJDK directory with the following command:

echo $JAVA_HOME

Open the core-site.xml file in a text editor:

sudo nano $HADOOP_HOME/etc/hadoop/core-site.xml

Add the following configuration to override the default values for the temporary directory and add your HDFS URL to replace the default local file system setting:

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/home/ashwin/tmpdata</value>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://127.0.0.1:9000</value>
</property>
</configuration>

Use the following command to open the hdfs-site.xml file for editing:

sudo nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml

Add the following configuration to the file and, if needed, adjust the NameNode and DataNode directories to your custom locations:

<configuration>
<property>
  <name>dfs.data.dir</name>
  <value>/home/ashwin/dfsdata/namenode</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/home/ashwin/dfsdata/datanode</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>
</configuration>

Use the following command to access the mapred-site.xml file and define MapReduce values:

sudo nano $HADOOP_HOME/etc/hadoop/mapred-site.xml

Add the following configuration to change the default MapReduce framework name value to yarn:

<configuration> 
<property> 
  <name>mapreduce.framework.name</name> 
  <value>yarn</value> 
</property> 
</configuration>

Open the yarn-site.xml file in a text editor:

sudo nano $HADOOP_HOME/etc/hadoop/yarn-site.xml

Append the following configuration to the file:

<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>127.0.0.1</value>
</property>
<property>
  <name>yarn.acl.enable</name>
  <value>0</value>
</property>
<property>
  <name>yarn.nodemanager.env-whitelist</name>   
  <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PERPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>

It is important to format the NameNode before starting Hadoop services for the first time:

hdfs namenode -format

Navigate to the hadoop-3.2.1/sbin directory and execute the following commands to start the NameNode and DataNode:

./start-all.sh

Type this simple command to check if all the daemons are active and running as Java processes:

jps

Done! 🙂

To study Apache Kafka Architecture in details, and how to install, deploy configure Apache kafka.

Ashwin Telmore — Thu, 17 Nov 2022 20:46:27 +0000

Apache Kafka is an open-source, distributed event streaming platform developed by the Apache Software Foundation. This is written in Scala and Java programming languages. You can install Kafka on any platform that supports Java programming language.

This tutorial provides you with step-by-step instructions to install Apache Kafka on Ubuntu 22.04 LTS Linux system. You will also learn to create topics in Kafka and run producer and consumer nodes.

Prerequisites
You must have sudo privileged account access to the Ubuntu 22.04 Linux system.

Step 1 – Installing Java
We can run the Apache Kafka server on systems that support Java. So make sure, you have Java installed on your Ubuntu system.

Use the following commands to install OpenJDK on your Ubuntu system from the official repositories.

sudo apt update  
sudo apt install default-jdk

Verify the current active Java version.

java --version

Output:

openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1)
OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1, mixed mode, sharing)

Step 2 – Download Latest Apache Kafka
You can download the latest Apache Kafka binary files from its official download page. Alternativaly you can download Kafka 3.2.0 with the below command.

wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz

Then extract the downloaded archive file and place them under /usr/local/kafka directory.


tar xzf kafka_2.13-3.2.0.tgz 
sudo mv kafka_2.13-3.2.0 /usr/local/kafka

Step 3 – Create Systemd Startup Scripts
Now, create systemd unit files for the Zookeeper and Kafka services. That will help you to start/stop the Kafka service in an easy way.

First, create a systemd unit file for Zookeeper:

sudo nano /etc/systemd/system/zookeeper.service

And add the following content:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save the file and close it.

Next, create a systemd unit file for the Kafka service:

sudo nano /etc/systemd/system/kafka.service

Add the below content. Make sure to set the correct JAVA_HOME path as per the Java installed on your system.

find /usr/lib/jvm/java-1.x.x-openjdk

sudo vim /etc/profile

export JAVA_HOME="path that you found"
export PATH=$JAVA_HOME/bin:$PATH

logout and login again, reboot, or use source /etc/profile to apply changes immediately in your current shell

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Save the file and close.

Reload the systemd daemon to apply new changes.

sudo systemctl daemon-reload

This will reload all the systemd files in the system environment.

Step 4 – Start Zookeeper and Kafka Services

Let’s start both services one by one. First, you need to start the ZooKeeper service and then start Kafka. Use the systemctl command to start a single-node ZooKeeper instance.

sudo systemctl start zookeeper 
sudo systemctl start kafka

Verify both of the services status:

sudo systemctl status zookeeper 
sudo systemctl status kafka

That’s it. You have successfully installed the Apache Kafka server on Ubuntu 22.04 system. Next, we will create topics in the Kafka server.

Program on the concept of array and operations on array in JavaScript.

Ashwin Telmore — Tue, 11 Oct 2022 08:16:36 +0000

Aim: Program on the concept of array and operations on array.

Requirements:

Working PC or Laptop ,
Installed any operating system
Any browser (Google Chrome)
Any code Editor(VS Code)

Theory:

In JavaScript, array is a single variable that is used to store different elements. It is often used when we want to store list of elements and access them by a single variable. Unlike most languages where array is a reference to the multiple variables, in JavaScript array is a single variable that stores multiple elements.

Declaration of an Array There are basically two ways to declare an array. Example:

var House = []; // method 1 
var House = new Array(); // method 2

Push function Definition and Usage The push() method adds new items to the end of an array.

The push() method changes the length of the array.

The push() method returns the new length.

Examples Add a new item to an array:

const fruits = ["1", "2", "3", "4"];
fruits.push("5");

Pop function Definition and Usage The pop() method removes (pops) the last element of an array.

The pop() method changes the original array.

The pop() method returns the removed element.

Examples Remove (pop) the last element:

const fruits = ["1", "2", "3", "4"];
fruits.pop();

Unshift function Definition and Usage The unshift() method adds new elements to the beginning of an array.

The unshift() method overwrites the original array.

Examples Shift (remove) the first element of the array:

const fruits = ["Banana", "Orange", "Apple", "Mango"];
fruits.shift();

Shift function Definition and Usage The shift() method removes the first item of an array.

The shift() method changes the original array.

The shift() method returns the shifted element.

Examples Shift (remove) the first element of the array:

const fruits = ["Banana", "Orange", "Apple", "Mango"];
fruits.shift();

Concat function Definition and Usage The concat() method concatenates (joins) two or more arrays.

The concat() method returns a new array, containing the joined arrays.

The concat() method does not change the existing arrays.

Examples Join two arrays:

const arr1 = ["Cecilie", "Lone"];
const arr2 = ["Emil", "Tobias", "Linus"];
const children = arr1.concat(arr2);

slice function Definition and Usage The slice() method returns selected elements in an array, as a new array.

The slice() method selects from a given start, up to a (not inclusive) given end.

The slice() method does not change the original array.

Examples Select elements:

const fruits = ["Banana", "Orange", "Lemon", "Apple", "Mango"];
const citrus = fruits.slice(1, 3);

Splice Definition and Usage The splice() method adds and/or removes array elements.

The splice() method overwrites the original array.

Examples At position 2, add 2 elements:

const fruits = ["Banana", "Orange", "Apple", "Mango"];
fruits.splice(2, 0, "Lemon", "Kiwi");

Filter function Definition and Usage The filter() method creates a new array filled with elements that pass a test provided by a function.

The filter() method does not execute the function for empty elements. The filter() method does not change the original array.

Example 1 Return an array of all values in ages[] that are 18 or over:

const ages = [32, 33, 16, 40];
const result = ages.filter(checkAdult);

function checkAdult(age) {
  return age >= 18;
}

Program code

var a = [1,2,3];
console.log('a :>> ', a);
console.log("push operations ")
a.push(4);
console.log('a :>> ', a);
a.push(5);
console.log('a :>> ', a);

console.log("pop operations ")
 console.log('a.pop() :>> ', a.pop()); 
 console.log('a.pop() :>> ', a.pop()); 
console.log('a :>> ', a);

console.log("shift operations ")
 console.log('a.shift() :>> ', a.shift()); 
console.log('a.shift() :>> ', a.shift()); 
console.log('a :>> ', a);

console.log("unshift operations ")
console.log('a.unshift(1) :>> '); 
console.log('a.unshift(2) :>> '); 
a.unshift(1);
a.unshift(2);
console.log('a :>> ', a);

Output:

Conclusion:

Result:

How to use FTP servers in Ubuntu

Ashwin Telmore — Tue, 11 Oct 2022 08:01:18 +0000

In This blog we will see how to install and configure an FTP server on Ubuntu 20.04 that you use to share files between your devices.

FTP stand for File Transfer Protocol which is a standard network protocol used to transfer files to and from a remote system. We will be installing vsftpd which stand for Very Secure Ftp Daemon, a stable, secure, and fast FTP server.

FTP is a very popular protocol, for more secure and faster data transfers

Installing vsftpd on Ubuntu 20.04

The vsftpd package is available in the Ubuntu repositories. To install it, run the following commands:

sudo apt update


sudo apt install vsftpd

The ftp service will automatically start once the installation process is complete. To verify it, print the service status:

sudo systemctl status vsftpd

The output should show that the vsftpd service is active and running:

 vsftpd.service - vsftpd FTP server
     Loaded: loaded (/lib/systemd/system/vsftpd.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-03-02 15:17:22 UTC; 3s ago
      .
      .
      .

Configuring vsftpd The vsftpd server configuration is stored in the /etc/vsftpd.conf file.

For more visit the vsftpd documentation page.

In the following sections, we will go over some important settings needed to configure a secure vsftpd installation.

Start by opening the vsftpd configuration file:

sudo vi /etc/vsftpd.conf

Paste these properties at last

listen=NO
listen_ipv6=YES
anonymous_enable=NO
local_enable=YES
write_enable=YES
dirmessage_enable=YES
use_localtime=YES
xferlog_enable=YES
connect_from_port_20=YES
chroot_local_user=YES
secure_chroot_dir=/var/run/vsftpd/empty
pam_service_name=vsftpd
user_sub_token=$USER
local_root=/home/$USER/ftp
pasv_min_port=30000
pasv_max_port=31000
userlist_enable=YES
userlist_file=/etc/vsftpd.user_list
userlist_deny=NO

Save the file and restart the vsftpd service for changes to take effect:

sudo systemctl restart vsftpd

Opening the Firewall If you are running a UFW firewall , youll need to allow FTP traffic.

To open port 21 (FTP command port), port 20 (FTP data port), and 30000-31000 (Passive ports range), run the following commands:

sudo ufw allow 20:21/tcp
sudo ufw allow 30000:31000/tcp

To avoid being locked out, make sure port 22 is open:

sudo ufw allow OpenSSH

Reload the UFW rules by disabling and re-enabling UFW:

sudo ufw disable
sudo ufw enable

To verify the changes run:

sudo ufw status

Output look like

Status: active
To Action From
-- ------ ----
20:21/tcp ALLOW Anywhere
30000:31000/tcp ALLOW Anywhere
OpenSSH ALLOW Anywhere
20:21/tcp (v6) ALLOW Anywhere (v6)
30000:31000/tcp (v6) ALLOW Anywhere (v6)
OpenSSH (v6) ALLOW Anywhere (v6)

Creating FTP User

To test the FTP server, we will create a new user. Create a new user named newuser:

sudo adduser newuser

Add the user to the allowed FTP users list:

echo "newuser" | sudo tee -a /etc/vsftpd.user_list

Create the FTP directory tree and set the correct permissions:

sudo mkdir -p /home/newuser/ftp

Give the require permission to the newuser.

sudo chmod 777 /home/newuser/ftp


sudo chown -R newuser: /home/newuser/ftp

Test your configuration Perform the following step in another windows/Linux system to test out

step 1: open Command line interface of your respective system and run this following command with replacing the ip_address of remote system

ftp ip_address

Enter the user name which you newly created with password and you will see the following output

C:\Users\window>ftp 192.168.29.128 
Connected to 192.168.29.128. 
220 (vsFTPd 3.0.3) 
200 Always in UTF8 mode. User (192.168.29.128:(none)): newuser
331 Please specify the password.
Password: 
230 Login successful.
ftp>

Done!😎

Read the next blog to upload and download the file through the ftp

How to access remote system by telnet server in 2022

Ashwin Telmore — Tue, 11 Oct 2022 07:40:16 +0000

Via telnet server we can access the any computer remotely. whoever telnet is not good option for the accessing the computer instead of telnet we should use ssh but in this blog we gona talk about telnet

here are the steps

Requirements

A server with Ubuntu 22.04 installed.
A non-root user with sudo privileges.
A static IP address 192.168.0.100 configure on your server.

Remote server configuration Install Telnet Server By default, the Telnet server package is available in the Ubuntu 20.04 default repository. You can install it by just running the following command:

sudo apt install telnetd -y

Once the installation is completed, you can check the status of Telnet service using the following command:

sudo systemctl status inetd Telnet is activation in the configuration file

open file /etc/xinetd.d/telnet by running following command

vim /etc/xinetd.d/telnet

Past following code here

service telnet
{
   flags = REUSE
   socket_type = stream
   wait = no
   user = root
   server = /usr/sbin/in.telnetd
   log_on_failure += USERID
   disable = yes
}

Restart the telnet

Now, we have to open port23 in the ufw firewall. Execute the below command:

ufw allow 23/tcp

The next step is to reload the firewall to apply the changes. To reload the firewall, execute the below command:

ufw reload

Verify that the service is started and running normally on the server. Typically, you can start the telnet service with the following command.

/etc/init.d/xinetd restart

Test Telnet Connection from Remote System Telnet server is now installed and listening on port 23. Its time to connect the Telnet server from the remote system.

Now, log in to another Ubuntu system and replace ip_address with your remote system which you want to access run the following command:

telnet ip_address

Done!