Hadoop

Hadoop

What Is Apache Hadoop?

The Apache™ Hadoop™ project develops open-source software for reliable, scalable, distributed computing.

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.


1.    Download and Install from apache website
       To start / stop hadoop use the following Commands /scripts on the hadoop installed directory:

a.  bin/start-dfs.sh

b.  bin/start-mapred.sh

c.  bin/stop-mapred.sh

d.  bin/stop-dfs.sh

2.       Check http://localhost:50030/jobtracker.jsp for Hadoop Services . 1 Node should be available for execution of MAPREDUCE jobs.
          Check http://localhost:50070/dfshealth.jsp for Namenode service.

3.       Start Putty login to hadoop server : haddop-user/hadoop

4.       Configuration file : hadoop-site.xml Set property there if required

The following settings are necessary to configure HDFS:

fs.default.name               protocol://servername:port hdfs://master.ora.org:8000

dfs.data.dir                         pathname /home/username/hdfs/data

dfs.name.dir                      pathname /home/username/hdfs/name

5.  One time format of HDFS is required for haddop to use the file system :

$ bin/hadoop namenode -format

What Is Map Reduce ?

The Mapreduce is a hadoop component which allows the programmer to split a large files into chunks and to group them together and store the output.

In SQL terminology Map is similler to Group by clause and Reduce can be similler to Union All clause.

A Mapreduce job splits the input data into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.

Lets take an example to understand the mapreduce task :
File 1 :
Hello World Bye World
File 2 :
Hello World  bye Data

the key value paise which is generated  < <word>, 1>.

< Hello, 1>
< World, 1>
< Bye, 1>
< World, 1>
The second map emits:
< Hello, 1>
< World, 1>
< bye, 1>
< Data, 1>
  

The output of the first map:
< Bye, 1>
< Hello, 1>
< World, 2>
The output of the second map:
< Bye, 1>
< World, 1>
< Hello, 1>
<Data, 1 >
 The final output of the reducer job can be :
< Bye, 2>
< Data, 1>
< Hello, 3>
< World, 2>