Step 1~3. 하둡(Hadoop) 설치, 환경설정, 예제(WordCount) 테스트 까지.

본 문서는 하둡(Hadoop)과 자바(JAVA)을 설치하고 기본적인 설정(의사분산모드) 후 예제 테스트까지하는 방법을 기술하고 있다.

본 문서의 내용은 본 문서가 작성된 일자 이전 대해서 확인 된 방법으로써 이후 변경된 내용을 담고 있지 않거나 혹은 일부는 추후 업데이트할 예정이다.

---

common에 있는 것으로 다운로드 (차이점은 모르겠음)

wget http://archive.apache.org/dist/hadoop/common/hadoop-1.0.3/hadoop-1.0.3.tar.gz

--wget http://archive.apache.org/dist/hadoop/core/hadoop-1.0.3/hadoop-1.0.3.tar.gz

tar xvfz hadoop-1.0.3.tar.gz

---

root@ailab-hadoop:/home/ailab# ssh-keygen -t rsa
\Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in \.
Your public key has been saved in \.pub.
The key fingerprint is:
1e:1d:d7:01:04:86:b3:d9:5b:78:6a:3c:5a:72:28:04 root@ailab-hadoop
The key's randomart image is:
+--[ RSA 2048]----+
|         .ooo..  |
|    E   o.   . . |
|     .   =... .  |
|      . o.ooo    |
|     .  So.=     |
|      ..o.O      |
|       ..* .     |
|        .        |
|                 |
+-----------------+

scp ~/.ssh/id_rsa.pub ailab@localhost:/home/ailab/

scp는 자신의 파일을 원격서버로 전송하거나 전송 받는 명령어인데. 위에서 생성한 id_rsa.pub을 localhost 서버의 ailab 계정의 /home/ailab/ 디렉토리로 복사라는 명령이고 복사한 후에 해당 서버에 접속해서

mv /home/ailab/id_rsa.pub ~/.ssh/authorized_keys 로 복사하면 자동로그인이 가능해 진다.

이건 앞서 http://redcarrot.tistory.com/103 에서 한 것과 같은 행위인데. 앞서는 로컬 컴퓨터에다 의사진행모드로 진행하기 위해 복사했던것이고, 이번 위의 방법은 실제 분산환경일 때 각 서버로 자동로그인이 가능하도록 하기 위한 방법이다.

---

vi $HADOOP_HOME/conf/core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

</configuration>

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>fs.default.name</name>
                <value>hdfs://localhost:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/home/ailab/hadoop-1.0.3/data/</value>
        </property>
</configuration>

vi $HADOOP_HOME/conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

</configuration>

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>        <!-- 1이면 의사모드 3이면 분산모드 -->
        </property>
        <property>
                <name>dfs.http.address</name>
                <value>localhost:50070</value>
        </property>
        <property>
                <name>dfs.secondary.http.address</name>
                <value>localhost:50090</value>
        </property>
</configuration>

vi $HADOOP_HOME/conf/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

</configuration>

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>mapred.job.tracker</name>
                <value>localhost:9001</value>
        </property>
</configuration>

scp /home/ailab/conf/* ailab@localhost:/home/ailab/hadoop-1.0.3/conf/

scp 명령을 통해 다른 모든 데이터노드 서버로 네임노드의 하둡 환경 설정 파일을 전송

---

네임노드 초기화
./bin/hadoop namenode -format

root@ailab-hadoop:/home/ailab/hadoop-1.0.3# ./bin/hadoop namenode -format
14/04/27 23:28:50 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ailab-hadoop/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.0.3
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by 'hortonfo' on Tue May  8 20:31:25 UTC 2012
************************************************************/
14/04/27 23:28:50 INFO util.GSet: VM type       = 64-bit
14/04/27 23:28:50 INFO util.GSet: 2% max memory = 17.77875 MB
14/04/27 23:28:50 INFO util.GSet: capacity      = 2^21 = 2097152 entries
14/04/27 23:28:50 INFO util.GSet: recommended=2097152, actual=2097152
14/04/27 23:28:50 INFO namenode.FSNamesystem: fsOwner=root
14/04/27 23:28:50 INFO namenode.FSNamesystem: supergroup=supergroup
14/04/27 23:28:50 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/04/27 23:28:50 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/04/27 23:28:50 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/04/27 23:28:50 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/04/27 23:28:50 INFO common.Storage: Image file of size 110 saved in 0 seconds.
14/04/27 23:28:50 INFO common.Storage: Storage directory /home/ailab/hadoop-1.0.3/data/dfs/name has been successfully formatted.
14/04/27 23:28:50 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ailab-hadoop/127.0.1.1
************************************************************/

하둡 실행
./bin/start-all.sh

root@ailab-hadoop:/home/ailab/hadoop-1.0.3# ./bin/start-all.sh
starting namenode, logging to /home/ailab/hadoop-1.0.3/libexec/../logs/hadoop-root-namenode-ailab-hadoop.out
localhost: starting datanode, logging to /home/ailab/hadoop-1.0.3/libexec/../logs/hadoop-root-datanode-ailab-hadoop.out
localhost: starting secondarynamenode, logging to /home/ailab/hadoop-1.0.3/libexec/../logs/hadoop-root-secondarynamenode-ailab-hadoop.out
starting jobtracker, logging to /home/ailab/hadoop-1.0.3/libexec/../logs/hadoop-root-jobtracker-ailab-hadoop.out
localhost: starting tasktracker, logging to /home/ailab/hadoop-1.0.3/libexec/../logs/hadoop-root-tasktracker-ailab-hadoop.out

실행 여부 확인

jps

root@ailab-hadoop:/home/ailab/hadoop-1.0.3# jps
11762 DataNode
12551 Jps
12412 TaskTracker
12054 SecondaryNameNode
12151 JobTracker
11482 NameNode\

웹UI를 통한 실행 확인

localhost:50070

네임노드IP:50070으로 접속시 HDFS용량, 데이터노드 개수 등 기본적인 HDFS의 상태 정보와 네임노드에 적재된 하둡 로그, HDFS에 저장된 파일도 조회 가능

---

WordCount 예제

root@ailab-hadoop:/home/ailab/hadoop-1.0.3# ./bin/hadoop fs -mkdir /input
root@ailab-hadoop:/home/ailab/hadoop-1.0.3# ./bin/hadoop fs -copyFromLocal README.txt /input
root@ailab-hadoop:/home/ailab/hadoop-1.0.3# ./bin/hadoop fs -ls /input
Found 1 items
-rw-r--r--   1 root supergroup       1366 2014-04-27 23:42 /input/README.txt

./bin/hadoop jar hadoop-examples-1.0.3.jar wordcount /input/README.txt README.txt.result2

root@ailab-hadoop:/home/ailab/hadoop-1.0.3# ./bin/hadoop jar hadoop-examples-1.0.3.jar wordcount /input/README.txt README.txt.result2
14/04/27 23:48:07 INFO input.FileInputFormat: Total input paths to process : 1
14/04/27 23:48:07 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/04/27 23:48:07 WARN snappy.LoadSnappy: Snappy native library not loaded
14/04/27 23:48:08 INFO mapred.JobClient: Running job: job_201404272331_0002
14/04/27 23:48:09 INFO mapred.JobClient:  map 0% reduce 0%
14/04/27 23:48:22 INFO mapred.JobClient:  map 100% reduce 0%
14/04/27 23:48:37 INFO mapred.JobClient:  map 100% reduce 100%
14/04/27 23:48:42 INFO mapred.JobClient: Job complete: job_201404272331_0002
14/04/27 23:48:42 INFO mapred.JobClient: Counters: 29
14/04/27 23:48:42 INFO mapred.JobClient:   Job Counters
14/04/27 23:48:42 INFO mapred.JobClient:     Launched reduce tasks=1
14/04/27 23:48:42 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=10812
14/04/27 23:48:42 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/04/27 23:48:42 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/04/27 23:48:42 INFO mapred.JobClient:     Launched map tasks=1
14/04/27 23:48:42 INFO mapred.JobClient:     Data-local map tasks=1
14/04/27 23:48:42 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=12727
14/04/27 23:48:42 INFO mapred.JobClient:   File Output Format Counters
14/04/27 23:48:42 INFO mapred.JobClient:     Bytes Written=1306
14/04/27 23:48:42 INFO mapred.JobClient:   FileSystemCounters
14/04/27 23:48:42 INFO mapred.JobClient:     FILE_BYTES_READ=1836
14/04/27 23:48:42 INFO mapred.JobClient:     HDFS_BYTES_READ=1469
14/04/27 23:48:42 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=46977
14/04/27 23:48:42 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=1306
14/04/27 23:48:42 INFO mapred.JobClient:   File Input Format Counters
14/04/27 23:48:42 INFO mapred.JobClient:     Bytes Read=1366
14/04/27 23:48:42 INFO mapred.JobClient:   Map-Reduce Framework
14/04/27 23:48:42 INFO mapred.JobClient:     Map output materialized bytes=1836
14/04/27 23:48:42 INFO mapred.JobClient:     Map input records=31
14/04/27 23:48:42 INFO mapred.JobClient:     Reduce shuffle bytes=0
14/04/27 23:48:42 INFO mapred.JobClient:     Spilled Records=262
14/04/27 23:48:42 INFO mapred.JobClient:     Map output bytes=2055
14/04/27 23:48:42 INFO mapred.JobClient:     CPU time spent (ms)=800
14/04/27 23:48:42 INFO mapred.JobClient:     Total committed heap usage (bytes)=260440064
14/04/27 23:48:42 INFO mapred.JobClient:     Combine input records=179
14/04/27 23:48:42 INFO mapred.JobClient:     SPLIT_RAW_BYTES=103
14/04/27 23:48:42 INFO mapred.JobClient:     Reduce input records=131
14/04/27 23:48:42 INFO mapred.JobClient:     Reduce input groups=131
14/04/27 23:48:42 INFO mapred.JobClient:     Combine output records=131
14/04/27 23:48:42 INFO mapred.JobClient:     Physical memory (bytes) snapshot=260243456
14/04/27 23:48:42 INFO mapred.JobClient:     Reduce output records=131
14/04/27 23:48:42 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=4862078976
14/04/27 23:48:42 INFO mapred.JobClient:     Map output records=179

결과 조회

./bin/hadoop fs -ls

root@ailab-hadoop:/home/ailab/hadoop-1.0.3# ./bin/hadoop fs -ls
Found 1 items
drwxr-xr-x   - root supergroup          0 2014-04-27 23:48 /user/root/README.txt.result2

./bin/hadoop fs -cat README.txt.result2/*

root@ailab-hadoop:/home/ailab/hadoop-1.0.3# ./bin/hadoop fs -cat README.txt.result2/*
(BIS),    1
(ECCN)    1
(TSU)    1
(see    1
5D002.C.1,    1
740.13)    1
<http://www.wassenaar.org/>    1
Administration    1
Apache    1
BEFORE    1
BIS    1
Bureau    1
Commerce,    1
Commodity    1
Control    1
Core    1
Department    1
ENC    1
Exception    1
Export    2
For    1
Foundation    1
Government    1
Hadoop    1
Hadoop,    1
Industry    1
Jetty    1
License    1
Number    1
Regulations,    1
SSL    1
Section    1
Security    1
See    1
Software    2
Technology    1
The    4
This    1
U.S.    1
Unrestricted    1
about    1
algorithms.    1
and    6
and/or    1
another    1
any    1
as    1
asymmetric    1
at:    2
both    1
by    1
check    1
classified    1
code    1
code.    1
concerning    1
country    1
country's    1
country,    1
cryptographic    3
currently    1
details    1
distribution    2
eligible    1
encryption    3
exception    1
export    1
following    1
for    3
form    1
from    1
functions    1
has    1
have    1
http://hadoop.apache.org/core/    1
http://wiki.apache.org/hadoop/    1
if    1
import,    2
in    1
included    1
includes    2
information    2
information.    1
is    1
it    1
latest    1
laws,    1
libraries    1
makes    1
manner    1
may    1
more    2
mortbay.org.    1
object    1
of    5
on    2
or    2
our    2
performing    1
permitted.    1
please    2
policies    1
possession,    2
project    1
provides    1
re-export    2
regulations    1
reside    1
restrictions    1
security    1
see    1
software    2
software,    2
software.    2
software:    1
source    1
the    8
this    3
to    2
under    1
use,    2
uses    1
using    2
visit    1
website    1
which    2
wiki,    1
with    1
written    1
you    1
your    1
cat: File does not exist: /user/root/README.txt.result2/_logs

---

Reference :시작하세요! 하둡 프로그래밍  : 기초부터 실무까지 하둡의 모든 것, 위키북스, 정재화 지음

2014/04/08 - [Study/Hadoop] - Step1. 우분투(Ubuntu) 기초

2014/04/08 - [Study/Hadoop] - Step2. 하둡(Hadoop) 설치 및 독자 모드

2014/04/10 - [Study/Hadoop] - Step3. 의사 분산 모드 (Pseudo Distributed Mode)