Hadoop version
HBase version
Compatible?
0.20.2 release |
0.90.2 |
NO |
0.20-append |
0.90.2 |
YES |
0.21.0 release |
0.90.2 |
NO |
0.22.x (in development) |
0.90.2 |
NO |
从上图可以看出,HBase0.90.2与Hadoop的主干版本0.20.0是不兼容的,虽然可以使用,但是在生产环境中会导致数据丢失。
比如在hbase的web界面会有如下提醒:
You are currently running the HMaster without HDFS append support enabled. This may result in data loss. Please see the HBase wiki for details.
As of today, Hadoop 0.20.2 is the latest stable release of Apache Hadoop that is marked as ready for production (neither 0.21 nor 0.22 are).
Unfortunately, Hadoop 0.20.2 release is not compatible with the latest stable version of HBase: if you run HBase on top of Hadoop 0.20.2, you risk to lose data! Hence HBase users are required to build their own Hadoop 0.20.x version if they want to run HBase on a production cluster of Hadoop. In this article, I describe how to build such a production-ready version of Hadoop 0.20.x that is compatible with HBase 0.90.2.
在Hbase0.20.2的官方book中也有提到:
This version of HBase will only run on Hadoop 0.20.x. It will not run on hadoop 0.21.x (nor 0.22.x). HBase will lose data unless it is running on an HDFS that has a durable sync
. Currently only the branch-0.20-append branch has this attribute []. No official releases have been made from this branch up to now so you will have to build your own Hadoop from the tip of this branch. Check it out using this url, branch-0.20-append. Scroll down in the Hadoop How To Release to the section Build Requirements for instruction on how to build Hadoop.
Or rather than build your own, you could use Cloudera's CDH3. CDH has the 0.20-append patches needed to add a durable sync (CDH3 betas will suffice; b2, b3, or b4).
所以本文就讨论如何使用编译hadoop的append分支,并整合进入Hadoop主干版本。
首先安装git工具。(是个类似于svn一样的版本控制工具)
$ apt-get install git
使用git获取源代码,并建立本地版本库,需要下载较长时间
$ git clone git://git.apache.org/hadoop-common.git
进入库内
$ cd hadoop-common
我们发现git到本地的库只可以看到hadoop的最新主干代码,实际上,git已经获取了所有版本,需要手动切换版本到append分支;
$ git checkout -t origin/branch-0.20-append
Branch branch-0.20-append set up to track remote branch branch-0.20-append from origin.
Switched to a new branch 'branch-0.20-append'
这样就切换到了append分支
我们在分支就可以准备进行编译:
首先在hadoop-common目录下创建 build.properties ,内容如下:
resolvers=internal
version=0.20-taotaosou-dfs(你需要指定的版本号,例子代表淘淘搜-分布式文件系统)
project.version=${version}
hadoop.version=${version}
hadoop-core.version=${version}
hadoop-hdfs.version=${version}
hadoop-mapred.version=${version}
在hadoop-common目录下,最后确认一下是否已经切换版本
git checkout branch-0.20-append
现在看,目录中内容全变了,切换到了append分支
下面开始编译,先安装ant哦
启动构建,需要较长时间完成(4分钟左右)
$ ant mvn-install
注意,如果需要重新运行该指令,你应该先清除生成的文件
rm -rf $HOME/.m2/repository
在hadoop-common目录下执行
ant clean-cache
编译完成之后,会进入测试阶段
# Optional: run the full test suite or just the core test suite
$ ant test
$ ant test-core
第一个 测试全部内容,第二个只测试核心功能
ant test 时间非常久,非服务器约10小时。
在哪里可以找到目标jar包呢?
$ find $HOME/.m2/repository -name "hadoop-*.jar"
.../repository/org/apache/hadoop/hadoop-examples/0.20-append-for-hbase/hadoop-examples-0.20-append-for-hbase.jar
.../repository/org/apache/hadoop/hadoop-test/0.20-append-for-hbase/hadoop-test-0.20-append-for-hbase.jar
.../repository/org/apache/hadoop/hadoop-tools/0.20-append-for-hbase/hadoop-tools-0.20-append-for-hbase.jar
.../repository/org/apache/hadoop/hadoop-streaming/0.20-append-for-hbase/hadoop-streaming-0.20-append-for-hbase.jar
.../repository/org/apache/hadoop/hadoop-core/0.20-append-for-hbase/hadoop-core-0.20-append-for-hbase.jar
接下来就是将新的jar替换旧的jar包(此处假设你已经架设好hadoop-0.20.2release版本)
1,替换旧的hadoop包;
2,替换hbase中lib文件夹中的包
请注意,替换jar包需要重新命名
Hadoop 0.20.2 release 版本的命名规则为 hadoop-VERSION-PACKAGE.jar,如:hadoop-0.20.2-examples.jar.
而新编译的版本命名规则为 hadoop-PACKAGE-VERSION.jar,如: hadoop-examples-0.20-append-for-hbase.jar.
所以你会以如下方式重命名:
hadoop-examples-0.20-append-for-hbase.jar --> hadoop-0.20-append-for-hbase-examples.jar
hadoop-test-0.20-append-for-hbase.jar --> hadoop-0.20-append-for-hbase-test.jar
hadoop-tools-0.20-append-for-hbase.jar --> hadoop-0.20-append-for-hbase-tools.jar
hadoop-streaming-0.20-append-for-hbase.jar --> hadoop-0.20-append-for-hbase-streaming.jar
hadoop-core-0.20-append-for-hbase.jar --> hadoop-0.20-append-for-hbase-core.jar
而与之相反,Hbase使用的命名规则为hadoop-PACKAGE-VERSION.jar ,所以提交到$HBASE_HOME/lib的jar包则不需要重命名,只需要保持原来的名称。
完成以上工作之后,新编译的包就可以使用了。
但是在测试过程中,你可能遇到一些test fail
比如:TestFileAppend4 总是会出错
但是幸运的是,这并不意味着不能使用,或许你还会遇到其他错误,但是,经过与hbase maillist联系,发现其实他们也是正常的。
所以有错误,也请放心,虽然你也或跟我一样感到不爽。
分享到:
相关推荐
win 7 64上编译 Hadoop 2.7.3 源码 的真实经历。
bin目录 hadoop.dll winutils.exe hadoop hadoop.cmd hadoop.exp hadoop.lib hadoop.pdb hdfs hdfs.cmd hdfs.dll hdfs.lib hdfs.pdb libwinutils.lib mapred mapred.cmd rcc winutils.pdb yarn yarn.cmd 亲测可用,...
hadoop调试工具hadoop.dll和hadoop.exp和winutils.exe 64位
hadoop.dll winutils.exe hadoop hadoop.cmd hadoop.exp hadoop.lib hadoop.pdb hdfs hdfs.cmd hdfs.dll hdfs.lib hdfs.pdb libwinutils.lib mapred mapred.cmd rcc winutils.pdb yarn yarn.cmd
win10编译过的hadoo2.6.4包括hadoop.dll和winutils.exe,解压后即可使用
hadoop的hadoop.dll和winutils.exe下载
hadoop2.6.0 hadoop.dll包括winutils.exe
解决hadoop版本不同而引起的org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
hadoop-2.7.3.tar.gz 下载 目前相对比较稳定的版本 hadoop tar 包下载hadoop-2.7.3.tar.gz 下载 hadoop tar 包下载
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). log4j:WARN Please initialize the log4j system properly. log4j:WARN See ...
hadoop2.7.2windows工具winutils.exe和hadoop.dll hadoop.exp libwinutils.lib等
Hadoop3.3.0在windows下部署开发环境或者远程开发hadoop集群,对应的 hadoop.dll和winutils.exe以及整个bin目录 ,以及安装方法 亲测可用,有什么问题可以在评论描述
经本人亲自测试,本地可用,hadoop-eclipse-plugin-2.7.4.jar放到eclipse的插件目录,hadoop.dll 请放到system32目录,eclipse版本我本地下载的是Oxygen.1a Release (4.7.1a)。
hadoop2.7.3的hadoop.dll和winutils.exe,哈哈哈哈哈哈哈
ambari-2.7.5 编译过程中四个大包下载很慢,所以需要提前下载,包含:hbase-2.0.2.3.1.4.0-315-bin.tar.gz ,hadoop-3.1.1.3.1.4.0-315.tar.gz , grafana-6.4.2.linux-amd64.tar.gz ,phoenix-5.0.0.3.1.4.0-315....
hadoop-3.2.4.tar.gz
为学习Hadoop亲手在Windows 7(64bit)编译
由于官网上提供的Hadoop源码是32位的,而目前大多系统为64位,所以需要编译对应版本的64位Hadoop,本人自己编译,亲测有效
hadoop2.6 hadoop.dll winutils.exe d
org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V 解决方案:下载本资源解压将hadoop.dll和winutils.exe文件复制到hadoop2.7.3的bin目录下即可解决。