Windows 11 WSL Ubuntu 环境：安装 Hadoop 完整指南

本文详细介绍了在WSL Ubuntu环境下安装Hadoop的完整流程。主要内容包括1) 环境准备安装WSL、Ubuntu及必要工具2) 配置SSH免密登录3) 下载安装Hadoop 3.3.6并配置环境变量4) 详细说明Hadoop核心配置文件的修改方法5) 提供启动/停止脚本及常见问题解决方案。指南采用国内镜像加速下载包含一键安装脚本适合Windows 11用户快速搭建Hadoop开发环境最低要求4GB内存和2GB硬盘空间。Hadoop 在 WSL Ubuntu 上的完整安装指南一、环境准备1.1 安装 WSL 和 Ubuntu以管理员身份打开 PowerShell 或 CMD执行powershell# 安装 WSL wsl --install # 重启电脑后打开 Microsoft Store 安装 Ubuntu 22.04 或 24.041.2 更新系统并安装必要软件打开 WSL 终端Ubuntu执行bash# 更新软件包 sudo apt update sudo apt upgrade -y # 安装必要工具 sudo apt install -y openjdk-11-jdk openssh-server wget curl nano net-tools # 验证 Java 安装 java -version # 应该显示 openjdk version 11.0.xx二、配置 SSH 免密登录bash# 1. 启动 SSH 服务 sudo service ssh start # 2. 生成 SSH 密钥 ssh-keygen -t rsa -P -f ~/.ssh/id_rsa # 3. 配置免密登录 cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys chmod 700 ~/.ssh # 4. 添加 localhost 到已知主机 ssh-keyscan localhost ~/.ssh/known_hosts 2/dev/null # 5. 测试免密登录应该不需要密码 ssh localhost exit三、下载并安装 Hadoop3.1 下载 Hadoop使用国内镜像bash# 创建下载目录 cd ~ mkdir -p downloads cd downloads # 使用清华镜像下载 Hadoop 3.3.6 wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz # 验证下载文件大小约 696MB ls -lh hadoop-3.3.6.tar.gz3.2 解压安装bash# 解压到 /usr/local sudo tar -xzvf hadoop-3.3.6.tar.gz -C /usr/local/ # 重命名目录 sudo mv /usr/local/hadoop-3.3.6 /usr/local/hadoop # 修改权限将 mumu 替换为你的用户名 sudo chown -R $(whoami):$(whoami) /usr/local/hadoop四、配置环境变量bash# 编辑 .bashrc nano ~/.bashrc在文件末尾添加bash# Java 环境 export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 # Hadoop 环境 export HADOOP_HOME/usr/local/hadoop export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin保存退出CtrlOEnterCtrlXbash# 使配置生效 source ~/.bashrc # 验证 hadoop version五、配置 Hadoop 核心文件进入配置目录bashcd $HADOOP_HOME/etc/hadoop5.1 配置 hadoop-env.shbash# 添加 JAVA_HOME 配置 echo export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 hadoop-env.sh5.2 配置 core-site.xmlbashcat core-site.xml EOF ?xml version1.0 encodingUTF-8? ?xml-stylesheet typetext/xsl hrefconfiguration.xsl? configuration property namefs.defaultFS/name valuehdfs://localhost:9000/value /property property namehadoop.tmp.dir/name value/home/$(whoami)/hadoop_tmp/value /property /configuration EOF5.3 配置 hdfs-site.xmlbashcat hdfs-site.xml EOF ?xml version1.0 encodingUTF-8? ?xml-stylesheet typetext/xsl hrefconfiguration.xsl? configuration property namedfs.replication/name value1/value /property property namedfs.namenode.name.dir/name value/home/$(whoami)/hadoop_tmp/dfs/name/value /property property namedfs.datanode.data.dir/name value/home/$(whoami)/hadoop_tmp/dfs/data/value /property /configuration EOF5.4 配置 yarn-site.xmlbashcat yarn-site.xml EOF ?xml version1.0? configuration property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.nodemanager.env-whitelist/name valueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME/value /property /configuration EOF5.5 配置 mapred-site.xmlbashcat mapred-site.xml EOF ?xml version1.0? ?xml-stylesheet typetext/xsl hrefconfiguration.xsl? configuration property namemapreduce.framework.name/name valueyarn/value /property /configuration EOF5.6 配置 yarn-env.shbashecho export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 yarn-env.sh六、创建数据目录并格式化bash# 创建数据目录 mkdir -p ~/hadoop_tmp/dfs/name mkdir -p ~/hadoop_tmp/dfs/data # 格式化 NameNode输入 Y 确认 hdfs namenode -format七、启动 Hadoopbash# 启动所有服务 start-all.sh # 等待 30 秒 sleep 30 # 验证进程应该看到 5 个核心进程 jps预期输出textNameNode DataNode SecondaryNameNode ResourceManager NodeManager Jps八、创建启动/停止脚本可选但推荐8.1 创建启动脚本bashmkdir -p ~/bin cat ~/bin/start-hadoop.sh EOF #!/bin/bash echo echo Starting Hadoop Cluster echo # 启动 SSH 服务 echo 1. Starting SSH service... sudo service ssh start # 启动 Hadoop echo 2. Starting Hadoop... start-all.sh # 等待服务启动 echo 3. Waiting 30 seconds for services... sleep 30 # 显示进程 echo 4. Checking Hadoop processes... jps echo echo echo ✅ Hadoop is ready! echo - HDFS UI: http://localhost:9870 echo - YARN UI: http://localhost:8088 echo EOF8.2 创建停止脚本bashcat ~/bin/stop-hadoop.sh EOF #!/bin/bash echo echo Stopping Hadoop Cluster echo stop-all.sh echo echo ✅ Hadoop stopped EOF8.3 设置权限和 PATHbashchmod x ~/bin/start-hadoop.sh ~/bin/stop-hadoop.sh echo export PATH$HOME/bin:$PATH ~/.bashrc source ~/.bashrc九、验证安装9.1 测试 HDFS 命令bash# 创建目录 hdfs dfs -mkdir -p /user/$(whoami)/test # 上传文件 echo Hello Hadoop test.txt hdfs dfs -put test.txt /user/$(whoami)/test/ # 查看文件 hdfs dfs -ls /user/$(whoami)/test/ hdfs dfs -cat /user/$(whoami)/test/test.txt9.2 访问 Web 界面在 Windows 浏览器中打开HDFS:http://localhost:9870/explorer.htmlYARN:http://localhost:8088十、日常使用启动 Hadoopbash# 方法一使用脚本 start-hadoop.sh # 方法二手动启动 sudo service ssh start start-all.sh停止 Hadoopbashstop-hadoop.sh常用命令速查操作命令查看进程jps列出 HDFS 文件hdfs dfs -ls /创建目录hdfs dfs -mkdir -p /path上传文件hdfs dfs -put local.txt /hdfs/path/下载文件hdfs dfs -get /hdfs/path/file.txt ./查看文件内容hdfs dfs -cat /hdfs/path/file.txt删除文件hdfs dfs -rm /hdfs/path/file.txt十一、常见问题解决问题 1Java 版本错误bash# 检查 Java 版本必须是 11 java -version # 如果不是 11切换版本 sudo update-alternatives --config java # 选择 Java 11问题 2SSH 连接失败bashsudo service ssh start ssh localhost exit问题 3NameNode 无法启动bash# 停止所有服务 stop-all.sh # 删除临时文件 rm -rf ~/hadoop_tmp/dfs/name/* ~/hadoop_tmp/dfs/data/* # 重新格式化 hdfs namenode -format # 重新启动 start-all.sh问题 4端口被占用bash# 检查端口 netstat -tlnp | grep 9870 netstat -tlnp | grep 8088 # 更换端口或关闭占用进程十二、系统要求项目最低要求推荐配置操作系统Windows 11启用 WSL2Windows 11 Pro内存4GB8GB硬盘空间2GBHadoop 1GB数据10GBWSL 版本WSL2WSL2Ubuntu 版本20.0422.04 或 24.04Java 版本Java 11Java 11快速复现命令汇总复制以下代码块在 WSL 终端中一次性执行注意替换用户名bash# 一键安装脚本 # 注意执行前请确保已安装 WSL 和 Ubuntu # 1. 安装基础软件 sudo apt update sudo apt install -y openjdk-11-jdk openssh-server wget curl nano # 2. 配置 SSH sudo service ssh start ssh-keygen -t rsa -P -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys # 3. 下载并安装 Hadoop cd ~ wget -P ~/downloads/ https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz sudo tar -xzvf ~/downloads/hadoop-3.3.6.tar.gz -C /usr/local/ sudo mv /usr/local/hadoop-3.3.6 /usr/local/hadoop sudo chown -R $(whoami):$(whoami) /usr/local/hadoop # 4. 配置环境变量追加到 .bashrc echo export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 ~/.bashrc echo export HADOOP_HOME/usr/local/hadoop ~/.bashrc echo export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop ~/.bashrc echo export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin ~/.bashrc source ~/.bashrc # 5. 配置 Hadoop 文件在配置目录中执行 cd $HADOOP_HOME/etc/hadoop echo export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 hadoop-env.sh echo export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 yarn-env.sh # 写入配置文件继续在配置目录执行注意替换用户名 cat core-site.xml EOF ?xml version1.0 encodingUTF-8? configuration property namefs.defaultFS/name valuehdfs://localhost:9000/value /property property namehadoop.tmp.dir/name value/home/$(whoami)/hadoop_tmp/value /property /configuration EOF cat hdfs-site.xml EOF ?xml version1.0 encodingUTF-8? configuration property namedfs.replication/name value1/value /property property namedfs.namenode.name.dir/name value/home/$(whoami)/hadoop_tmp/dfs/name/value /property property namedfs.datanode.data.dir/name value/home/$(whoami)/hadoop_tmp/dfs/data/value /property /configuration EOF cat yarn-site.xml EOF ?xml version1.0? configuration property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.nodemanager.env-whitelist/name valueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME/value /property /configuration EOF cat mapred-site.xml EOF ?xml version1.0? configuration property namemapreduce.framework.name/name valueyarn/value /property /configuration EOF # 6. 创建数据目录并格式化 mkdir -p ~/hadoop_tmp/dfs/name ~/hadoop_tmp/dfs/data hdfs namenode -format # 7. 启动 Hadoop start-all.sh sleep 30 jps恭喜按照以上步骤你可以在任何 Windows 11 电脑上成功安装 Hadoop