Hadoop学习笔记之HDFS

这篇具有很好参考价值的文章主要介绍了Hadoop学习笔记之HDFS。希望对大家有所帮助。如果存在错误或未考虑完全的地方,请大家不吝赐教,您也可以点击"举报违法"按钮提交疑问。

HDFS (Hadoop Distributed File System)

分布式存储系统

支持海量数据的存储,成百上千的计算机组成存储集群,HDFS可以运行在低成本的硬件之上,具有的高容错、高可靠性、高可扩展性、高吞吐率等特征,非常适合大规模数据集上的应用。

优点

  • 高容错性
  • 适合批处理
  • 适合大数据处理
  • 流式文件访问
  • 可构建在廉价机器上

缺点

  • 不适合低延迟数据访问
  • 不适合小文件存取
  • 不适合并发写入、文件随机修改

HDFS操作

命令操作HDFS

# 显示目录 / 下的文件
hdfs dfs -ls /
# 新建文件夹,绝对路径
hdfs dfs -mkdir /test
# 上传文件
hdfs dfs -put test.txt /test/
# 下载文件
hdfs dfs -get /test/test.txt
# 输出文件内容
hdfs dfs -cat /test/test.txt

Web端操作HDFS

打开http://192.168.9.200:9870/,可以对HDFS进行操作

Hadoop学习笔记之HDFS

文件在这里管理

Hadoop学习笔记之HDFS

可视化操作还是简单一些

Hadoop学习笔记之HDFS

Java操作HDFS

  1. 添加依赖
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>3.3.2</version>
</dependency>
  1. 建立连接
public void setUp() throws Exception {
    System.out.println("开始建立与HDFS的连接");
    configuration = new Configuration();
    fileSystem = FileSystem.get(new URI(HDFS_PATH), configuration, "hadoop");
}
  1. 文件操作
    Configuration configuration = null;
    FileSystem fileSystem = null;
    public static final String HDFS_PATH = "hdfs://192.168.9.200:9000";

    /**
     * 在 hdfs中新建文件夹
     *
     * @throws Exception
     */
    @Test
    public void mkdir() throws Exception {
        fileSystem.mkdirs(new Path("/JavaDemo/test"));
    }

    /**
     * 创建文件
     *
     * @throws Exception
     */

    @Test
    public void create() throws Exception {
        FSDataOutputStream outputStream = fileSystem.create(new Path("/JavaDemo/test/haha.txt"));
        outputStream.write("hello bigdata from javaDemo".getBytes());
        outputStream.flush();
        outputStream.close();
    }


    /**
     * 查看文件 hdfs -fs -cat file
     *
     * @throws Exception
     */
    @Test
    public void cat() throws Exception {
        FSDataInputStream in = fileSystem.open(new Path("/JavaDemo/test/haha.txt"));
        IOUtils.copyBytes(in, System.out, 1024);
        in.close();
    }

    /**
     * 重命名文件
     *
     * @throws Exception
     */
    @Test
    public void rename() throws Exception {
        Path oldPath = new Path("/JavaDemo/test/haha.txt");
        Path newPath = new Path("/JavaDemo/test/hehe.txt");
        fileSystem.rename(oldPath, newPath);
    }

    /**
     * 上传文件到HDFS
     *
     * @throws Exception
     */
    @Test
    public void copyFromLocalFile() throws Exception {
        Path loacalPath = new Path("hello.txt");
        Path hdfsPath = new Path("/");
        fileSystem.copyFromLocalFile(loacalPath, hdfsPath);
    }


    /**
     * 上传文件到HDFS带进度信息
     *
     * @throws Exception
     */

    @Test
    public void copyFromLocalFileWithProgress() throws Exception {
        InputStream in = new BufferedInputStream(Files.newInputStream(new File("hbase-2.2.7-bin.tar.gz").toPath()));
        FSDataOutputStream ouput = fileSystem.create(new Path("/JavaDemo/test/hbase-2.2.7-bin.tar.gz"), () -> {
            System.out.print(".");
        });
        IOUtils.copyBytes(in, ouput, 4096);
    }

    /**
     * 下载文件到HDFS
     *
     * @throws Exception
     */
    @Test
    public void copyToLocalFile() throws Exception {
        Path hdfsPath = new Path("/JavaDemo/test/haha.txt");
        Path loacalPath = new Path("./haha.txt");
        //  useRawLocalFileSystem
        fileSystem.copyToLocalFile(false, hdfsPath, loacalPath, true);
    }

    /**
     * 查看某个目录下所有文件
     *
     * @throws Exception
     */
    @Test
    public void listFiles() throws Exception {
        FileStatus[] fileStatuses = fileSystem.listStatus(new Path("/"));
        for (FileStatus f : fileStatuses) {
            String isDir = f.isDirectory() ? "文件夹" : "文件";
            short replication = f.getReplication();
            long len = f.getLen();
            String path = f.getPath().toString();

            System.out.println(isDir + "\t" + replication + "\t" + len + "\t" + path);

        }
    }

    /**
     * 删除文件
     *
     * @throws Exception
     */
    @Test
    public void delete() throws IOException {
        fileSystem.delete(new Path("/JavaDemo/haha.txt"), true);
    }
  1. 关闭连接
public void tearDown() {
    configuration = null;
    fileSystem = null;
    System.out.println("关闭与HDFS的连接");
}

完整测试文件

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.IOUtils;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.io.*;
import java.net.URI;
import java.nio.file.Files;

/**
 * @author Gettler
 * Java 操作HDFS
 */
public class HdfsDemo {
    Configuration configuration = null;
    FileSystem fileSystem = null;
    public static final String HDFS_PATH = "hdfs://192.168.9.200:9000";

    /**
     * 在 hdfs中新建文件夹
     *
     * @throws Exception
     */
    @Test
    public void mkdir() throws Exception {
        fileSystem.mkdirs(new Path("/JavaDemo/test"));
    }

    /**
     * 创建文件
     *
     * @throws Exception
     */

    @Test
    public void create() throws Exception {
        FSDataOutputStream outputStream = fileSystem.create(new Path("/JavaDemo/test/haha.txt"));
        outputStream.write("hello bigdata from javaDemo".getBytes());
        outputStream.flush();
        outputStream.close();
    }


    /**
     * 查看文件 hdfs -fs -cat file
     *
     * @throws Exception
     */
    @Test
    public void cat() throws Exception {
        FSDataInputStream in = fileSystem.open(new Path("/JavaDemo/test/haha.txt"));
        IOUtils.copyBytes(in, System.out, 1024);
        in.close();
    }

    /**
     * 重命名文件
     *
     * @throws Exception
     */
    @Test
    public void rename() throws Exception {
        Path oldPath = new Path("/JavaDemo/test/haha.txt");
        Path newPath = new Path("/JavaDemo/test/hehe.txt");
        fileSystem.rename(oldPath, newPath);
    }

    /**
     * 上传文件到HDFS
     *
     * @throws Exception
     */
    @Test
    public void copyFromLocalFile() throws Exception {
        Path loacalPath = new Path("hello.txt");
        Path hdfsPath = new Path("/");
        fileSystem.copyFromLocalFile(loacalPath, hdfsPath);
    }


    /**
     * 上传文件到HDFS带进度信息
     *
     * @throws Exception
     */

    @Test
    public void copyFromLocalFileWithProgress() throws Exception {
        InputStream in = new BufferedInputStream(Files.newInputStream(new File("hbase-2.2.7-bin.tar.gz").toPath()));
        FSDataOutputStream ouput = fileSystem.create(new Path("/JavaDemo/test/hbase-2.2.7-bin.tar.gz"), () -> {
            System.out.print(".");
        });
        IOUtils.copyBytes(in, ouput, 4096);
    }

    /**
     * 下载文件到HDFS
     *
     * @throws Exception
     */
    @Test
    public void copyToLocalFile() throws Exception {
        Path hdfsPath = new Path("/JavaDemo/test/haha.txt");
        Path loacalPath = new Path("./haha.txt");
        //  useRawLocalFileSystem
        fileSystem.copyToLocalFile(false, hdfsPath, loacalPath, true);
    }

    /**
     * 查看某个目录下所有文件
     *
     * @throws Exception
     */
    @Test
    public void listFiles() throws Exception {
        FileStatus[] fileStatuses = fileSystem.listStatus(new Path("/"));
        for (FileStatus f : fileStatuses) {
            String isDir = f.isDirectory() ? "文件夹" : "文件";
            short replication = f.getReplication();
            long len = f.getLen();
            String path = f.getPath().toString();

            System.out.println(isDir + "\t" + replication + "\t" + len + "\t" + path);

        }
    }

    /**
     * 删除文件
     *
     * @throws Exception
     */
    @Test
    public void delete() throws IOException {
        fileSystem.delete(new Path("/JavaDemo/haha.txt"), true);
    }


    //测试之前执行的代码
    @Before
    public void setUp() throws Exception {
        System.out.println("开始建立与HDFS的连接");
        configuration = new Configuration();
        fileSystem = FileSystem.get(new URI(HDFS_PATH), configuration, "hadoop");
    }

    //测试之完执行的代码
    @After
    public void tearDown() {
        configuration = null;
        fileSystem = null;
        System.out.println("关闭与HDFS的连接");
    }
}

9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss99999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999文章来源地址https://www.toymoban.com/news/detail-468973.html

到了这里,关于Hadoop学习笔记之HDFS的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处: 如若内容造成侵权/违法违规/事实不符,请点击违法举报进行投诉反馈,一经查实,立即删除!

领支付宝红包 赞助服务器费用

相关文章

  • 【Distributed】分布式监控系统zabbix应用(二)

    分担 server 的集中式压力; 解决多机房之间的网络延时问题。 结构:agent —— proxy —— server 2.1 设置 zabbix 的下载源,安装 zabbix-proxy 2.2 部署数据库   要求 MySQL 5.7 或 Mariadb 10.5 及以上版本 2.3 导入数据库信息 2.4 修改 zabbix-proxy 配置文件 2.5 启动 zabbix-proxy 2.6 在所有主机上

    2024年02月13日
    浏览(58)
  • Clickhouse分布式表引擎(Distributed)写入核心原理解析

    Clickhouse分布式表引擎(Distributed)写入核心原理解析 Clickhouse分布式表引擎(Distributed)查询核心原理解析 Distributed表引擎是分布式表的代名词,它自身不存储任何数据,而是作为数据分片的透明代理,能够自动路由数据至集群中的各个节点 ,所以Distributed表引擎需要和其他数

    2023年04月27日
    浏览(48)
  • GlusterFs 分布式复制卷(Distributed-Replicate)性能测试

    目录 fio工具参数解释 Glusterfs 和NFS 性能测试 顺序写: 随机写: 顺序读: 随机读: 随机读写: 参数说明: 测试结论: 与NFS对比 压测对比结果 NFS和GlusterFs的优缺点 NFS的优点 NFS的缺点 GlusterFS的优点 GlusterFS的缺点 NFS与GlusterFS对比 1. 功能对比 2. 吞吐量对比 3. 可靠性对比 4

    2024年02月12日
    浏览(42)
  • 大数据学习02-Hadoop分布式集群部署

    操作系统:centos7 软件环境:jdk8、hadoop-2.8.5 1.下载VMware,建议支持正版 2.安装到Widows目录下任意位置即可,安装目录自定义。打开VMware,界面如下: 3.创建虚拟机 创建虚拟机—选择自定义 这一步按照默认的配置就好 选择系统,安装程序光盘映像文件iso,这里需要下载cenos镜像

    2024年02月16日
    浏览(63)
  • Hadoop学习1:概述、单体搭建、伪分布式搭建

    Hadoop: 分布式系统基础架构   解决问题: 海量数据存储、海量数据的分析计算   官网: https://hadoop.apache.org/   HDFS(Hadoop Distributed File System): 分布式文件系统,用于存储数据   Hadoop的默认配置【core-site.xml】: https://hadoop.apache.org/docs/r3.3.6/hadoop-project-dist/hadoop-common/c

    2024年03月15日
    浏览(51)
  • FPGA原理与结构(6)——分布式RAM(Distributed RAM,DRAM)

    系列文章目录:FPGA原理与结构(0)——目录与传送门 目录 一、RAM概述 1、RAM基本概念 2、FPGA中RAM的分类 二、DRAM详解 1、FPGA资源         2、DRAM的配置形式 2.1 Single-Port(单端口) 2.2 Dual-Port(双端口) 2.3 Quad-Port(四端口) 2.4 Simple Dual-Port(简单双端口) 2.5 更大深度  

    2024年02月08日
    浏览(35)
  • pytorch 进行分布式调试debug torch.distributed.launch 三种方式

    一. pytorch 分布式调试debug torch.distributed.launch 三种方式 1. 方式1:ipdb调试(建议) 参考之前的博客:python调试器 ipdb 注意:pytorch 分布式调试只能使用侵入式调试,也即是在你需要打断点的地方(或者在主程序的第一行)添加下面的代码: 当进入pdb调试后,跟原先使用pdb调试

    2024年02月07日
    浏览(33)
  • 分布式链路追踪——Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

    要解决的问题 如何记录请求经过多个分布式服务的信息,以便分析问题所在? 如何保证这些信息得到完整的追踪? 如何尽可能不影响服务性能? 当用户请求到达前端A,将会发送rpc请求给中间层B、C;B可以立刻作出反应,但是C需要后端服务D、E的配合才能应答 一个简单有用

    2024年02月12日
    浏览(41)
  • 39学习分布式计算框架 Hadoop 的高可用方案,如 NameNode 集群、ZooKeeper

    Hadoop 是一个分布式计算框架,用于存储和处理大数据。在 Hadoop 集群中,NameNode 是一个关键组件,它负责管理 Hadoop 分布式文件系统(HDFS)中的文件和目录。为了确保高可用性,需要使用多个 NameNode 节点进行冗余备份,并使用 ZooKeeper 进行故障检测和自动故障切换。 以下是学

    2023年04月26日
    浏览(53)
  • [论文笔记] Gemini: A Computation-Centric Distributed Graph Processing System

    Gemini: 以计算为中心的分布式图处理系统 [Paper] [Slides] [Code] OSDI’16 提出了 Gemini, 一个分布式图处理系统, 应用了多种针对计算性能的优化以在 效率之上构建可扩展性 . Gemini 采用: 稀疏-稠密信号槽 抽象, 将混合推拉计算模型扩展到分布式场景 基于分块的划分 (chunk-based partiti

    2024年02月15日
    浏览(41)

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

博客赞助

微信扫一扫打赏

请作者喝杯咖啡吧~博客赞助

支付宝扫一扫领取红包,优惠每天领

二维码1

领取红包

二维码2

领红包