flink源码分析-获取JVM最大堆内存

这篇具有很好参考价值的文章主要介绍了flink源码分析-获取JVM最大堆内存。希望对大家有所帮助。如果存在错误或未考虑完全的地方,请大家不吝赐教,您也可以点击"举报违法"按钮提交疑问。

flink版本: flink-1.11.2

代码位置: org.apache.flink.runtime.util.EnvironmentInformation#getMaxJvmHeapMemory

如果设置了-Xmx参数,就返回这个参数,如果没设置就返回机器物理内存的1/4.  这里主要看各个机器内存的获取方法。

	/**
	 * The maximum JVM heap size, in bytes.
	 *
	 * <p>This method uses the <i>-Xmx</i> value of the JVM, if set. If not set, it returns (as
	 * a heuristic) 1/4th of the physical memory size.
	 *
	 * @return The maximum JVM heap size, in bytes.
	 */
	public static long getMaxJvmHeapMemory() {
		final long maxMemory = Runtime.getRuntime().maxMemory();
		if(maxMemory != Long.MAX_VALUE) {
			// we have the proper max memory
			return maxMemory;
		} else {
			// max JVM heap size is not set - use the heuristic to use 1/4th of the physical memory
			final long physicalMemory = Hardware.getSizeOfPhysicalMemory();
			if(physicalMemory != -1) {
				// got proper value for physical memory
				return physicalMemory / 4;
			} else {
				throw new RuntimeException(
					"Could not determine the amount of free memory.\n" + "Please set the maximum memory for the JVM, e.g. -Xmx512M for 512 megabytes.");
			}
		}
	}

进入getSizeOfPhysicalMemory()方法,里面有获取各种操作系统物理内存的方法:

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.flink.runtime.util;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.lang.management.ManagementFactory;
import java.lang.management.OperatingSystemMXBean;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.apache.flink.util.OperatingSystem;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * Convenience class to extract hardware specifics of the computer executing the running JVM.
 */
public class Hardware {

	private static final Logger LOG = LoggerFactory.getLogger(Hardware.class);

	private static final String LINUX_MEMORY_INFO_PATH = "/proc/meminfo";

	private static final Pattern LINUX_MEMORY_REGEX = Pattern.compile("^MemTotal:\\s*(\\d+)\\s+kB$");

	// ------------------------------------------------------------------------
	
	/**
	 * Gets the number of CPU cores (hardware contexts) that the JVM has access to.
	 * 
	 * @return The number of CPU cores.
	 */
	public static int getNumberCPUCores() {

		// TODO_MA 注释: 获取 Cpu Cores
		return Runtime.getRuntime().availableProcessors();
	}

	/**
	 * Returns the size of the physical memory in bytes.
	 * 
	 * @return the size of the physical memory in bytes or {@code -1}, if
	 *         the size could not be determined.
	 */
	public static long getSizeOfPhysicalMemory() {
		// first try if the JVM can directly tell us what the system memory is
		// this works only on Oracle JVMs
		try {
			Class<?> clazz = Class.forName("com.sun.management.OperatingSystemMXBean");
			Method method = clazz.getMethod("getTotalPhysicalMemorySize");
			OperatingSystemMXBean operatingSystemMXBean = ManagementFactory.getOperatingSystemMXBean();

			// someone may install different beans, so we need to check whether the bean
			// is in fact the sun management bean
			if (clazz.isInstance(operatingSystemMXBean)) {
				return (Long) method.invoke(operatingSystemMXBean);
			}
		}
		catch (ClassNotFoundException e) {
			// this happens on non-Oracle JVMs, do nothing and use the alternative code paths
		}
		catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
			LOG.warn("Access to physical memory size: " +
					"com.sun.management.OperatingSystemMXBean incompatibly changed.", e);
		}

		// we now try the OS specific access paths
		switch (OperatingSystem.getCurrentOperatingSystem()) {
			case LINUX:
				return getSizeOfPhysicalMemoryForLinux();

			case WINDOWS:
				return getSizeOfPhysicalMemoryForWindows();

			case MAC_OS:
				return getSizeOfPhysicalMemoryForMac();

			case FREE_BSD:
				return getSizeOfPhysicalMemoryForFreeBSD();

			case UNKNOWN:
				LOG.error("Cannot determine size of physical memory for unknown operating system");
				return -1;

			default:
				LOG.error("Unrecognized OS: " + OperatingSystem.getCurrentOperatingSystem());
				return -1;
		}
	}

	/**
	 * Returns the size of the physical memory in bytes on a Linux-based
	 * operating system.
	 * 
	 * @return the size of the physical memory in bytes or {@code -1}, if
	 *         the size could not be determined
	 */
	private static long getSizeOfPhysicalMemoryForLinux() {
		try (BufferedReader lineReader = new BufferedReader(new FileReader(LINUX_MEMORY_INFO_PATH))) {
			String line;
			while ((line = lineReader.readLine()) != null) {
				Matcher matcher = LINUX_MEMORY_REGEX.matcher(line);
				if (matcher.matches()) {
					String totalMemory = matcher.group(1);
					return Long.parseLong(totalMemory) * 1024L; // Convert from kilobyte to byte
				}
			}
			// expected line did not come
			LOG.error("Cannot determine the size of the physical memory for Linux host (using '/proc/meminfo'). " +
					"Unexpected format.");
			return -1;
		}
		catch (NumberFormatException e) {
			LOG.error("Cannot determine the size of the physical memory for Linux host (using '/proc/meminfo'). " +
					"Unexpected format.");
			return -1;
		}
		catch (Throwable t) {
			LOG.error("Cannot determine the size of the physical memory for Linux host (using '/proc/meminfo') ", t);
			return -1;
		}
	}

	/**
	 * Returns the size of the physical memory in bytes on a Mac OS-based
	 * operating system
	 * 
	 * @return the size of the physical memory in bytes or {@code -1}, if
	 *         the size could not be determined
	 */
	private static long getSizeOfPhysicalMemoryForMac() {
		BufferedReader bi = null;
		try {
			Process proc = Runtime.getRuntime().exec("sysctl hw.memsize");

			bi = new BufferedReader(new InputStreamReader(proc.getInputStream()));

			String line;
			while ((line = bi.readLine()) != null) {
				if (line.startsWith("hw.memsize")) {
					long memsize = Long.parseLong(line.split(":")[1].trim());
					bi.close();
					proc.destroy();
					return memsize;
				}
			}

		} catch (Throwable t) {
			LOG.error("Cannot determine physical memory of machine for MacOS host", t);
			return -1;
		} finally {
			if (bi != null) {
				try {
					bi.close();
				} catch (IOException ignored) {}
			}
		}
		return -1;
	}

	/**
	 * Returns the size of the physical memory in bytes on FreeBSD.
	 * 
	 * @return the size of the physical memory in bytes or {@code -1}, if
	 *         the size could not be determined
	 */
	private static long getSizeOfPhysicalMemoryForFreeBSD() {
		BufferedReader bi = null;
		try {
			Process proc = Runtime.getRuntime().exec("sysctl hw.physmem");

			bi = new BufferedReader(new InputStreamReader(proc.getInputStream()));

			String line;
			while ((line = bi.readLine()) != null) {
				if (line.startsWith("hw.physmem")) {
					long memsize = Long.parseLong(line.split(":")[1].trim());
					bi.close();
					proc.destroy();
					return memsize;
				}
			}
			
			LOG.error("Cannot determine the size of the physical memory for FreeBSD host " +
					"(using 'sysctl hw.physmem').");
			return -1;
		}
		catch (Throwable t) {
			LOG.error("Cannot determine the size of the physical memory for FreeBSD host " +
					"(using 'sysctl hw.physmem')", t);
			return -1;
		}
		finally {
			if (bi != null) {
				try {
					bi.close();
				} catch (IOException ignored) {}
			}
		}
	}

	/**
	 * Returns the size of the physical memory in bytes on Windows.
	 * 
	 * @return the size of the physical memory in bytes or {@code -1}, if
	 *         the size could not be determined
	 */
	private static long getSizeOfPhysicalMemoryForWindows() {
		BufferedReader bi = null;
		try {
			Process proc = Runtime.getRuntime().exec("wmic memorychip get capacity");

			bi = new BufferedReader(new InputStreamReader(proc.getInputStream()));

			String line = bi.readLine();
			if (line == null) {
				return -1L;
			}

			if (!line.startsWith("Capacity")) {
				return -1L;
			}

			long sizeOfPhyiscalMemory = 0L;
			while ((line = bi.readLine()) != null) {
				if (line.isEmpty()) {
					continue;
				}

				line = line.replaceAll(" ", "");
				sizeOfPhyiscalMemory += Long.parseLong(line);
			}
			return sizeOfPhyiscalMemory;
		}
		catch (Throwable t) {
			LOG.error("Cannot determine the size of the physical memory for Windows host " +
					"(using 'wmic memorychip')", t);
			return -1L;
		}
		finally {
			if (bi != null) {
				try {
					bi.close();
				} catch (Throwable ignored) {}
			}
		}
	}

	// --------------------------------------------------------------------------------------------

	private Hardware() {}
}

另外注意try catch的这种用法:文章来源地址https://www.toymoban.com/news/detail-616760.html

		try (BufferedReader lineReader = new BufferedReader(new FileReader(LINUX_MEMORY_INFO_PATH))) {
			String line;
			while ((line = lineReader.readLine()) != null) {
				Matcher matcher = LINUX_MEMORY_REGEX.matcher(line);
				if (matcher.matches()) {
					String totalMemory = matcher.group(1);
					return Long.parseLong(totalMemory) * 1024L; // Convert from kilobyte to byte
				}
			}
			// expected line did not come
			LOG.error("Cannot determine the size of the physical memory for Linux host (using '/proc/meminfo'). " +
					"Unexpected format.");
			return -1;
		}
		catch (NumberFormatException e) {
			LOG.error("Cannot determine the size of the physical memory for Linux host (using '/proc/meminfo'). " +
					"Unexpected format.");
			return -1;
		}

到了这里,关于flink源码分析-获取JVM最大堆内存的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处: 如若内容造成侵权/违法违规/事实不符,请点击违法举报进行投诉反馈,一经查实,立即删除!

领支付宝红包 赞助服务器费用

相关文章

  • 用jprofiler来分析 jvm 堆 内存泄露,fullgc

    jvm 命令和工具_个人渣记录仅为自己搜索用的博客-CSDN博客   方法1: 重新设置堆后,重启,复现.   方法2:  切割  官网文档 JProfiler Help - HPROF snapshots heap walker教学: 用jprofile查看hprof文件_hprof jprofile_java老张的博客-CSDN博客 通识教学: JVM监控及诊断工具GUI篇之JProfiler_每天都要进步一

    2024年02月08日
    浏览(99)
  • SpringBoot自主监控,获取服务信息、JVM、CPU、内存、磁盘、堆、线程、GC等

    1. 简介   在日常开发中一些关键的业务服务,期望在高并发状态下可以正常工作,或在异常情况时可以记录当时的性能信息,所以就需要进行监控。常见的监控例如: Prometheus 可以实现这个需求,如果需要更加简单方便的自主监控能力,可以引入本博客中的方案。 2. 相关博

    2024年02月15日
    浏览(47)
  • 一文深度讲解JVM 内存分析工具 MAT及实践(建议收藏)

    1. 前言 熟练掌握 MAT 是 Java 高手的必备能力,但实践时大家往往需面对众多功能,眼花缭乱不知如何下手,小编也没有找到一篇完善的教学素材,所以整理本文帮大家系统掌握 MAT 分析工具。 本文详细讲解 MAT 众多内存分析工具功能,这些功能组合使用异常强大,熟练使用几

    2024年02月09日
    浏览(45)
  • JVM逃逸分析原理解析:优化Java程序性能和内存利用效率

    在Java开发中,性能和内存利用效率一直是开发者关注的焦点。为了提高Java程序的执行效率,JVM引入了逃逸分析技术。本文将详细解析JVM逃逸分析的原理,帮助读者深入理解其工作机制。 逃逸分析是一种用于确定对象在方法的生命周期内是否逃逸出方法外部范围的技术。在

    2024年01月20日
    浏览(62)
  • JVM内存模型/运行时数据区域

    java虚拟机管理这块内存,所以我们也叫运行时数据区域 这里按线程是否共享来分类,所谓线程不共享就是每个线程里面都会配一套 程序计数器 栈, 互相不干涉。 而方法区和堆是线程所有共享 意味着只有一个(这里注意堆是实际概念 方法区是一个虚拟概念) 注意:程序计

    2024年01月18日
    浏览(55)
  • Kotlin协程的JVM实现源码分析(下)

    协程 根据 是否保存切换 调用栈 ,分为: 有栈协程(stackful coroutine) 无栈协程(stackless coroutine) 在代码上的区别是:是否可在普通函数里调用,并暂停其执行。 Kotlin协程,必须在挂起函数中调用和恢复,属于 无栈协程 。 常见的语言,协程实现: 有栈协程 :Go、Lua 无栈

    2024年01月23日
    浏览(44)
  • Kotlin协程的JVM实现源码分析(上)

    本文从协程的启动 launch 源码入手分析,协程JVM实现分为两篇: 协程启动和执行源码分析 无栈协程 和 Continuation 基本环境: IntelliJ IDEA 2023.3.2 Kotlin 1.8.20 kotlinx-coroutines-core 1.7.3 gradle 8.2 以 GlobalScope.launch 启动协程分析: 调用关系: CoroutineScope.launch - StandaloneCoroutine.start - Corou

    2024年01月19日
    浏览(38)
  • JVM—内存管理(运行时数据区)、垃圾回收

    当JVM类加载器加载完字节码文件之后,会交给执行引擎执行,在执行的过程中会有一块JVM内存区域来存放程序运行过程中的数据,也就是我们图中放的运行时数据区,那这一块运行时数据区究竟帮我们做了哪些工作?我们常说的线上内存泄漏和内存溢出是因为什么?我们今儿

    2024年02月13日
    浏览(41)
  • 区分什么是Java内存模型(JMM)和 JVM运行时数据区

    Java的内存区域和内存模型是不一样的东西,内存区域是指 JVM 运行时将数据分区域存储,强调对内存空间的划分 。 而内存模型(Java Memory Model,简称 JMM )是 定义了线程和主内存之间的抽象关系,即 JMM 定义了 JVM 在计算机内存(RAM)中的工作方式 ,如果我们要想深入了解Java并

    2024年02月11日
    浏览(50)

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

博客赞助

微信扫一扫打赏

请作者喝杯咖啡吧~博客赞助

支付宝扫一扫领取红包,优惠每天领

二维码1

领取红包

二维码2

领红包