• 1. JVM性能问题排查案例国际网站技术部 Hatter Jiang
  • 2. Agenda案例1 - 应用不时Load飙高 案例2 - Perm区GC频繁 性能调优工具及原理介绍 JTop BTrace
  • 3. 案例1:应用不时Load飙高
  • 4. 奇怪的现象不定时Load飙高 20以上,有时候40以上 重启后长时间Load正常 早上重启应用,一天表示都正常
  • 5. 查看哪个线程占用大量CPU?jtop http://code.google.com/p/hatter-source-code/wiki/jtop 查看占用CPU最高的几个线程 没有明显异常 但,发现有大量XStream加载类
  • 6. 测试一下Xstream的代价 public static void main(String[] args) { for (int i = 0; i < 100000; i++) { XStream xs = new XStream(); xs.toXML(new Object()); } }
  • 7. jstat –gcutil S0 S1 E O P YGC YGCT FGC FGCT GCT 80.32 0.00 98.07 68.92 28.74 78 3.450 2 0.078 3.528 0.00 80.32 26.02 75.17 28.74 85 3.831 2 0.078 3.909 0.00 80.33 4.00 80.52 28.74 91 4.185 2 0.078 4.262 80.81 0.00 54.04 84.99 28.74 96 4.497 2 0.078 4.575 0.00 80.70 64.05 89.45 28.74 101 4.817 2 0.078 4.895
  • 8. NEW一个XStream占用多少内存?通过classmexer.jar分析内存占用 1,163,232 41,832 11,776 11,776 11,776See also: http://sizeof.sourceforge.net/http://www.javamex.com/classmexer/
  • 9. 如何计算对象大小?java.lang.instrument.Instrumentation#getObjectSize
  • 10. 问题解决将XStream改成static搞定private static final JaxbXstreamWrapper jx = new JaxbXstreamWrapper(new HashMap() { { this.put("actionTrace", ActionTrace.class); } });
  • 11. 解决后的Load
  • 12. 案例2:Perm 区GC频繁
  • 13. Perm 中有什么?Class 字符串常量 等等 … …
  • 14. 查看Class加载情况java -verbose:class +TraceClassLoading,+TraceClassUnloading [Loaded java.lang.Object from /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar] [Loaded java.io.Serializable from /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar] [Loaded java.lang.Comparable from /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar] [Loaded java.lang.CharSequence from /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar] [Loaded java.lang.String from /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar] java –jar jflagall.jar –flag +ClassVerbose
  • 15. 通过BTrace查看btrace ClassLoaderDefine.java
  • 16. 字符串常量加载情况http://code.google.com/p/hatter-source-code/wiki/permstat 以下来自测试结果(非线上分析结果): COUNT SIZE AVERAGE(byte) DIFF COUNT DIFF SIZE 691399 42.26M 64 - - 932381 56.96M 64 240982 14.71M
  • 17. 定位String.internBTrace不工作? BTrace currently does not support tracing native methods. 如果JDK这么做我会很喜欢 public native String intern0(); public String intern() { return intern0(); }
  • 18. 我的方法:先找到所有的String.intern调用通过分析字节码 http://code.google.com/p/hatter-source-code/wiki/invokefind 为什么分析字节码?而不是源代码? [INFO] Match found: [oracle/jdbc/driver/OraclePreparedStatement.setStringForClobAtName(Ljava/lang/String;Ljava/lang/String;)V]: INVOKEVIRTUAL java/lang/String.intern ()Ljava/lang/String; [INFO] Match found: [com/thoughtworks/xstream/converters/basic/StringConverter.fromString(Ljava/lang/String;)Ljava/lang/Object;]: INVOKEVIRTUAL java/lang/String.intern ()Ljava/lang/String;
  • 19. 然后:通过BTrace定位btrace StringConverterTrace.java
  • 20. 可以使用:跟踪调用String.intern
  • 21. 找到罪魁祸首public class StringConverter extends AbstractBasicConverter { public boolean canConvert(Class type) { return type.equals(String.class); } protected Object fromString(String str) { return str.intern(); } }
  • 22. 为什么String.intern会使用Perm在Sun(Oracle) JDK6及以前版本Handle java_lang_String::create_tenured_from_unicode(jchar* unicode, int length, TRAPS) { return basic_create_from_unicode(unicode, length, true, CHECK_NH); }
  • 23. 解决后的Perm GC
  • 24. 诊断方法及诊断工具HotSpot问题诊断 https://code.google.com/p/hatter-source-code/wiki/Blog_HotSpotDiagnosis 诊断工具 https://code.google.com/p/hatter-source-code/wiki/Study_Java_Diagnosis
  • 25. jtopjava –jar jtop.jar –A –C
  • 26. JVM内存信息java.lang.management.MemoryMXBean #getHeapMemoryUsage() #getNonHeapMemoryUsage() Heap Memory: INIT=536870912 USED=125937240 COMMITED=534708224 MAX=1069416448 New+Eden+Old NonHeap Memory: INIT=24317952 USED=80337408 COMMITED=80744448 MAX=318767104 Code Cache+Perm
  • 27. GC次数/时间java.lang.management.GarbageCollectorMXBean #getCollectionCount() #getCollectionTime() GC ParNew VALID [Par Eden Space, Par Survivor Space] GC=62 GCT=921 GC ConcurrentMarkSweep VALID [Par Eden Space, Par Survivor Space, CMS Old Gen, CMS Perm Gen] GC=0 GCT=0
  • 28. 类加载/卸载数java.lang.management.ClassLoadingMXBean #getTotalLoadedClassCount() #getLoadedClassCount() #getUnloadedClassCount() ClassLoading LOADED=9094 TOTAL_LOADED=9094 UNLOADED=0
  • 29. Java线程情况java.lang.management.ThreadMXBean ThreadInfo[] dumpAllThreads(...) Total threads: 22 CPU=74 (3.37%) USER=61 (2.79%) NEW=0 RUNNABLE=5 BLOCKED=0 WAITING=9 TIMED_WAITING=8 TERMINATED=0
  • 30. 和宿主JVM通讯ManagementFactory.getPlatformMBeanServer().registerMBean(jTopMXBean, new ObjectName(JTOP_MXBEAN_NAME));
  • 31. BTrace etc. 原理介绍Attach API http://docs.oracle.com/javase/6/docs/technotes/guides/attach/index.html Instrumentation http://docs.oracle.com/javase/6/docs/technotes/guides/instrumentation/index.html ASM http://asm.ow2.org/ ServerSocket, RMI etc... http://code.google.com/p/hatter-source-code/wiki/Study_Java_ByteCode
  • 32. 被测试程序public class Test { public static void main(String[] args) throws Exception { for (int i = 0; i < 10000; i++) { print(i); Thread.sleep(2000); } } public static void print(int i) { System.out.println(i); } }
  • 33. Btrace脚本@BTrace public class Trace { @OnMethod(clazz = "Test", method = "print") public static void m0(AnyType[] args) { println("\n==== Test#print ===="); printArray(args); } } http://kenai.com/projects/btrace/pages/UserGuide
  • 34. BTrace测试$ java Test 0 1 2 3 4 5 $ btrace 1251 Trace.java ==== Test#print ==== [13, ] ==== Test#print ==== [14, ] ==== Test#print ==== [15, ] 程序输出BTrace输出
  • 35. classdump导出字节码$ sudo java -jar classdumpall.jar 1340 -filter Test [INFO] Output directory: . [INFO] Dump class: Test @ sun.misc.Launcher$AppClassLoader [INFO] jar: /Users/hatterjiang/Dropbox/Work@Alibaba/... %e5%9f%b9%e8%ae%ad%ef%bc%b0%ef%bc%b0%ef%bc%b4/JavaOpt/类名加载该类的ClassLoader该类的源Jar/目录名
  • 36. 反编译Btrace增强后的代码 public static void print(int paramInt) { $btrace$Trace$m0(new Object[] { Integer.valueOf(paramInt) }); System.out.println(paramInt); } private static void $btrace$Trace$m0(Object[] paramArrayOfObject) { if (!BTraceRuntime.enter(Trace.runtime)) return; try { BTraceUtils.println("\n==== Test#print ===="); BTraceUtils.printArray(paramArrayOfObject); BTraceRuntime.leave(); return; } catch (Throwable localThrowable) { BTraceRuntime.handleException(localThrowable); } }http://java.decompiler.free.fr/
  • 37. Linux/Unix性能工具top vmstat pstack gdb -ex "set pagination 0" -ex "thread apply all bt" --batch -p poor man's profiler http://poormansprofiler.org/
  • 38. JDK性能工具jstat jstack jinfo jmap jconsole/jvisualvm
  • 39. 第三方性能工具MAT http://www.eclipse.org/mat/ Btrace http://kenai.com/projects/btrace CRaSH http://julienviet.com/crash/ TProfiler https://github.com/taobao/TProfiler HouseMD https://github.com/zhongl/HouseMD jtop http://code.google.com/p/hatter-source-code/wiki/jtop
  • 40. 深入学习Troubleshooting Guide for Java SE 6 with HotSpot VM http://www.oracle.com/technetwork/java/javase/tsg-vm-149989.pdf Overview of Java SE Monitoring and Management http://docs.oracle.com/javase/6/docs/technotes/guides/management/overview.html Java SE 6 Performance White Paper http://java.sun.com/performance/reference/whitepapers/6_performance.html
  • 41. Q&AGoogle talk/Mail:jht5945@gmail.comhttp://code.hatter.mehttp://hatter.in