欢迎访问Spring Cloud中国社区

《重新定义Spring Cloud实战》由Spring Cloud中国社区倾力打造,基于Spring Cloud的Finchley.RELEASE版本,本书内容宽度足够广、深度足够深,而且立足于生产实践,直接从生产实践出发,包含大量生产实践的配置。欢迎加微信Software_King进群答疑,国内谁在使用Spring Cloud?欢迎登记

Feign Decoder姿势不对导致CPU100%

admin · 3月前 · 990 ·

一个线上服务偶尔卡顿,分析发现是loadClass导致的线程阻塞,而loadClass的原因与Feign配置有关。

1.现象

最近线上的一个服务偶尔出现卡顿,表现为不特定时刻出现几分钟的异常,在这段时间内响应时间激增.

2.分析

首先查看日志,找到慢请求,发现在服务间Feign调用后会出现一段时间的间隔。对Feign的Logger和HttpMessageConverterExtractor开启DEBUG日志,看到在Feign输出HTTP返回数据后到Jackson反序列化之间有几秒的间隔。

  1. 2019-12-20 10:25:55.493|*,*,*|DEBUG|feign.slf4j.Slf4jLogger:(72)|[ServiceName#methodName] <--- END HTTP (1571-byte body)
  2. 2019-12-20 10:25:56.830|*,*,*|DEBUG|org.springframework.web.client.HttpMessageConverterExtractor:(100)|Reading to [com.*.DataModel<com.*.BusinessModel>]

这点很奇怪,在GC日志中也没有Stop-The-World出现。用jstack打印堆栈,发现有几个和Feign相关的线程处于BLOCKED状态:

  1. "http-nio-8080-exec-276" #3021 daemon prio=5 os_prio=0 tid=0x00007fbd501a8800 nid=0xdea waiting for monitor entry [0x00007fbcbd76d000]
  2. java.lang.Thread.State: BLOCKED (on object monitor)
  3. at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
  4. - waiting to lock <0x00000006c6ed91d0> (a java.lang.Object)
  5. at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93)
  6. at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  7. at java.lang.Class.forName0(Native Method)
  8. at java.lang.Class.forName(Class.java:348)
  9. at org.springframework.util.ClassUtils.forName(ClassUtils.java:276)
  10. at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:797)
  11. at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:650)
  12. at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:633)
  13. at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.<init>(MappingJackson2HttpMessageConverter.java:59)
  14. at org.springframework.http.converter.support.AllEncompassingFormHttpMessageConverter.<init>(AllEncompassingFormHttpMessageConverter.java:76)
  15. at org.springframework.web.servlet.config.annotation.WebMvcConfigurationSupport.addDefaultHttpMessageConverters(WebMvcConfigurationSupport.java:796)
  16. at org.springframework.web.servlet.config.annotation.WebMvcConfigurationSupport.getMessageConverters(WebMvcConfigurationSupport.java:748)
  17. at org.springframework.boot.autoconfigure.http.HttpMessageConverters$1.defaultMessageConverters(HttpMessageConverters.java:185)
  18. at org.springframework.boot.autoconfigure.http.HttpMessageConverters.getDefaultConverters(HttpMessageConverters.java:188)
  19. at org.springframework.boot.autoconfigure.http.HttpMessageConverters.<init>(HttpMessageConverters.java:105)
  20. at org.springframework.boot.autoconfigure.http.HttpMessageConverters.<init>(HttpMessageConverters.java:92)
  21. at org.springframework.boot.autoconfigure.http.HttpMessageConverters.<init>(HttpMessageConverters.java:80)
  22. at com.yirendai.app.fortune.support.config.FeignConfig.lambda$feignDecoder$0(FeignConfig.java:26)
  23. at com.yirendai.app.fortune.support.config.FeignConfig$$Lambda$536/1366741625.getObject(Unknown Source)
  24. at org.springframework.cloud.openfeign.support.SpringDecoder.decode(SpringDecoder.java:57)
  25. at org.springframework.cloud.openfeign.support.ResponseEntityDecoder.decode(ResponseEntityDecoder.java:62)
  26. at feign.optionals.OptionalDecoder.decode(OptionalDecoder.java:36)
  27. at feign.SynchronousMethodHandler.decode(SynchronousMethodHandler.java:178)
  28. at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:142)
  29. at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80)
  30. at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
  31. at com.sun.proxy.$Proxy178.searchUserTag(Unknown Source)

查看registerWellKnownModulesIfAvailable处的代码,可以看到其逻辑为若classpath中有JodaTime的LocalDate,则加载Jackson对应的JodaModule(这个项目中没有引用)。

  1. if (ClassUtils.isPresent("org.joda.time.LocalDate", this.moduleClassLoader)) {
  2. try {
  3. Class<? extends Module> jodaModuleClass = (Class<? extends Module>)
  4. ClassUtils.forName("com.fasterxml.jackson.datatype.joda.JodaModule", this.moduleClassLoader);
  5. Module jodaModule = BeanUtils.instantiateClass(jodaModuleClass);
  6. modulesToRegister.set(jodaModule.getTypeId(), jodaModule);
  7. } catch (ClassNotFoundException ex) {
  8. // jackson-datatype-joda not available
  9. }
  10. }

LaunchedURLClassLoader.loadClass将调用ClassLoader.loadClass来加载类,加载时需要获取锁,因此在并发环境下,可能导致线程BLOCKED状态。

  1. protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
  2. synchronized (getClassLoadingLock(name)) {
  3. // First, check if the class has already been loaded
  4. Class<?> c = findLoadedClass(name);
  5. if (c == null) {
  6. // 省略类加载代码
  7. }
  8. if (resolve) {
  9. resolveClass(c);
  10. }
  11. return c;
  12. }
  13. }
  14. protected Object getClassLoadingLock(String className) {
  15. Object lock = this;
  16. if (parallelLockMap != null) {
  17. Object newLock = new Object();
  18. lock = parallelLockMap.putIfAbsent(className, newLock);
  19. if (lock == null) {
  20. lock = newLock;
  21. }
  22. }
  23. return lock;
  24. }

依据以上信息,出现卡顿的流程大致如下:

Feign请求时会初始化MappingJackson2HttpMessageConverter时尝试加载JodaModule。
而这个类并不在classpath中,因此无法在findLoadedClass中找到,每次都需要重新加载。
执行loadClass时需要加锁,在线上高并发场景下会导致线程BLOCKED状态。

3. 解决方法

3.1 解决方式一:避免ClassLoader反复加载

可以看出卡顿的直接原因是反复尝试加载不在classpath中的JodaModule,因此将这个依赖添加到工程中。加载一次后,再次调用可以通过findLoadedClass获得,减少加载类导致的资源消耗,从而减少BLOCKED的出现。

  1. <dependency>
  2. <groupId>com.fasterxml.jackson.datatype</groupId>
  3. <artifactId>jackson-datatype-joda</artifactId>
  4. <version>x.x.x</version>
  5. </dependency>

3.2 解决方式二:避免HttpMessageConverters重复初始化

但是还有另一个问题需要考虑:为什么每次请求都会初始化MappingJackson2HttpMessageConverter?查看SpringDecoder代码,可以看到每次反序列化response时会调用ObjectFactory<HttpMessageConverters>来获取converters。

  1. @Override
  2. public Object decode(final Response response, Type type) throws IOException, FeignException {
  3. if (type instanceof Class || type instanceof ParameterizedType || type instanceof WildcardType) {
  4. ({ "unchecked", "rawtypes" })
  5. HttpMessageConverterExtractor<?> extractor = new HttpMessageConverterExtractor(
  6. type, this.messageConverters.getObject().getConverters());
  7. return extractor.extractData(new FeignResponseAdapter(response));
  8. }
  9. throw new DecodeException(response.status(),
  10. "type is not an instance of Class or ParameterizedType: " + type,
  11. response.request());
  12. }

而在FeignConfig中配置的这个ObjectFactory的实现是new一个HttpMessageConverters对象。

  1. @Bean
  2. public Decoder feignDecoder() {
  3. ObjectMapper mapper = new ObjectMapper()
  4. .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
  5. HttpMessageConverter jacksonConverter = new MappingJackson2HttpMessageConverter(mapper);
  6. ObjectFactory<HttpMessageConverters> objectFactory = () -> new HttpMessageConverters(jacksonConverter);
  7. return new OptionalDecoder(new ResponseEntityDecoder(new SpringDecoder(objectFactory)));
  8. }

HttpMessageConverters的构造方法会默认执行getDefaultConverters。其逻辑可查看WebMvcConfigurationSupport代码,其中AllEncompassingFormHttpMessageConverter的构造函数会创建MappingJackson2HttpMessageConverter对象。

  1. public HttpMessageConverters(HttpMessageConverter<?>... additionalConverters) {
  2. this(Arrays.asList(additionalConverters));
  3. }
  4. public HttpMessageConverters(Collection<HttpMessageConverter<?>> additionalConverters) {
  5. this(true, additionalConverters);
  6. }
  7. public HttpMessageConverters(boolean addDefaultConverters, Collection<HttpMessageConverter<?>> converters) {
  8. List<HttpMessageConverter<?>> combined = getCombinedConverters(converters,
  9. addDefaultConverters ? getDefaultConverters() : Collections.emptyList());
  10. combined = postProcessConverters(combined);
  11. this.converters = Collections.unmodifiableList(combined);
  12. }

这就是每一个请求都会初始化MappingJackson2HttpMessageConverter并触发loadClass的原因,因此每一个Feign请求的开销都很大。由于我们只需要使用自定义的MappingJackson2HttpMessageConverter来执行反序列化,可以想办法避免执行getDefaultConverters
第一种方法是指定HttpMessageConverters的构造方法参数addDefaultConverters为false

  1. ObjectFactory<HttpMessageConverters> objectFactory = () -> new HttpMessageConverters(false, Collections.singletonList(jacksonConverter));

第二种方法则是使用Feign的JacksonDecoder:

  1. @Bean
  2. public Decoder feignDecoder() {
  3. ObjectMapper mapper = new ObjectMapper()
  4. .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
  5. return new JacksonDecoder(mapper);
  6. }
  1. <dependency>
  2. <groupId>io.github.openfeign</groupId>
  3. <artifactId>feign-jackson</artifactId>
  4. <version>x.x.x</version>
  5. </dependency>