My first native memory leak in Playtika

That day I received heapdump from production as it is usually done when application runs out of memory. I was 100% confident that was leaking Appdynamics agent (we got that issue several times already on multiple services), so I was like 'eehh, i won't even bother opening MAT', but still I had to attach report and analysis to jira ticket. Was I surprised when I actually saw no real leak suspicions? I looked into AppDynamics and saw quite strange picture of garbage collection, so my first resort was -- G1 will do the trick. It didn't help. (well, that actually increased lifetime of an application for 1 more day :)). Okay, what next? We checked java version on production and decided that we should update it, cause it was about half a year old. That also didn't help. At this point I started looking into Automatic memory leak detection in Appdynamics. When it was turned on it clearly said that half of memory was occupied by netty's Recycler objects. We turned it off, it didn't help. The most strange thing for me was that I was not quite sure that it was heap memory leak or not. Our architect found a great article about native memory leak detection using jemalloc: Using jemalloc to get to the bottom of a memory leak. I tried that at stage environment and no luck -- it showed me nothing! I was desperate. So we decided to try that on production! After one day of running we got a nice report:

The boldest lines are the code blocks that have allocated more native memory then others. The picture clearly states -- I should search for MethodHandles and Exceptions. 5 minutes searching for the first one in the project libraries and I knew that feign and spring-data-commons are using that for supporting default methods on their interfaces. We were not using default methods in our Feign clients, but we were using extensively default methods in our Spring Data Repositories. Ahhha!! That was it -- DATACMNS-1074. It also explained the second bold line with exceptions. Fix was simple (we didn't want to update libraries) -- remove default methods.
Thanks Jason Evans for jemalloc.
BTW j stands for Jason not java ;)