Package org.apache.hadoop.hbase.regionserver

Class DefaultHeapMemoryTuner

java.lang.Object
org.apache.hadoop.hbase.regionserver.DefaultHeapMemoryTuner
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable, HeapMemoryTuner
@Private class DefaultHeapMemoryTuner extends Object implements HeapMemoryTuner
The default implementation for the HeapMemoryTuner. This will do statistical checks on number of evictions, cache misses and flushes to decide whether there should be changes in the heap size of memstore/block cache. During each tuner operation tuner takes a step which can either be INCREASE_BLOCK_CACHE_SIZE (increase block cache size), INCREASE_MEMSTORE_SIZE (increase memstore size) and by default it is NEUTRAL (no change). We say block cache is sufficient when there is no block cache eviction at all or major amount of memory allocated to block cache is empty, similarly we say memory allocated for memstore is sufficient when there is no memstore flushes because of heap pressure or major amount of memory allocated to memstore is empty. If both are sufficient we do nothing, if exactly one of them is found to be sufficient we decrease its size by step and increase the other by same amount. If none of them is sufficient we do statistical analysis on number of cache misses and flushes to determine tuner direction. Based on these statistics we decide the tuner direction. If we are not confident about which step direction to take we do nothing and wait for next iteration. On expectation we will be tuning for at least 10% tuner calls. The number of past periods to consider for statistics calculation can be specified in config by hbase.regionserver.heapmemory.autotuner.lookup.periods. Also these many initial calls to tuner will be ignored (cache is warming up and we leave the system to reach steady state). After the tuner takes a step, in next call we insure that last call was indeed helpful and did not do us any harm. If not then we revert the previous step. The step size is dynamic and it changes based on current and past few tuning directions and their step sizes. We maintain a parameter decayingAvgTunerStepSize which is sum of past tuner steps with sign(positive for increase in memstore and negative for increase in block cache). But rather than simple sum it is calculated by giving more priority to the recent tuning steps. When last few tuner steps were NETURAL then we assume we are restarting the tuning process and step size is updated to maximum allowed size which can be specified in config by hbase.regionserver.heapmemory.autotuner.step.max. If in a particular tuning operation the step direction is opposite to what indicated by decayingTunerStepSizeSum we decrease the step size by half. Step size does not change in other tuning operations. When step size gets below a certain threshold then the following tuner operations are considered to be neutral. The minimum step size can be specified in config by hbase.regionserver.heapmemory.autotuner.step.min.