Package org.apache.hadoop.hbase.regionserver.compactions

Class CompactionConfiguration

java.lang.Object

org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration

@Private
public class CompactionConfiguration
extends Object

Compaction configuration for a particular instance of HStore. Takes into account both global settings and ones set on the column family/store. Control knobs for default compaction algorithm:

maxCompactSize - upper bound on file size to be included in minor compactions minCompactSize - lower bound below which compaction is selected without ratio test minFilesToCompact - lower bound on number of files in any minor compaction maxFilesToCompact - upper bound on number of files in any minor compaction compactionRatio - Ratio used for compaction minLocalityToForceCompact - Locality threshold for a store file to major compact (HBASE-11195)

Set parameter as "hbase.hstore.compaction.<attribute>"

Field Summary

Fields

Modifier and Type	Field	Description
private String	coldWindowStoragePolicy
static final String	COMPACTION_POLICY_CLASS_FOR_DATE_TIERED_WINDOWS_KEY
private final String	compactionPolicyForDateTieredWindow
private final double	compactionRatio
(package private) org.apache.hadoop.conf.Configuration	conf
static final String	DATE_TIERED_COLD_WINDOW_STORAGE_POLICY_KEY	Windows older than warm age belong to COLD_WINDOW
static final String	DATE_TIERED_COMPACTION_WINDOW_FACTORY_CLASS_KEY
static final String	DATE_TIERED_HOT_WINDOW_AGE_MILLIS_KEY
static final String	DATE_TIERED_HOT_WINDOW_STORAGE_POLICY_KEY
static final String	DATE_TIERED_INCOMING_WINDOW_MIN_KEY
static final String	DATE_TIERED_MAX_AGE_MILLIS_KEY
static final String	DATE_TIERED_SINGLE_OUTPUT_FOR_MINOR_COMPACTION_KEY
static final String	DATE_TIERED_STORAGE_POLICY_ENABLE_KEY
static final String	DATE_TIERED_WARM_WINDOW_AGE_MILLIS_KEY
static final String	DATE_TIERED_WARM_WINDOW_STORAGE_POLICY_KEY
private final String	dateTieredCompactionWindowFactory
private final int	dateTieredIncomingWindowMin
private final long	dateTieredMaxStoreFileAgeMillis
private final boolean	dateTieredSingleOutputForMinorCompaction
private final boolean	dateTieredStoragePolicyEnable
private static final Class<? extends RatioBasedCompactionPolicy>	DEFAULT_COMPACTION_POLICY_CLASS_FOR_DATE_TIERED_WINDOWS
private static final Class<? extends CompactionWindowFactory>	DEFAULT_DATE_TIERED_COMPACTION_WINDOW_FACTORY_CLASS
static final String	HBASE_HFILE_COMPACTION_DISCHARGER_THREAD_COUNT
static final String	HBASE_HSTORE_COMPACTION_MAX_KEY
static final String	HBASE_HSTORE_COMPACTION_MAX_SIZE_KEY
static final String	HBASE_HSTORE_COMPACTION_MAX_SIZE_OFFPEAK_KEY
static final String	HBASE_HSTORE_COMPACTION_MIN_KEY
static final String	HBASE_HSTORE_COMPACTION_MIN_SIZE_KEY
static final String	HBASE_HSTORE_COMPACTION_RATIO_KEY
static final String	HBASE_HSTORE_COMPACTION_RATIO_OFFPEAK_KEY
static final String	HBASE_HSTORE_MIN_LOCALITY_TO_SKIP_MAJOR_COMPACT
static final String	HBASE_HSTORE_OFFPEAK_END_HOUR
static final String	HBASE_HSTORE_OFFPEAK_START_HOUR
private long	hotWindowAgeMillis
private String	hotWindowStoragePolicy
private static final org.slf4j.Logger	LOG
private final float	majorCompactionJitter
private final long	majorCompactionPeriod
private final long	maxCompactSize	Since all these properties can change online, they are volatile
private final int	maxFilesToCompact
private final long	minCompactSize
private int	minFilesToCompact	This one can be update
private final float	minLocalityToForceCompact
private final double	offPeakCompactionRatio
private final long	offPeakMaxCompactSize
(package private) StoreConfigInformation	storeConfigInfo
private final long	throttlePoint
private long	warmWindowAgeMillis
private String	warmWindowStoragePolicy

Constructor Summary

Constructors

Constructor	Description
CompactionConfiguration(org.apache.hadoop.conf.Configuration conf, StoreConfigInformation storeConfigInfo)

Method Summary

Modifier and Type	Method	Description
String	getColdWindowStoragePolicy()
String	getCompactionPolicyForDateTieredWindow()
double	getCompactionRatio()	Returns Ratio used for compaction
double	getCompactionRatioOffPeak()	Returns Off peak Ratio used for compaction
String	getDateTieredCompactionWindowFactory()
int	getDateTieredIncomingWindowMin()
long	getDateTieredMaxStoreFileAgeMillis()
long	getHotWindowAgeMillis()
String	getHotWindowStoragePolicy()
float	getMajorCompactionJitter()
long	getMajorCompactionPeriod()
long	getMaxCompactSize()	Returns upper bound on file size to be included in minor compactions
long	getMaxCompactSize(boolean mayUseOffpeak)
int	getMaxFilesToCompact()	Returns upper bound on number of files to be included in minor compactions
long	getMinCompactSize()	Returns lower bound below which compaction is selected without ratio test
int	getMinFilesToCompact()	Returns upper bound on number of files to be included in minor compactions
float	getMinLocalityToForceCompact()
long	getOffPeakMaxCompactSize()
long	getThrottlePoint()	Returns ThrottlePoint used for classifying small and large compactions
long	getWarmWindowAgeMillis()
String	getWarmWindowStoragePolicy()
boolean	isDateTieredStoragePolicyEnable()
void	setMinFilesToCompact(int threshold)	Set upper bound on number of files to be included in minor compactions
String	toString()
boolean	useDateTieredSingleOutputForMinorCompaction()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

HBASE_HSTORE_COMPACTION_RATIO_KEY

public static final String HBASE_HSTORE_COMPACTION_RATIO_KEY

See Also:

• Constant Field Values

HBASE_HSTORE_COMPACTION_RATIO_OFFPEAK_KEY

public static final String HBASE_HSTORE_COMPACTION_RATIO_OFFPEAK_KEY

See Also:

• Constant Field Values

HBASE_HSTORE_COMPACTION_MIN_KEY

public static final String HBASE_HSTORE_COMPACTION_MIN_KEY

See Also:

• Constant Field Values

HBASE_HSTORE_COMPACTION_MIN_SIZE_KEY

public static final String HBASE_HSTORE_COMPACTION_MIN_SIZE_KEY

See Also:

• Constant Field Values

HBASE_HSTORE_COMPACTION_MAX_KEY

public static final String HBASE_HSTORE_COMPACTION_MAX_KEY

See Also:

• Constant Field Values

HBASE_HSTORE_COMPACTION_MAX_SIZE_KEY

public static final String HBASE_HSTORE_COMPACTION_MAX_SIZE_KEY

See Also:

• Constant Field Values

HBASE_HSTORE_COMPACTION_MAX_SIZE_OFFPEAK_KEY

public static final String HBASE_HSTORE_COMPACTION_MAX_SIZE_OFFPEAK_KEY

See Also:

• Constant Field Values

HBASE_HSTORE_OFFPEAK_END_HOUR

public static final String HBASE_HSTORE_OFFPEAK_END_HOUR

See Also:

• Constant Field Values

HBASE_HSTORE_OFFPEAK_START_HOUR

public static final String HBASE_HSTORE_OFFPEAK_START_HOUR

See Also:

• Constant Field Values

HBASE_HSTORE_MIN_LOCALITY_TO_SKIP_MAJOR_COMPACT

public static final String HBASE_HSTORE_MIN_LOCALITY_TO_SKIP_MAJOR_COMPACT

See Also:

• Constant Field Values

HBASE_HFILE_COMPACTION_DISCHARGER_THREAD_COUNT

public static final String HBASE_HFILE_COMPACTION_DISCHARGER_THREAD_COUNT

See Also:

• Constant Field Values

DATE_TIERED_MAX_AGE_MILLIS_KEY

public static final String DATE_TIERED_MAX_AGE_MILLIS_KEY

See Also:

• Constant Field Values

DATE_TIERED_INCOMING_WINDOW_MIN_KEY

public static final String DATE_TIERED_INCOMING_WINDOW_MIN_KEY

See Also:

• Constant Field Values

COMPACTION_POLICY_CLASS_FOR_DATE_TIERED_WINDOWS_KEY

public static final String COMPACTION_POLICY_CLASS_FOR_DATE_TIERED_WINDOWS_KEY

See Also:

• Constant Field Values

DATE_TIERED_SINGLE_OUTPUT_FOR_MINOR_COMPACTION_KEY

public static final String DATE_TIERED_SINGLE_OUTPUT_FOR_MINOR_COMPACTION_KEY

See Also:

• Constant Field Values

DEFAULT_COMPACTION_POLICY_CLASS_FOR_DATE_TIERED_WINDOWS

private static final Class<? extends RatioBasedCompactionPolicy> DEFAULT_COMPACTION_POLICY_CLASS_FOR_DATE_TIERED_WINDOWS

DATE_TIERED_COMPACTION_WINDOW_FACTORY_CLASS_KEY

public static final String DATE_TIERED_COMPACTION_WINDOW_FACTORY_CLASS_KEY

See Also:

• Constant Field Values

DEFAULT_DATE_TIERED_COMPACTION_WINDOW_FACTORY_CLASS

private static final Class<? extends CompactionWindowFactory> DEFAULT_DATE_TIERED_COMPACTION_WINDOW_FACTORY_CLASS

DATE_TIERED_STORAGE_POLICY_ENABLE_KEY

public static final String DATE_TIERED_STORAGE_POLICY_ENABLE_KEY

See Also:

• Constant Field Values

DATE_TIERED_HOT_WINDOW_AGE_MILLIS_KEY

public static final String DATE_TIERED_HOT_WINDOW_AGE_MILLIS_KEY

See Also:

• Constant Field Values

DATE_TIERED_HOT_WINDOW_STORAGE_POLICY_KEY

public static final String DATE_TIERED_HOT_WINDOW_STORAGE_POLICY_KEY

See Also:

• Constant Field Values

DATE_TIERED_WARM_WINDOW_AGE_MILLIS_KEY

public static final String DATE_TIERED_WARM_WINDOW_AGE_MILLIS_KEY

See Also:

• Constant Field Values

DATE_TIERED_WARM_WINDOW_STORAGE_POLICY_KEY

public static final String DATE_TIERED_WARM_WINDOW_STORAGE_POLICY_KEY

See Also:

• Constant Field Values

DATE_TIERED_COLD_WINDOW_STORAGE_POLICY_KEY

public static final String DATE_TIERED_COLD_WINDOW_STORAGE_POLICY_KEY

Windows older than warm age belong to COLD_WINDOW

See Also:

• Constant Field Values

conf

org.apache.hadoop.conf.Configuration conf

storeConfigInfo

StoreConfigInformation storeConfigInfo

offPeakCompactionRatio

private final double offPeakCompactionRatio

maxCompactSize

private final long maxCompactSize

Since all these properties can change online, they are volatile

offPeakMaxCompactSize

private final long offPeakMaxCompactSize

minCompactSize

private final long minCompactSize

minFilesToCompact

private int minFilesToCompact

This one can be update

maxFilesToCompact

private final int maxFilesToCompact

compactionRatio

private final double compactionRatio

throttlePoint

private final long throttlePoint

majorCompactionPeriod

private final long majorCompactionPeriod

majorCompactionJitter

private final float majorCompactionJitter

minLocalityToForceCompact

private final float minLocalityToForceCompact

dateTieredMaxStoreFileAgeMillis

private final long dateTieredMaxStoreFileAgeMillis

dateTieredIncomingWindowMin

private final int dateTieredIncomingWindowMin

compactionPolicyForDateTieredWindow

private final String compactionPolicyForDateTieredWindow

dateTieredSingleOutputForMinorCompaction

private final boolean dateTieredSingleOutputForMinorCompaction

dateTieredCompactionWindowFactory

private final String dateTieredCompactionWindowFactory

dateTieredStoragePolicyEnable

private final boolean dateTieredStoragePolicyEnable

hotWindowAgeMillis

private long hotWindowAgeMillis

warmWindowAgeMillis

private long warmWindowAgeMillis

hotWindowStoragePolicy

private String hotWindowStoragePolicy

warmWindowStoragePolicy

private String warmWindowStoragePolicy

coldWindowStoragePolicy

private String coldWindowStoragePolicy

Constructor Details

CompactionConfiguration

CompactionConfiguration(org.apache.hadoop.conf.Configuration conf,
 StoreConfigInformation storeConfigInfo)

Method Details

toString

public String toString()

Overrides:

toString in class Object

getMinCompactSize

public long getMinCompactSize()

Returns lower bound below which compaction is selected without ratio test

getMaxCompactSize

public long getMaxCompactSize()

Returns upper bound on file size to be included in minor compactions

getMinFilesToCompact

public int getMinFilesToCompact()

Returns upper bound on number of files to be included in minor compactions

setMinFilesToCompact

public void setMinFilesToCompact(int threshold)

Set upper bound on number of files to be included in minor compactions

Parameters:

threshold - value to set to

getMaxFilesToCompact

public int getMaxFilesToCompact()

Returns upper bound on number of files to be included in minor compactions

getCompactionRatio

public double getCompactionRatio()

Returns Ratio used for compaction

getCompactionRatioOffPeak

public double getCompactionRatioOffPeak()

Returns Off peak Ratio used for compaction

getThrottlePoint

public long getThrottlePoint()

Returns ThrottlePoint used for classifying small and large compactions

getMajorCompactionPeriod

public long getMajorCompactionPeriod()

Returns:

Major compaction period from compaction. Major compactions are selected periodically according to this parameter plus jitter

getMajorCompactionJitter

public float getMajorCompactionJitter()

Returns:

Major the jitter fraction, the fraction within which the major compaction period is randomly chosen from the majorCompactionPeriod in each store.

getMinLocalityToForceCompact

public float getMinLocalityToForceCompact()

Returns:

Block locality ratio, the ratio at which we will include old regions with a single store file for major compaction. Used to improve block locality for regions that haven't had writes in a while but are still being read.

getOffPeakMaxCompactSize

public long getOffPeakMaxCompactSize()

getMaxCompactSize

public long getMaxCompactSize(boolean mayUseOffpeak)

getDateTieredMaxStoreFileAgeMillis

public long getDateTieredMaxStoreFileAgeMillis()

getDateTieredIncomingWindowMin

public int getDateTieredIncomingWindowMin()

getCompactionPolicyForDateTieredWindow

public String getCompactionPolicyForDateTieredWindow()

useDateTieredSingleOutputForMinorCompaction

public boolean useDateTieredSingleOutputForMinorCompaction()

getDateTieredCompactionWindowFactory

public String getDateTieredCompactionWindowFactory()

isDateTieredStoragePolicyEnable

public boolean isDateTieredStoragePolicyEnable()

getHotWindowAgeMillis

public long getHotWindowAgeMillis()

getWarmWindowAgeMillis

public long getWarmWindowAgeMillis()

getHotWindowStoragePolicy

public String getHotWindowStoragePolicy()

getWarmWindowStoragePolicy

public String getWarmWindowStoragePolicy()

getColdWindowStoragePolicy

public String getColdWindowStoragePolicy()