Package org.apache.hadoop.hbase.io.hfile

Class CacheConfig

java.lang.Object

org.apache.hadoop.hbase.io.hfile.CacheConfig

All Implemented Interfaces:

ConfigurationObserver, PropagatingConfigurationObserver

@Private
public class CacheConfig
extends Object
implements PropagatingConfigurationObserver

Stores all of the cache objects and configuration for a single HFile.

Field Summary

Fields

Modifier and Type	Field	Description
private final BlockCache	blockCache
static final String	BUCKETCACHE_PERSIST_INTERVAL_KEY	Configuration key to set interval for persisting bucket cache to disk.
private final ByteBuffAllocator	byteBuffAllocator
static final String	CACHE_BLOCKS_ON_WRITE_KEY	Configuration key to cache data blocks on write.
static final String	CACHE_BLOOM_BLOCKS_ON_WRITE_KEY	Configuration key to cache compound bloom filter blocks on write.
static final String	CACHE_COMPACTED_BLOCKS_ON_WRITE_KEY	Configuration key to cache blocks when a compacted file is written
static final String	CACHE_COMPACTED_BLOCKS_ON_WRITE_THRESHOLD_KEY	Configuration key to determine total size in bytes of compacted files beyond which we do not cache blocks on compaction
static final String	CACHE_DATA_BLOCKS_COMPRESSED_KEY	Configuration key to cache data blocks in compressed and/or encrypted format.
static final String	CACHE_DATA_ON_READ_KEY	Configuration key to cache data blocks on read.
static final String	CACHE_INDEX_BLOCKS_ON_WRITE_KEY	Configuration key to cache leaf and intermediate-level index blocks on write.
private boolean	cacheBloomsOnWrite	Whether compound bloom filter blocks should be cached on write
private final boolean	cacheCompactedDataOnWrite	Whether data blocks should be cached when compacted file is written
private long	cacheCompactedDataOnWriteThreshold	Determine threshold beyond which we do not cache blocks on compaction
private final boolean	cacheDataCompressed	Whether data blocks should be stored in compressed and/or encrypted form in the cache
private boolean	cacheDataOnRead	Whether blocks should be cached on read (default is on if there is a cache but this can be turned off on a per-family or per-request basis).
private boolean	cacheDataOnWrite	Whether data blocks should be cached when new files are written
private boolean	cacheIndexesOnWrite	Whether index blocks should be cached when new files are written
static final boolean	DEFAULT_CACHE_BLOOMS_ON_WRITE
static final boolean	DEFAULT_CACHE_COMPACTED_BLOCKS_ON_WRITE
static final long	DEFAULT_CACHE_COMPACTED_BLOCKS_ON_WRITE_THRESHOLD
static final boolean	DEFAULT_CACHE_DATA_COMPRESSED
static final boolean	DEFAULT_CACHE_DATA_ON_READ
static final boolean	DEFAULT_CACHE_DATA_ON_WRITE
static final boolean	DEFAULT_CACHE_INDEXES_ON_WRITE
static final boolean	DEFAULT_EVICT_ON_CLOSE
static final boolean	DEFAULT_EVICT_ON_SPLIT
static final boolean	DEFAULT_IN_MEMORY
static final double	DEFAULT_PREFETCH_HEAP_USAGE_THRESHOLD
static final boolean	DEFAULT_PREFETCH_ON_OPEN
static final CacheConfig	DISABLED	Disabled cache configuration
static final boolean	DROP_BEHIND_CACHE_COMPACTION_DEFAULT
static final String	DROP_BEHIND_CACHE_COMPACTION_KEY
private final boolean	dropBehindCompaction
static final String	EVICT_BLOCKS_ON_CLOSE_KEY	Configuration key to evict all blocks of a given file from the block cache when the file is closed.
static final String	EVICT_BLOCKS_ON_SPLIT_KEY
private boolean	evictOnClose	Whether blocks of a file should be evicted when the file is closed
private final double	heapUsageThreshold
private final boolean	inMemory	Whether blocks should be flagged as in-memory when being cached
private static final org.slf4j.Logger	LOG
static final String	PREFETCH_BLOCKS_ON_OPEN_KEY	Configuration key to prefetch all blocks of a given file into the block cache when the file is opened.
static final String	PREFETCH_HEAP_USAGE_THRESHOLD	Configuration key to set the heap usage threshold limit once prefetch threads should be interrupted.
private final boolean	prefetchOnOpen	Whether data blocks should be prefetched into the cache

Constructor Summary

Constructors

Modifier	Constructor	Description
private	CacheConfig()
	CacheConfig(org.apache.hadoop.conf.Configuration conf)	Create a cache configuration using the specified configuration object and defaults for family level settings.
	CacheConfig(org.apache.hadoop.conf.Configuration conf, ColumnFamilyDescriptor family, BlockCache blockCache, ByteBuffAllocator byteBuffAllocator)	Create a cache configuration using the specified configuration object and family descriptor.
	CacheConfig(org.apache.hadoop.conf.Configuration conf, BlockCache blockCache)
	CacheConfig(CacheConfig cacheConf)	Constructs a cache configuration copied from the specified configuration.

Method Summary

Modifier and Type	Method	Description
void	deregisterChildren(ConfigurationManager manager)	Needs to be called to deregister the children from the manager.
void	enableCacheOnWrite()	Enable cache on write including: cacheDataOnWrite cacheIndexesOnWrite cacheBloomsOnWrite
Optional<BlockCache>	getBlockCache()	Returns the block cache.
ByteBuffAllocator	getByteBuffAllocator()
long	getCacheCompactedBlocksOnWriteThreshold()	Returns total file size in bytes threshold for caching while writing during compaction
private long	getCacheCompactedBlocksOnWriteThreshold(org.apache.hadoop.conf.Configuration conf)
double	getHeapUsageThreshold()
boolean	isCombinedBlockCache()
boolean	isHeapUsageBelowThreshold()	Checks if the current heap usage is below the threshold configured by "hbase.rs.prefetchheapusage" (0.8 by default).
boolean	isInMemory()	Returns true if blocks in this file should be flagged as in-memory
void	onConfigurationChange(org.apache.hadoop.conf.Configuration conf)	This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.
void	registerChildren(ConfigurationManager manager)	Needs to be called to register the children to the manager.
void	setCacheDataOnWrite(boolean cacheDataOnWrite)
void	setEvictOnClose(boolean evictOnClose)	Only used for testing.
boolean	shouldCacheBlockOnRead(BlockType.BlockCategory category)	Should we cache a block of a particular category? We always cache important blocks such as index blocks, as long as the block cache is available.
boolean	shouldCacheBloomsOnWrite()
boolean	shouldCacheCompactedBlocksOnWrite()	Returns true if blocks should be cached while writing during compaction, false if not
boolean	shouldCacheCompressed(BlockType.BlockCategory category)	Returns true if this BlockType.BlockCategory should be compressed in blockcache, false otherwise
boolean	shouldCacheDataCompressed()	Returns true if data blocks should be compressed in the cache, false if not
boolean	shouldCacheDataOnRead()	Returns whether the DATA blocks of this HFile should be cached on read or not (we always cache the meta blocks, the INDEX and BLOOM blocks).
boolean	shouldCacheDataOnWrite()
boolean	shouldCacheIndexesOnWrite()
boolean	shouldDropBehindCompaction()
boolean	shouldEvictOnClose()
boolean	shouldLockOnCacheMiss(BlockType blockType)	If we make sure the block could not be cached, we will not acquire the lock otherwise we will acquire lock
boolean	shouldPrefetchOnOpen()	Returns true if blocks should be prefetched into the cache on open, false if not
boolean	shouldReadBlockFromCache(BlockType blockType)	Return true if we may find this type of block in block cache.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

DISABLED

public static final CacheConfig DISABLED

Disabled cache configuration

CACHE_DATA_ON_READ_KEY

public static final String CACHE_DATA_ON_READ_KEY

Configuration key to cache data blocks on read. Bloom blocks and index blocks are always be cached if the block cache is enabled.

See Also:

• Constant Field Values

CACHE_BLOCKS_ON_WRITE_KEY

public static final String CACHE_BLOCKS_ON_WRITE_KEY

Configuration key to cache data blocks on write. There are separate switches for bloom blocks and non-root index blocks.

See Also:

• Constant Field Values

CACHE_INDEX_BLOCKS_ON_WRITE_KEY

public static final String CACHE_INDEX_BLOCKS_ON_WRITE_KEY

Configuration key to cache leaf and intermediate-level index blocks on write.

See Also:

• Constant Field Values

CACHE_BLOOM_BLOCKS_ON_WRITE_KEY

public static final String CACHE_BLOOM_BLOCKS_ON_WRITE_KEY

Configuration key to cache compound bloom filter blocks on write.

See Also:

• Constant Field Values

CACHE_DATA_BLOCKS_COMPRESSED_KEY

public static final String CACHE_DATA_BLOCKS_COMPRESSED_KEY

Configuration key to cache data blocks in compressed and/or encrypted format.

See Also:

• Constant Field Values

EVICT_BLOCKS_ON_CLOSE_KEY

public static final String EVICT_BLOCKS_ON_CLOSE_KEY

Configuration key to evict all blocks of a given file from the block cache when the file is closed.

See Also:

• Constant Field Values

EVICT_BLOCKS_ON_SPLIT_KEY

public static final String EVICT_BLOCKS_ON_SPLIT_KEY

See Also:

• Constant Field Values

PREFETCH_BLOCKS_ON_OPEN_KEY

public static final String PREFETCH_BLOCKS_ON_OPEN_KEY

Configuration key to prefetch all blocks of a given file into the block cache when the file is opened.

See Also:

• Constant Field Values

CACHE_COMPACTED_BLOCKS_ON_WRITE_KEY

public static final String CACHE_COMPACTED_BLOCKS_ON_WRITE_KEY

Configuration key to cache blocks when a compacted file is written

See Also:

• Constant Field Values

CACHE_COMPACTED_BLOCKS_ON_WRITE_THRESHOLD_KEY

public static final String CACHE_COMPACTED_BLOCKS_ON_WRITE_THRESHOLD_KEY

Configuration key to determine total size in bytes of compacted files beyond which we do not cache blocks on compaction

See Also:

• Constant Field Values

DROP_BEHIND_CACHE_COMPACTION_KEY

public static final String DROP_BEHIND_CACHE_COMPACTION_KEY

See Also:

• Constant Field Values

BUCKETCACHE_PERSIST_INTERVAL_KEY

public static final String BUCKETCACHE_PERSIST_INTERVAL_KEY

Configuration key to set interval for persisting bucket cache to disk.

See Also:

• Constant Field Values

PREFETCH_HEAP_USAGE_THRESHOLD

public static final String PREFETCH_HEAP_USAGE_THRESHOLD

Configuration key to set the heap usage threshold limit once prefetch threads should be interrupted.

See Also:

• Constant Field Values

DEFAULT_CACHE_DATA_ON_READ

public static final boolean DEFAULT_CACHE_DATA_ON_READ

See Also:

• Constant Field Values

DEFAULT_CACHE_DATA_ON_WRITE

public static final boolean DEFAULT_CACHE_DATA_ON_WRITE

See Also:

• Constant Field Values

DEFAULT_IN_MEMORY

public static final boolean DEFAULT_IN_MEMORY

See Also:

• Constant Field Values

DEFAULT_CACHE_INDEXES_ON_WRITE

public static final boolean DEFAULT_CACHE_INDEXES_ON_WRITE

See Also:

• Constant Field Values

DEFAULT_CACHE_BLOOMS_ON_WRITE

public static final boolean DEFAULT_CACHE_BLOOMS_ON_WRITE

See Also:

• Constant Field Values

DEFAULT_EVICT_ON_CLOSE

public static final boolean DEFAULT_EVICT_ON_CLOSE

See Also:

• Constant Field Values

DEFAULT_EVICT_ON_SPLIT

public static final boolean DEFAULT_EVICT_ON_SPLIT

See Also:

• Constant Field Values

DEFAULT_CACHE_DATA_COMPRESSED

public static final boolean DEFAULT_CACHE_DATA_COMPRESSED

See Also:

• Constant Field Values

DEFAULT_PREFETCH_ON_OPEN

public static final boolean DEFAULT_PREFETCH_ON_OPEN

See Also:

• Constant Field Values

DEFAULT_CACHE_COMPACTED_BLOCKS_ON_WRITE

public static final boolean DEFAULT_CACHE_COMPACTED_BLOCKS_ON_WRITE

See Also:

• Constant Field Values

DROP_BEHIND_CACHE_COMPACTION_DEFAULT

public static final boolean DROP_BEHIND_CACHE_COMPACTION_DEFAULT

See Also:

• Constant Field Values

DEFAULT_CACHE_COMPACTED_BLOCKS_ON_WRITE_THRESHOLD

public static final long DEFAULT_CACHE_COMPACTED_BLOCKS_ON_WRITE_THRESHOLD

See Also:

• Constant Field Values

DEFAULT_PREFETCH_HEAP_USAGE_THRESHOLD

public static final double DEFAULT_PREFETCH_HEAP_USAGE_THRESHOLD

See Also:

• Constant Field Values

cacheDataOnRead

private volatile boolean cacheDataOnRead

Whether blocks should be cached on read (default is on if there is a cache but this can be turned off on a per-family or per-request basis). If off we will STILL cache meta blocks; i.e. INDEX and BLOOM types. This cannot be disabled.

inMemory

private final boolean inMemory

Whether blocks should be flagged as in-memory when being cached

cacheDataOnWrite

private volatile boolean cacheDataOnWrite

Whether data blocks should be cached when new files are written

cacheIndexesOnWrite

private boolean cacheIndexesOnWrite

Whether index blocks should be cached when new files are written

cacheBloomsOnWrite

private boolean cacheBloomsOnWrite

Whether compound bloom filter blocks should be cached on write

evictOnClose

private volatile boolean evictOnClose

Whether blocks of a file should be evicted when the file is closed

cacheDataCompressed

private final boolean cacheDataCompressed

Whether data blocks should be stored in compressed and/or encrypted form in the cache

prefetchOnOpen

private final boolean prefetchOnOpen

Whether data blocks should be prefetched into the cache

cacheCompactedDataOnWrite

private final boolean cacheCompactedDataOnWrite

Whether data blocks should be cached when compacted file is written

cacheCompactedDataOnWriteThreshold

private long cacheCompactedDataOnWriteThreshold

Determine threshold beyond which we do not cache blocks on compaction

dropBehindCompaction

private final boolean dropBehindCompaction

blockCache

private final BlockCache blockCache

byteBuffAllocator

private final ByteBuffAllocator byteBuffAllocator

heapUsageThreshold

private final double heapUsageThreshold

Constructor Details

CacheConfig

public CacheConfig(org.apache.hadoop.conf.Configuration conf)

Create a cache configuration using the specified configuration object and defaults for family level settings. Only use if no column family context.

Parameters:

conf - hbase configuration

CacheConfig

public CacheConfig(org.apache.hadoop.conf.Configuration conf,
 BlockCache blockCache)

CacheConfig

public CacheConfig(org.apache.hadoop.conf.Configuration conf,
 ColumnFamilyDescriptor family,
 BlockCache blockCache,
 ByteBuffAllocator byteBuffAllocator)

Create a cache configuration using the specified configuration object and family descriptor.

Parameters:

conf - hbase configuration

family - column family configuration

CacheConfig

public CacheConfig(CacheConfig cacheConf)

Constructs a cache configuration copied from the specified configuration.

CacheConfig

private CacheConfig()

Method Details

shouldCacheDataOnRead

public boolean shouldCacheDataOnRead()

Returns whether the DATA blocks of this HFile should be cached on read or not (we always cache the meta blocks, the INDEX and BLOOM blocks).

Returns:

true if blocks should be cached on read, false if not

shouldDropBehindCompaction

public boolean shouldDropBehindCompaction()

shouldCacheBlockOnRead

public boolean shouldCacheBlockOnRead(BlockType.BlockCategory category)

Should we cache a block of a particular category? We always cache important blocks such as index blocks, as long as the block cache is available.

isInMemory

public boolean isInMemory()

Returns true if blocks in this file should be flagged as in-memory

shouldCacheDataOnWrite

public boolean shouldCacheDataOnWrite()

Returns:

true if data blocks should be written to the cache when an HFile is written, false if not

setCacheDataOnWrite

public void setCacheDataOnWrite(boolean cacheDataOnWrite)

Parameters:

cacheDataOnWrite - whether data blocks should be written to the cache when an HFile is written

enableCacheOnWrite

public void enableCacheOnWrite()

Enable cache on write including: cacheDataOnWrite cacheIndexesOnWrite cacheBloomsOnWrite

shouldCacheIndexesOnWrite

public boolean shouldCacheIndexesOnWrite()

Returns:

true if index blocks should be written to the cache when an HFile is written, false if not

shouldCacheBloomsOnWrite

public boolean shouldCacheBloomsOnWrite()

Returns:

true if bloom blocks should be written to the cache when an HFile is written, false if not

shouldEvictOnClose

public boolean shouldEvictOnClose()

Returns:

true if blocks should be evicted from the cache when an HFile reader is closed, false if not

setEvictOnClose

public void setEvictOnClose(boolean evictOnClose)

Only used for testing.

Parameters:

evictOnClose - whether blocks should be evicted from the cache when an HFile reader is closed

shouldCacheDataCompressed

public boolean shouldCacheDataCompressed()

Returns true if data blocks should be compressed in the cache, false if not

shouldCacheCompressed

public boolean shouldCacheCompressed(BlockType.BlockCategory category)

Returns true if this BlockType.BlockCategory should be compressed in blockcache, false otherwise

shouldPrefetchOnOpen

public boolean shouldPrefetchOnOpen()

Returns true if blocks should be prefetched into the cache on open, false if not

shouldCacheCompactedBlocksOnWrite

public boolean shouldCacheCompactedBlocksOnWrite()

Returns true if blocks should be cached while writing during compaction, false if not

getCacheCompactedBlocksOnWriteThreshold

public long getCacheCompactedBlocksOnWriteThreshold()

Returns total file size in bytes threshold for caching while writing during compaction

shouldReadBlockFromCache

public boolean shouldReadBlockFromCache(BlockType blockType)

Return true if we may find this type of block in block cache.

TODO: today family.isBlockCacheEnabled() only means cacheDataOnRead, so here we consider lots of other configurations such as cacheDataOnWrite. We should fix this in the future, cacheDataOnWrite should honor the CF level isBlockCacheEnabled configuration.

isHeapUsageBelowThreshold

public boolean isHeapUsageBelowThreshold()

Checks if the current heap usage is below the threshold configured by "hbase.rs.prefetchheapusage" (0.8 by default).

shouldLockOnCacheMiss

public boolean shouldLockOnCacheMiss(BlockType blockType)

If we make sure the block could not be cached, we will not acquire the lock otherwise we will acquire lock

getBlockCache

public Optional<BlockCache> getBlockCache()

Returns the block cache.

Returns:

the block cache, or null if caching is completely disabled

isCombinedBlockCache

public boolean isCombinedBlockCache()

getByteBuffAllocator

public ByteBuffAllocator getByteBuffAllocator()

getHeapUsageThreshold

public double getHeapUsageThreshold()

getCacheCompactedBlocksOnWriteThreshold

private long getCacheCompactedBlocksOnWriteThreshold(org.apache.hadoop.conf.Configuration conf)

toString

public String toString()

Overrides:

toString in class Object

onConfigurationChange

public void onConfigurationChange(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: ConfigurationObserver

This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.

Specified by:

onConfigurationChange in interface ConfigurationObserver

registerChildren

public void registerChildren(ConfigurationManager manager)

Description copied from interface: PropagatingConfigurationObserver

Needs to be called to register the children to the manager.

Specified by:

registerChildren in interface PropagatingConfigurationObserver

Parameters:

manager - : to register to

deregisterChildren

public void deregisterChildren(ConfigurationManager manager)

Description copied from interface: PropagatingConfigurationObserver

Needs to be called to deregister the children from the manager.

Specified by:

deregisterChildren in interface PropagatingConfigurationObserver

Parameters:

manager - : to deregister from