Package org.apache.hadoop.hbase.regionserver

Class HStore

java.lang.Object

org.apache.hadoop.hbase.regionserver.HStore

All Implemented Interfaces:

ConfigurationObserver, PropagatingConfigurationObserver, HeapSize, Store, StoreConfigInformation

Direct Known Subclasses:

HMobStore

@Private
public class HStore
extends Object
implements Store, HeapSize, StoreConfigInformation, PropagatingConfigurationObserver

A Store holds a column family in a Region. Its a memstore and a set of zero or more StoreFiles, which stretch backwards over time.

There's no reason to consider append-logging at this level; all logging and locking is handled at the HRegion level. Store just provides services to manage sets of StoreFiles. One of the most important of those services is compaction services where files are aggregated once they pass a configurable threshold.

Locking and transactions are handled at a higher level. This API should not be called directly but by an HRegion manager.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static final class	HStore.StoreFileWriterCreationTracker
private final class	HStore.StoreFlusherImpl

Field Summary

Fields

Modifier and Type	Field	Description
(package private) final ReentrantLock	archiveLock	Lock specific to archiving compacted store files.
static final String	BLOCK_STORAGE_POLICY_KEY
static final String	BLOCKING_STOREFILES_KEY
private long	blockingFileCount
private final Set<ChangedReadersObserver>	changedReaderObservers
static final String	COMPACTCHECKER_INTERVAL_MULTIPLIER_KEY
private AtomicLong	compactedCellsCount
private AtomicLong	compactedCellsSize
private int	compactionCheckMultiplier
protected org.apache.hadoop.conf.Configuration	conf
private final AtomicInteger	currentParallelPutCount	Use this counter to track concurrent puts.
private HFileDataBlockEncoder	dataBlockEncoder
static final long	DEEP_OVERHEAD
static final String	DEFAULT_BLOCK_STORAGE_POLICY
static final int	DEFAULT_BLOCKING_STOREFILE_COUNT
static final int	DEFAULT_COMPACTCHECKER_INTERVAL_MULTIPLIER
private static final int	DEFAULT_FLUSH_RETRIES_NUMBER
private final List<HStoreFile>	filesCompacting
static final long	FIXED_OVERHEAD
private AtomicLong	flushedCellsCount
private AtomicLong	flushedCellsSize
private AtomicLong	flushedOutputFileSize
private int	flushRetriesNumber
(package private) boolean	forceMajor
private long	lastCompactSize
private static final org.slf4j.Logger	LOG
private AtomicLong	majorCompactedCellsCount
private AtomicLong	majorCompactedCellsSize
protected final MemStore	memstore
static final String	MEMSTORE_CLASS_NAME
private LongAdder	memstoreOnlyRowReadsCount
private LongAdder	mixedRowReadsCount
private static final AtomicBoolean	offPeakCompactionTracker
private OffPeakHours	offPeakHours
private final int	parallelPutCountPrintThreshold
private int	pauseTime
private final HRegion	region
private ScanInfo	scanInfo
private static final int	SPLIT_REGION_COMPACTION_PRIORITY
private final StoreContext	storeContext
(package private) final StoreEngine<?,?,?,?>	storeEngine
private final Supplier<HStore.StoreFileWriterCreationTracker>	storeFileWriterCreationTrackerFactory
private final Set<HStore.StoreFileWriterCreationTracker>	storeFileWriterCreationTrackers
private AtomicLong	storeSize
private AtomicLong	totalUncompressedBytes
private final boolean	verifyBulkLoads
private final boolean	warmup

Fields inherited from interface org.apache.hadoop.hbase.regionserver.Store

NO_PRIORITY, PRIORITY_USER

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	HStore(HRegion region, ColumnFamilyDescriptor family, org.apache.hadoop.conf.Configuration confParam, boolean warmup)	Constructor

Method Summary

Modifier and Type	Method	Description
void	add(Iterable<ExtendedCell> cells, MemStoreSizing memstoreSizing)	Adds the specified value to the memstore
void	add(ExtendedCell cell, MemStoreSizing memstoreSizing)	Adds a value to the memstore
void	addChangedReaderObserver(ChangedReadersObserver o)
private void	addToCompactingFiles(Collection<HStoreFile> filesToAdd)	Adds the files to compacting files.
boolean	areWritesEnabled()
void	assertBulkLoadHFileOk(org.apache.hadoop.fs.Path srcPath)	This throws a WrongRegionException if the HFile does not fit in this region, or an InvalidHFileException if the HFile is not valid.
org.apache.hadoop.fs.Path	bulkLoadHFile(byte[] family, String srcPathStr, org.apache.hadoop.fs.Path dstPath)
private void	bulkLoadHFile(HStoreFile sf)
void	bulkLoadHFile(StoreFileInfo fileInfo)
void	cancelRequestedCompaction(CompactionContext compaction)
boolean	canSplit()	Returns whether this store is splittable, i.e., no reference file in this store.
private static void	clearAndClose(List<KeyValueScanner> scanners)
private void	clearCompactedfiles(List<HStoreFile> filesToRemove)
org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile>	close()	Close all the readers We don't need to worry about subsequent requests because the Region holds a write lock that will prevent any more reads or writes.
void	closeAndArchiveCompactedFiles()	Closes and archives the compacted files under this store
private org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile>	closeWithoutLock()
List<HStoreFile>	compact(CompactionContext compaction, ThroughputController throughputController, User user)	Compact the StoreFiles.
private boolean	completeFlush(List<HStoreFile> sfs, long snapshotId)
protected CacheConfig	createCacheConf(ColumnFamilyDescriptor family)	Creates the cache config.
StoreFlushContext	createFlushContext(long cacheFlushId, FlushLifeCycleTracker tracker)
protected KeyValueScanner	createScanner(Scan scan, ScanInfo scanInfo, NavigableSet<byte[]> targetCols, long readPt)
protected StoreEngine<?,?,?,?>	createStoreEngine(HStore store, org.apache.hadoop.conf.Configuration conf, CellComparator kvComparator)	Creates the store engine configured for the given Store.
void	deleteChangedReaderObserver(ChangedReadersObserver o)
void	deregisterChildren(ConfigurationManager manager)	Needs to be called to deregister the children from the manager.
static long	determineTTLFromFamily(ColumnFamilyDescriptor family)	Returns TTL in seconds of the specified family
protected List<HStoreFile>	doCompaction(CompactionRequestImpl cr, Collection<HStoreFile> filesToCompact, User user, long compactionStartTime, List<org.apache.hadoop.fs.Path> newFiles)
private void	finishCompactionRequest(CompactionRequestImpl cr)
protected List<org.apache.hadoop.fs.Path>	flushCache(long logCacheFlushId, MemStoreSnapshot snapshot, MonitoredTask status, ThroughputController throughputController, FlushLifeCycleTracker tracker, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)	Write out current snapshot.
OptionalDouble	getAvgStoreFileAge()	Returns Average age of store files in this store
long	getBlockingFileCount()	The number of files required before flushes for this store will be blocked.
long	getBloomFilterEligibleRequestsCount()	Returns count of requests which could have used bloom filters, but they weren't configured or loaded.
long	getBloomFilterNegativeResultsCount()	Returns count of negative results for bloom filter requests for this store.
long	getBloomFilterRequestsCount()	Returns count of bloom filter results for this store.
CacheConfig	getCacheConfig()	Used for tests.
ColumnFamilyDescriptor	getColumnFamilyDescriptor()
String	getColumnFamilyName()
long	getCompactedCellsCount()	Returns The number of cells processed during minor compactions
long	getCompactedCellsSize()	Returns The total amount of data processed during minor compactions, in bytes
Collection<HStoreFile>	getCompactedFiles()
int	getCompactedFilesCount()	Returns Count of compacted store files
long	getCompactionCheckMultiplier()
double	getCompactionPressure()	This value can represent the degree of emergency of compaction for this store.
CompactionProgress	getCompactionProgress()	getter for CompactionProgress object
int	getCompactPriority()
CellComparator	getComparator()
RegionCoprocessorHost	getCoprocessorHost()
int	getCurrentParallelPutCount()
HFileDataBlockEncoder	getDataBlockEncoder()	Returns the data block encoder
private InetSocketAddress[]	getFavoredNodes()
org.apache.hadoop.fs.FileSystem	getFileSystem()
MemStoreSize	getFlushableSize()
long	getFlushedCellsCount()	Returns The number of cells flushed to disk
long	getFlushedCellsSize()	Returns The total size of data flushed to disk, in bytes
long	getFlushedOutputFileSize()	Returns The total size of out output files on disk, in bytes
long	getHFilesSize()	Returns The size of only the store files which are HFiles, in bytes.
HRegion	getHRegion()
long	getLastCompactSize()	Returns aggregate size of all HStores used in the last compaction
long	getMajorCompactedCellsCount()	Returns The number of cells processed during major compactions
long	getMajorCompactedCellsSize()	Returns The total amount of data processed during major compactions, in bytes
int	getMaxCompactedStoreFileRefCount()	Returns get maximum ref count of storeFile among all compacted HStore Files for the HStore
OptionalLong	getMaxMemStoreTS()	Returns The maximum memstoreTS in all store files.
OptionalLong	getMaxSequenceId()	Returns The maximum sequence id in all store files.
OptionalLong	getMaxStoreFileAge()	Returns Max age of store files in this store
private MemStore	getMemstore()	Returns MemStore Instance to use in this store.
long	getMemStoreFlushSize()	Returns Gets the Memstore flush size for the region that this store works with.
long	getMemstoreOnlyRowReadsCount()	Returns the number of read requests purely from the memstore.
MemStoreSize	getMemStoreSize()	Returns The size of this store's memstore.
OptionalLong	getMinStoreFileAge()	Returns Min age of store files in this store
long	getMixedRowReadsCount()	Returns the number of read requests from the files under this store.
long	getNumHFiles()	Returns Number of HFiles in this store
long	getNumReferenceFiles()	Returns Number of reference files in this store
protected OffPeakHours	getOffPeakHours()
org.apache.hadoop.conf.Configuration	getReadOnlyConfiguration()
HRegionFileSystem	getRegionFileSystem()
RegionInfo	getRegionInfo()	Returns the parent region info hosting this store
ScanInfo	getScanInfo()
KeyValueScanner	getScanner(Scan scan, NavigableSet<byte[]> targetCols, long readPt)	Return a scanner for both the memstore and the HStore files.
List<KeyValueScanner>	getScanners(boolean cacheBlocks, boolean isGet, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher, byte[] startRow, byte[] stopRow, long readPt, boolean onlyLatestVersion)	Get all scanners with no filtering based on TTL (that happens further down the line).
List<KeyValueScanner>	getScanners(boolean cacheBlocks, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher, byte[] startRow, boolean includeStartRow, byte[] stopRow, boolean includeStopRow, long readPt, boolean onlyLatestVersion)	Get all scanners with no filtering based on TTL (that happens further down the line).
List<KeyValueScanner>	getScanners(List<HStoreFile> files, boolean cacheBlocks, boolean isGet, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher, byte[] startRow, byte[] stopRow, long readPt, boolean includeMemstoreScanner, boolean onlyLatestVersion)	Create scanners on the given files and if needed on the memstore with no filtering based on TTL (that happens further down the line).
List<KeyValueScanner>	getScanners(List<HStoreFile> files, boolean cacheBlocks, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher, byte[] startRow, boolean includeStartRow, byte[] stopRow, boolean includeStopRow, long readPt, boolean includeMemstoreScanner, boolean onlyLatestVersion)	Create scanners on the given files and if needed on the memstore with no filtering based on TTL (that happens further down the line).
long	getSize()	Returns aggregate size of HStore
long	getSmallestReadPoint()
MemStoreSize	getSnapshotSize()	Returns size of the memstore snapshot
Optional<byte[]>	getSplitPoint()	Determines if Store should be split.
(package private) StoreContext	getStoreContext()
StoreEngine<?,?,?,?>	getStoreEngine()	Returns the StoreEngine that is backing this concrete implementation of Store.
private LongStream	getStoreFileAgeStream()
Collection<HStoreFile>	getStorefiles()	Returns All store files.
Set<org.apache.hadoop.fs.Path>	getStoreFilesBeingWritten()	Return the storefiles which are currently being written to.
int	getStorefilesCount()	Returns Count of store files
private long	getStorefilesFieldSize(ToLongFunction<StoreFileReader> f)
(package private) long	getStoreFileSize(HStoreFile file)	Computes the length of a store file without succumbing to any errors along the way.
long	getStorefilesRootLevelIndexSize()	Returns The size of the store file root-level indexes, in bytes.
long	getStorefilesSize()	Returns The size of the store files, in bytes.
long	getStoreFileTtl()	Returns Gets the cf-specific time-to-live for store files.
int	getStoreRefCount()
long	getStoreSizeUncompressed()	Returns The size of the store files, in bytes, uncompressed.
TableName	getTableName()
private long	getTotalSize(Collection<HStoreFile> sfs)
long	getTotalStaticBloomSize()	Returns the total byte size of all Bloom filter bit arrays.
long	getTotalStaticIndexSize()	Returns the total size of all index blocks in the data block indexes, including the root level, intermediate levels, and the leaf level for multi-level indexes, or just the root level for single-level indexes.
boolean	hasReferences()	Returns true if the store has any underlying reference files to older HFiles
boolean	hasTooManyStoreFiles()	Returns Whether this store has too many store files.
long	heapSize()	Return the approximate 'exclusive deep size' of implementing object.
private StoreContext	initializeStoreContext(ColumnFamilyDescriptor family)
boolean	isPrimaryReplicaStore()
boolean	isSloppyMemStore()	Returns true if the memstore may need some extra memory space
private void	logCompactionEndMessage(CompactionRequestImpl cr, List<HStoreFile> sfs, long now, long compactionStartTime)	Log a very elaborate compaction completion message.
boolean	needsCompaction()	See if there's too much store files in this store
private void	notifyChangedReadersObservers(List<HStoreFile> sfs)	Notify all observers that set of Readers has changed.
void	onConfigurationChange(org.apache.hadoop.conf.Configuration conf)	This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.
private void	postRefreshStoreFiles()
void	postSnapshotOperation()	Perform tasks needed after the completion of snapshot operation.
Pair<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path>	preBulkLoadHFile(String srcPathStr, long seqNum)	This method should only be called from Region.
Long	preFlushSeqIDEstimation()
void	preSnapshotOperation()	Sets the store up for a region level snapshot operation.
List<KeyValueScanner>	recreateScanners(List<KeyValueScanner> currentFileScanners, boolean cacheBlocks, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher, byte[] startRow, boolean includeStartRow, byte[] stopRow, boolean includeStopRow, long readPt, boolean includeMemstoreScanner)	Recreates the scanners on the current list of active store file scanners
void	refreshStoreFiles()	Checks the underlying store files, and opens the files that have not been opened, and removes the store file readers for store files no longer available.
void	refreshStoreFiles(Collection<String> newFiles)	Replaces the store files that the store has with the given files.
protected void	refreshStoreSizeAndTotalBytes()	Update counts.
void	registerChildren(ConfigurationManager manager)	Needs to be called to register the children to the manager.
private void	removeCompactedfiles(Collection<HStoreFile> compactedfiles, boolean evictOnClose)	Archives and removes the compacted files
private void	removeUnneededFiles()
(package private) void	replaceStoreFiles(Collection<HStoreFile> compactedFiles, Collection<HStoreFile> result, boolean writeCompactionMarker)
void	replayCompactionMarker(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.CompactionDescriptor compaction, boolean pickCompactionFiles, boolean removeFiles)	Call to complete a compaction.
(package private) void	reportArchivedFilesForQuota(List<? extends StoreFile> archivedFiles, List<Long> fileSizes)
Optional<CompactionContext>	requestCompaction()
Optional<CompactionContext>	requestCompaction(int priority, CompactionLifeCycleTracker tracker, User user)
(package private) void	setDataBlockEncoderInTest(HFileDataBlockEncoder blockEncoder)	Should be used only in tests.
(package private) void	setScanInfo(ScanInfo scanInfo)	Set scan info, used by test
private void	setStoragePolicyFromFileName(List<org.apache.hadoop.fs.Path> newFiles)
boolean	shouldPerformMajorCompaction()	Tests whether we should run a major compaction.
void	startReplayingFromWAL()	This message intends to inform the MemStore that next coming updates are going to be part of the replaying edits from WAL
void	stopReplayingFromWAL()	This message intends to inform the MemStore that the replaying edits from WAL are done
boolean	throttleCompaction(long compactionSize)
long	timeOfOldestEdit()	When was the last edit done in the memstore
String	toString()
void	triggerMajorCompaction()
HStoreFile	tryCommitRecoveredHFile(org.apache.hadoop.fs.Path path)
void	updateCompactedMetrics(boolean isMajor, CompactionProgress progress)
(package private) void	updateMetricsStore(boolean memstoreRead)
(package private) void	updateSpaceQuotaAfterFileReplacement(RegionSizeStore sizeStore, RegionInfo regionInfo, Collection<HStoreFile> oldFiles, Collection<HStoreFile> newFiles)	Updates the space quota usage for this region, removing the size for files compacted away and adding in the size for new files.
void	upsert(Iterable<ExtendedCell> cells, long readpoint, MemStoreSizing memstoreSizing)	Adds or replaces the specified KeyValues.
(package private) int	versionsToReturn(int wantedVersions)
private void	writeCompactionWalRecord(Collection<HStoreFile> filesCompacted, Collection<HStoreFile> newFiles)	Writes the compaction WAL record.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

MEMSTORE_CLASS_NAME

public static final String MEMSTORE_CLASS_NAME

See Also:

• Constant Field Values

COMPACTCHECKER_INTERVAL_MULTIPLIER_KEY

public static final String COMPACTCHECKER_INTERVAL_MULTIPLIER_KEY

See Also:

• Constant Field Values

BLOCKING_STOREFILES_KEY

public static final String BLOCKING_STOREFILES_KEY

See Also:

• Constant Field Values

BLOCK_STORAGE_POLICY_KEY

public static final String BLOCK_STORAGE_POLICY_KEY

See Also:

• Constant Field Values

DEFAULT_BLOCK_STORAGE_POLICY

public static final String DEFAULT_BLOCK_STORAGE_POLICY

See Also:

• Constant Field Values

DEFAULT_COMPACTCHECKER_INTERVAL_MULTIPLIER

public static final int DEFAULT_COMPACTCHECKER_INTERVAL_MULTIPLIER

See Also:

• Constant Field Values

DEFAULT_BLOCKING_STOREFILE_COUNT

public static final int DEFAULT_BLOCKING_STOREFILE_COUNT

See Also:

• Constant Field Values

SPLIT_REGION_COMPACTION_PRIORITY

private static final int SPLIT_REGION_COMPACTION_PRIORITY

See Also:

• Constant Field Values

LOG

private static final org.slf4j.Logger LOG

memstore

protected final MemStore memstore

region

private final HRegion region

conf

protected org.apache.hadoop.conf.Configuration conf

lastCompactSize

private long lastCompactSize

forceMajor

volatile boolean forceMajor

storeSize

private AtomicLong storeSize

totalUncompressedBytes

private AtomicLong totalUncompressedBytes

memstoreOnlyRowReadsCount

private LongAdder memstoreOnlyRowReadsCount

mixedRowReadsCount

private LongAdder mixedRowReadsCount

archiveLock

final ReentrantLock archiveLock

Lock specific to archiving compacted store files. This avoids races around the combination of retrieving the list of compacted files and moving them to the archive directory. Since this is usually a background process (other than on close), we don't want to handle this with the store write lock, which would block readers and degrade performance. Locked by: - CompactedHFilesDispatchHandler via closeAndArchiveCompactedFiles() - close()

verifyBulkLoads

private final boolean verifyBulkLoads

currentParallelPutCount

private final AtomicInteger currentParallelPutCount

Use this counter to track concurrent puts. If TRACE-log is enabled, if we are over the threshold set by hbase.region.store.parallel.put.print.threshold (Default is 50) we will log a message that identifies the Store experience this high-level of concurrency.

parallelPutCountPrintThreshold

private final int parallelPutCountPrintThreshold

scanInfo

private ScanInfo scanInfo

filesCompacting

private final List<HStoreFile> filesCompacting

changedReaderObservers

private final Set<ChangedReadersObserver> changedReaderObservers

dataBlockEncoder

private HFileDataBlockEncoder dataBlockEncoder

storeEngine

final StoreEngine<?,?,?,?> storeEngine

offPeakCompactionTracker

private static final AtomicBoolean offPeakCompactionTracker

offPeakHours

private volatile OffPeakHours offPeakHours

DEFAULT_FLUSH_RETRIES_NUMBER

private static final int DEFAULT_FLUSH_RETRIES_NUMBER

See Also:

• Constant Field Values

flushRetriesNumber

private int flushRetriesNumber

pauseTime

private int pauseTime

blockingFileCount

private long blockingFileCount

compactionCheckMultiplier

private int compactionCheckMultiplier

flushedCellsCount

private AtomicLong flushedCellsCount

compactedCellsCount

private AtomicLong compactedCellsCount

majorCompactedCellsCount

private AtomicLong majorCompactedCellsCount

flushedCellsSize

private AtomicLong flushedCellsSize

flushedOutputFileSize

private AtomicLong flushedOutputFileSize

compactedCellsSize

private AtomicLong compactedCellsSize

majorCompactedCellsSize

private AtomicLong majorCompactedCellsSize

storeContext

private final StoreContext storeContext

storeFileWriterCreationTrackers

private final Set<HStore.StoreFileWriterCreationTracker> storeFileWriterCreationTrackers

storeFileWriterCreationTrackerFactory

private final Supplier<HStore.StoreFileWriterCreationTracker> storeFileWriterCreationTrackerFactory

warmup

private final boolean warmup

FIXED_OVERHEAD

public static final long FIXED_OVERHEAD

DEEP_OVERHEAD

public static final long DEEP_OVERHEAD

Constructor Details

HStore

protected HStore(HRegion region,
 ColumnFamilyDescriptor family,
 org.apache.hadoop.conf.Configuration confParam,
 boolean warmup)
          throws IOException

Constructor

Parameters:

family - HColumnDescriptor for this column

confParam - configuration object failed. Can be null.

Throws:

IOException

Method Details

initializeStoreContext

private StoreContext initializeStoreContext(ColumnFamilyDescriptor family)
                                     throws IOException

Throws:

IOException

getFavoredNodes

private InetSocketAddress[] getFavoredNodes()

getMemstore

private MemStore getMemstore()

Returns MemStore Instance to use in this store.

createCacheConf

protected CacheConfig createCacheConf(ColumnFamilyDescriptor family)

Creates the cache config.

Parameters:

family - The current column family.

createStoreEngine

protected StoreEngine<?,?,?,?> createStoreEngine(HStore store,
 org.apache.hadoop.conf.Configuration conf,
 CellComparator kvComparator)
                                          throws IOException

Creates the store engine configured for the given Store.

Parameters:

store - The store. An unfortunate dependency needed due to it being passed to coprocessors via the compactor.

conf - Store configuration.

kvComparator - KVComparator for storeFileManager.

Returns:

StoreEngine to use.

Throws:

IOException

determineTTLFromFamily

public static long determineTTLFromFamily(ColumnFamilyDescriptor family)

Returns TTL in seconds of the specified family

getStoreContext

StoreContext getStoreContext()

getColumnFamilyName

public String getColumnFamilyName()

Specified by:

getColumnFamilyName in interface Store

Specified by:

getColumnFamilyName in interface StoreConfigInformation

getTableName

public TableName getTableName()

Specified by:

getTableName in interface Store

getFileSystem

public org.apache.hadoop.fs.FileSystem getFileSystem()

Specified by:

getFileSystem in interface Store

getRegionFileSystem

public HRegionFileSystem getRegionFileSystem()

getStoreFileTtl

public long getStoreFileTtl()

Description copied from interface: StoreConfigInformation

Returns Gets the cf-specific time-to-live for store files.

Specified by:

getStoreFileTtl in interface StoreConfigInformation

getMemStoreFlushSize

public long getMemStoreFlushSize()

Description copied from interface: StoreConfigInformation

Returns Gets the Memstore flush size for the region that this store works with.

Specified by:

getMemStoreFlushSize in interface StoreConfigInformation

getFlushableSize

public MemStoreSize getFlushableSize()

Specified by:

getFlushableSize in interface Store

Returns:

The amount of memory we could flush from this memstore; usually this is equal to Store.getMemStoreSize() unless we are carrying snapshots and then it will be the size of outstanding snapshots.

getSnapshotSize

public MemStoreSize getSnapshotSize()

Description copied from interface: Store

Returns size of the memstore snapshot

Specified by:

getSnapshotSize in interface Store

getCompactionCheckMultiplier

public long getCompactionCheckMultiplier()

Specified by:

getCompactionCheckMultiplier in interface StoreConfigInformation

Returns:

Gets the cf-specific compaction check frequency multiplier. The need for compaction (outside of normal checks during flush, open, etc.) will be ascertained every multiplier * HConstants.THREAD_WAKE_FREQUENCY milliseconds.

getBlockingFileCount

public long getBlockingFileCount()

Description copied from interface: StoreConfigInformation

The number of files required before flushes for this store will be blocked.

Specified by:

getBlockingFileCount in interface StoreConfigInformation

getColumnFamilyDescriptor

public ColumnFamilyDescriptor getColumnFamilyDescriptor()

Specified by:

getColumnFamilyDescriptor in interface Store

getMaxSequenceId

public OptionalLong getMaxSequenceId()

Description copied from interface: Store

Returns The maximum sequence id in all store files.

Specified by:

getMaxSequenceId in interface Store

getMaxMemStoreTS

public OptionalLong getMaxMemStoreTS()

Description copied from interface: Store

Returns The maximum memstoreTS in all store files.

Specified by:

getMaxMemStoreTS in interface Store

getDataBlockEncoder

public HFileDataBlockEncoder getDataBlockEncoder()

Returns the data block encoder

setDataBlockEncoderInTest

void setDataBlockEncoderInTest(HFileDataBlockEncoder blockEncoder)

Should be used only in tests.

Parameters:

blockEncoder - the block delta encoder to use

postRefreshStoreFiles

private void postRefreshStoreFiles()
                            throws IOException

Throws:

IOException

refreshStoreFiles

public void refreshStoreFiles()
                       throws IOException

Description copied from interface: Store

Checks the underlying store files, and opens the files that have not been opened, and removes the store file readers for store files no longer available. Mainly used by secondary region replicas to keep up to date with the primary region files.

Specified by:

refreshStoreFiles in interface Store

Throws:

IOException

refreshStoreFiles

public void refreshStoreFiles(Collection<String> newFiles)
                       throws IOException

Replaces the store files that the store has with the given files. Mainly used by secondary region replicas to keep up to date with the primary region files.

Throws:

IOException

startReplayingFromWAL

public void startReplayingFromWAL()

This message intends to inform the MemStore that next coming updates are going to be part of the replaying edits from WAL

stopReplayingFromWAL

public void stopReplayingFromWAL()

This message intends to inform the MemStore that the replaying edits from WAL are done

add

public void add(ExtendedCell cell,
 MemStoreSizing memstoreSizing)

Adds a value to the memstore

add

public void add(Iterable<ExtendedCell> cells,
 MemStoreSizing memstoreSizing)

Adds the specified value to the memstore

timeOfOldestEdit

public long timeOfOldestEdit()

Description copied from interface: Store

When was the last edit done in the memstore

Specified by:

timeOfOldestEdit in interface Store

getStorefiles

public Collection<HStoreFile> getStorefiles()

Returns All store files.

Specified by:

getStorefiles in interface Store

getCompactedFiles

public Collection<HStoreFile> getCompactedFiles()

Specified by:

getCompactedFiles in interface Store

assertBulkLoadHFileOk

public void assertBulkLoadHFileOk(org.apache.hadoop.fs.Path srcPath)
                           throws IOException

This throws a WrongRegionException if the HFile does not fit in this region, or an InvalidHFileException if the HFile is not valid.

Throws:

IOException

preBulkLoadHFile

public Pair<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path> preBulkLoadHFile(String srcPathStr,
 long seqNum)
                                                                           throws IOException

This method should only be called from Region. It is assumed that the ranges of values in the HFile fit within the stores assigned region. (assertBulkLoadHFileOk checks this)

Parameters:

seqNum - sequence Id associated with the HFile

Throws:

IOException

bulkLoadHFile

public org.apache.hadoop.fs.Path bulkLoadHFile(byte[] family,
 String srcPathStr,
 org.apache.hadoop.fs.Path dstPath)
                                        throws IOException

Throws:

IOException

bulkLoadHFile

public void bulkLoadHFile(StoreFileInfo fileInfo)
                   throws IOException

Throws:

IOException

bulkLoadHFile

private void bulkLoadHFile(HStoreFile sf)
                    throws IOException

Throws:

IOException

closeWithoutLock

private org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile> closeWithoutLock()
                                                                                                        throws IOException

Throws:

IOException

close

public org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile> close()
                                                                                            throws IOException

Close all the readers We don't need to worry about subsequent requests because the Region holds a write lock that will prevent any more reads or writes.

Returns:

the StoreFiles that were previously being used.

Throws:

IOException - on failure

flushCache

protected List<org.apache.hadoop.fs.Path> flushCache(long logCacheFlushId,
 MemStoreSnapshot snapshot,
 MonitoredTask status,
 ThroughputController throughputController,
 FlushLifeCycleTracker tracker,
 Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)
                                              throws IOException

Write out current snapshot. Presumes StoreFlusherImpl.prepare() has been called previously.

Parameters:

logCacheFlushId - flush sequence number

Returns:

The path name of the tmp file to which the store was flushed

Throws:

IOException - if exception occurs during process

tryCommitRecoveredHFile

public HStoreFile tryCommitRecoveredHFile(org.apache.hadoop.fs.Path path)
                                   throws IOException

Throws:

IOException

getTotalSize

private long getTotalSize(Collection<HStoreFile> sfs)

completeFlush

private boolean completeFlush(List<HStoreFile> sfs,
 long snapshotId)
                       throws IOException

Throws:

IOException

notifyChangedReadersObservers

private void notifyChangedReadersObservers(List<HStoreFile> sfs)
                                    throws IOException

Notify all observers that set of Readers has changed.

Throws:

IOException

getScanners

public List<KeyValueScanner> getScanners(boolean cacheBlocks,
 boolean isGet,
 boolean usePread,
 boolean isCompaction,
 ScanQueryMatcher matcher,
 byte[] startRow,
 byte[] stopRow,
 long readPt,
 boolean onlyLatestVersion)
                                  throws IOException

Get all scanners with no filtering based on TTL (that happens further down the line).

Parameters:

cacheBlocks - cache the blocks or not

usePread - true to use pread, false if not

isCompaction - true if the scanner is created for compaction

matcher - the scan query matcher

startRow - the start row

stopRow - the stop row

readPt - the read point of the current scan

Returns:

all scanners for this store

Throws:

IOException

getScanners

public List<KeyValueScanner> getScanners(boolean cacheBlocks,
 boolean usePread,
 boolean isCompaction,
 ScanQueryMatcher matcher,
 byte[] startRow,
 boolean includeStartRow,
 byte[] stopRow,
 boolean includeStopRow,
 long readPt,
 boolean onlyLatestVersion)
                                  throws IOException

Get all scanners with no filtering based on TTL (that happens further down the line).

Parameters:

cacheBlocks - cache the blocks or not

usePread - true to use pread, false if not

isCompaction - true if the scanner is created for compaction

matcher - the scan query matcher

startRow - the start row

includeStartRow - true to include start row, false if not

stopRow - the stop row

includeStopRow - true to include stop row, false if not

readPt - the read point of the current scan

Returns:

all scanners for this store

Throws:

IOException

clearAndClose

private static void clearAndClose(List<KeyValueScanner> scanners)

getScanners

public List<KeyValueScanner> getScanners(List<HStoreFile> files,
 boolean cacheBlocks,
 boolean isGet,
 boolean usePread,
 boolean isCompaction,
 ScanQueryMatcher matcher,
 byte[] startRow,
 byte[] stopRow,
 long readPt,
 boolean includeMemstoreScanner,
 boolean onlyLatestVersion)
                                  throws IOException

Create scanners on the given files and if needed on the memstore with no filtering based on TTL (that happens further down the line).

Parameters:

files - the list of files on which the scanners has to be created

cacheBlocks - cache the blocks or not

usePread - true to use pread, false if not

isCompaction - true if the scanner is created for compaction

matcher - the scan query matcher

startRow - the start row

stopRow - the stop row

readPt - the read point of the current scan

includeMemstoreScanner - true if memstore has to be included

Returns:

scanners on the given files and on the memstore if specified

Throws:

IOException

getScanners

public List<KeyValueScanner> getScanners(List<HStoreFile> files,
 boolean cacheBlocks,
 boolean usePread,
 boolean isCompaction,
 ScanQueryMatcher matcher,
 byte[] startRow,
 boolean includeStartRow,
 byte[] stopRow,
 boolean includeStopRow,
 long readPt,
 boolean includeMemstoreScanner,
 boolean onlyLatestVersion)
                                  throws IOException

Create scanners on the given files and if needed on the memstore with no filtering based on TTL (that happens further down the line).

Parameters:

files - the list of files on which the scanners has to be created

cacheBlocks - ache the blocks or not

usePread - true to use pread, false if not

isCompaction - true if the scanner is created for compaction

matcher - the scan query matcher

startRow - the start row

includeStartRow - true to include start row, false if not

stopRow - the stop row

includeStopRow - true to include stop row, false if not

readPt - the read point of the current scan

includeMemstoreScanner - true if memstore has to be included

Returns:

scanners on the given files and on the memstore if specified

Throws:

IOException

addChangedReaderObserver

public void addChangedReaderObserver(ChangedReadersObserver o)

Parameters:

o - Observer who wants to know about changes in set of Readers

deleteChangedReaderObserver

public void deleteChangedReaderObserver(ChangedReadersObserver o)

Parameters:

o - Observer no longer interested in changes in set of Readers.

compact

public List<HStoreFile> compact(CompactionContext compaction,
 ThroughputController throughputController,
 User user)
                         throws IOException

Compact the StoreFiles. This method may take some time, so the calling thread must be able to block for long periods.

During this time, the Store can work as usual, getting values from StoreFiles and writing new StoreFiles from the memstore. Existing StoreFiles are not destroyed until the new compacted StoreFile is completely written-out to disk.

The compactLock prevents multiple simultaneous compactions. The structureLock prevents us from interfering with other write operations.

We don't want to hold the structureLock for the whole time, as a compact() can be lengthy and we want to allow cache-flushes during this period.

Compaction event should be idempotent, since there is no IO Fencing for the region directory in hdfs. A region server might still try to complete the compaction after it lost the region. That is why the following events are carefully ordered for a compaction: 1. Compaction writes new files under region/.tmp directory (compaction output) 2. Compaction atomically moves the temporary file under region directory 3. Compaction appends a WAL edit containing the compaction input and output files. Forces sync on WAL. 4. Compaction deletes the input files from the region directory. Failure conditions are handled like this: - If RS fails before 2, compaction wont complete. Even if RS lives on and finishes the compaction later, it will only write the new data file to the region directory. Since we already have this data, this will be idempotent but we will have a redundant copy of the data. - If RS fails between 2 and 3, the region will have a redundant copy of the data. The RS that failed won't be able to finish sync() for WAL because of lease recovery in WAL. - If RS fails after 3, the region region server who opens the region will pick up the the compaction marker from the WAL and replay it by removing the compaction input files. Failed RS can also attempt to delete those files, but the operation will be idempotent See HBASE-2231 for details.

Parameters:

compaction - compaction details obtained from requestCompaction()

Returns:

Storefile we compacted into or null if we failed or opted out early.

Throws:

IOException

doCompaction

protected List<HStoreFile> doCompaction(CompactionRequestImpl cr,
 Collection<HStoreFile> filesToCompact,
 User user,
 long compactionStartTime,
 List<org.apache.hadoop.fs.Path> newFiles)
                                 throws IOException

Throws:

IOException

setStoragePolicyFromFileName

private void setStoragePolicyFromFileName(List<org.apache.hadoop.fs.Path> newFiles)
                                   throws IOException

Throws:

IOException

writeCompactionWalRecord

private void writeCompactionWalRecord(Collection<HStoreFile> filesCompacted,
 Collection<HStoreFile> newFiles)
                               throws IOException

Writes the compaction WAL record.

Parameters:

filesCompacted - Files compacted (input).

newFiles - Files from compaction.

Throws:

IOException

replaceStoreFiles

void replaceStoreFiles(Collection<HStoreFile> compactedFiles,
 Collection<HStoreFile> result,
 boolean writeCompactionMarker)
                throws IOException

Throws:

IOException

updateSpaceQuotaAfterFileReplacement

void updateSpaceQuotaAfterFileReplacement(RegionSizeStore sizeStore,
 RegionInfo regionInfo,
 Collection<HStoreFile> oldFiles,
 Collection<HStoreFile> newFiles)

Updates the space quota usage for this region, removing the size for files compacted away and adding in the size for new files.

Parameters:

sizeStore - The object tracking changes in region size for space quotas.

regionInfo - The identifier for the region whose size is being updated.

oldFiles - Files removed from this store's region.

newFiles - Files added to this store's region.

logCompactionEndMessage

private void logCompactionEndMessage(CompactionRequestImpl cr,
 List<HStoreFile> sfs,
 long now,
 long compactionStartTime)

Log a very elaborate compaction completion message.

Parameters:

cr - Request.

sfs - Resulting files.

compactionStartTime - Start time.

replayCompactionMarker

public void replayCompactionMarker(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.CompactionDescriptor compaction,
 boolean pickCompactionFiles,
 boolean removeFiles)
                            throws IOException

Call to complete a compaction. Its for the case where we find in the WAL a compaction that was not finished. We could find one recovering a WAL after a regionserver crash. See HBASE-2231.

Throws:

IOException

hasReferences

public boolean hasReferences()

Description copied from interface: Store

Returns true if the store has any underlying reference files to older HFiles

Specified by:

hasReferences in interface Store

getCompactionProgress

public CompactionProgress getCompactionProgress()

getter for CompactionProgress object

Returns:

CompactionProgress object; can be null

shouldPerformMajorCompaction

public boolean shouldPerformMajorCompaction()
                                     throws IOException

Description copied from interface: Store

Tests whether we should run a major compaction. For example, if the configured major compaction interval is reached.

Specified by:

shouldPerformMajorCompaction in interface Store

Returns:

true if we should run a major compaction.

Throws:

IOException

requestCompaction

public Optional<CompactionContext> requestCompaction()
                                              throws IOException

Throws:

IOException

requestCompaction

public Optional<CompactionContext> requestCompaction(int priority,
 CompactionLifeCycleTracker tracker,
 User user)
                                              throws IOException

Throws:

IOException

addToCompactingFiles

private void addToCompactingFiles(Collection<HStoreFile> filesToAdd)

Adds the files to compacting files. filesCompacting must be locked.

removeUnneededFiles

private void removeUnneededFiles()
                          throws IOException

Throws:

IOException

cancelRequestedCompaction

public void cancelRequestedCompaction(CompactionContext compaction)

finishCompactionRequest

private void finishCompactionRequest(CompactionRequestImpl cr)

refreshStoreSizeAndTotalBytes

protected void refreshStoreSizeAndTotalBytes()
                                      throws IOException

Update counts.

Throws:

IOException

versionsToReturn

int versionsToReturn(int wantedVersions)

canSplit

public boolean canSplit()

Description copied from interface: Store

Returns whether this store is splittable, i.e., no reference file in this store.

Specified by:

canSplit in interface Store

getSplitPoint

public Optional<byte[]> getSplitPoint()

Determines if Store should be split.

getLastCompactSize

public long getLastCompactSize()

Description copied from interface: Store

Returns aggregate size of all HStores used in the last compaction

Specified by:

getLastCompactSize in interface Store

getSize

public long getSize()

Description copied from interface: Store

Returns aggregate size of HStore

Specified by:

getSize in interface Store

triggerMajorCompaction

public void triggerMajorCompaction()

getScanner

public KeyValueScanner getScanner(Scan scan,
 NavigableSet<byte[]> targetCols,
 long readPt)
                           throws IOException

Return a scanner for both the memstore and the HStore files. Assumes we are not in a compaction.

Parameters:

scan - Scan to apply when scanning the stores

targetCols - columns to scan

Returns:

a scanner over the current key values

Throws:

IOException - on failure

createScanner

protected KeyValueScanner createScanner(Scan scan,
 ScanInfo scanInfo,
 NavigableSet<byte[]> targetCols,
 long readPt)
                                 throws IOException

Throws:

IOException

recreateScanners

public List<KeyValueScanner> recreateScanners(List<KeyValueScanner> currentFileScanners,
 boolean cacheBlocks,
 boolean usePread,
 boolean isCompaction,
 ScanQueryMatcher matcher,
 byte[] startRow,
 boolean includeStartRow,
 byte[] stopRow,
 boolean includeStopRow,
 long readPt,
 boolean includeMemstoreScanner)
                                       throws IOException

Recreates the scanners on the current list of active store file scanners

Parameters:

currentFileScanners - the current set of active store file scanners

cacheBlocks - cache the blocks or not

usePread - use pread or not

isCompaction - is the scanner for compaction

matcher - the scan query matcher

startRow - the scan's start row

includeStartRow - should the scan include the start row

stopRow - the scan's stop row

includeStopRow - should the scan include the stop row

readPt - the read point of the current scane

includeMemstoreScanner - whether the current scanner should include memstorescanner

Returns:

list of scanners recreated on the current Scanners

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

getStorefilesCount

public int getStorefilesCount()

Description copied from interface: Store

Returns Count of store files

Specified by:

getStorefilesCount in interface Store

getCompactedFilesCount

public int getCompactedFilesCount()

Description copied from interface: Store

Returns Count of compacted store files

Specified by:

getCompactedFilesCount in interface Store

getStoreFileAgeStream

private LongStream getStoreFileAgeStream()

getMaxStoreFileAge

public OptionalLong getMaxStoreFileAge()

Description copied from interface: Store

Returns Max age of store files in this store

Specified by:

getMaxStoreFileAge in interface Store

getMinStoreFileAge

public OptionalLong getMinStoreFileAge()

Description copied from interface: Store

Returns Min age of store files in this store

Specified by:

getMinStoreFileAge in interface Store

getAvgStoreFileAge

public OptionalDouble getAvgStoreFileAge()

Description copied from interface: Store

Returns Average age of store files in this store

Specified by:

getAvgStoreFileAge in interface Store

getNumReferenceFiles

public long getNumReferenceFiles()

Description copied from interface: Store

Returns Number of reference files in this store

Specified by:

getNumReferenceFiles in interface Store

getNumHFiles

public long getNumHFiles()

Description copied from interface: Store

Returns Number of HFiles in this store

Specified by:

getNumHFiles in interface Store

getStoreSizeUncompressed

public long getStoreSizeUncompressed()

Description copied from interface: Store

Returns The size of the store files, in bytes, uncompressed.

Specified by:

getStoreSizeUncompressed in interface Store

getStorefilesSize

public long getStorefilesSize()

Description copied from interface: Store

Returns The size of the store files, in bytes.

Specified by:

getStorefilesSize in interface Store

getHFilesSize

public long getHFilesSize()

Description copied from interface: Store

Returns The size of only the store files which are HFiles, in bytes.

Specified by:

getHFilesSize in interface Store

getStorefilesFieldSize

private long getStorefilesFieldSize(ToLongFunction<StoreFileReader> f)

getStorefilesRootLevelIndexSize

public long getStorefilesRootLevelIndexSize()

Description copied from interface: Store

Returns The size of the store file root-level indexes, in bytes.

Specified by:

getStorefilesRootLevelIndexSize in interface Store

getTotalStaticIndexSize

public long getTotalStaticIndexSize()

Description copied from interface: Store

Returns the total size of all index blocks in the data block indexes, including the root level, intermediate levels, and the leaf level for multi-level indexes, or just the root level for single-level indexes.

Specified by:

getTotalStaticIndexSize in interface Store

Returns:

the total size of block indexes in the store

getTotalStaticBloomSize

public long getTotalStaticBloomSize()

Description copied from interface: Store

Returns the total byte size of all Bloom filter bit arrays. For compound Bloom filters even the Bloom blocks currently not loaded into the block cache are counted.

Specified by:

getTotalStaticBloomSize in interface Store

Returns:

the total size of all Bloom filters in the store

getMemStoreSize

public MemStoreSize getMemStoreSize()

Description copied from interface: Store

Returns The size of this store's memstore.

Specified by:

getMemStoreSize in interface Store

getCompactPriority

public int getCompactPriority()

Specified by:

getCompactPriority in interface Store

throttleCompaction

public boolean throttleCompaction(long compactionSize)

getHRegion

public HRegion getHRegion()

getCoprocessorHost

public RegionCoprocessorHost getCoprocessorHost()

getRegionInfo

public RegionInfo getRegionInfo()

Description copied from interface: Store

Returns the parent region info hosting this store

Specified by:

getRegionInfo in interface Store

Specified by:

getRegionInfo in interface StoreConfigInformation

areWritesEnabled

public boolean areWritesEnabled()

Specified by:

areWritesEnabled in interface Store

getSmallestReadPoint

public long getSmallestReadPoint()

Specified by:

getSmallestReadPoint in interface Store

Returns:

The smallest mvcc readPoint across all the scanners in this region. Writes older than this readPoint, are included in every read operation.

upsert

public void upsert(Iterable<ExtendedCell> cells,
 long readpoint,
 MemStoreSizing memstoreSizing)

Adds or replaces the specified KeyValues.

For each KeyValue specified, if a cell with the same row, family, and qualifier exists in MemStore, it will be replaced. Otherwise, it will just be inserted to MemStore.

This operation is atomic on each KeyValue (row/family/qualifier) but not necessarily atomic across all of them.

Parameters:

readpoint - readpoint below which we can safely remove duplicate KVs

createFlushContext

public StoreFlushContext createFlushContext(long cacheFlushId,
 FlushLifeCycleTracker tracker)

needsCompaction

public boolean needsCompaction()

Description copied from interface: Store

See if there's too much store files in this store

Specified by:

needsCompaction in interface Store

Returns:

true if number of store files is greater than the number defined in minFilesToCompact

getCacheConfig

public CacheConfig getCacheConfig()

Used for tests.

Returns:

cache configuration for this Store.

heapSize

public long heapSize()

Description copied from interface: HeapSize

Return the approximate 'exclusive deep size' of implementing object. Includes count of payload and hosting object sizings.

Specified by:

heapSize in interface HeapSize

getComparator

public CellComparator getComparator()

Specified by:

getComparator in interface Store

getScanInfo

public ScanInfo getScanInfo()

setScanInfo

void setScanInfo(ScanInfo scanInfo)

Set scan info, used by test

Parameters:

scanInfo - new scan info to use for test

hasTooManyStoreFiles

public boolean hasTooManyStoreFiles()

Description copied from interface: Store

Returns Whether this store has too many store files.

Specified by:

hasTooManyStoreFiles in interface Store

getFlushedCellsCount

public long getFlushedCellsCount()

Description copied from interface: Store

Returns The number of cells flushed to disk

Specified by:

getFlushedCellsCount in interface Store

getFlushedCellsSize

public long getFlushedCellsSize()

Description copied from interface: Store

Returns The total size of data flushed to disk, in bytes

Specified by:

getFlushedCellsSize in interface Store

getFlushedOutputFileSize

public long getFlushedOutputFileSize()

Description copied from interface: Store

Returns The total size of out output files on disk, in bytes

Specified by:

getFlushedOutputFileSize in interface Store

getCompactedCellsCount

public long getCompactedCellsCount()

Description copied from interface: Store

Returns The number of cells processed during minor compactions

Specified by:

getCompactedCellsCount in interface Store

getCompactedCellsSize

public long getCompactedCellsSize()

Description copied from interface: Store

Returns The total amount of data processed during minor compactions, in bytes

Specified by:

getCompactedCellsSize in interface Store

getMajorCompactedCellsCount

public long getMajorCompactedCellsCount()

Description copied from interface: Store

Returns The number of cells processed during major compactions

Specified by:

getMajorCompactedCellsCount in interface Store

getMajorCompactedCellsSize

public long getMajorCompactedCellsSize()

Description copied from interface: Store

Returns The total amount of data processed during major compactions, in bytes

Specified by:

getMajorCompactedCellsSize in interface Store

updateCompactedMetrics

public void updateCompactedMetrics(boolean isMajor,
 CompactionProgress progress)

getStoreEngine

public StoreEngine<?,?,?,?> getStoreEngine()

Returns the StoreEngine that is backing this concrete implementation of Store.

Returns:

Returns the StoreEngine object used internally inside this HStore object.

getOffPeakHours

protected OffPeakHours getOffPeakHours()

onConfigurationChange

public void onConfigurationChange(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: ConfigurationObserver

This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.

Specified by:

onConfigurationChange in interface ConfigurationObserver

registerChildren

public void registerChildren(ConfigurationManager manager)

Needs to be called to register the children to the manager.

Specified by:

registerChildren in interface PropagatingConfigurationObserver

Parameters:

manager - : to register to

deregisterChildren

public void deregisterChildren(ConfigurationManager manager)

Needs to be called to deregister the children from the manager.

Specified by:

deregisterChildren in interface PropagatingConfigurationObserver

Parameters:

manager - : to deregister from

getCompactionPressure

public double getCompactionPressure()

Description copied from interface: Store

This value can represent the degree of emergency of compaction for this store. It should be greater than or equal to 0.0, any value greater than 1.0 means we have too many store files.

• if getStorefilesCount <= getMinFilesToCompact, return 0.0
• return (getStorefilesCount - getMinFilesToCompact) / (blockingFileCount - getMinFilesToCompact)

And for striped stores, we should calculate this value by the files in each stripe separately and return the maximum value.

It is similar to Store.getCompactPriority() except that it is more suitable to use in a linear formula.

Specified by:

getCompactionPressure in interface Store

isPrimaryReplicaStore

public boolean isPrimaryReplicaStore()

Specified by:

isPrimaryReplicaStore in interface Store

preSnapshotOperation

public void preSnapshotOperation()

Sets the store up for a region level snapshot operation.

See Also:

• postSnapshotOperation()

postSnapshotOperation

public void postSnapshotOperation()

Perform tasks needed after the completion of snapshot operation.

See Also:

• preSnapshotOperation()

closeAndArchiveCompactedFiles

public void closeAndArchiveCompactedFiles()
                                   throws IOException

Closes and archives the compacted files under this store

Throws:

IOException

removeCompactedfiles

private void removeCompactedfiles(Collection<HStoreFile> compactedfiles,
 boolean evictOnClose)
                           throws IOException

Archives and removes the compacted files

Parameters:

compactedfiles - The compacted files in this store that are not active in reads

evictOnClose - true if blocks should be evicted from the cache when an HFile reader is closed, false if not

Throws:

IOException

getStoreFileSize

long getStoreFileSize(HStoreFile file)

Computes the length of a store file without succumbing to any errors along the way. If an error is encountered, the implementation returns 0 instead of the actual size.

Parameters:

file - The file to compute the size of.

Returns:

The size in bytes of the provided file.

preFlushSeqIDEstimation

public Long preFlushSeqIDEstimation()

isSloppyMemStore

public boolean isSloppyMemStore()

Description copied from interface: Store

Returns true if the memstore may need some extra memory space

Specified by:

isSloppyMemStore in interface Store

clearCompactedfiles

private void clearCompactedfiles(List<HStoreFile> filesToRemove)
                          throws IOException

Throws:

IOException

reportArchivedFilesForQuota

void reportArchivedFilesForQuota(List<? extends StoreFile> archivedFiles,
 List<Long> fileSizes)

getCurrentParallelPutCount

public int getCurrentParallelPutCount()

Specified by:

getCurrentParallelPutCount in interface Store

getStoreRefCount

public int getStoreRefCount()

getMaxCompactedStoreFileRefCount

public int getMaxCompactedStoreFileRefCount()

Returns get maximum ref count of storeFile among all compacted HStore Files for the HStore

getMemstoreOnlyRowReadsCount

public long getMemstoreOnlyRowReadsCount()

Description copied from interface: Store

Returns the number of read requests purely from the memstore.

Specified by:

getMemstoreOnlyRowReadsCount in interface Store

getMixedRowReadsCount

public long getMixedRowReadsCount()

Description copied from interface: Store

Returns the number of read requests from the files under this store.

Specified by:

getMixedRowReadsCount in interface Store

getReadOnlyConfiguration

public org.apache.hadoop.conf.Configuration getReadOnlyConfiguration()

Specified by:

getReadOnlyConfiguration in interface Store

Returns:

a read only configuration of this store; throws UnsupportedOperationException if you try to set a configuration.

updateMetricsStore

void updateMetricsStore(boolean memstoreRead)

getStoreFilesBeingWritten

public Set<org.apache.hadoop.fs.Path> getStoreFilesBeingWritten()

Return the storefiles which are currently being written to. Mainly used by BrokenStoreFileCleaner to prevent deleting the these files as they are not present in SFT yet.

getBloomFilterRequestsCount

public long getBloomFilterRequestsCount()

Description copied from interface: Store

Returns count of bloom filter results for this store.

Specified by:

getBloomFilterRequestsCount in interface Store

getBloomFilterNegativeResultsCount

public long getBloomFilterNegativeResultsCount()

Description copied from interface: Store

Returns count of negative results for bloom filter requests for this store.

Specified by:

getBloomFilterNegativeResultsCount in interface Store

getBloomFilterEligibleRequestsCount

public long getBloomFilterEligibleRequestsCount()

Description copied from interface: Store

Returns count of requests which could have used bloom filters, but they weren't configured or loaded.

Specified by:

getBloomFilterEligibleRequestsCount in interface Store