Package org.apache.hadoop.hbase.regionserver

Class CompactingMemStore

java.lang.Object

org.apache.hadoop.hbase.regionserver.AbstractMemStore

org.apache.hadoop.hbase.regionserver.CompactingMemStore

All Implemented Interfaces:

Closeable, AutoCloseable, MemStore

@Private
public class CompactingMemStore
extends AbstractMemStore

A memstore implementation which supports in-memory compaction. A compaction pipeline is added between the active set and the snapshot data structures; it consists of a list of segments that are subject to compaction. Like the snapshot, all pipeline segments are read-only; updates only affect the active set. To ensure this property we take advantage of the existing blocking mechanism -- the active set is pushed to the pipeline while holding the region's updatesLock in exclusive mode. Periodically, a compaction is applied in the background to all pipeline segments resulting in a single read-only component. The ``old'' segments are discarded when no scanner is reading them.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	CompactingMemStore.IndexType	Types of indexes (part of immutable segments) to be used after flattening, compaction, or merge are applied.
private class	CompactingMemStore.InMemoryCompactionRunnable	The in-memory-flusher thread performs the flush asynchronously.

Field Summary

Fields

Modifier and Type	Field	Description
protected final AtomicBoolean	allowCompaction
static final String	COMPACTING_MEMSTORE_TYPE_DEFAULT
static final String	COMPACTING_MEMSTORE_TYPE_KEY
protected MemStoreCompactor	compactor
private boolean	compositeSnapshot
static final long	DEEP_OVERHEAD
static final int	IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT
static final String	IN_MEMORY_CONPACTION_POOL_SIZE_KEY
private static final int	IN_MEMORY_FLUSH_MULTIPLIER
static final String	IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY
private CompactingMemStore.IndexType	indexType
private final AtomicBoolean	inMemoryCompactionInProgress
private long	inmemoryFlushSize
private boolean	inWalReplay
private static final org.slf4j.Logger	LOG
private CompactionPipeline	pipeline
private HStore	store

Fields inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore

FIXED_OVERHEAD, regionServices, snapshot, snapshotId

Constructor Summary

Constructors

Constructor	Description
CompactingMemStore(org.apache.hadoop.conf.Configuration conf, CellComparator c, HStore store, RegionServicesForStores regionServices, MemoryCompactionPolicy compactionPolicy)

Method Summary

Modifier and Type	Method	Description
protected boolean	checkAndAddToActiveSize(MutableSegment currActive, Cell cellToAdd, MemStoreSizing memstoreSizing)	Check whether anything need to be done based on the current active set size.
protected List<KeyValueScanner>	createList(int capacity)
protected MemStoreCompactor	createMemStoreCompactor(MemoryCompactionPolicy compactionPolicy)
void	debug()
void	flattenOneSegment(long requesterVersion, MemStoreCompactionStrategy.Action action)
(package private) void	flushInMemory()
protected void	flushInMemory(MutableSegment currActive)
String	getFamilyName()
private byte[]	getFamilyNameInBytes()
MemStoreSize	getFlushableSize()	Flush will first clear out the data in snapshot if any (It will take a second flush invocation to clear the current Cell set).
VersionedSegmentsList	getImmutableSegments()
CompactingMemStore.IndexType	getIndexType()
(package private) long	getInmemoryFlushSize()
private Segment	getLastSegment()
(package private) Cell	getNextRow(Cell cell)
private ThreadPoolExecutor	getPool()
private RegionServicesForStores	getRegionServices()
List<KeyValueScanner>	getScanners(long readPt)	This method is protected under HStore#lock read lock.
protected List<Segment>	getSegments()	Returns an ordered list of segments from most recent to oldest in memstore
long	getSmallestReadPoint()
HStore	getStore()
boolean	hasImmutableSegments()
protected long	heapSize()
private void	initInmemoryFlushSize(org.apache.hadoop.conf.Configuration conf)
(package private) void	inMemoryCompaction()
(package private) boolean	isMemStoreFlushingInMemory()
boolean	isSloppy()
protected long	keySize()	Returns The total size of cells in this memstore.
protected void	postUpdate(MutableSegment currentActive)	Issue any post update synchronization and tests
long	preFlushSeqIDEstimation()	This method is called before the flush is executed.
protected boolean	preUpdate(MutableSegment currentActive, Cell cell, MemStoreSizing memstoreSizing)	Issue any synchronization and test needed before applying the update For compacting memstore this means checking the update can increase the size without overflow
protected void	pushActiveToPipeline(MutableSegment currActive, boolean checkEmpty)	NOTE: When flushInMemory(MutableSegment) calls this method, due to concurrent writes and because we first add cell size to currActive.getDataSize and then actually add cell to currActive.cellSet, it is possible that currActive.getDataSize could not accommodate cellToAdd but currActive.cellSet is still empty if pending writes which not yet add cells to currActive.cellSet,so for flushInMemory(MutableSegment),checkEmpty parameter is false.
private void	pushPipelineToSnapshot()
private void	pushTailToSnapshot()
private void	pushToSnapshot(List<ImmutableSegment> segments)
void	setCompositeSnapshot(boolean useCompositeSnapshot)
(package private) void	setIndexType(CompactingMemStore.IndexType type)
void	setInMemoryCompactionCompleted()
protected boolean	setInMemoryCompactionFlag()
MemStoreSize	size()
protected boolean	sizeAddedPreOperation()
MemStoreSnapshot	snapshot()	Push the current active memstore segment into the pipeline and create a snapshot of the tail of current compaction pipeline Snapshot must be cleared by call to AbstractMemStore.clearSnapshot(long).
void	startReplayingFromWAL()	This message intends to inform the MemStore that next coming updates are going to be part of the replaying edits from WAL
private void	stopCompaction()	The request to cancel the compaction asynchronous task (caused by in-memory flush) The compaction may still happen if the request was sent too late Non-blocking request
void	stopReplayingFromWAL()	This message intends to inform the MemStore that the replaying edits from WAL are done
boolean	swapCompactedSegments(VersionedSegmentsList versionedList, ImmutableSegment result, boolean merge)
protected boolean	swapPipelineWithNull(VersionedSegmentsList segments)
private void	tryFlushInMemoryAndCompactingAsync(MutableSegment currActive)	Try to flush the currActive in memory and submit the background CompactingMemStore.InMemoryCompactionRunnable to RegionServicesForStores.getInMemoryCompactionPool().
void	updateLowestUnflushedSequenceIdInWAL(boolean onlyIfGreater)	Updates the wal with the lowest sequence id (oldest entry) that is still in memory

Methods inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore

add, add, addToScanners, addToScanners, clearSnapshot, close, doAdd, doClearSnapShot, dump, getActive, getComparator, getConfiguration, getLowest, getNextRow, getSnapshot, getSnapshotSize, resetActive, resetTimeOfOldestEdit, timeOfOldestEdit, toString, upsert

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

COMPACTING_MEMSTORE_TYPE_KEY

public static final String COMPACTING_MEMSTORE_TYPE_KEY

See Also:

• Constant Field Values

COMPACTING_MEMSTORE_TYPE_DEFAULT

public static final String COMPACTING_MEMSTORE_TYPE_DEFAULT

IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY

public static final String IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY

See Also:

• Constant Field Values

IN_MEMORY_FLUSH_MULTIPLIER

private static final int IN_MEMORY_FLUSH_MULTIPLIER

See Also:

• Constant Field Values

IN_MEMORY_CONPACTION_POOL_SIZE_KEY

public static final String IN_MEMORY_CONPACTION_POOL_SIZE_KEY

See Also:

• Constant Field Values

IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT

public static final int IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT

See Also:

• Constant Field Values

LOG

private static final org.slf4j.Logger LOG

store

private HStore store

pipeline

private CompactionPipeline pipeline

compactor

protected MemStoreCompactor compactor

inmemoryFlushSize

private long inmemoryFlushSize

inMemoryCompactionInProgress

private final AtomicBoolean inMemoryCompactionInProgress

inWalReplay

private boolean inWalReplay

allowCompaction

protected final AtomicBoolean allowCompaction

compositeSnapshot

private boolean compositeSnapshot

indexType

private CompactingMemStore.IndexType indexType

DEEP_OVERHEAD

public static final long DEEP_OVERHEAD

Constructor Details

CompactingMemStore

public CompactingMemStore(org.apache.hadoop.conf.Configuration conf,
 CellComparator c,
 HStore store,
 RegionServicesForStores regionServices,
 MemoryCompactionPolicy compactionPolicy)
                   throws IOException

Throws:

IOException

Method Details

createMemStoreCompactor

protected MemStoreCompactor createMemStoreCompactor(MemoryCompactionPolicy compactionPolicy)
                                             throws IllegalArgumentIOException

Throws:

IllegalArgumentIOException

initInmemoryFlushSize

private void initInmemoryFlushSize(org.apache.hadoop.conf.Configuration conf)

size

public MemStoreSize size()

Returns:

Total memory occupied by this MemStore. This won't include any size occupied by the snapshot. We assume the snapshot will get cleared soon. This is not thread safe and the memstore may be changed while computing its size. It is the responsibility of the caller to make sure this doesn't happen.

preFlushSeqIDEstimation

public long preFlushSeqIDEstimation()

This method is called before the flush is executed.

Returns:

an estimation (lower bound) of the unflushed sequence id in memstore after the flush is executed. if memstore will be cleared returns HConstants.NO_SEQNUM.

isSloppy

public boolean isSloppy()

snapshot

public MemStoreSnapshot snapshot()

Push the current active memstore segment into the pipeline and create a snapshot of the tail of current compaction pipeline Snapshot must be cleared by call to AbstractMemStore.clearSnapshot(long). AbstractMemStore.clearSnapshot(long).

Returns:

MemStoreSnapshot

getFlushableSize

public MemStoreSize getFlushableSize()

Description copied from interface: MemStore

Flush will first clear out the data in snapshot if any (It will take a second flush invocation to clear the current Cell set). If snapshot is empty, current Cell set will be flushed.

Returns:

On flush, how much memory we will clear.

setInMemoryCompactionCompleted

public void setInMemoryCompactionCompleted()

setInMemoryCompactionFlag

protected boolean setInMemoryCompactionFlag()

keySize

protected long keySize()

Description copied from class: AbstractMemStore

Returns The total size of cells in this memstore. We will not consider cells in the snapshot

Specified by:

keySize in class AbstractMemStore

heapSize

protected long heapSize()

Specified by:

heapSize in class AbstractMemStore

Returns:

The total heap size of cells in this memstore. We will not consider cells in the snapshot

updateLowestUnflushedSequenceIdInWAL

public void updateLowestUnflushedSequenceIdInWAL(boolean onlyIfGreater)

Description copied from class: AbstractMemStore

Updates the wal with the lowest sequence id (oldest entry) that is still in memory

Specified by:

updateLowestUnflushedSequenceIdInWAL in class AbstractMemStore

Parameters:

onlyIfGreater - a flag that marks whether to update the sequence id no matter what or only if it is greater than the previous sequence id

startReplayingFromWAL

public void startReplayingFromWAL()

This message intends to inform the MemStore that next coming updates are going to be part of the replaying edits from WAL

stopReplayingFromWAL

public void stopReplayingFromWAL()

This message intends to inform the MemStore that the replaying edits from WAL are done

preUpdate

protected boolean preUpdate(MutableSegment currentActive,
 Cell cell,
 MemStoreSizing memstoreSizing)

Issue any synchronization and test needed before applying the update For compacting memstore this means checking the update can increase the size without overflow

Specified by:

preUpdate in class AbstractMemStore

Parameters:

currentActive - the segment to be updated

cell - the cell to be added

memstoreSizing - object to accumulate region size changes

Returns:

true iff can proceed with applying the update

postUpdate

protected void postUpdate(MutableSegment currentActive)

Description copied from class: AbstractMemStore

Issue any post update synchronization and tests

Specified by:

postUpdate in class AbstractMemStore

Parameters:

currentActive - updated segment

sizeAddedPreOperation

protected boolean sizeAddedPreOperation()

Specified by:

sizeAddedPreOperation in class AbstractMemStore

getSegments

protected List<Segment> getSegments()

Description copied from class: AbstractMemStore

Returns an ordered list of segments from most recent to oldest in memstore

Specified by:

getSegments in class AbstractMemStore

setCompositeSnapshot

public void setCompositeSnapshot(boolean useCompositeSnapshot)

swapCompactedSegments

public boolean swapCompactedSegments(VersionedSegmentsList versionedList,
 ImmutableSegment result,
 boolean merge)

flattenOneSegment

public void flattenOneSegment(long requesterVersion,
 MemStoreCompactionStrategy.Action action)

Parameters:

requesterVersion - The caller must hold the VersionedList of the pipeline with version taken earlier. This version must be passed as a parameter here. The flattening happens only if versions match.

setIndexType

void setIndexType(CompactingMemStore.IndexType type)

getIndexType

public CompactingMemStore.IndexType getIndexType()

hasImmutableSegments

public boolean hasImmutableSegments()

getImmutableSegments

public VersionedSegmentsList getImmutableSegments()

getSmallestReadPoint

public long getSmallestReadPoint()

getStore

public HStore getStore()

getFamilyName

public String getFamilyName()

getScanners

public List<KeyValueScanner> getScanners(long readPt)
                                  throws IOException

This method is protected under HStore#lock read lock.

Returns:

scanner over the memstore. This might include scanner over the snapshot when one is present.

Throws:

IOException

createList

protected List<KeyValueScanner> createList(int capacity)

checkAndAddToActiveSize

protected boolean checkAndAddToActiveSize(MutableSegment currActive,
 Cell cellToAdd,
 MemStoreSizing memstoreSizing)

Check whether anything need to be done based on the current active set size. The method is invoked upon every addition to the active set. For CompactingMemStore, flush the active set to the read-only memory if it's size is above threshold

Parameters:

currActive - intended segment to update

cellToAdd - cell to be added to the segment

memstoreSizing - object to accumulate changed size

Returns:

true if the cell can be added to the currActive

tryFlushInMemoryAndCompactingAsync

private void tryFlushInMemoryAndCompactingAsync(MutableSegment currActive)

Try to flush the currActive in memory and submit the background CompactingMemStore.InMemoryCompactionRunnable to RegionServicesForStores.getInMemoryCompactionPool(). Just one thread can do the actual flushing in memory.

Parameters:

currActive - current Active Segment to be flush in memory.

flushInMemory

void flushInMemory()

flushInMemory

protected void flushInMemory(MutableSegment currActive)

inMemoryCompaction

void inMemoryCompaction()

getLastSegment

private Segment getLastSegment()

getFamilyNameInBytes

private byte[] getFamilyNameInBytes()

getPool

private ThreadPoolExecutor getPool()

stopCompaction

private void stopCompaction()

The request to cancel the compaction asynchronous task (caused by in-memory flush) The compaction may still happen if the request was sent too late Non-blocking request

pushActiveToPipeline

protected void pushActiveToPipeline(MutableSegment currActive,
 boolean checkEmpty)

NOTE: When flushInMemory(MutableSegment) calls this method, due to concurrent writes and because we first add cell size to currActive.getDataSize and then actually add cell to currActive.cellSet, it is possible that currActive.getDataSize could not accommodate cellToAdd but currActive.cellSet is still empty if pending writes which not yet add cells to currActive.cellSet,so for flushInMemory(MutableSegment),checkEmpty parameter is false. But if snapshot() called this method,because there is no pending write,checkEmpty parameter could be true.

pushTailToSnapshot

private void pushTailToSnapshot()

pushPipelineToSnapshot

private void pushPipelineToSnapshot()

swapPipelineWithNull

protected boolean swapPipelineWithNull(VersionedSegmentsList segments)

pushToSnapshot

private void pushToSnapshot(List<ImmutableSegment> segments)

getRegionServices

private RegionServicesForStores getRegionServices()

isMemStoreFlushingInMemory

boolean isMemStoreFlushingInMemory()

getNextRow

Cell getNextRow(Cell cell)

Parameters:

cell - Find the row that comes after this one. If null, we return the first.

Returns:

Next row or null if none found.

getInmemoryFlushSize

long getInmemoryFlushSize()

debug

public void debug()