Package org.apache.hadoop.hbase.regionserver

Class MemStoreFlusher

java.lang.Object

org.apache.hadoop.hbase.regionserver.MemStoreFlusher

All Implemented Interfaces:

ConfigurationObserver, FlushRequester

@Private
public class MemStoreFlusher
extends Object
implements FlushRequester, ConfigurationObserver

Thread that flushes cache on request NOTE: This class extends Thread rather than Chore because the sleep time can be interrupted when there is something to do, rather than the Chore sleep time which is invariant.

See Also:

• FlushRequester

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private class	MemStoreFlusher.FlushHandler
(package private) static interface	MemStoreFlusher.FlushQueueEntry
(package private) static class	MemStoreFlusher.FlushRegionEntry	Datastructure used in the flush queue.

Field Summary

Fields

Modifier and Type	Field	Description
private long	blockingWaitTime
private final Object	blockSignal
private org.apache.hadoop.conf.Configuration	conf
private final AtomicInteger	flusherIdGen
private ThreadFactory	flusherThreadFactory
private MemStoreFlusher.FlushHandler[]	flushHandlers
private final BlockingQueue<MemStoreFlusher.FlushQueueEntry>	flushQueue
private List<FlushRequestListener>	flushRequestListeners
private final ReentrantReadWriteLock	lock
private static final org.slf4j.Logger	LOG
protected final Map<Region,MemStoreFlusher.FlushRegionEntry>	regionsInQueue
private final HRegionServer	server
private final long	threadWakeFrequency
private final LongAdder	updatesBlockedMsHighWater
private static final MemStoreFlusher.FlushQueueEntry	WAKEUPFLUSH_INSTANCE	Singleton instance inserted into flush queue used for signaling.
private AtomicBoolean	wakeupPending

Constructor Summary

Constructors

Constructor	Description
MemStoreFlusher(org.apache.hadoop.conf.Configuration conf, HRegionServer server)

Method Summary

Modifier and Type	Method	Description
String	dumpQueue()
private boolean	flushOneForGlobalPressure(FlushType flushType)	The memstore across all regions has exceeded the low water mark.
private boolean	flushRegion(HRegion region, boolean emergencyFlush, List<byte[]> families, FlushLifeCycleTracker tracker)	Flush a region.
private boolean	flushRegion(MemStoreFlusher.FlushRegionEntry fqe)	A flushRegion that checks store file count.
private HRegion	getBiggestMemStoreOfRegionReplica(SortedMap<Long,HRegion> regionsBySize, Set<HRegion> excludedRegions)
private HRegion	getBiggestMemStoreRegion(SortedMap<Long,HRegion> regionsBySize, Set<HRegion> excludedRegions, boolean checkStoreFileCount)
int	getFlusherCount()
int	getFlushQueueSize()
private int	getHandlerCount(org.apache.hadoop.conf.Configuration conf)
private static long	getMemStoreDataSize(HRegion r)	Returns Return memstore data size or null if r is null
private static long	getMemStoreHeapSize(HRegion r)	Returns Return memstore heap size or null if r is null
private static long	getMemStoreOffHeapSize(HRegion r)	Returns Return memstore offheap size or null if r is null
private int	getStoreFileCount(Region region)
LongAdder	getUpdatesBlockedMsHighWater()
(package private) void	interruptIfNecessary()	Only interrupt once it's done with a run through the work loop.
private FlushType	isAboveHighWaterMark()	Return the FlushType if global memory usage is above the high watermark
private FlushType	isAboveLowWaterMark()	Return the FlushType if we're above the low watermark
(package private) boolean	isAlive()
private boolean	isTooManyStoreFiles(Region region)
private void	logMsg(String type, long val, long max)
private void	notifyFlushRequest(Region region, boolean emergencyFlush)
void	onConfigurationChange(org.apache.hadoop.conf.Configuration newConf)	This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.
void	reclaimMemStoreMemory()	Check if the regionserver's memstore memory usage is greater than the limit.
private boolean	refreshStoreFilesAndReclaimMemory(Region region)
void	registerFlushRequestListener(FlushRequestListener listener)	Register a MemstoreFlushListener
boolean	requestDelayedFlush(HRegion r, long delay)	Tell the listener the cache needs to be flushed after a delay
boolean	requestFlush(HRegion r, List<byte[]> families, FlushLifeCycleTracker tracker)	Tell the listener the cache needs to be flushed.
boolean	requestFlush(HRegion r, FlushLifeCycleTracker tracker)	Tell the listener the cache needs to be flushed.
void	setGlobalMemStoreLimit(long globalMemStoreSize)	Sets the global memstore limit to a new size.
(package private) void	shutdown()
(package private) void	start(Thread.UncaughtExceptionHandler eh)
private void	startFlushHandlerThreads(MemStoreFlusher.FlushHandler[] flushHandlers, int start, int end)
private void	stopFlushHandlerThreads(MemStoreFlusher.FlushHandler[] flushHandlers, int start, int end)
String	toString()
boolean	unregisterFlushRequestListener(FlushRequestListener listener)	Unregister the listener from MemstoreFlushListeners
private void	wakeupFlushThread()
private void	wakeUpIfBlocking()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

conf

private org.apache.hadoop.conf.Configuration conf

flushQueue

private final BlockingQueue<MemStoreFlusher.FlushQueueEntry> flushQueue

regionsInQueue

protected final Map<Region,MemStoreFlusher.FlushRegionEntry> regionsInQueue

wakeupPending

private AtomicBoolean wakeupPending

threadWakeFrequency

private final long threadWakeFrequency

server

private final HRegionServer server

lock

private final ReentrantReadWriteLock lock

blockSignal

private final Object blockSignal

blockingWaitTime

private long blockingWaitTime

updatesBlockedMsHighWater

private final LongAdder updatesBlockedMsHighWater

flushHandlers

private MemStoreFlusher.FlushHandler[] flushHandlers

flusherIdGen

private final AtomicInteger flusherIdGen

flusherThreadFactory

private ThreadFactory flusherThreadFactory

flushRequestListeners

private List<FlushRequestListener> flushRequestListeners

WAKEUPFLUSH_INSTANCE

private static final MemStoreFlusher.FlushQueueEntry WAKEUPFLUSH_INSTANCE

Singleton instance inserted into flush queue used for signaling.

Constructor Details

MemStoreFlusher

public MemStoreFlusher(org.apache.hadoop.conf.Configuration conf,
 HRegionServer server)

Method Details

getUpdatesBlockedMsHighWater

public LongAdder getUpdatesBlockedMsHighWater()

flushOneForGlobalPressure

private boolean flushOneForGlobalPressure(FlushType flushType)

The memstore across all regions has exceeded the low water mark. Pick one region to flush and flush it synchronously (this is called from the flush thread)

Returns:

true if successful

getMemStoreOffHeapSize

private static long getMemStoreOffHeapSize(HRegion r)

Returns Return memstore offheap size or null if r is null

getMemStoreHeapSize

private static long getMemStoreHeapSize(HRegion r)

Returns Return memstore heap size or null if r is null

getMemStoreDataSize

private static long getMemStoreDataSize(HRegion r)

Returns Return memstore data size or null if r is null

wakeupFlushThread

private void wakeupFlushThread()

getBiggestMemStoreRegion

private HRegion getBiggestMemStoreRegion(SortedMap<Long,HRegion> regionsBySize,
 Set<HRegion> excludedRegions,
 boolean checkStoreFileCount)

getBiggestMemStoreOfRegionReplica

private HRegion getBiggestMemStoreOfRegionReplica(SortedMap<Long,HRegion> regionsBySize,
 Set<HRegion> excludedRegions)

refreshStoreFilesAndReclaimMemory

private boolean refreshStoreFilesAndReclaimMemory(Region region)

isAboveHighWaterMark

private FlushType isAboveHighWaterMark()

Return the FlushType if global memory usage is above the high watermark

isAboveLowWaterMark

private FlushType isAboveLowWaterMark()

Return the FlushType if we're above the low watermark

requestFlush

public boolean requestFlush(HRegion r,
 FlushLifeCycleTracker tracker)

Description copied from interface: FlushRequester

Tell the listener the cache needs to be flushed.

Specified by:

requestFlush in interface FlushRequester

Parameters:

r - the Region requesting the cache flush

Returns:

true if our region is added into the queue, false otherwise

requestFlush

public boolean requestFlush(HRegion r,
 List<byte[]> families,
 FlushLifeCycleTracker tracker)

Description copied from interface: FlushRequester

Tell the listener the cache needs to be flushed.

Specified by:

requestFlush in interface FlushRequester

Parameters:

r - the Region requesting the cache flush

families - stores of region to flush, if null then use flush policy

Returns:

true if our region is added into the queue, false otherwise

requestDelayedFlush

public boolean requestDelayedFlush(HRegion r,
 long delay)

Description copied from interface: FlushRequester

Tell the listener the cache needs to be flushed after a delay

Specified by:

requestDelayedFlush in interface FlushRequester

Parameters:

r - the Region requesting the cache flush

delay - after how much time should the flush happen

Returns:

true if our region is added into the queue, false otherwise

getFlushQueueSize

public int getFlushQueueSize()

interruptIfNecessary

void interruptIfNecessary()

Only interrupt once it's done with a run through the work loop.

start

void start(Thread.UncaughtExceptionHandler eh)

isAlive

boolean isAlive()

shutdown

void shutdown()

flushRegion

private boolean flushRegion(MemStoreFlusher.FlushRegionEntry fqe)

A flushRegion that checks store file count. If too many, puts the flush on delay queue to retry later.

Returns:

true if the region was successfully flushed, false otherwise. If false, there will be accompanying log messages explaining why the region was not flushed.

flushRegion

private boolean flushRegion(HRegion region,
 boolean emergencyFlush,
 List<byte[]> families,
 FlushLifeCycleTracker tracker)

Flush a region.

Parameters:

region - Region to flush.

emergencyFlush - Set if we are being force flushed. If true the region needs to be removed from the flush queue. If false, when we were called from the main flusher run loop and we got the entry to flush by calling poll on the flush queue (which removed it).

families - stores of region to flush.

Returns:

true if the region was successfully flushed, false otherwise. If false, there will be accompanying log messages explaining why the region was not flushed.

notifyFlushRequest

private void notifyFlushRequest(Region region,
 boolean emergencyFlush)

wakeUpIfBlocking

private void wakeUpIfBlocking()

isTooManyStoreFiles

private boolean isTooManyStoreFiles(Region region)

getStoreFileCount

private int getStoreFileCount(Region region)

reclaimMemStoreMemory

public void reclaimMemStoreMemory()

Check if the regionserver's memstore memory usage is greater than the limit. If so, flush regions with the biggest memstores until we're down to the lower limit. This method blocks callers until we're down to a safe amount of memstore consumption.

logMsg

private void logMsg(String type,
 long val,
 long max)

toString

public String toString()

Overrides:

toString in class Object

dumpQueue

public String dumpQueue()

registerFlushRequestListener

public void registerFlushRequestListener(FlushRequestListener listener)

Register a MemstoreFlushListener

Specified by:

registerFlushRequestListener in interface FlushRequester

unregisterFlushRequestListener

public boolean unregisterFlushRequestListener(FlushRequestListener listener)

Unregister the listener from MemstoreFlushListeners

Specified by:

unregisterFlushRequestListener in interface FlushRequester

Returns:

true when passed listener is unregistered successfully.

setGlobalMemStoreLimit

public void setGlobalMemStoreLimit(long globalMemStoreSize)

Sets the global memstore limit to a new size.

Specified by:

setGlobalMemStoreLimit in interface FlushRequester

getHandlerCount

private int getHandlerCount(org.apache.hadoop.conf.Configuration conf)

onConfigurationChange

public void onConfigurationChange(org.apache.hadoop.conf.Configuration newConf)

Description copied from interface: ConfigurationObserver

This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.

Specified by:

onConfigurationChange in interface ConfigurationObserver

startFlushHandlerThreads

private void startFlushHandlerThreads(MemStoreFlusher.FlushHandler[] flushHandlers,
 int start,
 int end)

stopFlushHandlerThreads

private void stopFlushHandlerThreads(MemStoreFlusher.FlushHandler[] flushHandlers,
 int start,
 int end)

getFlusherCount

public int getFlusherCount()