Package org.apache.hadoop.hbase.regionserver

Class StripeStoreFileManager

java.lang.Object

org.apache.hadoop.hbase.regionserver.StripeStoreFileManager

All Implemented Interfaces:

StripeCompactionPolicy.StripeInformationProvider, StoreFileManager

@Private
public class StripeStoreFileManager
extends Object
implements StoreFileManager, StripeCompactionPolicy.StripeInformationProvider

Stripe implementation of StoreFileManager. Not thread safe - relies on external locking (in HStore). Collections that this class returns are immutable or unique to the call, so they should be safe. Stripe store splits the key space of the region into non-overlapping stripes, as well as some recent files that have all the keys (level 0). Each stripe contains a set of files. When L0 is compacted, it's split into the files corresponding to existing stripe boundaries, that can thus be added to stripes. When scan or get happens, it only has to read the files from the corresponding stripes. See StripeCompationPolicy on how the stripes are determined; this class doesn't care. This class should work together with StripeCompactionPolicy and StripeCompactor. With regard to how they work, we make at least the following (reasonable) assumptions: - Compaction produces one file per new stripe (if any); that is easy to change. - Compaction has one contiguous set of stripes both in and out, except if L0 is involved.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private class	StripeStoreFileManager.CompactionOrFlushMergeCopy	Non-static helper class for merging compaction or flush results.
private static class	StripeStoreFileManager.KeyBeforeConcatenatedLists	An extension of ConcatenatedLists that has several peculiar properties.
private static class	StripeStoreFileManager.State	The state class.

Field Summary

Fields

Modifier and Type	Field	Description
private final int	blockingFileCount
private final CellComparator	cellComparator
private StripeStoreConfig	config
private HashMap<HStoreFile,byte[]>	fileEnds
private HashMap<HStoreFile,byte[]>	fileStarts	Cached file metadata (or overrides as the case may be)
(package private) static final byte[]	INVALID_KEY
private static final byte[]	INVALID_KEY_IN_MAP	Normally invalid key is null, but in the map null is the result for "no key"; so use the following constant value in these maps instead.
private static final org.slf4j.Logger	LOG
private static final Bytes.RowEndKeyComparator	MAP_COMPARATOR
static final byte[]	OPEN_KEY	The key value used for range boundary, indicating that the boundary is open (i.e.
private StripeStoreFileManager.State	state
static final byte[]	STRIPE_END_KEY
static final byte[]	STRIPE_START_KEY	The file metadata fields that contain the stripe information.

Constructor Summary

Constructors

Constructor	Description
StripeStoreFileManager(CellComparator kvComparator, org.apache.hadoop.conf.Configuration conf, StripeStoreConfig config)

Method Summary

Modifier and Type	Method	Description
void	addCompactionResults(Collection<HStoreFile> compactedFiles, Collection<HStoreFile> results)	Adds only the new compaction results into the structure.
org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile>	clearCompactedFiles()	Clears all the compacted files and returns them.
org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile>	clearFiles()	Clears all the files currently in use and returns them.
private void	debugDumpState(String string)
private byte[]	endOf(HStoreFile sf)
private void	ensureEdgeStripeMetadata(ArrayList<HStoreFile> stripe, boolean isFirst)
private void	ensureLevel0Metadata(HStoreFile sf)
private Collection<HStoreFile>	findExpiredFiles(org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile> stripe, long maxTs, List<HStoreFile> filesCompacting, Collection<HStoreFile> expiredStoreFiles)
private final int	findStripeForRow(byte[] row, boolean isStart)	Finds the stripe index for the stripe containing a row provided externally for get/scan.
private final int	findStripeIndexByEndRow(byte[] endRow)	Finds the stripe index by end row.
Iterator<HStoreFile>	getCandidateFilesForRowKeyBefore(KeyValue targetKey)	See StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue) for details on this methods.
Collection<HStoreFile>	getCompactedfiles()	List of compacted files inside this store that needs to be excluded in reads because further new reads will be using only the newly created files out of compaction.
int	getCompactedFilesCount()	Returns the number of compacted files.
double	getCompactionPressure()
final byte[]	getEndRow(int stripeIndex)	Gets the end row for a given stripe.
Collection<HStoreFile>	getFilesForScan(byte[] startRow, boolean includeStartRow, byte[] stopRow, boolean includeStopRow)	Gets the store files to scan for a Scan or Get request.
List<HStoreFile>	getLevel0Files()	Returns Level 0 files.
private double	getMidStripeSplitRatio(long smallerSize, long largerSize, long lastLargerSize)
Optional<byte[]>	getSplitPoint()	Override of getSplitPoint that determines the split point as the boundary between two stripes, unless it causes significant imbalance between split sides' sizes.
private Optional<byte[]>	getSplitPointFromAllFiles()
final byte[]	getStartRow(int stripeIndex)	Gets the start row for a given stripe.
int	getStoreCompactionPriority()	Returns The store compaction priority.
Comparator<HStoreFile>	getStoreFileComparator()
int	getStorefileCount()	Returns the number of files currently in use.
Collection<HStoreFile>	getStorefiles()	Gets the snapshot of the store files currently in use.
List<byte[]>	getStripeBoundaries()	Returns All stripe boundaries; including the open ones on both ends.
int	getStripeCount()	Returns Stripe count.
private long	getStripeFilesSize(int stripeIndex)	Gets the total size of all files in the stripe.
ArrayList<org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile>>	getStripes()	Returns The stripes.
Collection<HStoreFile>	getUnneededFiles(long maxTs, List<HStoreFile> filesCompacting)
private static void	insertFileIntoStripe(ArrayList<HStoreFile> stripe, HStoreFile sf)	Inserts a file in the correct place (by seqnum) in a stripe copy.
void	insertNewFiles(Collection<HStoreFile> sfs)	Adds new files, either for from MemStore flush or bulk insert, into the structure.
private static final boolean	isInvalid(byte[] key)	Checks whether the key is invalid (e.g.
private static final boolean	isOpen(byte[] key)	Checks whether the key indicates an open interval boundary (i.e.
private static final boolean	isOpen(Cell key)
void	loadFiles(List<HStoreFile> storeFiles)	Loads the initial store files into empty StoreFileManager.
private void	loadUnclassifiedStoreFiles(List<HStoreFile> storeFiles)	Loads initial store files that were picked up from some physical location pertaining to this store (presumably).
private void	markCompactedAway(Collection<HStoreFile> compactedFiles)
private final int	nonOpenRowCompare(byte[] k1, byte[] k2)	Compare two keys.
private final int	nonOpenRowCompare(Cell k1, byte[] k2)
void	removeCompactedFiles(Collection<HStoreFile> compactedFiles)	Remove the compacted files
private final boolean	rowEquals(byte[] k1, byte[] k2)	Compare two keys for equality.
private byte[]	startOf(HStoreFile sf)
Iterator<HStoreFile>	updateCandidateFilesForRowKeyBefore(Iterator<HStoreFile> candidateFiles, KeyValue targetKey, Cell candidate)	See StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue) and StoreFileManager.updateCandidateFilesForRowKeyBefore(Iterator, KeyValue, Cell) for details on this methods.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

STRIPE_START_KEY

public static final byte[] STRIPE_START_KEY

The file metadata fields that contain the stripe information.

STRIPE_END_KEY

public static final byte[] STRIPE_END_KEY

MAP_COMPARATOR

private static final Bytes.RowEndKeyComparator MAP_COMPARATOR

OPEN_KEY

public static final byte[] OPEN_KEY

The key value used for range boundary, indicating that the boundary is open (i.e. +-inf).

INVALID_KEY

static final byte[] INVALID_KEY

state

private StripeStoreFileManager.State state

fileStarts

private HashMap<HStoreFile,byte[]> fileStarts

Cached file metadata (or overrides as the case may be)

fileEnds

private HashMap<HStoreFile,byte[]> fileEnds

INVALID_KEY_IN_MAP

private static final byte[] INVALID_KEY_IN_MAP

Normally invalid key is null, but in the map null is the result for "no key"; so use the following constant value in these maps instead. Note that this is a constant and we use it to compare by reference when we read from the map.

cellComparator

private final CellComparator cellComparator

config

private StripeStoreConfig config

blockingFileCount

private final int blockingFileCount

Constructor Details

StripeStoreFileManager

public StripeStoreFileManager(CellComparator kvComparator,
 org.apache.hadoop.conf.Configuration conf,
 StripeStoreConfig config)

Method Details

loadFiles

public void loadFiles(List<HStoreFile> storeFiles)

Description copied from interface: StoreFileManager

Loads the initial store files into empty StoreFileManager.

Specified by:

loadFiles in interface StoreFileManager

Parameters:

storeFiles - The files to load.

getStorefiles

public Collection<HStoreFile> getStorefiles()

Description copied from interface: StoreFileManager

Gets the snapshot of the store files currently in use. Can be used for things like metrics and checks; should not assume anything about relations between store files in the list.

Specified by:

getStorefiles in interface StoreFileManager

Specified by:

getStorefiles in interface StripeCompactionPolicy.StripeInformationProvider

Returns:

The list of StoreFiles.

getCompactedfiles

public Collection<HStoreFile> getCompactedfiles()

Description copied from interface: StoreFileManager

List of compacted files inside this store that needs to be excluded in reads because further new reads will be using only the newly created files out of compaction. These compacted files will be deleted/cleared once all the existing readers on these compacted files are done.

Specified by:

getCompactedfiles in interface StoreFileManager

Returns:

the list of compacted files

getCompactedFilesCount

public int getCompactedFilesCount()

Description copied from interface: StoreFileManager

Returns the number of compacted files.

Specified by:

getCompactedFilesCount in interface StoreFileManager

Returns:

The number of files.

insertNewFiles

public void insertNewFiles(Collection<HStoreFile> sfs)

Description copied from interface: StoreFileManager

Adds new files, either for from MemStore flush or bulk insert, into the structure.

Specified by:

insertNewFiles in interface StoreFileManager

Parameters:

sfs - New store files.

clearFiles

public org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile> clearFiles()

Description copied from interface: StoreFileManager

Clears all the files currently in use and returns them.

Specified by:

clearFiles in interface StoreFileManager

Returns:

The files previously in use.

clearCompactedFiles

public org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile> clearCompactedFiles()

Description copied from interface: StoreFileManager

Clears all the compacted files and returns them. This method is expected to be accessed single threaded.

Specified by:

clearCompactedFiles in interface StoreFileManager

Returns:

The files compacted previously.

getStorefileCount

public int getStorefileCount()

Description copied from interface: StoreFileManager

Returns the number of files currently in use.

Specified by:

getStorefileCount in interface StoreFileManager

Returns:

The number of files.

getCandidateFilesForRowKeyBefore

public Iterator<HStoreFile> getCandidateFilesForRowKeyBefore(KeyValue targetKey)

See StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue) for details on this methods.

Specified by:

getCandidateFilesForRowKeyBefore in interface StoreFileManager

Parameters:

targetKey - The key that is the basis of the search.

Returns:

The files that may have the key less than or equal to targetKey, in reverse order of new-ness, and preference for target key.

updateCandidateFilesForRowKeyBefore

public Iterator<HStoreFile> updateCandidateFilesForRowKeyBefore(Iterator<HStoreFile> candidateFiles,
 KeyValue targetKey,
 Cell candidate)

See StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue) and StoreFileManager.updateCandidateFilesForRowKeyBefore(Iterator, KeyValue, Cell) for details on this methods.

Specified by:

updateCandidateFilesForRowKeyBefore in interface StoreFileManager

Parameters:

candidateFiles - The candidate files not yet checked for better candidates - return value from StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue), with some files already removed.

targetKey - The key to search for.

candidate - The current best candidate found.

Returns:

The list to replace candidateFiles.

getSplitPoint

public Optional<byte[]> getSplitPoint()
                               throws IOException

Override of getSplitPoint that determines the split point as the boundary between two stripes, unless it causes significant imbalance between split sides' sizes. In that case, the split boundary will be chosen from the middle of one of the stripes to minimize imbalance.

Specified by:

getSplitPoint in interface StoreFileManager

Returns:

The split point, or null if no split is possible.

Throws:

IOException

getSplitPointFromAllFiles

private Optional<byte[]> getSplitPointFromAllFiles()
                                            throws IOException

Throws:

IOException

getMidStripeSplitRatio

private double getMidStripeSplitRatio(long smallerSize,
 long largerSize,
 long lastLargerSize)

getFilesForScan

public Collection<HStoreFile> getFilesForScan(byte[] startRow,
 boolean includeStartRow,
 byte[] stopRow,
 boolean includeStopRow)

Description copied from interface: StoreFileManager

Gets the store files to scan for a Scan or Get request.

Specified by:

getFilesForScan in interface StoreFileManager

Parameters:

startRow - Start row of the request.

stopRow - Stop row of the request.

Returns:

The list of files that are to be read for this request.

addCompactionResults

public void addCompactionResults(Collection<HStoreFile> compactedFiles,
 Collection<HStoreFile> results)

Description copied from interface: StoreFileManager

Adds only the new compaction results into the structure.

Specified by:

addCompactionResults in interface StoreFileManager

Parameters:

compactedFiles - The input files for the compaction.

results - The resulting files for the compaction.

markCompactedAway

private void markCompactedAway(Collection<HStoreFile> compactedFiles)

removeCompactedFiles

public void removeCompactedFiles(Collection<HStoreFile> compactedFiles)

Description copied from interface: StoreFileManager

Remove the compacted files

Specified by:

removeCompactedFiles in interface StoreFileManager

Parameters:

compactedFiles - the list of compacted files

getStoreCompactionPriority

public int getStoreCompactionPriority()

Description copied from interface: StoreFileManager

Returns The store compaction priority.

Specified by:

getStoreCompactionPriority in interface StoreFileManager

getStripeFilesSize

private long getStripeFilesSize(int stripeIndex)

Gets the total size of all files in the stripe.

Parameters:

stripeIndex - Stripe index.

Returns:

Size.

loadUnclassifiedStoreFiles

private void loadUnclassifiedStoreFiles(List<HStoreFile> storeFiles)

Loads initial store files that were picked up from some physical location pertaining to this store (presumably). Unlike adding files after compaction, assumes empty initial sets, and is forgiving with regard to stripe constraints - at worst, many/all files will go to level 0.

Parameters:

storeFiles - Store files to add.

ensureEdgeStripeMetadata

private void ensureEdgeStripeMetadata(ArrayList<HStoreFile> stripe,
 boolean isFirst)

ensureLevel0Metadata

private void ensureLevel0Metadata(HStoreFile sf)

debugDumpState

private void debugDumpState(String string)

isOpen

private static final boolean isOpen(byte[] key)

Checks whether the key indicates an open interval boundary (i.e. infinity).

isOpen

private static final boolean isOpen(Cell key)

isInvalid

private static final boolean isInvalid(byte[] key)

Checks whether the key is invalid (e.g. from an L0 file, or non-stripe-compacted files).

rowEquals

private final boolean rowEquals(byte[] k1,
 byte[] k2)

Compare two keys for equality.

nonOpenRowCompare

private final int nonOpenRowCompare(byte[] k1,
 byte[] k2)

Compare two keys. Keys must not be open (isOpen(row) == false).

nonOpenRowCompare

private final int nonOpenRowCompare(Cell k1,
 byte[] k2)

findStripeIndexByEndRow

private final int findStripeIndexByEndRow(byte[] endRow)

Finds the stripe index by end row.

findStripeForRow

private final int findStripeForRow(byte[] row,
 boolean isStart)

Finds the stripe index for the stripe containing a row provided externally for get/scan.

getStartRow

public final byte[] getStartRow(int stripeIndex)

Description copied from interface: StripeCompactionPolicy.StripeInformationProvider

Gets the start row for a given stripe.

Specified by:

getStartRow in interface StripeCompactionPolicy.StripeInformationProvider

Parameters:

stripeIndex - Stripe index.

Returns:

Start row. May be an open key.

getEndRow

public final byte[] getEndRow(int stripeIndex)

Description copied from interface: StripeCompactionPolicy.StripeInformationProvider

Gets the end row for a given stripe.

Specified by:

getEndRow in interface StripeCompactionPolicy.StripeInformationProvider

Parameters:

stripeIndex - Stripe index.

Returns:

End row. May be an open key.

startOf

private byte[] startOf(HStoreFile sf)

endOf

private byte[] endOf(HStoreFile sf)

insertFileIntoStripe

private static void insertFileIntoStripe(ArrayList<HStoreFile> stripe,
 HStoreFile sf)

Inserts a file in the correct place (by seqnum) in a stripe copy.

Parameters:

stripe - Stripe copy to insert into.

sf - File to insert.

getLevel0Files

public List<HStoreFile> getLevel0Files()

Description copied from interface: StripeCompactionPolicy.StripeInformationProvider

Returns Level 0 files.

Specified by:

getLevel0Files in interface StripeCompactionPolicy.StripeInformationProvider

getStripeBoundaries

public List<byte[]> getStripeBoundaries()

Description copied from interface: StripeCompactionPolicy.StripeInformationProvider

Returns All stripe boundaries; including the open ones on both ends.

Specified by:

getStripeBoundaries in interface StripeCompactionPolicy.StripeInformationProvider

getStripes

public ArrayList<org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile>> getStripes()

Description copied from interface: StripeCompactionPolicy.StripeInformationProvider

Returns The stripes.

Specified by:

getStripes in interface StripeCompactionPolicy.StripeInformationProvider

getStripeCount

public int getStripeCount()

Description copied from interface: StripeCompactionPolicy.StripeInformationProvider

Returns Stripe count.

Specified by:

getStripeCount in interface StripeCompactionPolicy.StripeInformationProvider

getUnneededFiles

public Collection<HStoreFile> getUnneededFiles(long maxTs,
 List<HStoreFile> filesCompacting)

Specified by:

getUnneededFiles in interface StoreFileManager

Parameters:

maxTs - Maximum expired timestamp.

filesCompacting - Files that are currently compacting.

Returns:

The files which don't have any necessary data according to TTL and other criteria.

findExpiredFiles

private Collection<HStoreFile> findExpiredFiles(org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile> stripe,
 long maxTs,
 List<HStoreFile> filesCompacting,
 Collection<HStoreFile> expiredStoreFiles)

getCompactionPressure

public double getCompactionPressure()

Specified by:

getCompactionPressure in interface StoreFileManager

Returns:

the compaction pressure used for compaction throughput tuning.

See Also:

• HStore.getCompactionPressure()

getStoreFileComparator

public Comparator<HStoreFile> getStoreFileComparator()

Specified by:

getStoreFileComparator in interface StoreFileManager

Returns:

the comparator used to sort storefiles. Usually, the HStoreFile.getMaxSequenceId() is the first priority.