Package org.apache.hadoop.hbase.regionserver

Class StoreFileScanner

java.lang.Object

org.apache.hadoop.hbase.regionserver.StoreFileScanner

All Implemented Interfaces:

Closeable, AutoCloseable, KeyValueScanner, Shipper

@LimitedPrivate("Phoenix")
@Evolving
public class StoreFileScanner
extends Object
implements KeyValueScanner

KeyValueScanner adaptor over the Reader. It also provides hooks into bloom filter things.

Field Summary

Fields

Modifier and Type	Field	Description
private final boolean	canOptimizeForNonNullColumn
private boolean	closed
private ExtendedCell	cur
private boolean	delayedReseek
private ExtendedCell	delayedSeekKV
private final boolean	enforceMVCC
private final boolean	hasMVCCInfo
private final HFileScanner	hfs
private final boolean	isFastSeekingEncoding
private Cell	previousRow
private final StoreFileReader	reader
private final long	readPt
private boolean	realSeekDone
private final long	scannerOrder
private static LongAdder	seekCount
private boolean	stopSkippingKVsIfNextRow

Fields inherited from interface org.apache.hadoop.hbase.regionserver.KeyValueScanner

NO_NEXT_INDEXED_KEY

Constructor Summary

Constructors

Constructor	Description
StoreFileScanner(StoreFileReader reader, HFileScanner hfs, boolean useMVCC, boolean hasMVCC, long readPt, long scannerOrder, boolean canOptimizeForNonNullColumn, boolean isFastSeekingEncoding)	Implements a KeyValueScanner on top of the specified HFileScanner

Method Summary

Modifier and Type	Method	Description
boolean	backwardSeek(ExtendedCell key)	Seek the scanner at or before the row of specified Cell, it firstly tries to seek the scanner at or after the specified Cell, return if peek KeyValue of scanner has the same row with specified Cell, otherwise seek the scanner at the first Cell of the row which is the previous row of specified KeyValue
void	close()	Close the KeyValue scanner.
void	enforceSeek()	Does the real seek operation in case it was skipped by seekToRowCol(KeyValue, boolean) (TODO: Whats this?).
(package private) CellComparator	getComparator()
org.apache.hadoop.fs.Path	getFilePath()
ExtendedCell	getNextIndexedKey()
(package private) StoreFileReader	getReader()
long	getScannerOrder()	Get the order of this KeyValueScanner.
static List<StoreFileScanner>	getScannersForCompaction(Collection<HStoreFile> files, boolean canUseDropBehind, long readPt)	Get scanners for compaction.
static List<StoreFileScanner>	getScannersForStoreFiles(Collection<HStoreFile> files, boolean cacheBlocks, boolean usePread, boolean isCompaction, boolean useDropBehind, long readPt)	Return an array of scanners corresponding to the given set of store files.
static List<StoreFileScanner>	getScannersForStoreFiles(Collection<HStoreFile> files, boolean cacheBlocks, boolean usePread, boolean isCompaction, boolean canUseDrop, ScanQueryMatcher matcher, long readPt)	Return an array of scanners corresponding to the given set of store files, And set the ScanQueryMatcher for each store file scanner for further optimization
(package private) static final long	getSeekCount()
(package private) static final void	instrument()
boolean	isFileScanner()	Returns true if this is a file scanner.
private boolean	isStillAtSeekTargetAfterSkippingNewerKvs(Cell seekKey)
ExtendedCell	next()	Return the next Cell in this scanner, iterating the scanner
ExtendedCell	peek()	Look at the next Cell in this scanner, but do not iterate scanner.
boolean	realSeekDone()	We optimize our store scanners by checking the most recent store file first, so we sometimes pretend we have done a seek but delay it until the store scanner bubbles up to the top of the key-value heap.
void	recordBlockSize(IntConsumer blockSizeConsumer)	Record the size of the current block in bytes, passing as an argument to the blockSizeConsumer.
boolean	requestSeek(ExtendedCell kv, boolean forward, boolean useBloom)	Pretend we have done a seek but don't do it yet, if possible.
boolean	reseek(ExtendedCell key)	Reseek the scanner at or after the specified KeyValue.
private boolean	reseekAtOrAfter(ExtendedCell seekKey)
(package private) static boolean	reseekAtOrAfter(HFileScanner s, ExtendedCell k)
boolean	seek(ExtendedCell key)	Seek the scanner at or after the specified KeyValue.
private boolean	seekAtOrAfter(ExtendedCell seekKey)
static boolean	seekAtOrAfter(HFileScanner s, ExtendedCell k)	Returns false if not found or if k is after the end.
private boolean	seekBefore(ExtendedCell seekKey)
private void	seekBeforeAndSaveKeyToPreviousRow(ExtendedCell seekKey)	Seeks before the seek target cell and saves the location to previousRow.
boolean	seekToLastRow()	Seek the scanner at the first KeyValue of last row
boolean	seekToPreviousRow(ExtendedCell originalKey)	Seek the scanner at the first Cell of the row which is the previous row of specified key
private boolean	seekToPreviousRowStateless(ExtendedCell originalKey)	This variant of the StoreFileScanner#seekToPreviousRow(Cell) method requires two seeks.
private boolean	seekToPreviousRowWithHint()	This variant of the StoreFileScanner#seekToPreviousRow(Cell) method requires one seek and one reseek.
private boolean	seekToPreviousRowWithoutHint(Cell originalKey)	This variant of the StoreFileScanner#seekToPreviousRow(Cell) method requires two seeks and one reseek.
protected void	setCurrentCell(ExtendedCell newVal)
void	shipped()	Called after a batch of rows scanned and set to be returned to client.
boolean	shouldUseScanner(Scan scan, HStore store, long oldestUnexpiredTS)	Allows to filter out scanners (both StoreFile and memstore) that we don't want to use based on criteria such as Bloom filters and timestamp ranges.
protected boolean	skipKVsNewerThanReadpoint()
private boolean	skipKvsNewerThanReadpointReversed()
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

reader

private final StoreFileReader reader

hfs

private final HFileScanner hfs

cur

private ExtendedCell cur

closed

private boolean closed

realSeekDone

private boolean realSeekDone

delayedReseek

private boolean delayedReseek

delayedSeekKV

private ExtendedCell delayedSeekKV

enforceMVCC

private final boolean enforceMVCC

hasMVCCInfo

private final boolean hasMVCCInfo

stopSkippingKVsIfNextRow

private boolean stopSkippingKVsIfNextRow

previousRow

private Cell previousRow

isFastSeekingEncoding

private final boolean isFastSeekingEncoding

seekCount

private static LongAdder seekCount

canOptimizeForNonNullColumn

private final boolean canOptimizeForNonNullColumn

readPt

private final long readPt

scannerOrder

private final long scannerOrder

Constructor Details

StoreFileScanner

public StoreFileScanner(StoreFileReader reader,
 HFileScanner hfs,
 boolean useMVCC,
 boolean hasMVCC,
 long readPt,
 long scannerOrder,
 boolean canOptimizeForNonNullColumn,
 boolean isFastSeekingEncoding)

Implements a KeyValueScanner on top of the specified HFileScanner

Parameters:

useMVCC - If true, scanner will filter out updates with MVCC larger than readPt.

readPt - MVCC value to use to filter out the updates newer than this scanner.

hasMVCC - Set to true if underlying store file reader has MVCC info.

scannerOrder - Order of the scanner relative to other scanners. See KeyValueScanner.getScannerOrder().

canOptimizeForNonNullColumn - true if we can make sure there is no null column, otherwise false. This is a hint for optimization.

isFastSeekingEncoding - true if the data block encoding can seek quickly from the beginning of a block (i.e. RIV1), otherwise false. This is a hint for optimization.

Method Details

getScannersForStoreFiles

public static List<StoreFileScanner> getScannersForStoreFiles(Collection<HStoreFile> files,
 boolean cacheBlocks,
 boolean usePread,
 boolean isCompaction,
 boolean useDropBehind,
 long readPt)
                                                       throws IOException

Return an array of scanners corresponding to the given set of store files.

Throws:

IOException

getScannersForStoreFiles

public static List<StoreFileScanner> getScannersForStoreFiles(Collection<HStoreFile> files,
 boolean cacheBlocks,
 boolean usePread,
 boolean isCompaction,
 boolean canUseDrop,
 ScanQueryMatcher matcher,
 long readPt)
                                                       throws IOException

Return an array of scanners corresponding to the given set of store files, And set the ScanQueryMatcher for each store file scanner for further optimization

Throws:

IOException

getScannersForCompaction

public static List<StoreFileScanner> getScannersForCompaction(Collection<HStoreFile> files,
 boolean canUseDropBehind,
 long readPt)
                                                       throws IOException

Get scanners for compaction. We will create a separated reader for each store file to avoid contention with normal read request.

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

peek

public ExtendedCell peek()

Description copied from interface: KeyValueScanner

Look at the next Cell in this scanner, but do not iterate scanner. NOTICE: The returned cell has not been passed into ScanQueryMatcher. So it may not be what the user need.

Specified by:

peek in interface KeyValueScanner

Returns:

the next Cell

next

public ExtendedCell next()
                  throws IOException

Description copied from interface: KeyValueScanner

Return the next Cell in this scanner, iterating the scanner

Specified by:

next in interface KeyValueScanner

Returns:

the next Cell

Throws:

IOException

seek

public boolean seek(ExtendedCell key)
             throws IOException

Description copied from interface: KeyValueScanner

Seek the scanner at or after the specified KeyValue.

Specified by:

seek in interface KeyValueScanner

Parameters:

key - seek value

Returns:

true if scanner has values left, false if end of scanner

Throws:

IOException

reseek

public boolean reseek(ExtendedCell key)
               throws IOException

Description copied from interface: KeyValueScanner

Reseek the scanner at or after the specified KeyValue. This method is guaranteed to seek at or after the required key only if the key comes after the current position of the scanner. Should not be used to seek to a key which may come before the current position.

Specified by:

reseek in interface KeyValueScanner

Parameters:

key - seek value (should be non-null)

Returns:

true if scanner has values left, false if end of scanner

Throws:

IOException

setCurrentCell

protected void setCurrentCell(ExtendedCell newVal)
                       throws IOException

Throws:

IOException

skipKVsNewerThanReadpoint

protected boolean skipKVsNewerThanReadpoint()
                                     throws IOException

Throws:

IOException

close

public void close()

Description copied from interface: KeyValueScanner

Close the KeyValue scanner.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface KeyValueScanner

seekAtOrAfter

public static boolean seekAtOrAfter(HFileScanner s,
 ExtendedCell k)
                             throws IOException

Returns false if not found or if k is after the end.

Throws:

IOException

reseekAtOrAfter

static boolean reseekAtOrAfter(HFileScanner s,
 ExtendedCell k)
                        throws IOException

Throws:

IOException

getScannerOrder

public long getScannerOrder()

Description copied from interface: KeyValueScanner

Get the order of this KeyValueScanner. This is only relevant for StoreFileScanners. This is required for comparing multiple files to find out which one has the latest data. StoreFileScanners are ordered from 0 (oldest) to newest in increasing order.

Specified by:

getScannerOrder in interface KeyValueScanner

See Also:

• KeyValueScanner.getScannerOrder()

requestSeek

public boolean requestSeek(ExtendedCell kv,
 boolean forward,
 boolean useBloom)
                    throws IOException

Pretend we have done a seek but don't do it yet, if possible. The hope is that we find requested columns in more recent files and won't have to seek in older files. Creates a fake key/value with the given row/column and the highest (most recent) possible timestamp we might get from this file. When users of such "lazy scanner" need to know the next KV precisely (e.g. when this scanner is at the top of the heap), they run enforceSeek().

Note that this function does guarantee that the current KV of this scanner will be advanced to at least the given KV. Because of this, it does have to do a real seek in cases when the seek timestamp is older than the highest timestamp of the file, e.g. when we are trying to seek to the next row/column and use OLDEST_TIMESTAMP in the seek key.

Specified by:

requestSeek in interface KeyValueScanner

forward - do a forward-only "reseek" instead of a random-access seek

useBloom - whether to enable multi-column Bloom filter optimization

Throws:

IOException

getReader

StoreFileReader getReader()

getComparator

CellComparator getComparator()

realSeekDone

public boolean realSeekDone()

Description copied from interface: KeyValueScanner

We optimize our store scanners by checking the most recent store file first, so we sometimes pretend we have done a seek but delay it until the store scanner bubbles up to the top of the key-value heap. This method is then used to ensure the top store file scanner has done a seek operation.

Specified by:

realSeekDone in interface KeyValueScanner

enforceSeek

public void enforceSeek()
                 throws IOException

Description copied from interface: KeyValueScanner

Does the real seek operation in case it was skipped by seekToRowCol(KeyValue, boolean) (TODO: Whats this?). Note that this function should be never called on scanners that always do real seek operations (i.e. most of the scanners). The easiest way to achieve this is to call KeyValueScanner.realSeekDone() first.

Specified by:

enforceSeek in interface KeyValueScanner

Throws:

IOException

isFileScanner

public boolean isFileScanner()

Description copied from interface: KeyValueScanner

Returns true if this is a file scanner. Otherwise a memory scanner is assumed.

Specified by:

isFileScanner in interface KeyValueScanner

recordBlockSize

public void recordBlockSize(IntConsumer blockSizeConsumer)

Description copied from interface: KeyValueScanner

Record the size of the current block in bytes, passing as an argument to the blockSizeConsumer. Implementations should ensure that blockSizeConsumer is only called once per block.

Specified by:

recordBlockSize in interface KeyValueScanner

Parameters:

blockSizeConsumer - to be called with block size in bytes, once per block.

getFilePath

public org.apache.hadoop.fs.Path getFilePath()

Specified by:

getFilePath in interface KeyValueScanner

Returns:

the file path if this is a file scanner, otherwise null.

See Also:

• KeyValueScanner.isFileScanner()

getSeekCount

static final long getSeekCount()

instrument

static final void instrument()

shouldUseScanner

public boolean shouldUseScanner(Scan scan,
 HStore store,
 long oldestUnexpiredTS)

Description copied from interface: KeyValueScanner

Allows to filter out scanners (both StoreFile and memstore) that we don't want to use based on criteria such as Bloom filters and timestamp ranges.

Specified by:

shouldUseScanner in interface KeyValueScanner

Parameters:

scan - the scan that we are selecting scanners for

store - the store we are performing the scan on.

oldestUnexpiredTS - the oldest timestamp we are interested in for this query, based on TTL

Returns:

true if the scanner should be included in the query

seekToPreviousRow

public boolean seekToPreviousRow(ExtendedCell originalKey)
                          throws IOException

Description copied from interface: KeyValueScanner

Seek the scanner at the first Cell of the row which is the previous row of specified key

Specified by:

seekToPreviousRow in interface KeyValueScanner

Parameters:

originalKey - seek value

Returns:

true if the scanner at the first valid Cell of previous row, false if not existing such Cell

Throws:

IOException

seekToPreviousRowWithHint

private boolean seekToPreviousRowWithHint()
                                   throws IOException

This variant of the StoreFileScanner#seekToPreviousRow(Cell) method requires one seek and one reseek. This method maintains state in previousRow which only makes sense in the context of a sequential row-by-row reverse scan. previousRow should be reset if that is not the case. The reasoning for why this method is faster than StoreFileScanner#seekToPreviousRowStateless(Cell) is that seeks are slower as they need to start from the beginning of the file, while reseeks go forward from the current position.

Throws:

IOException

seekToPreviousRowWithoutHint

private boolean seekToPreviousRowWithoutHint(Cell originalKey)
                                      throws IOException

This variant of the StoreFileScanner#seekToPreviousRow(Cell) method requires two seeks and one reseek. The extra expense/seek is with the intent of speeding up subsequent calls by using the seekToPreviousRowWithHint() which this method seeds the state for by setting previousRow

Throws:

IOException

seekToPreviousRowStateless

private boolean seekToPreviousRowStateless(ExtendedCell originalKey)
                                    throws IOException

This variant of the StoreFileScanner#seekToPreviousRow(Cell) method requires two seeks. It should be used if the cost for seeking is lower i.e. when using a fast seeking data block encoding like RIV1.

Throws:

IOException

seekBefore

private boolean seekBefore(ExtendedCell seekKey)
                    throws IOException

Throws:

IOException

seekBeforeAndSaveKeyToPreviousRow

private void seekBeforeAndSaveKeyToPreviousRow(ExtendedCell seekKey)
                                        throws IOException

Seeks before the seek target cell and saves the location to previousRow. If there doesn't exist a KV in this file before the seek target cell, reposition the scanner at the beginning of the storefile (in preparation to a reseek at or after the seek key) and set the previousRow to null. If previousRow is ever non-null and then transitions to being null again via this method, that's because there doesn't exist a row before the seek target in the storefile (i.e. we're at the beginning of the storefile)

Throws:

IOException

seekAtOrAfter

private boolean seekAtOrAfter(ExtendedCell seekKey)
                       throws IOException

Throws:

IOException

reseekAtOrAfter

private boolean reseekAtOrAfter(ExtendedCell seekKey)
                         throws IOException

Throws:

IOException

isStillAtSeekTargetAfterSkippingNewerKvs

private boolean isStillAtSeekTargetAfterSkippingNewerKvs(Cell seekKey)
                                                  throws IOException

Throws:

IOException

skipKvsNewerThanReadpointReversed

private boolean skipKvsNewerThanReadpointReversed()
                                           throws IOException

Throws:

IOException

seekToLastRow

public boolean seekToLastRow()
                      throws IOException

Description copied from interface: KeyValueScanner

Seek the scanner at the first KeyValue of last row

Specified by:

seekToLastRow in interface KeyValueScanner

Returns:

true if scanner has values left, false if the underlying data is empty

Throws:

IOException

backwardSeek

public boolean backwardSeek(ExtendedCell key)
                     throws IOException

Description copied from interface: KeyValueScanner

Seek the scanner at or before the row of specified Cell, it firstly tries to seek the scanner at or after the specified Cell, return if peek KeyValue of scanner has the same row with specified Cell, otherwise seek the scanner at the first Cell of the row which is the previous row of specified KeyValue

Specified by:

backwardSeek in interface KeyValueScanner

Parameters:

key - seek KeyValue

Returns:

true if the scanner is at the valid KeyValue, false if such KeyValue does not exist

Throws:

IOException

getNextIndexedKey

public ExtendedCell getNextIndexedKey()

Specified by:

getNextIndexedKey in interface KeyValueScanner

Returns:

the next key in the index, usually the first key of next block OR a key that falls between last key of current block and first key of next block.. see HFileWriterImpl#getMidpoint, or null if not known.

shipped

public void shipped()
             throws IOException

Description copied from interface: Shipper

Called after a batch of rows scanned and set to be returned to client. Any in between cleanup can be done here.

Specified by:

shipped in interface Shipper

Throws:

IOException