Package org.apache.hadoop.hbase.regionserver.compactions

Class Compactor<T extends CellSink>

java.lang.Object

org.apache.hadoop.hbase.regionserver.compactions.Compactor<T>

Direct Known Subclasses:

AbstractMultiOutputCompactor, DefaultCompactor

@Private
public abstract class Compactor<T extends CellSink>
extends Object

A compactor is a compaction algorithm associated a given policy. Base class also contains reusable parts for implementing compactors (what is common and what isn't is evolving).

Compactions might be concurrent against a given store and the Compactor is shared among them. Do not put mutable state into class fields. All Compactor class fields should be final or effectively final. 'keepSeqIdPeriod' is an exception to this rule because unit tests may set it.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static interface	Compactor.CellSinkFactory<S>
protected static class	Compactor.FileDetails	The sole reason this class exists is that java has no ref/out/pointer parameters.
protected static interface	Compactor.InternalScannerFactory

Field Summary

Fields

Modifier and Type	Field	Description
protected static final long	COMPACTION_PROGRESS_LOG_INTERVAL
protected final int	compactionKVMax
protected final long	compactScannerSizeLimit
protected final org.apache.hadoop.conf.Configuration	conf
protected final Compactor.InternalScannerFactory	defaultScannerFactory
protected final boolean	dropCacheMajor
protected final boolean	dropCacheMinor
protected int	keepSeqIdPeriod	specify how many days to keep MVCC values during major compaction
private static final org.slf4j.Logger	LOG
protected static final String	MAJOR_COMPACTION_DROP_CACHE
protected final Compression.Algorithm	majorCompactionCompression
protected static final String	MINOR_COMPACTION_DROP_CACHE
protected final Compression.Algorithm	minorCompactionCompression
private final Set<CompactionProgress>	progressSet
protected final HStore	store

Constructor Summary

Constructors

Constructor	Description
Compactor(org.apache.hadoop.conf.Configuration conf, HStore store)

Method Summary

Modifier and Type	Method	Description
protected abstract void	abortWriter(T writer)
protected abstract List<org.apache.hadoop.fs.Path>	commitWriter(T writer, Compactor.FileDetails fd, CompactionRequestImpl request)
protected final List<org.apache.hadoop.fs.Path>	compact(CompactionRequestImpl request, Compactor.InternalScannerFactory scannerFactory, Compactor.CellSinkFactory<T> sinkFactory, ThroughputController throughputController, User user)
private List<StoreFileScanner>	createFileScanners(Collection<HStoreFile> filesToCompact, long smallestReadPoint, boolean useDropBehind)	Creates file scanners for compaction.
protected final CreateStoreFileWriterParams	createParams(Compactor.FileDetails fd, boolean shouldDropBehind, boolean major, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)
protected InternalScanner	createScanner(HStore store, ScanInfo scanInfo, List<StoreFileScanner> scanners, long smallestReadPoint, long earliestPutTs, byte[] dropDeletesFromRow, byte[] dropDeletesToRow)
protected InternalScanner	createScanner(HStore store, ScanInfo scanInfo, List<StoreFileScanner> scanners, ScanType scanType, long smallestReadPoint, long earliestPutTs)
protected final StoreFileWriter	createWriter(Compactor.FileDetails fd, boolean shouldDropBehind, boolean major, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)	Creates a writer for a new file.
protected final StoreFileWriter	createWriter(Compactor.FileDetails fd, boolean shouldDropBehind, String fileStoragePolicy, boolean major, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)
private Compactor.FileDetails	getFileDetails(Collection<HStoreFile> filesToCompact, boolean allFiles, boolean major)	Extracts some details about the files to compact that are commonly needed by compactors.
CompactionProgress	getProgress()	Return the aggregate progress for all currently active compactions.
private long	getSmallestReadPoint()
boolean	isCompacting()
protected boolean	performCompaction(Compactor.FileDetails fd, InternalScanner scanner, CellSink writer, long smallestReadPoint, boolean cleanSeqId, ThroughputController throughputController, CompactionRequestImpl request, CompactionProgress progress)	Performs the compaction.
private InternalScanner	postCompactScannerOpen(CompactionRequestImpl request, ScanType scanType, InternalScanner scanner, User user)	Calls coprocessor, if any, to create scanners - after normal scanner creation.
private ScanInfo	preCompactScannerOpen(CompactionRequestImpl request, ScanType scanType, User user)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

COMPACTION_PROGRESS_LOG_INTERVAL

protected static final long COMPACTION_PROGRESS_LOG_INTERVAL

See Also:

• Constant Field Values

conf

protected final org.apache.hadoop.conf.Configuration conf

store

protected final HStore store

compactionKVMax

protected final int compactionKVMax

compactScannerSizeLimit

protected final long compactScannerSizeLimit

majorCompactionCompression

protected final Compression.Algorithm majorCompactionCompression

minorCompactionCompression

protected final Compression.Algorithm minorCompactionCompression

keepSeqIdPeriod

protected int keepSeqIdPeriod

specify how many days to keep MVCC values during major compaction

MAJOR_COMPACTION_DROP_CACHE

protected static final String MAJOR_COMPACTION_DROP_CACHE

See Also:

• Constant Field Values

MINOR_COMPACTION_DROP_CACHE

protected static final String MINOR_COMPACTION_DROP_CACHE

See Also:

• Constant Field Values

dropCacheMajor

protected final boolean dropCacheMajor

dropCacheMinor

protected final boolean dropCacheMinor

progressSet

private final Set<CompactionProgress> progressSet

defaultScannerFactory

protected final Compactor.InternalScannerFactory defaultScannerFactory

Constructor Details

Compactor

Compactor(org.apache.hadoop.conf.Configuration conf,
 HStore store)

Method Details

getFileDetails

private Compactor.FileDetails getFileDetails(Collection<HStoreFile> filesToCompact,
 boolean allFiles,
 boolean major)
                                      throws IOException

Extracts some details about the files to compact that are commonly needed by compactors.

Parameters:

filesToCompact - Files.

allFiles - Whether all files are included for compaction

Returns:

The result.

Throws:

IOException

createFileScanners

private List<StoreFileScanner> createFileScanners(Collection<HStoreFile> filesToCompact,
 long smallestReadPoint,
 boolean useDropBehind)
                                           throws IOException

Creates file scanners for compaction.

Parameters:

filesToCompact - Files.

Returns:

Scanners.

Throws:

IOException

getSmallestReadPoint

private long getSmallestReadPoint()

createParams

protected final CreateStoreFileWriterParams createParams(Compactor.FileDetails fd,
 boolean shouldDropBehind,
 boolean major,
 Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)

createWriter

protected final StoreFileWriter createWriter(Compactor.FileDetails fd,
 boolean shouldDropBehind,
 boolean major,
 Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)
                                      throws IOException

Creates a writer for a new file.

Parameters:

fd - The file details.

Returns:

Writer for a new StoreFile

Throws:

IOException - if creation failed

createWriter

protected final StoreFileWriter createWriter(Compactor.FileDetails fd,
 boolean shouldDropBehind,
 String fileStoragePolicy,
 boolean major,
 Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)
                                      throws IOException

Throws:

IOException

preCompactScannerOpen

private ScanInfo preCompactScannerOpen(CompactionRequestImpl request,
 ScanType scanType,
 User user)
                                throws IOException

Throws:

IOException

postCompactScannerOpen

private InternalScanner postCompactScannerOpen(CompactionRequestImpl request,
 ScanType scanType,
 InternalScanner scanner,
 User user)
                                        throws IOException

Calls coprocessor, if any, to create scanners - after normal scanner creation.

Parameters:

request - Compaction request.

scanType - Scan type.

scanner - The default scanner created for compaction.

Returns:

Scanner scanner to use (usually the default); null if compaction should not proceed.

Throws:

IOException

compact

protected final List<org.apache.hadoop.fs.Path> compact(CompactionRequestImpl request,
 Compactor.InternalScannerFactory scannerFactory,
 Compactor.CellSinkFactory<T> sinkFactory,
 ThroughputController throughputController,
 User user)
                                                 throws IOException

Throws:

IOException

commitWriter

protected abstract List<org.apache.hadoop.fs.Path> commitWriter(T writer,
 Compactor.FileDetails fd,
 CompactionRequestImpl request)
                                                         throws IOException

Throws:

IOException

abortWriter

protected abstract void abortWriter(T writer)
                             throws IOException

Throws:

IOException

performCompaction

protected boolean performCompaction(Compactor.FileDetails fd,
 InternalScanner scanner,
 CellSink writer,
 long smallestReadPoint,
 boolean cleanSeqId,
 ThroughputController throughputController,
 CompactionRequestImpl request,
 CompactionProgress progress)
                             throws IOException

Performs the compaction.

Parameters:

fd - FileDetails of cell sink writer

scanner - Where to read from.

writer - Where to write to.

smallestReadPoint - Smallest read point.

cleanSeqId - When true, remove seqId(used to be mvcc) value which is <= smallestReadPoint

request - compaction request.

progress - Progress reporter.

Returns:

Whether compaction ended; false if it was interrupted for some reason.

Throws:

IOException

createScanner

protected InternalScanner createScanner(HStore store,
 ScanInfo scanInfo,
 List<StoreFileScanner> scanners,
 ScanType scanType,
 long smallestReadPoint,
 long earliestPutTs)
                                 throws IOException

Parameters:

store - store

scanners - Store file scanners.

scanType - Scan type.

smallestReadPoint - Smallest MVCC read point.

earliestPutTs - Earliest put across all files.

Returns:

A compaction scanner.

Throws:

IOException

createScanner

protected InternalScanner createScanner(HStore store,
 ScanInfo scanInfo,
 List<StoreFileScanner> scanners,
 long smallestReadPoint,
 long earliestPutTs,
 byte[] dropDeletesFromRow,
 byte[] dropDeletesToRow)
                                 throws IOException

Parameters:

store - The store.

scanners - Store file scanners.

smallestReadPoint - Smallest MVCC read point.

earliestPutTs - Earliest put across all files.

dropDeletesFromRow - Drop deletes starting with this row, inclusive. Can be null.

dropDeletesToRow - Drop deletes ending with this row, exclusive. Can be null.

Returns:

A compaction scanner.

Throws:

IOException

getProgress

public CompactionProgress getProgress()

Return the aggregate progress for all currently active compactions.

isCompacting

public boolean isCompacting()