Package org.apache.hadoop.hbase.io.hfile
package org.apache.hadoop.hbase.io.hfile
Provides implementations of Enabling
See the HBase Reference Guide Enable BucketCache.
HFile and HFile
BlockCache. Caches are configured (and instantiated)
by CacheConfig. See head of the
CacheConfig class for constants that define
cache options and configuration keys to use setting cache options. Cache implementations
include the default, native on-heap LruBlockCache and a
BucketCache that has a bunch of deploy formats
including acting as a L2 for LruBlockCache -- when a block is evicted from LruBlockCache, it
goes to the BucketCache and when we search a block, we look in both places -- or, the
most common deploy type,
using CombinedBlockCache, BucketCache is used as
a host for data blocks with meta blocks in an instance of LruBlockCache. BucketCache
can also be onheap, offheap, and file-backed.
Which BlockCache should I use?
By default LruBlockCache is on. If you would like to cache more, and offheap (offheap usually means less GC headache), try enabling * BucketCache. Fetching will always be slower when fetching from BucketCache but latencies tend to be less erratic over time (roughly because GC is less). See Nick Dimiduk's BlockCache 101 for some numbers.Enabling BucketCache
See the HBase Reference Guide Enable BucketCache.-
ClassDescriptionSnapshot of block cache age in cache.Block cache interface.Enum of all built in external block caches.Cache Key for use with implementations of
BlockCacheIterator over an array of BlockCache CachedBlocks.Utilty for aggregating counts in CachedBlocks and toString/toJSON CachedBlocks and BlockCaches.Little data structure to hold counts for a file.Use one of these to keep a running account of cached blocks by file.Allows for defining different compression rate predicates on its implementing classes.Various types of HFile blocks.BlockWithScanInfo is wrapper class for HFileBlock with other attributes.Cacheable is an interface that allows for an object to be cached.CacheableDeserializer<T extends Cacheable>Interface for a deserializer.This class is used to manage the identifiers forCacheableDeserializer.Stores all of the cache objects and configuration for a single HFile.Class that implements cache metrics.Utility methods to compute and validate checksums.CombinedBlockCache is an abstraction layer that combinesFirstLevelBlockCacheandBucketCache.A Bloom filter implementation built on top ofBloomFilterChunk, encapsulating a set of fixed-size Bloom filters written out at the time ofHFilegeneration into the data block stream, and loaded on demand at query time.Adds methods required for writing a compound Bloom filter to the data section of anHFileto theCompoundBloomFilterclass.A Bloom filter chunk enqueued for writingThis exception is thrown when attempts to read an HFile fail due to corruption or truncation issues.TheByteBuffAllocatorwon't allocate pooled heapByteBuffnow; at the same time, if allocate an off-heapByteBufffrom allocator, then it must be a pooled one.In-memory BlockCache that may be backed by secondary layer(s).TheHFilehas a fixed trailer which contains offsets to other variable parts of the file.File format for hbase.An abstraction used by the block index.An interface used by clients to open and iterate anHFile.API required to write anHFileThis variety of ways to construct writers is used throughout the code, and we want to be able to swap writer implementations.Cacheable Blocks of anHFileversion 2 file.Iterator for readingHFileBlocks in load-on-open-section, such as root data index block, meta index block, file info block etc.Something that can be written into a block.An HFile block reader with iteration ability.Reads version 2 HFile blocks from the filesystem.Data-structure to use caching the header of the NEXT block.Unified version 2HFileblock writer.Provides functionality to write (HFileBlockIndex.BlockIndexWriter) and read BlockIndexReader single-level and multi-level block indexes.A single chunk of the block index in the process of writing.The reader will always hold the root level index in the memory.Writes the block index into the output stream.An implementation of the BlockIndexReader that deals with block keys which are plain byte[] like MetaBlock or the Bloom Block for ROW bloom.An implementation of the BlockIndexReader that deals with block keys which are the key part of a cell like the Data block index or the ROW_COL bloom blocks This needs a comparator to work with the CellsRead-only HFile Context Information.A builder that helps in building up the HFileContextControls what kind of data block encoding is used.Do different kinds of data block encoding according to column family options.Controls what kind of index block encoding is used.Do different kinds of index block encoding according to column family options.Metadata Map of attributes for HFile written out as HFile Trailer.Implementation ofHFile.Readerto deal with pread.Implements pretty-printing functionality forHFiles.Holds a Histogram and supporting min/max and range buckets for analyzing distribution of key bytes, value bytes, row bytes, and row columns.Simple reporter which collects registered histograms for printing to an output stream inHFilePrettyPrinter.SimpleReporter.report().A builder forHFilePrettyPrinter.SimpleReporterinstances.Implementation that can handle all hfile versions ofHFile.Reader.Scanner that operates on encoded data blocks.An exception thrown when an operation requiring a scanner to be seeked is invoked on a scanner that is not seeked.A scanner allows you to position yourself within a HFile and scan through it.Implementation ofHFile.Readerto deal with stream read do not perform any prefetch operations (HFilePreadReader will do this).Common functionality needed by all versions ofHFilewriters.An on heap block cache implementation extended LruBlockCache and only cache index block.A way to write "inline" blocks into anHFile.Thrown when an invalid HFile format is detectedThis realisation improve performance of classical LRU cache up to 3 times via reduce GC job.A block cache implementation that is memory-aware usingHeapSize, memory-bound using an LRU eviction algorithm, and concurrent: backed by aConcurrentHashMapand with a non-blocking eviction thread giving constant-timeLruBlockCache.cacheBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, org.apache.hadoop.hbase.io.hfile.Cacheable, boolean)andLruBlockCache.getBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, boolean, boolean, boolean)operations.Represents an entry in theLruBlockCache.A memory-bound queue that will grow until an element brings total size >= maxSize.Class to store blocks into memcached.Class to encode and decode an HFileBlock to and from memecached's resulting byte arrays.Does not perform any kind of encoding/decoding.Does not perform any kind of encoding/decoding.This BlockCompressedSizePredicator implementation adjusts the block size limit based on the compression rate of the block contents read so far.Carries the information on some of the meta data about the HFile ReaderA builder that helps in building up the ReaderContextBlockCache which is resizable.TheByteBuffAllocatorwon't allocate pooled heapByteBuffnow; at the same time, if allocate an off-heapByteBufffrom allocator, then it must be a pooled one.A block cache that is memory-aware usingHeapSize, memory bounded using the W-TinyLFU eviction algorithm, and concurrent.This BlockCompressedSizePredicator implementation doesn't actually performs any predicate and simply returns true onshouldFinishBlock.