Package org.apache.hadoop.hbase.io.hfile

Class HFileBlock.FSReaderImpl

java.lang.Object
org.apache.hadoop.hbase.io.hfile.HFileBlock.FSReaderImpl
All Implemented Interfaces:
HFileBlock.FSReader
Enclosing class:
HFileBlock
static class HFileBlock.FSReaderImpl extends Object implements HFileBlock.FSReader
Reads version 2 HFile blocks from the filesystem.

  • Field Details

  • Constructor Details

  • Method Details

    • blockRange

      public HFileBlock.BlockIterator blockRange(long startOffset, long endOffset)
      Description copied from interface: HFileBlock.FSReader
      Creates a block iterator over the given portion of the HFile. The iterator returns blocks starting with offset such that offset <= startOffset < endOffset. Returned blocks are always unpacked. Used when no hfile index available; e.g. reading in the hfile index blocks themselves on file open.
      Specified by:
      blockRange in interface HFileBlock.FSReader
      Parameters:
      startOffset - the offset of the block to start iteration with
      endOffset - the offset to end iteration at (exclusive)
      Returns:
      an iterator of blocks between the two given offsets

    • readAtOffset

      protected boolean readAtOffset(org.apache.hadoop.fs.FSDataInputStream istream, ByteBuff dest, int size, boolean peekIntoNextBlock, long fileOffset, boolean pread) throws IOException
      Does a positional read or a seek and read into the given byte buffer. We need take care that we will call the ByteBuff.release() for every exit to deallocate the ByteBuffers, otherwise the memory leak may happen.
      Parameters:
      dest - destination buffer
      size - size of read
      peekIntoNextBlock - whether to read the next block's on-disk size
      fileOffset - position in the stream to read at
      pread - whether we should do a positional read
      istream - The input source of data
      Returns:
      true to indicate the destination buffer include the next block header, otherwise only include the current block data without the next block header.
      Throws:
      IOException - if any IO error happen.

    • readBlockData

      public HFileBlock readBlockData(long offset, long onDiskSizeWithHeaderL, boolean pread, boolean updateMetrics, boolean intoHeap) throws IOException
      Reads a version 2 block (version 1 blocks not supported and not expected). Tries to do as little memory allocation as possible, using the provided on-disk size.
      Specified by:
      readBlockData in interface HFileBlock.FSReader
      Parameters:
      offset - the offset in the stream to read at
      onDiskSizeWithHeaderL - the on-disk size of the block, including the header, or -1 if unknown; i.e. when iterating over blocks reading in the file metadata info.
      pread - whether to use a positional read
      updateMetrics - whether to update the metrics
      intoHeap - allocate ByteBuff of block from heap or off-heap.
      Returns:
      the newly read block
      Throws:
      IOException
      See Also:

    • checkOnDiskSizeWithHeader

      private boolean checkOnDiskSizeWithHeader(int value)
      Check that value read from a block header seems reasonable, within a large margin of error.
      Returns:
      true if the value is safe to proceed, false otherwise.

    • checkCallerProvidedOnDiskSizeWithHeader

      private boolean checkCallerProvidedOnDiskSizeWithHeader(long value)
      Check that value provided by the calling context seems reasonable, within a large margin of error.
      Returns:
      true if the value is safe to proceed, false otherwise.

    • getCachedHeader

      private ByteBuff getCachedHeader(long offset)
      Check atomic reference cache for this block's header. Cache only good if next read coming through is next in sequence in the block. We read next block's header on the tail of reading the previous block to save a seek. Otherwise, we have to do a seek to read the header before we can pull in the block OR we have to backup the stream because we over-read (the next block's header).
      Returns:
      The cached block header or null if not found.
      See Also:

    • cacheNextBlockHeader

      private void cacheNextBlockHeader(long offset, ByteBuff onDiskBlock, int onDiskSizeWithHeader, int headerLength)
      Save away the next blocks header in atomic reference.
      See Also:

    • invalidateNextBlockHeader

      private void invalidateNextBlockHeader()
      Clear the cached value when its integrity is suspect.

    • getNextBlockOnDiskSize

      private int getNextBlockOnDiskSize(ByteBuff onDiskBlock, int onDiskSizeWithHeader)

    • allocate

      private ByteBuff allocate(int size, boolean intoHeap)

    • readBlockDataInternal

      protected HFileBlock readBlockDataInternal(org.apache.hadoop.fs.FSDataInputStream is, long offset, long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum, boolean updateMetrics, boolean intoHeap) throws IOException
      Reads a version 2 block.
      Parameters:
      offset - the offset in the stream to read at.
      onDiskSizeWithHeaderL - the on-disk size of the block, including the header and checksums if present or -1 if unknown (as a long). Can be -1 if we are doing raw iteration of blocks as when loading up file metadata; i.e. the first read of a new file. Usually non-null gotten from the file index.
      pread - whether to use a positional read
      verifyChecksum - Whether to use HBase checksums. If HBase checksum is switched off, then use HDFS checksum. Can also flip on/off reading same file if we hit a troublesome patch in an hfile.
      updateMetrics - whether need to update the metrics.
      intoHeap - allocate the ByteBuff of block from heap or off-heap.
      Returns:
      the HFileBlock or null if there is a HBase checksum mismatch
      Throws:
      IOException

    • setIncludesMemStoreTS

      public void setIncludesMemStoreTS(boolean includesMemstoreTS)
      Specified by:
      setIncludesMemStoreTS in interface HFileBlock.FSReader

    • setDataBlockEncoder

      public void setDataBlockEncoder(HFileDataBlockEncoder encoder, org.apache.hadoop.conf.Configuration conf)
      Specified by:
      setDataBlockEncoder in interface HFileBlock.FSReader

    • getBlockDecodingContext

      public HFileBlockDecodingContext getBlockDecodingContext()
      Description copied from interface: HFileBlock.FSReader
      Get a decoder for BlockType.ENCODED_DATA blocks from this file.
      Specified by:
      getBlockDecodingContext in interface HFileBlock.FSReader

    • getDefaultBlockDecodingContext

      public HFileBlockDecodingContext getDefaultBlockDecodingContext()
      Description copied from interface: HFileBlock.FSReader
      Get the default decoder for blocks from this file.
      Specified by:
      getDefaultBlockDecodingContext in interface HFileBlock.FSReader

    • validateChecksum

      private boolean validateChecksum(long offset, ByteBuff data, int hdrSize)
      Generates the checksum for the header as well as the data and then validates it. If the block doesn't uses checksum, returns false.
      Returns:
      True if checksum matches, else false.

    • closeStreams

      public void closeStreams() throws IOException
      Description copied from interface: HFileBlock.FSReader
      Closes the backing streams
      Specified by:
      closeStreams in interface HFileBlock.FSReader
      Throws:
      IOException

    • unbufferStream

      public void unbufferStream()
      Description copied from interface: HFileBlock.FSReader
      To close the stream's socket. Note: This can be concurrently called from multiple threads and implementation should take care of thread safety.
      Specified by:
      unbufferStream in interface HFileBlock.FSReader

    • toString

      public String toString()
      Overrides:
      toString in class Object