Package org.apache.hadoop.hbase.io.hfile

Class ChecksumUtil

java.lang.Object
org.apache.hadoop.hbase.io.hfile.ChecksumUtil
@Private public class ChecksumUtil extends Object
Utility methods to compute and validate checksums.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    CHECKSUM_BUF_SIZE
     
    private static boolean
    generateExceptions
    This is used by unit tests to make checksum failures throw an exception instead of returning null.
    static final org.slf4j.Logger
    LOG
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    ChecksumUtil()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    (package private) static void
    generateChecksums(byte[] indata, int startOffset, int endOffset, byte[] outdata, int outOffset, ChecksumType checksumType, int bytesPerChecksum)
    Generates a checksum for all the data in indata.
    static void
    generateExceptionForChecksumFailureForTest(boolean value)
    Mechanism to throw an exception in case of hbase checksum failure.
    (package private) static long
    numBytes(long datasize, int bytesPerChecksum)
    Returns the number of bytes needed to store the checksums for a specified data size
    (package private) static long
    numChunks(long datasize, int bytesPerChecksum)
    Returns the number of checksum chunks needed to store the checksums for a specified data size
    (package private) static boolean
    validateChecksum(ByteBuff buf, String pathName, long offset, int hdrSize)
    Validates that the data in the specified HFileBlock matches the checksum.
    private static boolean
    verifyChunkedSums(org.apache.hadoop.util.DataChecksum dataChecksum, ByteBuff data, ByteBuff checksums, String pathName)
    Like the hadoop's DataChecksum.verifyChunkedSums(ByteBuffer, ByteBuffer, String, long), this method will also verify checksum of each chunk in data.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

    • CHECKSUM_BUF_SIZE

      public static final int CHECKSUM_BUF_SIZE
      See Also:

    • generateExceptions

      private static boolean generateExceptions
      This is used by unit tests to make checksum failures throw an exception instead of returning null. Returning a null value from checksum validation will cause the higher layer to retry that read with hdfs-level checksums. Instead, we would like checksum failures to cause the entire unit test to fail.

  • Constructor Details

  • Method Details

    • generateChecksums

      static void generateChecksums(byte[] indata, int startOffset, int endOffset, byte[] outdata, int outOffset, ChecksumType checksumType, int bytesPerChecksum) throws IOException
      Generates a checksum for all the data in indata. The checksum is written to outdata.
      Parameters:
      indata - input data stream
      startOffset - starting offset in the indata stream from where to compute checkums from
      endOffset - ending offset in the indata stream upto which checksums needs to be computed
      outdata - the output buffer where checksum values are written
      outOffset - the starting offset in the outdata where the checksum values are written
      checksumType - type of checksum
      bytesPerChecksum - number of bytes per checksum value
      Throws:
      IOException

    • verifyChunkedSums

      private static boolean verifyChunkedSums(org.apache.hadoop.util.DataChecksum dataChecksum, ByteBuff data, ByteBuff checksums, String pathName)
      Like the hadoop's DataChecksum.verifyChunkedSums(ByteBuffer, ByteBuffer, String, long), this method will also verify checksum of each chunk in data. the difference is: this method can accept ByteBuff as arguments, we can not add it in hadoop-common so defined here.
      Parameters:
      dataChecksum - to calculate the checksum.
      data - as the input
      checksums - to compare
      pathName - indicate that the data is read from which file.
      Returns:
      a flag indicate the checksum match or mismatch.
      See Also:
      • DataChecksum.verifyChunkedSums(ByteBuffer, ByteBuffer, String, long)

    • validateChecksum

      static boolean validateChecksum(ByteBuff buf, String pathName, long offset, int hdrSize)
      Validates that the data in the specified HFileBlock matches the checksum. Generates the checksums for the data and then validate that it matches those stored in the end of the data.
      Parameters:
      buf - Contains the data in following order: HFileBlock header, data, checksums.
      pathName - Path of the HFile to which the data belongs. Only used for logging.
      offset - offset of the data being validated. Only used for logging.
      hdrSize - Size of the block header in data. Only used for logging.
      Returns:
      True if checksum matches, else false.

    • numBytes

      static long numBytes(long datasize, int bytesPerChecksum)
      Returns the number of bytes needed to store the checksums for a specified data size
      Parameters:
      datasize - number of bytes of data
      bytesPerChecksum - number of bytes in a checksum chunk
      Returns:
      The number of bytes needed to store the checksum values

    • numChunks

      static long numChunks(long datasize, int bytesPerChecksum)
      Returns the number of checksum chunks needed to store the checksums for a specified data size
      Parameters:
      datasize - number of bytes of data
      bytesPerChecksum - number of bytes in a checksum chunk
      Returns:
      The number of checksum chunks

    • generateExceptionForChecksumFailureForTest

      public static void generateExceptionForChecksumFailureForTest(boolean value)
      Mechanism to throw an exception in case of hbase checksum failure. This is used by unit tests only.
      Parameters:
      value - Setting this to true will cause hbase checksum verification failures to generate exceptions.