Package org.apache.hadoop.hbase.io.hfile.bucket

Class BucketAllocator.IndexStatistics

java.lang.Object
org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator.IndexStatistics
Enclosing class:
BucketAllocator
static class BucketAllocator.IndexStatistics extends Object
Statistics to give a glimpse into the distribution of BucketCache objects. Each configured bucket size, denoted by BucketAllocator.BucketSizeInfo, gets an IndexStatistic. A BucketSizeInfo allocates blocks of a configured size from claimed buckets. If you have a bucket size of 512k, the corresponding BucketSizeInfo will always allocate chunks of 512k at a time regardless of actual request.

Over time, as a BucketSizeInfo gets more allocations, it will claim more buckets from the total pool of completelyFreeBuckets. As blocks are freed from a BucketSizeInfo, those buckets may be returned to the completelyFreeBuckets pool.

The IndexStatistics help visualize how these buckets are currently distributed, through counts of items, bytes, and fullBuckets. Additionally, mismatches between block sizes and bucket sizes can manifest in inefficient cache usage. These typically manifest in three ways:

1. Allocation failures, because block size is larger than max bucket size. These show up in logs and can be alleviated by adding larger bucket sizes if appropriate.
2. Memory fragmentation, because blocks are typically smaller than the bucket size. See fragmentationBytes() for details.
3. Memory waste, because a bucket's itemSize is not a perfect divisor of bucketCapacity. see wastedBytes() for details.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    private int
    completelyFreeBuckets
     
    private long
    fragmentationBytes
     
    private long
    freeCount
     
    private int
    fullBuckets
     
    private long
    itemSize
     
    private long
    totalCount
     
    private long
    usedCount
     
    private long
    wastedBytes
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    IndexStatistics()
     
    IndexStatistics(long free, long used, long itemSize, int fullBuckets, int completelyFreeBuckets, long wastedBytes, long fragmentationBytes)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    completelyFreeBuckets()
    How many buckets are currently claimed by this bucket size but as yet totally unused.
    long
    fragmentationBytes()
    Every time you allocate blocks in these buckets where the block size is less than the bucket size, fragmentation increases by that difference.
    long
    freeBytes()
    How many more bytes can be allocated from the currently claimed blocks of this bucket size
    long
    freeCount()
    How many more items can be allocated from the currently claimed blocks of this bucket size
    int
    fullBuckets()
    How many buckets have been completely filled by blocks for this bucket size.
    long
    itemSize()
    This bucket size can only allocate items of this size, even if the requested allocation size is smaller.
    void
    setTo(long free, long used, long itemSize, int fullBuckets, int completelyFreeBuckets, long wastedBytes, long fragmentationBytes)
     
    long
    totalBytes()
    Combined totalCount() * itemSize()
    long
    totalCount()
    Combined freeCount() + usedCount()
    long
    usedBytes()
    How many bytes are currently taking up space in this bucket size's buckets Note: If your items are less than the bucket size of this bucket, the actual used bytes by items will be lower than this value.
    long
    usedCount()
    How many items are currently taking up space in this bucket size's buckets
    long
    wastedBytes()
    If BucketAllocator.bucketCapacity is not perfectly divisible by this itemSize(), the remainder will be unusable by in buckets of this size.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

  • Constructor Details

    • IndexStatistics

      public IndexStatistics(long free, long used, long itemSize, int fullBuckets, int completelyFreeBuckets, long wastedBytes, long fragmentationBytes)

    • IndexStatistics

      public IndexStatistics()

  • Method Details

    • freeCount

      public long freeCount()
      How many more items can be allocated from the currently claimed blocks of this bucket size

    • usedCount

      public long usedCount()
      How many items are currently taking up space in this bucket size's buckets

    • totalCount

      public long totalCount()
      Combined freeCount() + usedCount()

    • freeBytes

      public long freeBytes()
      How many more bytes can be allocated from the currently claimed blocks of this bucket size

    • usedBytes

      public long usedBytes()
      How many bytes are currently taking up space in this bucket size's buckets Note: If your items are less than the bucket size of this bucket, the actual used bytes by items will be lower than this value. But since a bucket size can only allocate items of a single size, this value is the true number of used bytes. The difference will be counted in fragmentationBytes().

    • totalBytes

      public long totalBytes()
      Combined totalCount() * itemSize()

    • itemSize

      public long itemSize()
      This bucket size can only allocate items of this size, even if the requested allocation size is smaller. The rest goes towards fragmentationBytes().

    • fullBuckets

      public int fullBuckets()
      How many buckets have been completely filled by blocks for this bucket size. These buckets can't accept any more blocks unless some existing are freed.

    • completelyFreeBuckets

      public int completelyFreeBuckets()
      How many buckets are currently claimed by this bucket size but as yet totally unused. These buckets are available for reallocation to other bucket sizes if those fill up.

    • wastedBytes

      public long wastedBytes()
      If BucketAllocator.bucketCapacity is not perfectly divisible by this itemSize(), the remainder will be unusable by in buckets of this size. A high value here may be optimized by trying to choose bucket sizes which can better divide BucketAllocator.bucketCapacity.

    • fragmentationBytes

      public long fragmentationBytes()
      Every time you allocate blocks in these buckets where the block size is less than the bucket size, fragmentation increases by that difference. You can reduce fragmentation by lowering the bucket size so that it is closer to the typical block size. This may have the consequence of bumping some blocks to the next larger bucket size, so experimentation may be needed.

    • setTo

      public void setTo(long free, long used, long itemSize, int fullBuckets, int completelyFreeBuckets, long wastedBytes, long fragmentationBytes)