Package org.apache.hadoop.hbase.io.hfile.bucket

Class BucketEntry

java.lang.Object

org.apache.hadoop.hbase.io.hfile.bucket.BucketEntry

All Implemented Interfaces:

HBaseReferenceCounted, org.apache.hbase.thirdparty.io.netty.util.ReferenceCounted

@Private
public class BucketEntry
extends Object
implements HBaseReferenceCounted

Item in cache. We expect this to be where most memory goes. Java uses 8 bytes just for object headers; after this, we want to use as little as possible - so we only use 8 bytes, but in order to do so we end up messing around with all this Java casting stuff. Offset stored as 5 bytes that make up the long. Doubt we'll see devices this big for ages. Offsets are divided by 256. So 5 bytes gives us 256TB or so.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
(package private) static interface	BucketEntry.BucketEntryHandler<T>

Field Summary

Fields

Modifier and Type	Field	Description
private long	accessCounter
(package private) final ByteBuffAllocator	allocator
private long	cachedTime	Time this block was cached.
(package private) static final Comparator<BucketEntry>	COMPARATOR
(package private) byte	deserializerIndex	The index of the deserializer that can deserialize this BucketEntry content.
private int	length
(package private) final AtomicBoolean	markedAsEvicted
private byte	offset1
private int	offsetBase
private int	onDiskSizeWithHeader
private BlockPriority	priority
private final RefCnt	refCnt	The RefCnt means how many paths are referring the BucketEntry, there are two cases: 1.If the IOEngine.usesSharedMemory() is false(eg.FileIOEngine),the refCnt is always 1 until this BucketEntry is evicted from BucketCache.backingMap.Even if the corresponding HFileBlock is referenced by RPC reading, the refCnt should not increase.

Constructor Summary

Constructors

Constructor	Description
BucketEntry(long offset, int length, int onDiskSizeWithHeader, long accessCounter, boolean inMemory, Function<BucketEntry,ByteBuffAllocator.Recycler> createRecycler, ByteBuffAllocator allocator)
BucketEntry(long offset, int length, int onDiskSizeWithHeader, long accessCounter, long cachedTime, boolean inMemory, Function<BucketEntry,ByteBuffAllocator.Recycler> createRecycler, ByteBuffAllocator allocator)

Method Summary

Modifier and Type	Method	Description
(package private) void	access(long accessCounter)	Block has been accessed.
(package private) CacheableDeserializer<Cacheable>	deserializerReference()
(package private) long	getAccessCounter()
long	getCachedTime()
int	getLength()
int	getOnDiskSizeWithHeader()
BlockPriority	getPriority()
(package private) boolean	isRpcRef()	Check whether have some RPC patch referring this block. For IOEngine.usesSharedMemory() is true(eg.ByteBufferIOEngine), there're two case: 1.
(package private) boolean	markAsEvicted()	The BucketCache will try to release its reference to this BucketEntry many times.
(package private) long	offset()
int	refCnt()
boolean	release()	We've three cases to release refCnt now: 1.
BucketEntry	retain()
(package private) void	setDeserializerReference(CacheableDeserializer<Cacheable> deserializer)
private void	setOffset(long value)
(package private) <T> T	withWriteLock(IdReadWriteLock<Long> offsetLock, BucketEntry.BucketEntryHandler<T> handler)
(package private) Cacheable	wrapAsCacheable(ByteBuffer[] buffers)
(package private) Cacheable	wrapAsCacheable(ByteBuff buf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.nio.HBaseReferenceCounted

release, retain, touch, touch

Field Details

COMPARATOR

static final Comparator<BucketEntry> COMPARATOR

offsetBase

private int offsetBase

length

private int length

onDiskSizeWithHeader

private int onDiskSizeWithHeader

offset1

private byte offset1

deserializerIndex

byte deserializerIndex

The index of the deserializer that can deserialize this BucketEntry content. See CacheableDeserializerIdManager for hosting of index to serializers.

accessCounter

private volatile long accessCounter

priority

private BlockPriority priority

refCnt

private final RefCnt refCnt

 The RefCnt means how many paths are referring the BucketEntry, there are two cases:
 1.If the IOEngine.usesSharedMemory() is false(eg.FileIOEngine),the refCnt is
   always 1 until this BucketEntry is evicted from BucketCache.backingMap.Even
   if the corresponding HFileBlock is referenced by RPC reading, the refCnt should not
   increase.

 2.If the IOEngine.usesSharedMemory() is true(eg.ByteBufferIOEngine),each RPC
   reading path is considering as one path, the BucketCache.backingMap reference is
   also considered a path. NOTICE that if two read RPC path hit the same BucketEntry,
   then the HFileBlocks the two RPC referred will share the same refCnt instance with
   the BucketEntry,so the refCnt will increase or decrease as the following:
   (1) when writerThread flush the block into IOEngine and add the bucketEntry into backingMap,
       the refCnt ++;
   (2) If BucketCache evict the block and move the bucketEntry out of backingMap, the refCnt--;
       it usually happen when HFile is closing or someone call the clearBucketCache by force.
   (3) The read RPC path start to refer the block which is backend by the memory area in
       bucketEntry, then refCnt ++ ;
   (4) The read RPC patch shipped the response, and release the block. then refCnt--;
    Once the refCnt decrease to zero, then the BucketAllocator will free the block area.
 

markedAsEvicted

final AtomicBoolean markedAsEvicted

allocator

final ByteBuffAllocator allocator

cachedTime

private long cachedTime

Time this block was cached. Presumes we are created just before we are added to the cache.

Constructor Details

BucketEntry

BucketEntry(long offset,
 int length,
 int onDiskSizeWithHeader,
 long accessCounter,
 boolean inMemory,
 Function<BucketEntry,ByteBuffAllocator.Recycler> createRecycler,
 ByteBuffAllocator allocator)

Parameters:

createRecycler - used to free this BucketEntry when refCnt becoming 0. NOTICE that ByteBuffAllocator.NONE could only be used for test.

BucketEntry

BucketEntry(long offset,
 int length,
 int onDiskSizeWithHeader,
 long accessCounter,
 long cachedTime,
 boolean inMemory,
 Function<BucketEntry,ByteBuffAllocator.Recycler> createRecycler,
 ByteBuffAllocator allocator)

Method Details

offset

long offset()

setOffset

private void setOffset(long value)

getLength

public int getLength()

deserializerReference

CacheableDeserializer<Cacheable> deserializerReference()

setDeserializerReference

void setDeserializerReference(CacheableDeserializer<Cacheable> deserializer)

getAccessCounter

long getAccessCounter()

access

void access(long accessCounter)

Block has been accessed. Update its local access counter.

getPriority

public BlockPriority getPriority()

getCachedTime

public long getCachedTime()

getOnDiskSizeWithHeader

public int getOnDiskSizeWithHeader()

markAsEvicted

boolean markAsEvicted()

The BucketCache will try to release its reference to this BucketEntry many times. we must make sure the idempotent, otherwise it'll decrease the RPC's reference count in advance, then for RPC memory leak happen.

Returns:

true if we deallocate this entry successfully.

isRpcRef

boolean isRpcRef()

Check whether have some RPC patch referring this block.
For IOEngine.usesSharedMemory() is true(eg.ByteBufferIOEngine), there're two case:
1. If current refCnt is greater than 1, there must be at least one referring RPC path;
2. If current refCnt is equal to 1 and the markedAtEvicted is true, the it means backingMap has released its reference, the remaining reference can only be from RPC path.
We use this check to decide whether we can free the block area: when cached size exceed the acceptable size, our eviction policy will choose those stale blocks without any RPC reference and the RPC referred block will be excluded.

For IOEngine.usesSharedMemory() is false(eg.FileIOEngine), refCnt is always 1 until it is evicted from BucketCache.backingMap, so isRpcRef() is always return false.

Returns:

true to indicate there're some RPC referring the block.

wrapAsCacheable

Cacheable wrapAsCacheable(ByteBuffer[] buffers)
                   throws IOException

Throws:

IOException

wrapAsCacheable

Cacheable wrapAsCacheable(ByteBuff buf)
                   throws IOException

Throws:

IOException

withWriteLock

<T> T withWriteLock(IdReadWriteLock<Long> offsetLock,
 BucketEntry.BucketEntryHandler<T> handler)

refCnt

public int refCnt()

Specified by:

refCnt in interface org.apache.hbase.thirdparty.io.netty.util.ReferenceCounted

retain

public BucketEntry retain()

Specified by:

retain in interface org.apache.hbase.thirdparty.io.netty.util.ReferenceCounted

release

public boolean release()

We've three cases to release refCnt now:
1. BucketCache#evictBlock, it will release the backingMap's reference by force because we're closing file or clear the bucket cache or some corruption happen. when all rpc references gone, then free the area in bucketAllocator.
2. BucketCache#returnBlock . when rpc shipped, we'll release the block, only when backingMap also release its refCnt (case.1 will do this) and no other rpc reference, then it will free the area in bucketAllocator.
3.evict those block without any rpc reference if cache size exceeded. we'll only free those blocks with zero rpc reference count.

Specified by:

release in interface org.apache.hbase.thirdparty.io.netty.util.ReferenceCounted

Returns:

true to indicate we've decreased to zero and do the de-allocation.