Package org.apache.hadoop.hbase.io.hfile.bucket

Class TestVerifyBucketCacheFile

java.lang.Object

org.apache.hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile

public class TestVerifyBucketCacheFile
extends Object

Basic test for check file's integrity before start BucketCache in fileIOEngine

Field Summary

Fields

Modifier and Type	Field	Description
(package private) final long	capacitySize
static final HBaseClassTestRule	CLASS_RULE
int	constructedBlockSize
int[]	constructedBlockSizes
(package private) final int	writerQLen
(package private) final int	writeThreads

Constructor Summary

Constructors

Constructor	Description
TestVerifyBucketCacheFile()

Method Summary

Modifier and Type	Method	Description
private void	cacheAndWaitUntilFlushedToBucket(org.apache.hadoop.hbase.io.hfile.bucket.BucketCache cache, org.apache.hadoop.hbase.io.hfile.BlockCacheKey cacheKey, org.apache.hadoop.hbase.io.hfile.Cacheable block)
static Iterable<Object[]>	data()
void	testBucketCacheRecovery()	When using persistent bucket cache, there may be crashes between persisting the backing map and syncing new blocks to the cache file itself, leading to an inconsistent state between the cache keys and the cached data.
private void	testChunkedBackingMapRecovery(int chunkSize, int numBlocks)
void	testCompletelyFilledChunks()
void	testModifiedBucketCacheFileData()	Test whether BucketCache is started normally after modifying the cache file.
void	testModifiedBucketCacheFileTime()	Test whether BucketCache is started normally after modifying the cache file's last modified time.
void	testPartiallyFilledChunks()
void	testRetrieveFromFile()	Test cache file or persistence file does not exist whether BucketCache starts normally (1) Start BucketCache and add some blocks, then shutdown BucketCache and persist cache to file.
void	testRetrieveFromFileAfterDelete()
void	testSingleChunk()
private void	waitUntilFlushedToBucket(org.apache.hadoop.hbase.io.hfile.bucket.BucketCache cache, org.apache.hadoop.hbase.io.hfile.BlockCacheKey cacheKey)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

CLASS_RULE

public static final HBaseClassTestRule CLASS_RULE

constructedBlockSize

public int constructedBlockSize

constructedBlockSizes

public int[] constructedBlockSizes

capacitySize

final long capacitySize

See Also:

• Constant Field Values

writeThreads

final int writeThreads

See Also:

• Constant Field Values

writerQLen

final int writerQLen

See Also:

• Constant Field Values

Constructor Details

TestVerifyBucketCacheFile

public TestVerifyBucketCacheFile()

Method Details

data

public static Iterable<Object[]> data()

testRetrieveFromFile

public void testRetrieveFromFile()
                          throws Exception

Test cache file or persistence file does not exist whether BucketCache starts normally (1) Start BucketCache and add some blocks, then shutdown BucketCache and persist cache to file. Restart BucketCache and it can restore cache from file. (2) Delete bucket cache file after shutdown BucketCache. Restart BucketCache and it can't restore cache from file, the cache file and persistence file would be deleted before BucketCache start normally. (3) Delete persistence file after shutdown BucketCache. Restart BucketCache and it can't restore cache from file, the cache file and persistence file would be deleted before BucketCache start normally.

Throws:

Exception - the exception

testRetrieveFromFileAfterDelete

public void testRetrieveFromFileAfterDelete()
                                     throws Exception

Throws:

Exception

testModifiedBucketCacheFileData

public void testModifiedBucketCacheFileData()
                                     throws Exception

Test whether BucketCache is started normally after modifying the cache file. Start BucketCache and add some blocks, then shutdown BucketCache and persist cache to file. Restart BucketCache after modify cache file's data, and it can't restore cache from file, the cache file and persistence file would be deleted before BucketCache start normally.

Throws:

Exception - the exception

testModifiedBucketCacheFileTime

public void testModifiedBucketCacheFileTime()
                                     throws Exception

Test whether BucketCache is started normally after modifying the cache file's last modified time. First Start BucketCache and add some blocks, then shutdown BucketCache and persist cache to file. Then Restart BucketCache after modify cache file's last modified time. HBASE-XXXX has modified persistence cache such that now we store extra 8 bytes at the end of each block in the cache, representing the nanosecond time the block has been cached. So in the event the cache file has failed checksum verification during loading time, we go through all the cached blocks in the cache map and validate the cached time long between what is in the map and the cache file. If that check fails, we pull the cache key entry out of the map. Since in this test we are only modifying the access time to induce a checksum error, the cache file content is still valid and the extra verification should validate that all cache keys in the map are still recoverable from the cache.

Throws:

Exception - the exception

testBucketCacheRecovery

public void testBucketCacheRecovery()
                             throws Exception

When using persistent bucket cache, there may be crashes between persisting the backing map and syncing new blocks to the cache file itself, leading to an inconsistent state between the cache keys and the cached data. This is to make sure the cache keys are updated accordingly, and the keys that are still valid do succeed in retrieve related block data from the cache without any corruption.

Throws:

Exception - the exception

testSingleChunk

public void testSingleChunk()
                     throws Exception

Throws:

Exception

testCompletelyFilledChunks

public void testCompletelyFilledChunks()
                                throws Exception

Throws:

Exception

testPartiallyFilledChunks

public void testPartiallyFilledChunks()
                               throws Exception

Throws:

Exception

testChunkedBackingMapRecovery

private void testChunkedBackingMapRecovery(int chunkSize,
 int numBlocks)
                                    throws Exception

Throws:

Exception

waitUntilFlushedToBucket

private void waitUntilFlushedToBucket(org.apache.hadoop.hbase.io.hfile.bucket.BucketCache cache,
 org.apache.hadoop.hbase.io.hfile.BlockCacheKey cacheKey)
                               throws InterruptedException

Throws:

InterruptedException

cacheAndWaitUntilFlushedToBucket

private void cacheAndWaitUntilFlushedToBucket(org.apache.hadoop.hbase.io.hfile.bucket.BucketCache cache,
 org.apache.hadoop.hbase.io.hfile.BlockCacheKey cacheKey,
 org.apache.hadoop.hbase.io.hfile.Cacheable block)
                                       throws InterruptedException

Throws:

InterruptedException