Package org.apache.hadoop.hbase

Class TestPartialResultsFromClientSide

java.lang.Object

org.apache.hadoop.hbase.TestPartialResultsFromClientSide

public class TestPartialResultsFromClientSide
extends Object

These tests are focused on testing how partial results appear to a client. Partial results are Results that contain only a portion of a row's complete list of cells. Partial results are formed when the server breaches its maximum result size when trying to service a client's RPC request. It is the responsibility of the scanner on the client side to recognize when partial results have been returned and to take action to form the complete results.

Unless the flag Scan.setAllowPartialResults(boolean) has been set to true, the caller of ResultScanner.next() should never see partial results.

Field Summary

Fields

Modifier and Type	Field	Description
private static long	CELL_HEAP_SIZE
static final HBaseClassTestRule	CLASS_RULE
private static byte[][]	FAMILIES
private static byte[]	FAMILY
private static final org.slf4j.Logger	LOG
private static final int	MINICLUSTER_SIZE
org.junit.rules.TestName	name
private static int	NUM_COLS
private static int	NUM_FAMILIES
private static int	NUM_QUALIFIERS
private static int	NUM_ROWS
private static byte[]	QUALIFIER
private static byte[][]	QUALIFIERS
private static byte[]	ROW
private static byte[][]	ROWS
private static org.apache.hadoop.hbase.client.Table	TABLE
private static org.apache.hadoop.hbase.TableName	TABLE_NAME	Table configuration
private static final HBaseTestingUtil	TEST_UTIL
private static long	timeout
private static byte[]	VALUE
private static int	VALUE_SIZE

Constructor Summary

Constructors

Constructor	Description
TestPartialResultsFromClientSide()

Method Summary

Modifier and Type	Method	Description
private void	assertCell(org.apache.hadoop.hbase.Cell cell, byte[] row, byte[] cf, byte[] cq)
(package private) static void	compareResults(org.apache.hadoop.hbase.client.Result r1, org.apache.hadoop.hbase.client.Result r2, String message)	Compares two results and fails the test if the results are different
private int	countCellsFromScanner(org.apache.hadoop.hbase.client.ResultScanner scanner)	Exhausts the scanner by calling next repetitively.
(package private) static ArrayList<org.apache.hadoop.hbase.Cell>	createKeyValuesForRow(byte[] row, byte[][] families, byte[][] qualifiers, byte[] value)	Make key values to represent each possible combination of family and qualifier in the specified row.
(package private) static ArrayList<org.apache.hadoop.hbase.client.Put>	createPuts(byte[][] rows, byte[][] families, byte[][] qualifiers, byte[] value)	Make puts to put the input value into each combination of row, family, and qualifier
(package private) static org.apache.hadoop.hbase.client.Table	createTestTable(org.apache.hadoop.hbase.TableName name, byte[][] rows, byte[][] families, byte[][] qualifiers, byte[] cellValue)
private long	getCellHeapSize()
private long	getResultSizeForNumberOfCells(int numberOfCells)
private void	moveRegion(org.apache.hadoop.hbase.client.Table table, int index)
static void	setUpBeforeClass()
static void	tearDownAfterClass()
void	testAllowPartialResults()	Ensure that we only see Results marked as partial when the allowPartial flag is set
void	testBatchingResultWhenRegionMove()
void	testCompleteResultWhenRegionMove()
void	testDontThrowUnknowScannerExceptionToClient()
void	testEquivalenceOfScanResults()	Ensure that the results returned from a scanner that retrieves all results in a single RPC call matches the results that are returned from a scanner that must incrementally combine partial results into complete results.
void	testEquivalenceOfScanResults(org.apache.hadoop.hbase.client.Table table, org.apache.hadoop.hbase.client.Scan scan1, org.apache.hadoop.hbase.client.Scan scan2)
void	testExceptionThrownOnMismatchedPartialResults()	When reconstructing the complete result from its partials we ensure that the row of each partial result is the same.
void	testExpectedNumberOfCellsPerPartialResult()	Setting the max result size allows us to control how many cells we expect to see on each call to next on the scanner.
void	testExpectedNumberOfCellsPerPartialResult(org.apache.hadoop.hbase.client.Scan baseScan)
void	testExpectedNumberOfCellsPerPartialResult(org.apache.hadoop.hbase.client.Scan baseScan, int expectedNumberOfCells)
void	testExpectedValuesOfPartialResults()	Ensure that the expected key values appear in a result returned from a scanner that is combining partial results into complete results
void	testExpectedValuesOfPartialResults(boolean reversed)
void	testMayHaveMoreCellsInRowReturnsTrueAndSetBatch()
void	testNoPartialResultsWhenRowFilterPresent()	When a scan has a filter where Filter.hasFilterRow() is true, the scanner should not return partial results.
void	testOrderingOfCellsInPartialResults()	Order of cells in partial results matches the ordering of cells from complete results
void	testOrderingOfCellsInPartialResults(org.apache.hadoop.hbase.client.Scan basePartialScan)
void	testPartialResultsAndBatch()	Test various combinations of batching and partial results for correctness
void	testPartialResultsAndBatch(int batch, int cellsPerPartialResult)
void	testPartialResultsAndCaching()	Examine the interaction between the maxResultSize and caching.
void	testPartialResultsAndCaching(int resultSizeRowLimit, int cachingRowLimit)
void	testPartialResultsReassembly()	Test the method Result.createCompleteResult(Iterable)
void	testPartialResultsReassembly(org.apache.hadoop.hbase.client.Scan scanBase)
void	testPartialResultsWithColumnFilter()	Test partial Result re-assembly in the presence of different filters.
void	testPartialResultsWithColumnFilter(org.apache.hadoop.hbase.filter.Filter filter)
void	testPartialResultWhenRegionMove()
void	testReadPointAndPartialResults()
void	testReversedCompleteResultWhenRegionMove()
void	testReversedPartialResultWhenRegionMove()
(package private) static void	verifyResult(org.apache.hadoop.hbase.client.Result result, List<org.apache.hadoop.hbase.Cell> expKvList, String msg)	Verifies that result contains all the key values within expKvList.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

CLASS_RULE

public static final HBaseClassTestRule CLASS_RULE

LOG

private static final org.slf4j.Logger LOG

TEST_UTIL

private static final HBaseTestingUtil TEST_UTIL

MINICLUSTER_SIZE

private static final int MINICLUSTER_SIZE

See Also:

• Constant Field Values

TABLE

private static org.apache.hadoop.hbase.client.Table TABLE

TABLE_NAME

private static org.apache.hadoop.hbase.TableName TABLE_NAME

Table configuration

NUM_ROWS

private static int NUM_ROWS

ROW

private static byte[] ROW

ROWS

private static byte[][] ROWS

NUM_FAMILIES

private static int NUM_FAMILIES

FAMILY

private static byte[] FAMILY

FAMILIES

private static byte[][] FAMILIES

NUM_QUALIFIERS

private static int NUM_QUALIFIERS

QUALIFIER

private static byte[] QUALIFIER

QUALIFIERS

private static byte[][] QUALIFIERS

VALUE_SIZE

private static int VALUE_SIZE

VALUE

private static byte[] VALUE

NUM_COLS

private static int NUM_COLS

CELL_HEAP_SIZE

private static long CELL_HEAP_SIZE

timeout

private static long timeout

name

public org.junit.rules.TestName name

Constructor Details

TestPartialResultsFromClientSide

public TestPartialResultsFromClientSide()

Method Details

setUpBeforeClass

public static void setUpBeforeClass()
                             throws Exception

Throws:

Exception

createTestTable

static org.apache.hadoop.hbase.client.Table createTestTable(org.apache.hadoop.hbase.TableName name,
 byte[][] rows,
 byte[][] families,
 byte[][] qualifiers,
 byte[] cellValue)
                                                     throws IOException

Throws:

IOException

tearDownAfterClass

public static void tearDownAfterClass()
                               throws Exception

Throws:

Exception

testExpectedValuesOfPartialResults

public void testExpectedValuesOfPartialResults()
                                        throws Exception

Ensure that the expected key values appear in a result returned from a scanner that is combining partial results into complete results

Throws:

Exception

testExpectedValuesOfPartialResults

public void testExpectedValuesOfPartialResults(boolean reversed)
                                        throws Exception

Throws:

Exception

testAllowPartialResults

public void testAllowPartialResults()
                             throws Exception

Ensure that we only see Results marked as partial when the allowPartial flag is set

Throws:

Exception

testEquivalenceOfScanResults

public void testEquivalenceOfScanResults()
                                  throws Exception

Ensure that the results returned from a scanner that retrieves all results in a single RPC call matches the results that are returned from a scanner that must incrementally combine partial results into complete results. A variety of scan configurations can be tested

Throws:

Exception

testEquivalenceOfScanResults

public void testEquivalenceOfScanResults(org.apache.hadoop.hbase.client.Table table,
 org.apache.hadoop.hbase.client.Scan scan1,
 org.apache.hadoop.hbase.client.Scan scan2)
                                  throws Exception

Throws:

Exception

testOrderingOfCellsInPartialResults

public void testOrderingOfCellsInPartialResults()
                                         throws Exception

Order of cells in partial results matches the ordering of cells from complete results

Throws:

Exception

testOrderingOfCellsInPartialResults

public void testOrderingOfCellsInPartialResults(org.apache.hadoop.hbase.client.Scan basePartialScan)
                                         throws Exception

Throws:

Exception

testExpectedNumberOfCellsPerPartialResult

public void testExpectedNumberOfCellsPerPartialResult()
                                               throws Exception

Setting the max result size allows us to control how many cells we expect to see on each call to next on the scanner. Test a variety of different sizes for correctness

Throws:

Exception

testExpectedNumberOfCellsPerPartialResult

public void testExpectedNumberOfCellsPerPartialResult(org.apache.hadoop.hbase.client.Scan baseScan)
                                               throws Exception

Throws:

Exception

testExpectedNumberOfCellsPerPartialResult

public void testExpectedNumberOfCellsPerPartialResult(org.apache.hadoop.hbase.client.Scan baseScan,
 int expectedNumberOfCells)
                                               throws Exception

Throws:

Exception

getCellHeapSize

private long getCellHeapSize()
                      throws Exception

Returns:

The approximate heap size of a cell in the test table. All cells should have approximately the same heap size, so the value is cached to avoid repeating the calculation

Throws:

Exception

getResultSizeForNumberOfCells

private long getResultSizeForNumberOfCells(int numberOfCells)
                                    throws Exception

Returns:

the result size that should be used in Scan.setMaxResultSize(long) if you want the server to return exactly numberOfCells cells

Throws:

Exception

testPartialResultsAndBatch

public void testPartialResultsAndBatch()
                                throws Exception

Test various combinations of batching and partial results for correctness

Throws:

Exception

testPartialResultsAndBatch

public void testPartialResultsAndBatch(int batch,
 int cellsPerPartialResult)
                                throws Exception

Throws:

Exception

testPartialResultsReassembly

public void testPartialResultsReassembly()
                                  throws Exception

Test the method Result.createCompleteResult(Iterable)

Throws:

Exception

testPartialResultsReassembly

public void testPartialResultsReassembly(org.apache.hadoop.hbase.client.Scan scanBase)
                                  throws Exception

Throws:

Exception

testExceptionThrownOnMismatchedPartialResults

public void testExceptionThrownOnMismatchedPartialResults()
                                                   throws IOException

When reconstructing the complete result from its partials we ensure that the row of each partial result is the same. If one of the rows differs, an exception is thrown.

Throws:

IOException

testNoPartialResultsWhenRowFilterPresent

public void testNoPartialResultsWhenRowFilterPresent()
                                              throws Exception

When a scan has a filter where Filter.hasFilterRow() is true, the scanner should not return partial results. The scanner cannot return partial results because the entire row needs to be read for the include/exclude decision to be made

Throws:

Exception

testPartialResultsAndCaching

public void testPartialResultsAndCaching()
                                  throws Exception

Examine the interaction between the maxResultSize and caching. If the caching limit is reached before the maxResultSize limit, we should not see partial results. On the other hand, if the maxResultSize limit is reached before the caching limit, it is likely that partial results will be seen.

Throws:

Exception

testPartialResultsAndCaching

public void testPartialResultsAndCaching(int resultSizeRowLimit,
 int cachingRowLimit)
                                  throws Exception

Parameters:

resultSizeRowLimit - The row limit that will be enforced through maxResultSize

cachingRowLimit - The row limit that will be enforced through caching

Throws:

Exception

createPuts

static ArrayList<org.apache.hadoop.hbase.client.Put> createPuts(byte[][] rows,
 byte[][] families,
 byte[][] qualifiers,
 byte[] value)
                                                         throws IOException

Make puts to put the input value into each combination of row, family, and qualifier

Parameters:

rows - the rows to use

families - the families to use

qualifiers - the qualifiers to use

value - the values to use

Returns:

the dot product of the given rows, families, qualifiers, and values

Throws:

IOException - if there is a problem creating one of the Put objects

createKeyValuesForRow

static ArrayList<org.apache.hadoop.hbase.Cell> createKeyValuesForRow(byte[] row,
 byte[][] families,
 byte[][] qualifiers,
 byte[] value)

Make key values to represent each possible combination of family and qualifier in the specified row.

Parameters:

row - the row to use

families - the families to use

qualifiers - the qualifiers to use

value - the values to use

Returns:

the dot product of the given families, qualifiers, and values for a given row

verifyResult

static void verifyResult(org.apache.hadoop.hbase.client.Result result,
 List<org.apache.hadoop.hbase.Cell> expKvList,
 String msg)

Verifies that result contains all the key values within expKvList. Fails the test otherwise

compareResults

static void compareResults(org.apache.hadoop.hbase.client.Result r1,
 org.apache.hadoop.hbase.client.Result r2,
 String message)

Compares two results and fails the test if the results are different

testReadPointAndPartialResults

public void testReadPointAndPartialResults()
                                    throws Exception

Throws:

Exception

countCellsFromScanner

private int countCellsFromScanner(org.apache.hadoop.hbase.client.ResultScanner scanner)
                           throws Exception

Exhausts the scanner by calling next repetitively. Once completely exhausted, close scanner and return total cell count

Parameters:

scanner - the scanner to exhaust

Returns:

the number of cells counted

Throws:

Exception - if there is a problem retrieving cells from the scanner

testPartialResultsWithColumnFilter

public void testPartialResultsWithColumnFilter()
                                        throws Exception

Test partial Result re-assembly in the presence of different filters. The Results from the partial scanner should match the Results returned from a scanner that receives all of the results in one RPC to the server. The partial scanner is tested with a variety of different result sizes (all of which are less than the size necessary to fetch an entire row)

Throws:

Exception

testPartialResultsWithColumnFilter

public void testPartialResultsWithColumnFilter(org.apache.hadoop.hbase.filter.Filter filter)
                                        throws Exception

Throws:

Exception

moveRegion

private void moveRegion(org.apache.hadoop.hbase.client.Table table,
 int index)
                 throws IOException

Throws:

IOException

assertCell

private void assertCell(org.apache.hadoop.hbase.Cell cell,
 byte[] row,
 byte[] cf,
 byte[] cq)

testPartialResultWhenRegionMove

public void testPartialResultWhenRegionMove()
                                     throws IOException

Throws:

IOException

testReversedPartialResultWhenRegionMove

public void testReversedPartialResultWhenRegionMove()
                                             throws IOException

Throws:

IOException

testCompleteResultWhenRegionMove

public void testCompleteResultWhenRegionMove()
                                      throws IOException

Throws:

IOException

testReversedCompleteResultWhenRegionMove

public void testReversedCompleteResultWhenRegionMove()
                                              throws IOException

Throws:

IOException

testBatchingResultWhenRegionMove

public void testBatchingResultWhenRegionMove()
                                      throws IOException

Throws:

IOException

testDontThrowUnknowScannerExceptionToClient

public void testDontThrowUnknowScannerExceptionToClient()
                                                 throws Exception

Throws:

Exception

testMayHaveMoreCellsInRowReturnsTrueAndSetBatch

public void testMayHaveMoreCellsInRowReturnsTrueAndSetBatch()
                                                     throws IOException

Throws:

IOException