Package org.apache.hadoop.hbase.mapreduce

Class IntegrationTestBulkLoad

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.IntegrationTestBase

org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

Direct Known Subclasses:

IntegrationTestFileBasedSFTBulkLoad

public class IntegrationTestBulkLoad
extends IntegrationTestBase

Test Bulk Load and MR on a distributed cluster. It starts an MR job that creates linked chains The format of rows is like this: Row Key -> Long L:<< Chain Id >> -> Row Key of the next link in the chain S:<< Chain Id >> -> The step in the chain that his link is. D:<< Chain Id >> -> Random Data. All chains start on row 0. All rk's are > 0. After creating the linked lists they are walked over using a TableMapper based Mapreduce Job. There are a few options exposed: hbase.IntegrationTestBulkLoad.chainLength The number of rows that will be part of each and every chain. hbase.IntegrationTestBulkLoad.numMaps The number of mappers that will be run. Each mapper creates on linked list chain. hbase.IntegrationTestBulkLoad.numImportRounds How many jobs will be run to create linked lists. hbase.IntegrationTestBulkLoad.tableName The name of the table. hbase.IntegrationTestBulkLoad.replicaCount How many region replicas to configure for the table under test.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	IntegrationTestBulkLoad.CompositeKeyComparator	Comparator used to order linkKeys so that they are passed to a reducer in order.
static class	IntegrationTestBulkLoad.EmptySplit
static class	IntegrationTestBulkLoad.FixedRecordReader<K,V>
static class	IntegrationTestBulkLoad.ITBulkLoadInputFormat
static class	IntegrationTestBulkLoad.LinkChain	Writable used as the value emitted from a pass over the hbase table.
static class	IntegrationTestBulkLoad.LinkedListCheckingMapper	Mapper to pass over the table.
static class	IntegrationTestBulkLoad.LinkedListCheckingReducer	Class that does the actual checking of the links.
static class	IntegrationTestBulkLoad.LinkedListCreationMapper	Mapper that creates a linked list of KeyValues.
static class	IntegrationTestBulkLoad.LinkKey	Writable class used as the key to group links in the linked list.
static class	IntegrationTestBulkLoad.NaturalKeyGroupingComparator	Comparator used to figure out if a linkKey should be grouped together.
static class	IntegrationTestBulkLoad.NaturalKeyPartitioner	Class to figure out what partition to send a link in the chain to.
static class	IntegrationTestBulkLoad.SlowMeCoproScanOperations

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.util.AbstractHBaseTool.OptionsOrderComparator

Field Summary

Fields

Modifier and Type	Field	Description
private static final byte[]	CHAIN_FAM
private static int	CHAIN_LENGTH
private static String	CHAIN_LENGTH_KEY
private boolean	check
private static final byte[]	DATA_FAM
private boolean	load
private static final org.slf4j.Logger	LOG
private static int	NUM_IMPORT_ROUNDS
private static String	NUM_IMPORT_ROUNDS_KEY
private static int	NUM_MAPS
private static String	NUM_MAPS_KEY
private static int	NUM_REPLICA_COUNT_DEFAULT
private static String	NUM_REPLICA_COUNT_KEY
private static final String	OPT_CHECK
private static final String	OPT_LOAD
private static String	ROUND_NUM_KEY
private static final byte[]	SORT_FAM
private static String	TABLE_NAME
private static String	TABLE_NAME_KEY

Fields inherited from class org.apache.hadoop.hbase.IntegrationTestBase

CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS, LONG_HELP_OPTION, options, SHORT_HELP_OPTION

Constructor Summary

Constructors

Constructor	Description
IntegrationTestBulkLoad()

Method Summary

Modifier and Type	Method	Description
protected void	addOptions()
protected Set<String>	getColumnFamilies()	Provides the name of the CFs that are protected from random Chaos monkey activity (alter)
private byte[][]	getSplits(int numRegions)
org.apache.hadoop.hbase.TableName	getTablename()	Provides the name of the table that is protected from random Chaos monkey activity
static org.apache.hadoop.hbase.TableName	getTableName(org.apache.hadoop.conf.Configuration conf)
private void	installSlowingCoproc()	Modify table getTableName() to carry IntegrationTestBulkLoad.SlowMeCoproScanOperations.
static void	main(String[] args)
protected void	processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)
private void	runCheck()	After adding data to the table start a mr job to
private void	runCheckWithRetry()
private void	runLinkedListMRJob(int iteration)
void	runLoad()
int	runTestFromCommandLine()
void	setUpCluster()
private void	setupTable()
void	testBulkLoad()

Methods inherited from class org.apache.hadoop.hbase.IntegrationTestBase

cleanUp, cleanUpCluster, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, loadMonkeyProperties, processBaseOptions, setUp, setUpMonkey, startMonkey

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOption, addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOption, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, getOptionAsDouble, getOptionAsInt, getOptionAsInt, getOptionAsLong, getOptionAsLong, newParser, parseArgs, parseInt, parseLong, printUsage, printUsage, processOldArgs, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

CHAIN_FAM

private static final byte[] CHAIN_FAM

SORT_FAM

private static final byte[] SORT_FAM

DATA_FAM

private static final byte[] DATA_FAM

CHAIN_LENGTH_KEY

private static String CHAIN_LENGTH_KEY

CHAIN_LENGTH

private static int CHAIN_LENGTH

NUM_MAPS_KEY

private static String NUM_MAPS_KEY

NUM_MAPS

private static int NUM_MAPS

NUM_IMPORT_ROUNDS_KEY

private static String NUM_IMPORT_ROUNDS_KEY

NUM_IMPORT_ROUNDS

private static int NUM_IMPORT_ROUNDS

ROUND_NUM_KEY

private static String ROUND_NUM_KEY

TABLE_NAME_KEY

private static String TABLE_NAME_KEY

TABLE_NAME

private static String TABLE_NAME

NUM_REPLICA_COUNT_KEY

private static String NUM_REPLICA_COUNT_KEY

NUM_REPLICA_COUNT_DEFAULT

private static int NUM_REPLICA_COUNT_DEFAULT

OPT_LOAD

private static final String OPT_LOAD

See Also:

• Constant Field Values

OPT_CHECK

private static final String OPT_CHECK

See Also:

• Constant Field Values

load

private boolean load

check

private boolean check

Constructor Details

IntegrationTestBulkLoad

public IntegrationTestBulkLoad()

Method Details

installSlowingCoproc

private void installSlowingCoproc()
                           throws IOException,
InterruptedException

Modify table getTableName() to carry IntegrationTestBulkLoad.SlowMeCoproScanOperations.

Throws:

IOException

InterruptedException

testBulkLoad

public void testBulkLoad()
                  throws Exception

Throws:

Exception

runLoad

public void runLoad()
             throws Exception

Throws:

Exception

getSplits

private byte[][] getSplits(int numRegions)

setupTable

private void setupTable()
                 throws IOException,
InterruptedException

Throws:

IOException

InterruptedException

runLinkedListMRJob

private void runLinkedListMRJob(int iteration)
                         throws Exception

Throws:

Exception

runCheckWithRetry

private void runCheckWithRetry()
                        throws IOException,
ClassNotFoundException,
InterruptedException

Throws:

IOException

ClassNotFoundException

InterruptedException

runCheck

private void runCheck()
               throws IOException,
ClassNotFoundException,
InterruptedException

After adding data to the table start a mr job to

Throws:

IOException

ClassNotFoundException

InterruptedException

setUpCluster

public void setUpCluster()
                  throws Exception

Specified by:

setUpCluster in class IntegrationTestBase

Throws:

Exception

addOptions

protected void addOptions()

Overrides:

addOptions in class IntegrationTestBase

processOptions

protected void processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)

Overrides:

processOptions in class IntegrationTestBase

runTestFromCommandLine

public int runTestFromCommandLine()
                           throws Exception

Specified by:

runTestFromCommandLine in class IntegrationTestBase

Throws:

Exception

getTablename

public org.apache.hadoop.hbase.TableName getTablename()

Description copied from class: IntegrationTestBase

Provides the name of the table that is protected from random Chaos monkey activity

Specified by:

getTablename in class IntegrationTestBase

Returns:

table to not delete.

getTableName

public static org.apache.hadoop.hbase.TableName getTableName(org.apache.hadoop.conf.Configuration conf)

getColumnFamilies

protected Set<String> getColumnFamilies()

Description copied from class: IntegrationTestBase

Provides the name of the CFs that are protected from random Chaos monkey activity (alter)

Specified by:

getColumnFamilies in class IntegrationTestBase

Returns:

set of cf names to protect.

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception