Package org.apache.hadoop.hbase.util

Class RegionMover

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.util.RegionMover

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

@Public
public class RegionMover
extends AbstractHBaseTool
implements Closeable

Tool for loading/unloading regions to/from given regionserver This tool can be run from Command line directly as a utility. Supports Ack/No Ack mode for loading/unloading operations.Ack mode acknowledges if regions are online after movement while noAck mode is best effort mode that improves performance but will still move on if region is stuck/not moved. Motivation behind noAck mode being RS shutdown where even if a Region is stuck, upon shutdown master will move it anyways. This can also be used by constructiong an Object using the builder and then calling load() or unload() methods for the desired operations.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	RegionMover.RegionMoverBuilder	Builder for Region mover.

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

AbstractHBaseTool.OptionsOrderComparator

Field Summary

Fields

Modifier and Type	Field	Description
private boolean	ack
private Admin	admin
private Connection	conn
static final int	DEFAULT_MOVE_RETRIES_MAX
static final int	DEFAULT_MOVE_WAIT_MAX
static final int	DEFAULT_SERVERSTART_WAIT_MAX
private String	designatedFile
private String	excludeFile
private String	filename
private String	hostname
private List<String>	isolateRegionIdArray
private String	loadUnload
private static final org.slf4j.Logger	LOG
private int	maxthreads
static final String	MOVE_RETRIES_MAX_KEY
static final String	MOVE_WAIT_MAX_KEY
private int	port
private RackManager	rackManager
private RegionMover.RegionMoverBuilder	rmbuilder
static final String	SERVERSTART_WAIT_MAX_KEY
private int	timeout

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS, LONG_HELP_OPTION, options, SHORT_HELP_OPTION

Constructor Summary

Constructors

Modifier	Constructor	Description
private	RegionMover()
private	RegionMover(RegionMover.RegionMoverBuilder builder)

Method Summary

Modifier and Type	Method	Description
protected void	addOptions()	Override this to add command-line options using AbstractHBaseTool.addOptWithArg(java.lang.String, java.lang.String) and similar methods.
void	close()
private void	deleteFile(String filename)
protected int	doWork()	The "main function" of the tool
(package private) Collection<ServerName>	filterRSGroupServers(RSGroupInfo rsgroup, Collection<ServerName> onlineServers)
private Optional<RegionInfo>	getMetaRegionInfoIfToBeMoved(List<RegionInfo> regionsToMove)
private Callable<Boolean>	getMetaRegionMovePlan()
private Callable<Boolean>	getNonMetaRegionsMovePlan()
private Callable<Boolean>	getRegionsMovePlan(boolean moveMetaRegion)
private ServerName	getTargetServer()
private boolean	ignoreRegionMoveFailure(ExecutionException e)
private void	includeExcludeRegionServers(String fileName, List<ServerName> regionServers, boolean isInclude)	Designates or excludes the servername whose hostname and port portion matches the list given in the file.
boolean	isolateRegions()	Isolated regions specified in isolateRegionIdArray on hostname in ack Mode and Unload regions from given hostname using ack/noAck mode and maxthreads.
boolean	load()	Loads the specified hostname with regions listed in the filename RegionMover Object has to be created using RegionMover(RegionMoverBuilder)
private void	loadRegions(List<RegionInfo> regionsToMove)
static void	main(String[] args)
protected void	processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)	This method is called to process the options after they have been parsed.
private List<RegionInfo>	readRegionsFromFile(String filename)
private List<String>	readServersFromFile(String filename)
private void	stripMaster(List<ServerName> regionServers)	Exclude master from list of RSs to move regions to
private ServerName	stripServer(List<ServerName> regionServers, String hostname, int port)	Remove the servername whose hostname and port portion matches from the passed array of servers.
private void	submitRegionMovesWhileUnloading(ServerName server, List<ServerName> regionServers, List<RegionInfo> movedRegions, List<RegionInfo> regionsToMove, boolean forceMoveRegionByAck)
boolean	unload()	Unload regions from given hostname using ack/noAck mode and maxthreads.In noAck mode we do not make sure that region is successfully online on the target region server,hence it is best effort.We do not unload regions to hostnames given in excludeFile.
boolean	unloadFromRack()	Unload regions from given hostname using ack/noAck mode and maxthreads.In noAck mode we do not make sure that region is successfully online on the target region server,hence it is best effort.We do not unload regions to hostnames given in excludeFile.
private boolean	unloadRegions(boolean unloadFromRack)
private boolean	unloadRegions(boolean unloadFromRack, List<String> isolateRegionIdArray)
private void	unloadRegions(ServerName server, List<ServerName> regionServers, List<RegionInfo> movedRegions, List<String> isolateRegionIdArray)
private void	waitMoveTasksToFinish(ExecutorService moveRegionsPool, List<Future<Boolean>> taskList, long timeoutInSeconds)
private boolean	waitTaskToFinish(ExecutorService pool, Future<Boolean> task, String operation)
private void	writeFile(String filename, List<RegionInfo> movedRegions)	Write the number of regions moved in the first line followed by regions moved in subsequent lines

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOption, addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOption, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, getConf, getOptionAsDouble, getOptionAsInt, getOptionAsInt, getOptionAsLong, getOptionAsLong, newParser, parseArgs, parseInt, parseLong, printUsage, printUsage, processOldArgs, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

MOVE_RETRIES_MAX_KEY

public static final String MOVE_RETRIES_MAX_KEY

See Also:

• Constant Field Values

MOVE_WAIT_MAX_KEY

public static final String MOVE_WAIT_MAX_KEY

See Also:

• Constant Field Values

SERVERSTART_WAIT_MAX_KEY

public static final String SERVERSTART_WAIT_MAX_KEY

See Also:

• Constant Field Values

DEFAULT_MOVE_RETRIES_MAX

public static final int DEFAULT_MOVE_RETRIES_MAX

See Also:

• Constant Field Values

DEFAULT_MOVE_WAIT_MAX

public static final int DEFAULT_MOVE_WAIT_MAX

See Also:

• Constant Field Values

DEFAULT_SERVERSTART_WAIT_MAX

public static final int DEFAULT_SERVERSTART_WAIT_MAX

See Also:

• Constant Field Values

LOG

private static final org.slf4j.Logger LOG

rmbuilder

private RegionMover.RegionMoverBuilder rmbuilder

ack

private boolean ack

maxthreads

private int maxthreads

timeout

private int timeout

isolateRegionIdArray

private List<String> isolateRegionIdArray

loadUnload

private String loadUnload

hostname

private String hostname

filename

private String filename

excludeFile

private String excludeFile

designatedFile

private String designatedFile

port

private int port

conn

private Connection conn

admin

private Admin admin

rackManager

private RackManager rackManager

Constructor Details

RegionMover

private RegionMover(RegionMover.RegionMoverBuilder builder)
             throws IOException

Throws:

IOException

RegionMover

private RegionMover()

Method Details

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

load

public boolean load()
             throws ExecutionException,
InterruptedException,
TimeoutException

Loads the specified hostname with regions listed in the filename RegionMover Object has to be created using RegionMover(RegionMoverBuilder)

Returns:

true if loading succeeded, false otherwise

Throws:

ExecutionException

InterruptedException

TimeoutException

getMetaRegionMovePlan

private Callable<Boolean> getMetaRegionMovePlan()

getNonMetaRegionsMovePlan

private Callable<Boolean> getNonMetaRegionsMovePlan()

getRegionsMovePlan

private Callable<Boolean> getRegionsMovePlan(boolean moveMetaRegion)

getMetaRegionInfoIfToBeMoved

private Optional<RegionInfo> getMetaRegionInfoIfToBeMoved(List<RegionInfo> regionsToMove)

loadRegions

private void loadRegions(List<RegionInfo> regionsToMove)
                  throws Exception

Throws:

Exception

unload

public boolean unload()
               throws InterruptedException,
ExecutionException,
TimeoutException

Unload regions from given hostname using ack/noAck mode and maxthreads.In noAck mode we do not make sure that region is successfully online on the target region server,hence it is best effort.We do not unload regions to hostnames given in excludeFile. If designatedFile is present with some contents, we will unload regions to hostnames provided in designatedFile

Returns:

true if unloading succeeded, false otherwise

Throws:

InterruptedException

ExecutionException

TimeoutException

unloadFromRack

public boolean unloadFromRack()
                       throws InterruptedException,
ExecutionException,
TimeoutException

Unload regions from given hostname using ack/noAck mode and maxthreads.In noAck mode we do not make sure that region is successfully online on the target region server,hence it is best effort.We do not unload regions to hostnames given in excludeFile. If designatedFile is present with some contents, we will unload regions to hostnames provided in designatedFile. While unloading regions, destination RegionServers are selected from different rack i.e regions should not move to any RegionServers that belong to same rack as source RegionServer.

Returns:

true if unloading succeeded, false otherwise

Throws:

InterruptedException

ExecutionException

TimeoutException

unloadRegions

private boolean unloadRegions(boolean unloadFromRack)
                       throws ExecutionException,
InterruptedException,
TimeoutException

Throws:

ExecutionException

InterruptedException

TimeoutException

isolateRegions

public boolean isolateRegions()
                       throws ExecutionException,
InterruptedException,
TimeoutException

Isolated regions specified in isolateRegionIdArray on hostname in ack Mode and Unload regions from given hostname using ack/noAck mode and maxthreads. In noAck mode we do not make sure that region is successfully online on the target region server,hence it is the best effort. We do not unload regions to hostnames given in excludeFile. If designatedFile is present with some contents, we will unload regions to hostnames provided in designatedFile

Returns:

true if region isolation succeeded, false otherwise

Throws:

ExecutionException

InterruptedException

TimeoutException

unloadRegions

private boolean unloadRegions(boolean unloadFromRack,
 List<String> isolateRegionIdArray)
                       throws InterruptedException,
ExecutionException,
TimeoutException

Throws:

InterruptedException

ExecutionException

TimeoutException

filterRSGroupServers

@Private
Collection<ServerName> filterRSGroupServers(RSGroupInfo rsgroup,
 Collection<ServerName> onlineServers)

unloadRegions

private void unloadRegions(ServerName server,
 List<ServerName> regionServers,
 List<RegionInfo> movedRegions,
 List<String> isolateRegionIdArray)
                    throws Exception

Throws:

Exception

submitRegionMovesWhileUnloading

private void submitRegionMovesWhileUnloading(ServerName server,
 List<ServerName> regionServers,
 List<RegionInfo> movedRegions,
 List<RegionInfo> regionsToMove,
 boolean forceMoveRegionByAck)
                                      throws Exception

Throws:

Exception

waitTaskToFinish

private boolean waitTaskToFinish(ExecutorService pool,
 Future<Boolean> task,
 String operation)
                          throws TimeoutException,
InterruptedException,
ExecutionException

Throws:

TimeoutException

InterruptedException

ExecutionException

waitMoveTasksToFinish

private void waitMoveTasksToFinish(ExecutorService moveRegionsPool,
 List<Future<Boolean>> taskList,
 long timeoutInSeconds)
                            throws Exception

Throws:

Exception

ignoreRegionMoveFailure

private boolean ignoreRegionMoveFailure(ExecutionException e)

getTargetServer

private ServerName getTargetServer()
                            throws Exception

Throws:

Exception

readRegionsFromFile

private List<RegionInfo> readRegionsFromFile(String filename)
                                      throws IOException

Throws:

IOException

writeFile

private void writeFile(String filename,
 List<RegionInfo> movedRegions)
                throws IOException

Write the number of regions moved in the first line followed by regions moved in subsequent lines

Throws:

IOException

deleteFile

private void deleteFile(String filename)

readServersFromFile

private List<String> readServersFromFile(String filename)
                                  throws IOException

Parameters:

filename - The file should have 'host:port' per line

Returns:

List of servers from the file in format 'hostname:port'.

Throws:

IOException

includeExcludeRegionServers

private void includeExcludeRegionServers(String fileName,
 List<ServerName> regionServers,
 boolean isInclude)
                                  throws IOException

Designates or excludes the servername whose hostname and port portion matches the list given in the file. Example:
If you want to designated RSs, suppose designatedFile has RS1, regionServers has RS1, RS2 and RS3. When we call includeExcludeRegionServers(designatedFile, regionServers, true), RS2 and RS3 are removed from regionServers list so that regions can move to only RS1. If you want to exclude RSs, suppose excludeFile has RS1, regionServers has RS1, RS2 and RS3. When we call includeExcludeRegionServers(excludeFile, servers, false), RS1 is removed from regionServers list so that regions can move to only RS2 and RS3.

Throws:

IOException

stripMaster

private void stripMaster(List<ServerName> regionServers)
                  throws IOException

Exclude master from list of RSs to move regions to

Throws:

IOException

stripServer

private ServerName stripServer(List<ServerName> regionServers,
 String hostname,
 int port)

Remove the servername whose hostname and port portion matches from the passed array of servers. Returns as side-effect the servername removed.

Returns:

server removed from list of Region Servers

addOptions

protected void addOptions()

Description copied from class: AbstractHBaseTool

Override this to add command-line options using AbstractHBaseTool.addOptWithArg(java.lang.String, java.lang.String) and similar methods.

Specified by:

addOptions in class AbstractHBaseTool

processOptions

protected void processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)

Description copied from class: AbstractHBaseTool

This method is called to process the options after they have been parsed.

Specified by:

processOptions in class AbstractHBaseTool

doWork

protected int doWork()
              throws Exception

Description copied from class: AbstractHBaseTool

The "main function" of the tool

Specified by:

doWork in class AbstractHBaseTool

Throws:

Exception

main

public static void main(String[] args)