Package org.apache.hadoop.hbase

Class HBaseCluster

java.lang.Object
org.apache.hadoop.hbase.HBaseCluster
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable
Direct Known Subclasses:
DistributedHBaseCluster, MiniHBaseCluster
@Public public abstract class HBaseCluster extends Object implements Closeable, org.apache.hadoop.conf.Configurable
This class defines methods that can help with managing HBase clusters from unit tests and system tests. There are 3 types of cluster deployments:
  • MiniHBaseCluster: each server is run in the same JVM in separate threads, used by unit tests
  • DistributedHBaseCluster: the cluster is pre-deployed, system and integration tests can interact with the cluster.
  • ProcessBasedLocalHBaseCluster: each server is deployed locally but in separate JVMs.

HBaseCluster unifies the way tests interact with the cluster, so that the same test can be run against a mini-cluster during unit test execution, or a distributed cluster having tens/hundreds of nodes during execution of integration tests.

HBaseCluster exposes client-side public interfaces to tests, so that tests does not assume running in a particular mode. Not all the tests are suitable to be run on an actual cluster, and some tests will still need to mock stuff and introspect internal state. For those use cases from unit tests, or if more control is needed, you can use the subclasses directly. In that sense, this class does not abstract away every interface that MiniHBaseCluster or DistributedHBaseCluster provide.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected org.apache.hadoop.conf.Configuration
    conf
     
    protected org.apache.hadoop.hbase.ClusterMetrics
    initialClusterStatus
    the status of the cluster before we begin
    (package private) static final org.slf4j.Logger
    LOG
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    HBaseCluster(org.apache.hadoop.conf.Configuration conf)
    Construct an HBaseCluster

  • Method Summary

    Modifier and Type
    Method
    Description
    abstract void
    close()
    Closes all the resources held open for this cluster.
    abstract org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService.BlockingInterface
    getAdminProtocol(org.apache.hadoop.hbase.ServerName serverName)
    Returns an AdminProtocol interface to the regionserver
    abstract org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.ClientService.BlockingInterface
    getClientProtocol(org.apache.hadoop.hbase.ServerName serverName)
    Returns a ClientProtocol interface to the regionserver
    abstract org.apache.hadoop.hbase.ClusterMetrics
    getClusterMetrics()
    Returns a ClusterMetrics for this HBase cluster.
    org.apache.hadoop.conf.Configuration
    getConf()
     
    org.apache.hadoop.hbase.ClusterMetrics
    getInitialClusterMetrics()
    Returns a ClusterStatus for this HBase cluster as observed at the starting of the HBaseCluster
    abstract org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.MasterService.BlockingInterface
    getMasterAdminService()
    Returns an MasterProtos.MasterService.BlockingInterface to the active master
    org.apache.hadoop.hbase.ServerName
    getServerHoldingMeta()
    Get the ServerName of region server serving the first hbase:meta region
    abstract org.apache.hadoop.hbase.ServerName
    getServerHoldingRegion(org.apache.hadoop.hbase.TableName tn, byte[] regionName)
    Get the ServerName of region server serving the specified region
    boolean
    isDistributedCluster()
     
    abstract boolean
    isKilledRS(org.apache.hadoop.hbase.ServerName serverName)
    Keeping track of killed servers and being able to check if a particular server was killed makes it possible to do fault tolerance testing for dead servers in a deterministic way.
    abstract void
    killDataNode(org.apache.hadoop.hbase.ServerName serverName)
    Kills the datanode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
    abstract void
    killJournalNode(org.apache.hadoop.hbase.ServerName serverName)
    Kills the journalnode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
    abstract void
    killMaster(org.apache.hadoop.hbase.ServerName serverName)
    Kills the master process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
    abstract void
    killNameNode(org.apache.hadoop.hbase.ServerName serverName)
    Kills the namenode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
    abstract void
    killRegionServer(org.apache.hadoop.hbase.ServerName serverName)
    Kills the region server process if this is a distributed cluster, otherwise this causes the region server to exit doing basic clean up only.
    abstract void
    killZkNode(org.apache.hadoop.hbase.ServerName serverName)
    Kills the zookeeper node process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
    boolean
    restoreClusterMetrics(org.apache.hadoop.hbase.ClusterMetrics desiredStatus)
    Restores the cluster to given state if this is a real cluster, otherwise does nothing.
    boolean
    restoreInitialStatus()
    Restores the cluster to it's initial state if this is a real cluster, otherwise does nothing.
    abstract void
    resumeRegionServer(org.apache.hadoop.hbase.ServerName serverName)
    Resume the region server
    void
    setConf(org.apache.hadoop.conf.Configuration conf)
     
    abstract void
    shutdown()
    Shut down the HBase cluster
    abstract void
    startDataNode(org.apache.hadoop.hbase.ServerName serverName)
    Starts a new datanode on the given hostname or if this is a mini/local cluster, silently logs warning message.
    abstract void
    startJournalNode(org.apache.hadoop.hbase.ServerName serverName)
    Starts a new journalnode on the given hostname or if this is a mini/local cluster, silently logs warning message.
    abstract void
    startMaster(String hostname, int port)
    Starts a new master on the given hostname or if this is a mini/local cluster, starts a master locally.
    abstract void
    startNameNode(org.apache.hadoop.hbase.ServerName serverName)
    Starts a new namenode on the given hostname or if this is a mini/local cluster, silently logs warning message.
    abstract void
    startRegionServer(String hostname, int port)
    Starts a new region server on the given hostname or if this is a mini/local cluster, starts a region server locally.
    abstract void
    startZkNode(String hostname, int port)
    Starts a new zookeeper node on the given hostname or if this is a mini/local cluster, silently logs warning message.
    abstract void
    stopDataNode(org.apache.hadoop.hbase.ServerName serverName)
    Stops the datanode if this is a distributed cluster, otherwise silently logs warning message.
    abstract void
    stopJournalNode(org.apache.hadoop.hbase.ServerName serverName)
    Stops the journalnode if this is a distributed cluster, otherwise silently logs warning message.
    abstract void
    stopMaster(org.apache.hadoop.hbase.ServerName serverName)
    Stops the given master, by attempting a gradual stop.
    abstract void
    stopNameNode(org.apache.hadoop.hbase.ServerName serverName)
    Stops the namenode if this is a distributed cluster, otherwise silently logs warning message.
    abstract void
    stopRegionServer(org.apache.hadoop.hbase.ServerName serverName)
    Stops the given region server, by attempting a gradual stop.
    abstract void
    stopZkNode(org.apache.hadoop.hbase.ServerName serverName)
    Stops the region zookeeper if this is a distributed cluster, otherwise silently logs warning message.
    abstract void
    suspendRegionServer(org.apache.hadoop.hbase.ServerName serverName)
    Suspend the region server
    boolean
    waitForActiveAndReadyMaster()
    Blocks until there is an active master and that master has completed initialization.
    abstract boolean
    waitForActiveAndReadyMaster(long timeout)
    Blocks until there is an active master and that master has completed initialization.
    void
    waitForDatanodesRegistered(int nbDN)
     
    abstract void
    waitForDataNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout)
    Wait for the specified datanode to join the cluster
    abstract void
    waitForDataNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout)
    Wait for the specified datanode to stop the thread / process.
    abstract void
    waitForJournalNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout)
    Wait for the specified journalnode to join the cluster
    abstract void
    waitForJournalNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout)
    Wait for the specified journalnode to stop
    abstract void
    waitForMasterToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout)
    Wait for the specified master to stop the thread / process.
    void
    waitForNamenodeAvailable()
    Wait for the namenode.
    abstract void
    waitForNameNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout)
    Wait for the specified namenode to join the cluster
    abstract void
    waitForNameNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout)
    Wait for the specified namenode to stop
    void
    waitForRegionServerToStart(String hostname, int port, long timeout)
    Wait for the specified region server to join the cluster
    abstract void
    waitForRegionServerToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout)
    Wait for the specified region server to stop the thread / process.
    abstract void
    waitForZkNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout)
    Wait for the specified zookeeper node to join the cluster
    abstract void
    waitForZkNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout)
    Wait for the specified zookeeper node to stop the thread / process.
    abstract void
    waitUntilShutDown()
    Wait for HBase Cluster to shut down.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • LOG

      static final org.slf4j.Logger LOG

    • conf

      protected org.apache.hadoop.conf.Configuration conf

    • initialClusterStatus

      protected org.apache.hadoop.hbase.ClusterMetrics initialClusterStatus
      the status of the cluster before we begin

  • Constructor Details

    • HBaseCluster

      public HBaseCluster(org.apache.hadoop.conf.Configuration conf)
      Construct an HBaseCluster
      Parameters:
      conf - Configuration to be used for cluster

  • Method Details

    • setConf

      public void setConf(org.apache.hadoop.conf.Configuration conf)
      Specified by:
      setConf in interface org.apache.hadoop.conf.Configurable

    • getConf

      public org.apache.hadoop.conf.Configuration getConf()
      Specified by:
      getConf in interface org.apache.hadoop.conf.Configurable

    • getClusterMetrics

      public abstract org.apache.hadoop.hbase.ClusterMetrics getClusterMetrics() throws IOException
      Returns a ClusterMetrics for this HBase cluster.
      Throws:
      IOException
      See Also:

    • getInitialClusterMetrics

      public org.apache.hadoop.hbase.ClusterMetrics getInitialClusterMetrics() throws IOException
      Returns a ClusterStatus for this HBase cluster as observed at the starting of the HBaseCluster
      Throws:
      IOException

    • getMasterAdminService

      public abstract org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.MasterService.BlockingInterface getMasterAdminService() throws IOException
      Returns an MasterProtos.MasterService.BlockingInterface to the active master
      Throws:
      IOException

    • getAdminProtocol

      public abstract org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService.BlockingInterface getAdminProtocol(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Returns an AdminProtocol interface to the regionserver
      Throws:
      IOException

    • getClientProtocol

      public abstract org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.ClientService.BlockingInterface getClientProtocol(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Returns a ClientProtocol interface to the regionserver
      Throws:
      IOException

    • startRegionServer

      public abstract void startRegionServer(String hostname, int port) throws IOException
      Starts a new region server on the given hostname or if this is a mini/local cluster, starts a region server locally.
      Parameters:
      hostname - the hostname to start the regionserver on
      Throws:
      IOException - if something goes wrong

    • killRegionServer

      public abstract void killRegionServer(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Kills the region server process if this is a distributed cluster, otherwise this causes the region server to exit doing basic clean up only.
      Throws:
      IOException - if something goes wrong

    • isKilledRS

      public abstract boolean isKilledRS(org.apache.hadoop.hbase.ServerName serverName)
      Keeping track of killed servers and being able to check if a particular server was killed makes it possible to do fault tolerance testing for dead servers in a deterministic way. A concrete example of such case is - killing servers and waiting for all regions of a particular table to be assigned. We can check for server column in META table and that its value is not one of the killed servers.

    • stopRegionServer

      public abstract void stopRegionServer(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Stops the given region server, by attempting a gradual stop.
      Throws:
      IOException - if something goes wrong

    • waitForRegionServerToStart

      public void waitForRegionServerToStart(String hostname, int port, long timeout) throws IOException
      Wait for the specified region server to join the cluster
      Throws:
      IOException - if something goes wrong or timeout occurs

    • waitForRegionServerToStop

      public abstract void waitForRegionServerToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) throws IOException
      Wait for the specified region server to stop the thread / process.
      Throws:
      IOException - if something goes wrong or timeout occurs

    • suspendRegionServer

      public abstract void suspendRegionServer(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Suspend the region server
      Parameters:
      serverName - the hostname to suspend the regionserver on
      Throws:
      IOException - if something goes wrong

    • resumeRegionServer

      public abstract void resumeRegionServer(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Resume the region server
      Parameters:
      serverName - the hostname to resume the regionserver on
      Throws:
      IOException - if something goes wrong

    • startZkNode

      public abstract void startZkNode(String hostname, int port) throws IOException
      Starts a new zookeeper node on the given hostname or if this is a mini/local cluster, silently logs warning message.
      Parameters:
      hostname - the hostname to start the regionserver on
      Throws:
      IOException - if something goes wrong

    • killZkNode

      public abstract void killZkNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Kills the zookeeper node process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
      Throws:
      IOException - if something goes wrong

    • stopZkNode

      public abstract void stopZkNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Stops the region zookeeper if this is a distributed cluster, otherwise silently logs warning message.
      Throws:
      IOException - if something goes wrong

    • waitForZkNodeToStart

      public abstract void waitForZkNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout) throws IOException
      Wait for the specified zookeeper node to join the cluster
      Throws:
      IOException - if something goes wrong or timeout occurs

    • waitForZkNodeToStop

      public abstract void waitForZkNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) throws IOException
      Wait for the specified zookeeper node to stop the thread / process.
      Throws:
      IOException - if something goes wrong or timeout occurs

    • startDataNode

      public abstract void startDataNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Starts a new datanode on the given hostname or if this is a mini/local cluster, silently logs warning message.
      Throws:
      IOException - if something goes wrong

    • killDataNode

      public abstract void killDataNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Kills the datanode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
      Throws:
      IOException - if something goes wrong

    • stopDataNode

      public abstract void stopDataNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Stops the datanode if this is a distributed cluster, otherwise silently logs warning message.
      Throws:
      IOException - if something goes wrong

    • waitForDataNodeToStart

      public abstract void waitForDataNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout) throws IOException
      Wait for the specified datanode to join the cluster
      Throws:
      IOException - if something goes wrong or timeout occurs

    • waitForDataNodeToStop

      public abstract void waitForDataNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) throws IOException
      Wait for the specified datanode to stop the thread / process.
      Throws:
      IOException - if something goes wrong or timeout occurs

    • startNameNode

      public abstract void startNameNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Starts a new namenode on the given hostname or if this is a mini/local cluster, silently logs warning message.
      Throws:
      IOException - if something goes wrong

    • killNameNode

      public abstract void killNameNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Kills the namenode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
      Throws:
      IOException - if something goes wrong

    • stopNameNode

      public abstract void stopNameNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Stops the namenode if this is a distributed cluster, otherwise silently logs warning message.
      Throws:
      IOException - if something goes wrong

    • waitForNameNodeToStart

      public abstract void waitForNameNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout) throws IOException
      Wait for the specified namenode to join the cluster
      Throws:
      IOException - if something goes wrong or timeout occurs

    • waitForNameNodeToStop

      public abstract void waitForNameNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) throws IOException
      Wait for the specified namenode to stop
      Throws:
      IOException - if something goes wrong or timeout occurs

    • startJournalNode

      public abstract void startJournalNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Starts a new journalnode on the given hostname or if this is a mini/local cluster, silently logs warning message.
      Throws:
      IOException - if something goes wrong

    • killJournalNode

      public abstract void killJournalNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Kills the journalnode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
      Throws:
      IOException - if something goes wrong

    • stopJournalNode

      public abstract void stopJournalNode(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Stops the journalnode if this is a distributed cluster, otherwise silently logs warning message.
      Throws:
      IOException - if something goes wrong

    • waitForJournalNodeToStart

      public abstract void waitForJournalNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout) throws IOException
      Wait for the specified journalnode to join the cluster
      Throws:
      IOException - if something goes wrong or timeout occurs

    • waitForJournalNodeToStop

      public abstract void waitForJournalNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) throws IOException
      Wait for the specified journalnode to stop
      Throws:
      IOException - if something goes wrong or timeout occurs

    • startMaster

      public abstract void startMaster(String hostname, int port) throws IOException
      Starts a new master on the given hostname or if this is a mini/local cluster, starts a master locally.
      Parameters:
      hostname - the hostname to start the master on
      Throws:
      IOException - if something goes wrong

    • killMaster

      public abstract void killMaster(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Kills the master process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
      Throws:
      IOException - if something goes wrong

    • stopMaster

      public abstract void stopMaster(org.apache.hadoop.hbase.ServerName serverName) throws IOException
      Stops the given master, by attempting a gradual stop.
      Throws:
      IOException - if something goes wrong

    • waitForMasterToStop

      public abstract void waitForMasterToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) throws IOException
      Wait for the specified master to stop the thread / process.
      Throws:
      IOException - if something goes wrong or timeout occurs

    • waitForActiveAndReadyMaster

      public boolean waitForActiveAndReadyMaster() throws IOException
      Blocks until there is an active master and that master has completed initialization.
      Returns:
      true if an active master becomes available. false if there are no masters left.
      Throws:
      IOException - if something goes wrong or timeout occurs

    • waitForActiveAndReadyMaster

      public abstract boolean waitForActiveAndReadyMaster(long timeout) throws IOException
      Blocks until there is an active master and that master has completed initialization.
      Parameters:
      timeout - the timeout limit in ms
      Returns:
      true if an active master becomes available. false if there are no masters left.
      Throws:
      IOException

    • waitUntilShutDown

      public abstract void waitUntilShutDown() throws IOException
      Wait for HBase Cluster to shut down.
      Throws:
      IOException

    • shutdown

      public abstract void shutdown() throws IOException
      Shut down the HBase cluster
      Throws:
      IOException

    • restoreInitialStatus

      public boolean restoreInitialStatus() throws IOException
      Restores the cluster to it's initial state if this is a real cluster, otherwise does nothing. This is a best effort restore. If the servers are not reachable, or insufficient permissions, etc. restoration might be partial.
      Returns:
      whether restoration is complete
      Throws:
      IOException

    • restoreClusterMetrics

      public boolean restoreClusterMetrics(org.apache.hadoop.hbase.ClusterMetrics desiredStatus) throws IOException
      Restores the cluster to given state if this is a real cluster, otherwise does nothing. This is a best effort restore. If the servers are not reachable, or insufficient permissions, etc. restoration might be partial.
      Returns:
      whether restoration is complete
      Throws:
      IOException

    • getServerHoldingMeta

      public org.apache.hadoop.hbase.ServerName getServerHoldingMeta() throws IOException
      Get the ServerName of region server serving the first hbase:meta region
      Throws:
      IOException

    • getServerHoldingRegion

      public abstract org.apache.hadoop.hbase.ServerName getServerHoldingRegion(org.apache.hadoop.hbase.TableName tn, byte[] regionName) throws IOException
      Get the ServerName of region server serving the specified region
      Parameters:
      regionName - Name of the region in bytes
      tn - Table name that has the region.
      Returns:
      ServerName that hosts the region or null
      Throws:
      IOException

    • isDistributedCluster

      public boolean isDistributedCluster()
      Returns:
      whether we are interacting with a distributed cluster as opposed to an in-process mini/local cluster.

    • close

      public abstract void close() throws IOException
      Closes all the resources held open for this cluster. Note that this call does not shutdown the cluster.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException
      See Also:

    • waitForNamenodeAvailable

      public void waitForNamenodeAvailable() throws InterruptedException
      Wait for the namenode.
      Throws:
      InterruptedException

    • waitForDatanodesRegistered

      public void waitForDatanodesRegistered(int nbDN) throws Exception
      Throws:
      Exception