Package org.apache.hadoop.hbase.replication

Class TableReplicationQueueStorage

java.lang.Object

org.apache.hadoop.hbase.replication.TableReplicationQueueStorage

All Implemented Interfaces:

ReplicationQueueStorage

@Private
public class TableReplicationQueueStorage
extends Object
implements ReplicationQueueStorage

HBase table based replication queue storage.

Field Summary

Fields

Modifier and Type	Field	Description
private final Connection	conn
static final byte[]	HFILE_REF_FAMILY
static final byte[]	LAST_SEQUENCE_ID_FAMILY
static final byte[]	QUEUE_FAMILY
private final TableName	tableName

Constructor Summary

Constructors

Constructor	Description
TableReplicationQueueStorage(Connection conn, TableName tableName)

Method Summary

Modifier and Type	Method	Description
void	addHFileRefs(String peerId, List<Pair<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path>> pairs)	Add new hfile references to the queue.
private void	addLastSeqIdsPut(org.apache.hadoop.hbase.shaded.protobuf.generated.MultiRowMutationProtos.MutateRowsRequest.Builder builder, String peerId, Map<String,Long> lastSeqIds, AsyncTable<?> table)
void	batchUpdateHFileRefs(String peerId, List<String> hfileRefs)	Add the given hfile refs to the given peer.
void	batchUpdateLastSequenceIds(List<ZKReplicationQueueStorageForMigration.ZkLastPushedSeqId> lastPushedSeqIds)	Update last pushed sequence id for the given regions and peers.
void	batchUpdateQueues(ServerName serverName, List<ReplicationQueueData> datas)	Update the replication queue datas for a given region server.
Map<String,ReplicationGroupOffset>	claimQueue(ReplicationQueueId queueId, ServerName targetServerName)	Change ownership for the queue identified by queueId and belongs to a dead region server.
Set<String>	getAllHFileRefs()	Load all hfile references in all replication queues.
List<String>	getAllPeersFromHFileRefsQueue()	Get list of all peers from hfile reference queue.
long	getLastSequenceId(String encodedRegionName, String peerId)	Read the max sequence id of the specific region for a given peer.
Map<String,ReplicationGroupOffset>	getOffsets(ReplicationQueueId queueId)	Get the current offset of all the WAL groups for a queue
private Map<String,ReplicationGroupOffset>	getOffsets0(Table table, ReplicationQueueId queueId)
List<String>	getReplicableHFiles(String peerId)	Get a list of all hfile references in the given peer.
boolean	hasData()	Whether the replication queue table exists.
List<ReplicationQueueId>	listAllQueueIds(String peerId)	Get a list of all queues for the specific peer.
List<ReplicationQueueId>	listAllQueueIds(String peerId, ServerName serverName)	Get a list of all queues for the specific region server and the specific peer
private void	listAllQueueIds(Table table, String peerId, ServerName serverName, List<ReplicationQueueId> queueIds)
private void	listAllQueueIds(Table table, Scan scan, List<ReplicationQueueId> queueIds)
List<ReplicationQueueId>	listAllQueueIds(ServerName serverName)	Get a list of all queues for the specific region server.
List<ReplicationQueueData>	listAllQueues()	Get a list of all queues and the offsets.
List<ServerName>	listAllReplicators()	Get a list of all region servers that have outstanding replication queues.
private org.apache.hbase.thirdparty.com.google.common.collect.ImmutableMap<String,ReplicationGroupOffset>	parseOffsets(Result result)
void	removeAllQueues(String peerId)	Remove all the replication queues for the given peer.
void	removeHFileRefs(String peerId, List<String> files)	Remove hfile references from the queue.
void	removeLastSequenceIds(String peerId)	Remove all the max sequence id record for the given peer.
void	removeLastSequenceIds(String peerId, List<String> encodedRegionNames)	Remove the max sequence id record for the given peer and regions.
void	removeLastSequenceIdsAndHFileRefsBefore(long ts)	Remove all the last sequence ids and hfile references data which are written before the given timestamp.
void	removePeerFromHFileRefs(String peerId)	Remove a peer from hfile reference queue.
void	removeQueue(ReplicationQueueId queueId)	Remove a replication queue
private <T extends Collection<String>> T	scanHFiles(Scan scan, Supplier<T> creator)
void	setLastSequenceIds(String peerId, Map<String,Long> lastSeqIds)	Set the max sequence id of a bunch of regions for a given peer.
void	setOffset(ReplicationQueueId queueId, String walGroup, ReplicationGroupOffset offset, Map<String,Long> lastSeqIds)	Set the current offset for a specific WAL group in a given queue.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

QUEUE_FAMILY

public static final byte[] QUEUE_FAMILY

LAST_SEQUENCE_ID_FAMILY

public static final byte[] LAST_SEQUENCE_ID_FAMILY

HFILE_REF_FAMILY

public static final byte[] HFILE_REF_FAMILY

conn

private final Connection conn

tableName

private final TableName tableName

Constructor Details

TableReplicationQueueStorage

public TableReplicationQueueStorage(Connection conn,
 TableName tableName)

Method Details

addLastSeqIdsPut

private void addLastSeqIdsPut(org.apache.hadoop.hbase.shaded.protobuf.generated.MultiRowMutationProtos.MutateRowsRequest.Builder builder,
 String peerId,
 Map<String,Long> lastSeqIds,
 AsyncTable<?> table)
                       throws IOException

Throws:

IOException

setOffset

public void setOffset(ReplicationQueueId queueId,
 String walGroup,
 ReplicationGroupOffset offset,
 Map<String,Long> lastSeqIds)
               throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Set the current offset for a specific WAL group in a given queue.

Specified by:

setOffset in interface ReplicationQueueStorage

Parameters:

queueId - the id of the queue

walGroup - the group of the WAL, can be empty if multi wal is not enabled

offset - the current offset of replication progress

lastSeqIds - map with {encodedRegionName, sequenceId} pairs for serial replication.

Throws:

ReplicationException

parseOffsets

private org.apache.hbase.thirdparty.com.google.common.collect.ImmutableMap<String,ReplicationGroupOffset> parseOffsets(Result result)

getOffsets0

private Map<String,ReplicationGroupOffset> getOffsets0(Table table,
 ReplicationQueueId queueId)
                                                throws IOException

Throws:

IOException

getOffsets

public Map<String,ReplicationGroupOffset> getOffsets(ReplicationQueueId queueId)
                                              throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Get the current offset of all the WAL groups for a queue

Specified by:

getOffsets in interface ReplicationQueueStorage

Parameters:

queueId - the id of the queue

Returns:

a map of all offsets of the WAL groups. The key the is WAL group and the value is the position.

Throws:

ReplicationException

listAllQueueIds

private void listAllQueueIds(Table table,
 Scan scan,
 List<ReplicationQueueId> queueIds)
                      throws IOException

Throws:

IOException

listAllQueueIds

private void listAllQueueIds(Table table,
 String peerId,
 ServerName serverName,
 List<ReplicationQueueId> queueIds)
                      throws IOException

Throws:

IOException

listAllQueueIds

public List<ReplicationQueueId> listAllQueueIds(String peerId)
                                         throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Get a list of all queues for the specific peer.

Specified by:

listAllQueueIds in interface ReplicationQueueStorage

Parameters:

peerId - the id of the peer

Returns:

a list of queueIds

Throws:

ReplicationException

listAllQueueIds

public List<ReplicationQueueId> listAllQueueIds(ServerName serverName)
                                         throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Get a list of all queues for the specific region server.

Specified by:

listAllQueueIds in interface ReplicationQueueStorage

Parameters:

serverName - the server name of the region server that owns the set of queues

Returns:

a list of queueIds

Throws:

ReplicationException

listAllQueueIds

public List<ReplicationQueueId> listAllQueueIds(String peerId,
 ServerName serverName)
                                         throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Get a list of all queues for the specific region server and the specific peer

Specified by:

listAllQueueIds in interface ReplicationQueueStorage

Parameters:

peerId - the id of the peer

serverName - the server name of the region server that owns the set of queues

Returns:

a list of queueIds

Throws:

ReplicationException

listAllQueues

public List<ReplicationQueueData> listAllQueues()
                                         throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Get a list of all queues and the offsets.

Specified by:

listAllQueues in interface ReplicationQueueStorage

Throws:

ReplicationException

listAllReplicators

public List<ServerName> listAllReplicators()
                                    throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Get a list of all region servers that have outstanding replication queues. These servers could be alive, dead or from a previous run of the cluster.

Specified by:

listAllReplicators in interface ReplicationQueueStorage

Returns:

a list of server names

Throws:

ReplicationException

claimQueue

public Map<String,ReplicationGroupOffset> claimQueue(ReplicationQueueId queueId,
 ServerName targetServerName)
                                              throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Change ownership for the queue identified by queueId and belongs to a dead region server.

Specified by:

claimQueue in interface ReplicationQueueStorage

Parameters:

queueId - the id of the queue

targetServerName - the name of the target region server

Returns:

the new PeerId and A SortedSet of WALs in its queue

Throws:

ReplicationException

removeQueue

public void removeQueue(ReplicationQueueId queueId)
                 throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Remove a replication queue

Specified by:

removeQueue in interface ReplicationQueueStorage

Parameters:

queueId - the id of the queue to remove

Throws:

ReplicationException

removeAllQueues

public void removeAllQueues(String peerId)
                     throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Remove all the replication queues for the given peer. Usually used when removing a peer.

Specified by:

removeAllQueues in interface ReplicationQueueStorage

Parameters:

peerId - the id of the peer

Throws:

ReplicationException

getLastSequenceId

public long getLastSequenceId(String encodedRegionName,
 String peerId)
                       throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Read the max sequence id of the specific region for a given peer. For serial replication, we need the max sequenced id to decide whether we can push the next entries.

Specified by:

getLastSequenceId in interface ReplicationQueueStorage

Parameters:

encodedRegionName - the encoded region name

peerId - peer id

Returns:

the max sequence id of the specific region for a given peer.

Throws:

ReplicationException

setLastSequenceIds

public void setLastSequenceIds(String peerId,
 Map<String,Long> lastSeqIds)
                        throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Set the max sequence id of a bunch of regions for a given peer. Will be called when setting up a serial replication peer.

Specified by:

setLastSequenceIds in interface ReplicationQueueStorage

Parameters:

peerId - peer id

lastSeqIds - map with {encodedRegionName, sequenceId} pairs for serial replication.

Throws:

ReplicationException

removeLastSequenceIds

public void removeLastSequenceIds(String peerId)
                           throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Remove all the max sequence id record for the given peer.

Specified by:

removeLastSequenceIds in interface ReplicationQueueStorage

Parameters:

peerId - peer id

Throws:

ReplicationException

removeLastSequenceIds

public void removeLastSequenceIds(String peerId,
 List<String> encodedRegionNames)
                           throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Remove the max sequence id record for the given peer and regions.

Specified by:

removeLastSequenceIds in interface ReplicationQueueStorage

Parameters:

peerId - peer id

encodedRegionNames - the encoded region names

Throws:

ReplicationException

removePeerFromHFileRefs

public void removePeerFromHFileRefs(String peerId)
                             throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Remove a peer from hfile reference queue.

Specified by:

removePeerFromHFileRefs in interface ReplicationQueueStorage

Parameters:

peerId - peer cluster id to be removed

Throws:

ReplicationException

addHFileRefs

public void addHFileRefs(String peerId,
 List<Pair<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path>> pairs)
                  throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Add new hfile references to the queue.

Specified by:

addHFileRefs in interface ReplicationQueueStorage

Parameters:

peerId - peer cluster id to which the hfiles need to be replicated

pairs - list of pairs of { HFile location in staging dir, HFile path in region dir which will be added in the queue }

Throws:

ReplicationException - if fails to add a hfile reference

removeHFileRefs

public void removeHFileRefs(String peerId,
 List<String> files)
                     throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Remove hfile references from the queue.

Specified by:

removeHFileRefs in interface ReplicationQueueStorage

Parameters:

peerId - peer cluster id from which this hfile references needs to be removed

files - list of hfile references to be removed

Throws:

ReplicationException

getAllPeersFromHFileRefsQueue

public List<String> getAllPeersFromHFileRefsQueue()
                                           throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Get list of all peers from hfile reference queue.

Specified by:

getAllPeersFromHFileRefsQueue in interface ReplicationQueueStorage

Returns:

a list of peer ids

Throws:

ReplicationException

scanHFiles

private <T extends Collection<String>> T scanHFiles(Scan scan,
 Supplier<T> creator)
                                             throws IOException

Throws:

IOException

getReplicableHFiles

public List<String> getReplicableHFiles(String peerId)
                                 throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Get a list of all hfile references in the given peer.

Specified by:

getReplicableHFiles in interface ReplicationQueueStorage

Parameters:

peerId - a String that identifies the peer

Returns:

a list of hfile references

Throws:

ReplicationException

getAllHFileRefs

public Set<String> getAllHFileRefs()
                            throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Load all hfile references in all replication queues. This method guarantees to return a snapshot which contains all hfile references at the start of this call. However, some newly created hfile references during the call may not be included.

Specified by:

getAllHFileRefs in interface ReplicationQueueStorage

Throws:

ReplicationException

hasData

public boolean hasData()
                throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Whether the replication queue table exists.

Specified by:

hasData in interface ReplicationQueueStorage

Returns:

Whether the replication queue table exists

Throws:

ReplicationException

batchUpdateQueues

public void batchUpdateQueues(ServerName serverName,
 List<ReplicationQueueData> datas)
                       throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Update the replication queue datas for a given region server.

Specified by:

batchUpdateQueues in interface ReplicationQueueStorage

Throws:

ReplicationException

batchUpdateLastSequenceIds

public void batchUpdateLastSequenceIds(List<ZKReplicationQueueStorageForMigration.ZkLastPushedSeqId> lastPushedSeqIds)
                                throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Update last pushed sequence id for the given regions and peers.

Specified by:

batchUpdateLastSequenceIds in interface ReplicationQueueStorage

Throws:

ReplicationException

batchUpdateHFileRefs

public void batchUpdateHFileRefs(String peerId,
 List<String> hfileRefs)
                          throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Add the given hfile refs to the given peer.

Specified by:

batchUpdateHFileRefs in interface ReplicationQueueStorage

Throws:

ReplicationException

removeLastSequenceIdsAndHFileRefsBefore

public void removeLastSequenceIdsAndHFileRefsBefore(long ts)
                                             throws ReplicationException

Description copied from interface: ReplicationQueueStorage

Remove all the last sequence ids and hfile references data which are written before the given timestamp.

The data of these two types are not used by replication directly.

For last sequence ids, we will check it in serial replication, to make sure that we will replicate all edits in order, so if there are stale data, the worst case is that we will stop replicating as we think we still need to finish previous ranges first, although actually we have already replicated them out.

For hfile references, it is just used by hfile cleaner to not remove these hfiles before we replicate them out, so if there are stale data, the worst case is that we can not remove these hfiles, although actually they have already been replicated out.

So it is OK for us to just bring up the cluster first, and then use this method to delete the stale data, i.e, the data which are written before a specific timestamp.

Specified by:

removeLastSequenceIdsAndHFileRefsBefore in interface ReplicationQueueStorage

Throws:

ReplicationException