Package org.apache.hadoop.hbase.snapshot

Class SnapshotDescriptionUtils

java.lang.Object
org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils
@Private public final class SnapshotDescriptionUtils extends Object
Utility class to help manage SnapshotDesriptions.

Snapshots are laid out on disk like this: 

 /hbase/.snapshots
          /.tmp                <---- working directory
          /[snapshot name]     <----- completed snapshot
A completed snapshot named 'completed' then looks like (multiple regions, servers, files, etc. signified by '...' on the same directory depth). 
 /hbase/.snapshots/completed
                   .snapshotinfo          <--- Description of the snapshot
                   .tableinfo             <--- Copy of the tableinfo
                    /.logs
                        /[server_name]
                            /... [log files]
                         ...
                   /[region name]           <---- All the region's information
                   .regioninfo              <---- Copy of the HRegionInfo
                      /[column family name]
                          /[hfile name]     <--- name of the hfile in the real region
                          ...
                      ...
                    ...
Utility methods in this class are useful for getting the correct locations for different parts of the snapshot, as well as moving completed snapshots into place (see completeSnapshot(org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path, org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.FileSystem, org.apache.hadoop.conf.Configuration), and writing the SnapshotProtos.SnapshotDescription to the working snapshot directory.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    SnapshotDescriptionUtils.CompletedSnaphotDirectoriesFilter
    Filter that only accepts completed snapshot directories

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final long
    DEFAULT_MAX_WAIT_TIME
    By default, wait 300 seconds for a snapshot to complete
    private static final org.slf4j.Logger
    LOG
     
    static final String
    MASTER_SNAPSHOT_TIMEOUT_MILLIS
     
    static final long
    NO_SNAPSHOT_START_TIME_SPECIFIED
    Default value if no start time is specified
    private static final long
    NO_SNAPSHOT_TTL_SPECIFIED
     
    static final String
    SNAPSHOT_CORRUPTED_FILE
     
    static final int
    SNAPSHOT_LAYOUT_VERSION
    Version of the fs layout for a snapshot.
    static final int
    SNAPSHOT_TIMEOUT_MILLIS_DEFAULT
    Deprecated.
    Use DEFAULT_MAX_WAIT_TIME instead.
    static final String
    SNAPSHOT_TIMEOUT_MILLIS_KEY
    Deprecated.
    Use MASTER_SNAPSHOT_TIMEOUT_MILLIS instead.
    static final String
    SNAPSHOT_TMP_DIR_NAME
    Temporary directory under the snapshot directory to store in-progress snapshots
    static final String
    SNAPSHOT_WORKING_DIR
    The configuration property that determines the filepath of the snapshot base working directory
    static final String
    SNAPSHOTINFO_FILE
    The file contains the snapshot basic information and it is under the directory of a snapshot.

  • Constructor Summary

    Constructors
    Modifier
    Constructor
    Description
    private
    SnapshotDescriptionUtils()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static void
    completeSnapshot(org.apache.hadoop.fs.Path snapshotDir, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileSystem workingDirFs, org.apache.hadoop.conf.Configuration conf)
    Commits the snapshot process by moving the working snapshot to the finalized filepath
    static org.apache.hadoop.fs.Path
    getCompletedSnapshotDir(String snapshotName, org.apache.hadoop.fs.Path rootDir)
    Get the directory for a completed snapshot.
    static org.apache.hadoop.fs.Path
    getCompletedSnapshotDir(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.fs.Path rootDir)
    Get the directory for a specified snapshot.
    static org.apache.hadoop.fs.Path
    getCorruptedFlagFileForSnapshot(org.apache.hadoop.fs.Path workingDir)
    Get the flag file path if the snapshot is corrupted
    private static org.apache.hadoop.fs.Path
    getDefaultWorkingSnapshotDir(org.apache.hadoop.fs.Path rootDir)
    Get the default working directory for snapshots - where they are built, where they are temporarily copied on export, etc.
    static long
    getMaxMasterTimeout(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription.Type type, long defaultMaxWaitTime)
     
    static org.apache.hadoop.fs.Path
    getSnapshotRootDir(org.apache.hadoop.fs.Path rootDir)
    Get the snapshot root directory.
    static final org.apache.hadoop.fs.Path
    getSnapshotsDir(org.apache.hadoop.fs.Path rootDir)
     
    private static final org.apache.hadoop.fs.Path
    getSpecifiedSnapshotDir(org.apache.hadoop.fs.Path snapshotsDir, String snapshotName)
    Get the directory within the given filepath to store the snapshot instance
    static org.apache.hadoop.fs.Path
    getWorkingSnapshotDir(String snapshotName, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.conf.Configuration conf)
    Get the directory to build a snapshot, before it is finalized
    static org.apache.hadoop.fs.Path
    getWorkingSnapshotDir(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.conf.Configuration conf)
    Get the general working directory for snapshots - where they are built, where they are temporarily copied on export, etc.
    static org.apache.hadoop.fs.Path
    getWorkingSnapshotDir(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.conf.Configuration conf)
    Get the directory to build a snapshot, before it is finalized
    static boolean
    isExpiredSnapshot(long snapshotTtl, long snapshotCreatedTime, long currentTime)
    Method to check whether TTL has expired for specified snapshot creation time and snapshot ttl.
    static boolean
    isSecurityAvailable(org.apache.hadoop.conf.Configuration conf)
     
    static boolean
    isSnapshotOwner(SnapshotDescription snapshot, User user)
    Check if the user is this table snapshot's owner
    static boolean
    isSubDirectoryOf(org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.fs.Path rootDir)
    Determines if the given workingDir is a subdirectory of the given "root directory"
    static boolean
    isWithinDefaultWorkingDir(org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.conf.Configuration conf)
    Determines if the given workingDir is a subdirectory of the default working snapshot directory
    static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription
    readSnapshotInfo(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path snapshotDir)
    Read in the HBaseProtos.SnapshotDescription stored for the snapshot in the passed directory
    (package private) static boolean
    shouldSkipRenameSnapshotDirectories(URI workingURI, URI rootURI)
     
    static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription
    validate(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.conf.Configuration conf)
    Convert the passed snapshot description into a 'full' snapshot description based on default parameters, if none have been supplied.
    private static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription
    writeAclToSnapshotDescription(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.conf.Configuration conf)
     
    static void
    writeSnapshotInfo(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.fs.FileSystem fs)
    Write the snapshot description into the working directory of a snapshot

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

  • Constructor Details

  • Method Details

    • getMaxMasterTimeout

      public static long getMaxMasterTimeout(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription.Type type, long defaultMaxWaitTime)
      Parameters:
      conf - Configuration from which to check for the timeout
      type - type of snapshot being taken
      defaultMaxWaitTime - Default amount of time to wait, if none is in the configuration
      Returns:
      the max amount of time the master should wait for a snapshot to complete

    • getSnapshotRootDir

      public static org.apache.hadoop.fs.Path getSnapshotRootDir(org.apache.hadoop.fs.Path rootDir)
      Get the snapshot root directory. All the snapshots are kept under this directory, i.e. ${hbase.rootdir}/.snapshot
      Parameters:
      rootDir - hbase root directory
      Returns:
      the base directory in which all snapshots are kept

    • getCompletedSnapshotDir

      public static org.apache.hadoop.fs.Path getCompletedSnapshotDir(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.fs.Path rootDir)
      Get the directory for a specified snapshot. This directory is a sub-directory of snapshot root directory and all the data files for a snapshot are kept under this directory.
      Parameters:
      snapshot - snapshot being taken
      rootDir - hbase root directory
      Returns:
      the final directory for the completed snapshot

    • getCompletedSnapshotDir

      public static org.apache.hadoop.fs.Path getCompletedSnapshotDir(String snapshotName, org.apache.hadoop.fs.Path rootDir)
      Get the directory for a completed snapshot. This directory is a sub-directory of snapshot root directory and all the data files for a snapshot are kept under this directory.
      Parameters:
      snapshotName - name of the snapshot being taken
      rootDir - hbase root directory
      Returns:
      the final directory for the completed snapshot

    • getWorkingSnapshotDir

      public static org.apache.hadoop.fs.Path getWorkingSnapshotDir(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.conf.Configuration conf)
      Get the general working directory for snapshots - where they are built, where they are temporarily copied on export, etc.
      Parameters:
      rootDir - root directory of the HBase installation
      conf - Configuration of the HBase instance
      Returns:
      Path to the snapshot tmp directory, relative to the passed root directory

    • getWorkingSnapshotDir

      public static org.apache.hadoop.fs.Path getWorkingSnapshotDir(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.conf.Configuration conf)
      Get the directory to build a snapshot, before it is finalized
      Parameters:
      snapshot - snapshot that will be built
      rootDir - root directory of the hbase installation
      conf - Configuration of the HBase instance
      Returns:
      Path where one can build a snapshot

    • getWorkingSnapshotDir

      public static org.apache.hadoop.fs.Path getWorkingSnapshotDir(String snapshotName, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.conf.Configuration conf)
      Get the directory to build a snapshot, before it is finalized
      Parameters:
      snapshotName - name of the snapshot
      rootDir - root directory of the hbase installation
      conf - Configuration of the HBase instance
      Returns:
      Path where one can build a snapshot

    • getCorruptedFlagFileForSnapshot

      public static org.apache.hadoop.fs.Path getCorruptedFlagFileForSnapshot(org.apache.hadoop.fs.Path workingDir)
      Get the flag file path if the snapshot is corrupted
      Parameters:
      workingDir - the directory where we build the specific snapshot
      Returns:
      Path snapshot corrupted flag file path

    • getSpecifiedSnapshotDir

      private static final org.apache.hadoop.fs.Path getSpecifiedSnapshotDir(org.apache.hadoop.fs.Path snapshotsDir, String snapshotName)
      Get the directory within the given filepath to store the snapshot instance
      Parameters:
      snapshotsDir - directory to store snapshot directory within
      snapshotName - name of the snapshot to take
      Returns:
      the final directory for the snapshot in the given filepath

    • getSnapshotsDir

      public static final org.apache.hadoop.fs.Path getSnapshotsDir(org.apache.hadoop.fs.Path rootDir)
      Parameters:
      rootDir - hbase root directory
      Returns:
      the directory for all completed snapshots;

    • isSubDirectoryOf

      public static boolean isSubDirectoryOf(org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.fs.Path rootDir)
      Determines if the given workingDir is a subdirectory of the given "root directory"
      Parameters:
      workingDir - a directory to check
      rootDir - root directory of the HBase installation
      Returns:
      true if the given workingDir is a subdirectory of the given root directory, false otherwise

    • isWithinDefaultWorkingDir

      public static boolean isWithinDefaultWorkingDir(org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.conf.Configuration conf) throws IOException
      Determines if the given workingDir is a subdirectory of the default working snapshot directory
      Parameters:
      workingDir - a directory to check
      conf - configuration for the HBase cluster
      Returns:
      true if the given workingDir is a subdirectory of the default working directory for snapshots, false otherwise
      Throws:
      IOException - if we can't get the root dir

    • getDefaultWorkingSnapshotDir

      private static org.apache.hadoop.fs.Path getDefaultWorkingSnapshotDir(org.apache.hadoop.fs.Path rootDir)
      Get the default working directory for snapshots - where they are built, where they are temporarily copied on export, etc.
      Parameters:
      rootDir - root directory of the HBase installation
      Returns:
      Path to the default snapshot tmp directory, relative to the passed root directory

    • validate

      public static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription validate(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.conf.Configuration conf) throws IllegalArgumentException, IOException
      Convert the passed snapshot description into a 'full' snapshot description based on default parameters, if none have been supplied. This resolves any 'optional' parameters that aren't supplied to their default values.
      Parameters:
      snapshot - general snapshot descriptor
      conf - Configuration to read configured snapshot defaults if snapshot is not complete
      Returns:
      a valid snapshot description
      Throws:
      IllegalArgumentException - if the SnapshotProtos.SnapshotDescription is not a complete SnapshotProtos.SnapshotDescription.
      IOException

    • writeSnapshotInfo

      public static void writeSnapshotInfo(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.fs.FileSystem fs) throws IOException
      Write the snapshot description into the working directory of a snapshot
      Parameters:
      snapshot - description of the snapshot being taken
      workingDir - working directory of the snapshot
      fs - FileSystem on which the snapshot should be taken
      Throws:
      IOException - if we can't reach the filesystem and the file cannot be cleaned up on failure

    • readSnapshotInfo

      public static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription readSnapshotInfo(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path snapshotDir) throws CorruptedSnapshotException
      Read in the HBaseProtos.SnapshotDescription stored for the snapshot in the passed directory
      Parameters:
      fs - filesystem where the snapshot was taken
      snapshotDir - directory where the snapshot was stored
      Returns:
      the stored snapshot description
      Throws:
      CorruptedSnapshotException - if the snapshot cannot be read

    • completeSnapshot

      public static void completeSnapshot(org.apache.hadoop.fs.Path snapshotDir, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileSystem workingDirFs, org.apache.hadoop.conf.Configuration conf) throws SnapshotCreationException, IOException
      Commits the snapshot process by moving the working snapshot to the finalized filepath
      Parameters:
      snapshotDir - The file path of the completed snapshots
      workingDir - The file path of the in progress snapshots
      fs - The file system of the completed snapshots
      workingDirFs - The file system of the in progress snapshots
      conf - Configuration
      Throws:
      SnapshotCreationException - if the snapshot could not be moved
      IOException - the filesystem could not be reached

    • shouldSkipRenameSnapshotDirectories

      static boolean shouldSkipRenameSnapshotDirectories(URI workingURI, URI rootURI)

    • isSnapshotOwner

      public static boolean isSnapshotOwner(SnapshotDescription snapshot, User user)
      Check if the user is this table snapshot's owner
      Parameters:
      snapshot - the table snapshot description
      user - the user
      Returns:
      true if the user is the owner of the snapshot, false otherwise or the snapshot owner field is not present.

    • isSecurityAvailable

      public static boolean isSecurityAvailable(org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • writeAclToSnapshotDescription

      private static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription writeAclToSnapshotDescription(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • isExpiredSnapshot

      public static boolean isExpiredSnapshot(long snapshotTtl, long snapshotCreatedTime, long currentTime)
      Method to check whether TTL has expired for specified snapshot creation time and snapshot ttl. NOTE: For backward compatibility (after the patch deployment on HMaster), any snapshot with ttl 0 is to be considered as snapshot to keep FOREVER. Default ttl value specified by HConstants.DEFAULT_SNAPSHOT_TTL
      Returns:
      true if ttl has expired, or, false, otherwise