Package org.apache.hadoop.hbase.wal

Class AbstractFSWALProvider<T extends AbstractFSWAL<?>>

java.lang.Object
org.apache.hadoop.hbase.wal.AbstractWALProvider
org.apache.hadoop.hbase.wal.AbstractFSWALProvider<T>
All Implemented Interfaces:
PeerActionListener, WALProvider
Direct Known Subclasses:
AsyncFSWALProvider, FSHLogProvider
@Private @Evolving public abstract class AbstractFSWALProvider<T extends AbstractFSWAL<?>> extends AbstractWALProvider
Base class of a WAL Provider that returns a single thread safe WAL that writes to Hadoop FS. By default, this implementation picks a directory in Hadoop FS based on a combination of
  • the HBase root directory
  • HConstants.HREGION_LOGDIR_NAME
  • the given factory's factoryId (usually identifying the regionserver by host:port)
It also uses the providerId to differentiate among files.

  • Field Details

  • Constructor Details

  • Method Details

    • doInit

      protected void doInit(WALFactory factory, org.apache.hadoop.conf.Configuration conf, String providerId) throws IOException
      Specified by:
      doInit in class AbstractWALProvider
      Parameters:
      factory - factory that made us, identity used for FS layout. may not be null
      conf - may not be null
      providerId - differentiate between providers from one factory, used for FS layout. may be null
      Throws:
      IOException

    • getWALs0

      protected List<WAL> getWALs0()
      Specified by:
      getWALs0 in class AbstractWALProvider

    • getWAL0

      protected T getWAL0(RegionInfo region) throws IOException
      Specified by:
      getWAL0 in class AbstractWALProvider
      Throws:
      IOException

    • createWAL

      protected abstract T createWAL() throws IOException
      Throws:
      IOException

    • doInit

      protected abstract void doInit(org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • shutdown0

      protected void shutdown0() throws IOException
      Specified by:
      shutdown0 in class AbstractWALProvider
      Throws:
      IOException

    • close0

      protected void close0() throws IOException
      Specified by:
      close0 in class AbstractWALProvider
      Throws:
      IOException

    • getNumLogFiles0

      protected long getNumLogFiles0()
      iff the given WALFactory is using the DefaultWALProvider for meta and/or non-meta, count the number of files (rolled and active). if either of them aren't, count 0 for that provider.
      Specified by:
      getNumLogFiles0 in class AbstractWALProvider

    • getLogFileSize0

      protected long getLogFileSize0()
      iff the given WALFactory is using the DefaultWALProvider for meta and/or non-meta, count the size of files (only rolled). if either of them aren't, count 0 for that provider.
      Specified by:
      getLogFileSize0 in class AbstractWALProvider

    • createRemoteWAL

      protected WAL createRemoteWAL(RegionInfo region, org.apache.hadoop.fs.FileSystem remoteFs, org.apache.hadoop.fs.Path remoteWALDir, String prefix, String suffix) throws IOException
      Specified by:
      createRemoteWAL in class AbstractWALProvider
      Throws:
      IOException

    • getNumRolledLogFiles

      public static int getNumRolledLogFiles(WAL wal)
      returns the number of rolled WAL files.

    • getLogFileSize

      public static long getLogFileSize(WAL wal)
      returns the size of rolled WAL files.

    • getCurrentFileName

      public static org.apache.hadoop.fs.Path getCurrentFileName(WAL wal)
      return the current filename from the current wal.

    • requestLogRoll

      static void requestLogRoll(WAL wal)
      request a log roll, but don't actually do it.

    • extractFileNumFromWAL

      public static long extractFileNumFromWAL(WAL wal)
      It returns the file create timestamp (the 'FileNum') from the file name. For name format see validateWALFilename(String) public until remaining tests move to o.a.h.h.wal
      Parameters:
      wal - must not be null
      Returns:
      the file number that is part of the WAL file name

    • validateWALFilename

      public static boolean validateWALFilename(String filename)
      A WAL file name is of the format: <wal-name>WAL_FILE_NAME_DELIMITER <file-creation-timestamp>[.<suffix>]. provider-name is usually made up of a server-name and a provider-id
      Parameters:
      filename - name of the file to validate
      Returns:
      true if the filename matches an WAL, false otherwise

    • getTimestamp

      public static long getTimestamp(String name)
      Split a WAL filename to get a start time. WALs usually have the time we start writing to them with as part of their name, usually the suffix. Sometimes there will be an extra suffix as when it is a WAL for the meta table. For example, WALs might look like this 10.20.20.171%3A60020.1277499063250 where 1277499063250 is the timestamp. Could also be a meta WAL which adds a '.meta' suffix or a synchronous replication WAL which adds a '.syncrep' suffix. Check for these. File also may have no timestamp on it. For example the recovered.edits files are WALs but are named in ascending order. Here is an example: 0000000000000016310. Allow for this.
      Parameters:
      name - Name of the WAL file.
      Returns:
      Timestamp or NO_TIMESTAMP.

    • getWALDirectoryName

      public static String getWALDirectoryName(String serverName)
      Construct the directory name for all WALs on a given server. Dir names currently look like this for WALs: hbase//WALs/kalashnikov.att.net,61634,1486865297088.
      Parameters:
      serverName - Server name formatted as described in ServerName
      Returns:
      the relative WAL directory name, e.g. .logs/1.example.org,60030,12345 if serverName passed is 1.example.org,60030,12345

    • getWALArchiveDirectoryName

      public static String getWALArchiveDirectoryName(org.apache.hadoop.conf.Configuration conf, String serverName)
      Construct the directory name for all old WALs on a given server. The default old WALs dir looks like: hbase/oldWALs. If you config hbase.separate.oldlogdir.by.regionserver to true, it looks like hbase//oldWALs/kalashnikov.att.net,61634,1486865297088.
      Parameters:
      serverName - Server name formatted as described in ServerName
      Returns:
      the relative WAL directory name

    • getArchivedWALFiles

      public static List<org.apache.hadoop.fs.Path> getArchivedWALFiles(org.apache.hadoop.conf.Configuration conf, ServerName serverName, String logPrefix) throws IOException
      List all the old wal files for a dead region server.

      Initially added for supporting replication, where we need to get the wal files to replicate for a dead region server.

      Throws:
      IOException

    • getWALFiles

      public static List<org.apache.hadoop.fs.Path> getWALFiles(org.apache.hadoop.conf.Configuration c, ServerName serverName) throws IOException
      List all the wal files for a logPrefix.
      Throws:
      IOException

    • getServerNameFromWALDirectoryName

      public static ServerName getServerNameFromWALDirectoryName(org.apache.hadoop.conf.Configuration conf, String path) throws IOException
      Pulls a ServerName out of a Path generated according to our layout rules. In the below layouts, this method ignores the format of the logfile component. Current format: [base directory for hbase]/hbase/.logs/ServerName/logfile or [base directory for hbase]/hbase/.logs/ServerName-splitting/logfile Expected to work for individual log files and server-specific directories.
      Returns:
      null if it's not a log file. Returns the ServerName of the region server that created this log file otherwise.
      Throws:
      IOException

    • getServerNameFromWALDirectoryName

      public static ServerName getServerNameFromWALDirectoryName(org.apache.hadoop.fs.Path logFile)
      This function returns region server name from a log file name which is in one of the following formats:
      • hdfs://<name node>/hbase/.logs/<server name>-splitting/...
      • hdfs://<name node>/hbase/.logs/<server name>/...
      Returns:
      null if the passed in logFile isn't a valid WAL file path

    • isMetaFile

      public static boolean isMetaFile(org.apache.hadoop.fs.Path p)

    • isMetaFile

      public static boolean isMetaFile(String p)
      Returns True if String ends in META_WAL_PROVIDER_ID

    • isArchivedLogFile

      public static boolean isArchivedLogFile(org.apache.hadoop.fs.Path p)

    • findArchivedLog

      public static org.apache.hadoop.fs.Path findArchivedLog(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) throws IOException
      Find the archived WAL file path if it is not able to locate in WALs dir.
      Parameters:
      path - - active WAL file path
      conf - - configuration
      Returns:
      archived path if exists, null - otherwise
      Throws:
      IOException - exception

    • recoverLease

      public static void recoverLease(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path)

    • addWALActionsListener

      public void addWALActionsListener(WALActionsListener listener)
      Description copied from interface: WALProvider
      Add a WALActionsListener.

      Notice that you must call this method before calling WALProvider.getWAL(RegionInfo) as this method will not effect the WAL which has already been created. And as long as we can only it when initialization, it is not thread safe.

      Specified by:
      addWALActionsListener in interface WALProvider
      Overrides:
      addWALActionsListener in class AbstractWALProvider

    • getWALNameGroupFromWALName

      private static String getWALNameGroupFromWALName(String name, int group)

    • getWALPrefixFromWALName

      public static String getWALPrefixFromWALName(String name)
      Get prefix of the log from its name, assuming WAL name in format of log_prefix.filenumber.log_suffix
      Parameters:
      name - Name of the WAL to parse
      Returns:
      prefix of the log
      Throws:
      IllegalArgumentException - if the name passed in is not a valid wal file name
      See Also:

    • parseServerNameFromWALName

      public static ServerName parseServerNameFromWALName(String name)
      Parse the server name from wal prefix. A wal's name is always started with a server name in non test code.
      Returns:
      the server name
      Throws:
      IllegalArgumentException - if the name passed in is not started with a server name