Package org.apache.hadoop.hbase.master.cleaner

Class CleanerChore<T extends FileCleanerDelegate>

java.lang.Object
org.apache.hadoop.hbase.ScheduledChore
org.apache.hadoop.hbase.master.cleaner.CleanerChore<T>
Type Parameters:
T - Cleaner delegate class that is dynamically loaded from configuration
All Implemented Interfaces:
Runnable
Direct Known Subclasses:
HFileCleaner, LogCleaner
@Private public abstract class CleanerChore<T extends FileCleanerDelegate> extends ScheduledChore
Abstract Cleaner that uses a chain of delegates to clean a directory of files

  • Field Details

  • Constructor Details

    • CleanerChore

      public CleanerChore(String name, int sleepPeriod, Stoppable s, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path oldFileDir, String confKey, DirScanPool pool)

    • CleanerChore

      public CleanerChore(String name, int sleepPeriod, Stoppable s, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path oldFileDir, String confKey, DirScanPool pool, Map<String,Object> params, List<org.apache.hadoop.fs.Path> excludePaths)
      Parameters:
      name - name of the chore being run
      sleepPeriod - the period of time to sleep between each run
      s - the stopper
      conf - configuration to use
      fs - handle to the FS
      oldFileDir - the path to the archived files
      confKey - configuration key for the classes to instantiate
      pool - the thread pool used to scan directories
      params - members could be used in cleaner

  • Method Details

    • calculatePoolSize

      static int calculatePoolSize(String poolSize)
      Calculate size for cleaner pool.
      Parameters:
      poolSize - size from configuration
      Returns:
      size of pool after calculation

    • validate

      protected abstract boolean validate(org.apache.hadoop.fs.Path file)
      Validate the file to see if it even belongs in the directory. If it is valid, then the file will go through the cleaner delegates, but otherwise the file is just deleted.
      Parameters:
      file - full Path of the file to be checked
      Returns:
      true if the file is valid, false otherwise

    • initCleanerChain

      private void initCleanerChain(String confKey)
      Instantiate and initialize all the file cleaners set in the configuration
      Parameters:
      confKey - key to get the file cleaner classes from the configuration

    • newFileCleaner

      private T newFileCleaner(String className, org.apache.hadoop.conf.Configuration conf)
      A utility method to create new instances of LogCleanerDelegate based on the class name of the LogCleanerDelegate.
      Parameters:
      className - fully qualified class name of the LogCleanerDelegate
      conf - used configuration
      Returns:
      the new instance

    • initialChore

      protected boolean initialChore()
      Description copied from class: ScheduledChore
      Override to run a task before we start looping.
      Overrides:
      initialChore in class ScheduledChore
      Returns:
      true if initial chore was successful

    • chore

      protected void chore()
      Description copied from class: ScheduledChore
      The task to execute on each scheduled execution of the Chore
      Specified by:
      chore in class ScheduledChore

    • preRunCleaner

      private void preRunCleaner()

    • postRunCleaner

      private void postRunCleaner()

    • triggerCleanerNow

      public CompletableFuture<Boolean> triggerCleanerNow() throws InterruptedException
      Trigger the cleaner immediately and return a CompletableFuture for getting the result. Return true means all the old files have been deleted, otherwise false.
      Throws:
      InterruptedException

    • sortByConsumedSpace

      private void sortByConsumedSpace(List<org.apache.hadoop.fs.FileStatus> dirs)
      Sort the given list in (descending) order of the space each element takes
      Parameters:
      dirs - the list to sort, element in it should be directory (not file)

    • checkAndDeleteFiles

      private boolean checkAndDeleteFiles(List<org.apache.hadoop.fs.FileStatus> files)
      Run the given files through each of the cleaners to see if it should be deleted, deleting it if necessary.
      Parameters:
      files - List of FileStatus for the files to check (and possibly delete)
      Returns:
      true iff successfully deleted all files

    • isEmptyDirDeletable

      private boolean isEmptyDirDeletable(org.apache.hadoop.fs.Path dir)
      Check if a empty directory with no subdirs or subfiles can be deleted
      Parameters:
      dir - Path of the directory
      Returns:
      True if the directory can be deleted, otherwise false

    • deleteFiles

      protected int deleteFiles(Iterable<org.apache.hadoop.fs.FileStatus> filesToDelete)
      Delete the given files
      Parameters:
      filesToDelete - files to delete
      Returns:
      number of deleted files

    • cleanup

      public void cleanup()
      Description copied from class: ScheduledChore
      Override to run cleanup tasks when the Chore encounters an error and must stop running
      Overrides:
      cleanup in class ScheduledChore

    • getChorePoolSize

      int getChorePoolSize()

    • setEnabled

      public boolean setEnabled(boolean enabled)

    • getEnabled

      public boolean getEnabled()

    • traverseAndDelete

      private void traverseAndDelete(org.apache.hadoop.fs.Path dir, boolean root, CompletableFuture<Boolean> result)
      Attempts to clean up a directory(its subdirectories, and files) in a ThreadPoolExecutor concurrently. We can get the final result by calling result.get().

    • shouldExclude

      private boolean shouldExclude(org.apache.hadoop.fs.FileStatus f)
      Check if a path should not perform clear

    • deleteAction

      private boolean deleteAction(CleanerChore.Action<Boolean> deletion, String type, org.apache.hadoop.fs.Path dir)
      Perform a delete on a specified type.
      Parameters:
      deletion - a delete
      type - possible values are 'files', 'subdirs', 'dirs'
      Returns:
      true if it deleted successfully, false otherwise