Package org.apache.hadoop.hbase.io.hfile

Class HFileInfo

java.lang.Object

org.apache.hadoop.hbase.io.hfile.HFileInfo

All Implemented Interfaces:

Map<byte[],byte[]>, SortedMap<byte[],byte[]>

@Private
public class HFileInfo
extends Object
implements SortedMap<byte[],byte[]>

Metadata Map of attributes for HFile written out as HFile Trailer. Created by the Writer and added to the tail of the file just before close. Metadata includes core attributes such as last key seen, comparator used writing the file, etc. Clients can add their own attributes via append(byte[], byte[], boolean) and they'll be persisted and available at read time. Reader creates the HFileInfo on open by reading the tail of the HFile. The parse of the HFile trailer also creates a HFileContext, a read-only data structure that includes bulk of the HFileInfo and extras that is safe to pass around when working on HFiles.

See Also:

• HFileContext

Nested Class Summary

Nested classes/interfaces inherited from interface java.util.Map

Map.Entry<K extends Object,V extends Object>

Field Summary

Fields

Modifier and Type	Field	Description
(package private) static final byte[]	AVG_KEY_LEN
(package private) static final byte[]	AVG_VALUE_LEN
private int	avgKeyLen	Average key length read from file info
private int	avgValueLen	Average value length read from file info
private Cell	biggestCell	Biggest Cell in the file, key only.
private HFileBlock.BlockIterator	blockIter	The iterator will track all blocks in load-on-open section, since we use the ByteBuffAllocator to manage the ByteBuffers in block now, so we must ensure that deallocate all ByteBuffers in the end.
(package private) static final byte[]	CREATE_TIME_TS
private HFileBlockIndex.CellBasedKeyBlockIndexReader	dataIndexReader
private boolean	decodeMemstoreTS
private HFileContext	hfileContext
private boolean	includesMemstoreTS
(package private) static final byte[]	KEY_OF_BIGGEST_CELL
(package private) static final byte[]	LASTKEY
private ExtendedCell	lastKeyCell	Last key in the file.
(package private) static final byte[]	LEN_OF_BIGGEST_CELL
private long	lenOfBiggestCell	Length of the biggest Cell
private List<HFileBlock>	loadOnOpenBlocks	Blocks read from the load-on-open section, excluding data root index, meta index, and file info.
private static final org.slf4j.Logger	LOG
private final SortedMap<byte[],byte[]>	map
(package private) static final int	MAX_MINOR_VERSION	Maximum minor version supported by this HFile format
static final byte[]	MAX_TAGS_LEN
private HFileBlockIndex.ByteArrayKeyBlockIndexReader	metaIndexReader
private static final int	MIN_V2_MINOR_VERSION_WITH_PB	We can read files whose major version is v2 IFF their minor version is at least 3.
(package private) static final String	RESERVED_PREFIX
(package private) static final byte[]	RESERVED_PREFIX_BYTES
(package private) static final byte[]	TAGS_COMPRESSED
private FixedFileTrailer	trailer

Constructor Summary

Constructors

Constructor	Description
HFileInfo()
HFileInfo(ReaderContext context, org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method	Description
HFileInfo	append(byte[] k, byte[] v, boolean checkPrefix)	Append the given key/value pair to the file info, optionally checking the key prefix.
private void	checkFileVersion(org.apache.hadoop.fs.Path path)	File version check is a little sloppy.
void	clear()
void	close()
Comparator<? super byte[]>	comparator()
boolean	containsKey(Object key)
boolean	containsValue(Object value)
private HFileContext	createHFileContext(org.apache.hadoop.fs.Path path, FixedFileTrailer trailer, org.apache.hadoop.conf.Configuration conf)
Set<Map.Entry<byte[],byte[]>>	entrySet()
boolean	equals(Object o)
byte[]	firstKey()
byte[]	get(Object key)
int	getAvgKeyLen()
int	getAvgValueLen()
HFileBlockIndex.CellBasedKeyBlockIndexReader	getDataBlockIndexReader()
HFileContext	getHFileContext()
String	getKeyOfBiggestCell()
ExtendedCell	getLastKeyCell()
long	getLenOfBiggestCell()
List<HFileBlock>	getLoadOnOpenBlocks()
int	getMajorVersion()
HFileBlockIndex.ByteArrayKeyBlockIndexReader	getMetaBlockIndexReader()
FixedFileTrailer	getTrailer()
int	hashCode()
SortedMap<byte[],byte[]>	headMap(byte[] toKey)
void	initMetaAndIndex(HFile.Reader reader)	should be called after initTrailerAndContext
void	initTrailerAndContext(ReaderContext context, org.apache.hadoop.conf.Configuration conf)
boolean	isDecodeMemstoreTS()
boolean	isEmpty()
static boolean	isReservedFileInfoKey(byte[] key)	Return true if the given file info key is reserved for internal use.
Set<byte[]>	keySet()
byte[]	lastKey()
private void	loadMetaInfo(HFileBlock.BlockIterator blockIter, HFileContext hfileContext)
(package private) void	parsePB(org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos.FileInfoProto fip)	Fill our map with content of the pb we read off disk
(package private) void	parseWritable(DataInputStream in)	Now parse the old Writable format.
byte[]	put(byte[] key, byte[] value)
void	putAll(Map<? extends byte[],? extends byte[]> m)
(package private) void	read(DataInputStream in)	Populate this instance with what we find on the passed in in stream.
byte[]	remove(Object key)
void	setTrailer(FixedFileTrailer trailer)
boolean	shouldIncludeMemStoreTS()
int	size()
SortedMap<byte[],byte[]>	subMap(byte[] fromKey, byte[] toKey)
SortedMap<byte[],byte[]>	tailMap(byte[] fromKey)
Collection<byte[]>	values()
(package private) void	write(DataOutputStream out)	Write out this instance on the passed in out stream.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Map

compute, computeIfAbsent, computeIfPresent, forEach, getOrDefault, merge, putIfAbsent, remove, replace, replace, replaceAll

Field Details

LOG

private static final org.slf4j.Logger LOG

RESERVED_PREFIX

static final String RESERVED_PREFIX

See Also:

• Constant Field Values

RESERVED_PREFIX_BYTES

static final byte[] RESERVED_PREFIX_BYTES

LASTKEY

static final byte[] LASTKEY

AVG_KEY_LEN

static final byte[] AVG_KEY_LEN

AVG_VALUE_LEN

static final byte[] AVG_VALUE_LEN

CREATE_TIME_TS

static final byte[] CREATE_TIME_TS

TAGS_COMPRESSED

static final byte[] TAGS_COMPRESSED

KEY_OF_BIGGEST_CELL

static final byte[] KEY_OF_BIGGEST_CELL

LEN_OF_BIGGEST_CELL

static final byte[] LEN_OF_BIGGEST_CELL

MAX_TAGS_LEN

public static final byte[] MAX_TAGS_LEN

map

private final SortedMap<byte[],byte[]> map

MIN_V2_MINOR_VERSION_WITH_PB

private static final int MIN_V2_MINOR_VERSION_WITH_PB

We can read files whose major version is v2 IFF their minor version is at least 3.

See Also:

• Constant Field Values

MAX_MINOR_VERSION

static final int MAX_MINOR_VERSION

Maximum minor version supported by this HFile format

See Also:

• Constant Field Values

lastKeyCell

private ExtendedCell lastKeyCell

Last key in the file. Filled in when we read in the file info

avgKeyLen

private int avgKeyLen

Average key length read from file info

avgValueLen

private int avgValueLen

Average value length read from file info

biggestCell

private Cell biggestCell

Biggest Cell in the file, key only. Filled in when we read in the file info

lenOfBiggestCell

private long lenOfBiggestCell

Length of the biggest Cell

includesMemstoreTS

private boolean includesMemstoreTS

decodeMemstoreTS

private boolean decodeMemstoreTS

loadOnOpenBlocks

private List<HFileBlock> loadOnOpenBlocks

Blocks read from the load-on-open section, excluding data root index, meta index, and file info.

blockIter

private HFileBlock.BlockIterator blockIter

The iterator will track all blocks in load-on-open section, since we use the ByteBuffAllocator to manage the ByteBuffers in block now, so we must ensure that deallocate all ByteBuffers in the end.

dataIndexReader

private HFileBlockIndex.CellBasedKeyBlockIndexReader dataIndexReader

metaIndexReader

private HFileBlockIndex.ByteArrayKeyBlockIndexReader metaIndexReader

trailer

private FixedFileTrailer trailer

hfileContext

private HFileContext hfileContext

Constructor Details

HFileInfo

public HFileInfo()

HFileInfo

public HFileInfo(ReaderContext context,
 org.apache.hadoop.conf.Configuration conf)
          throws IOException

Throws:

IOException

Method Details

append

public HFileInfo append(byte[] k,
 byte[] v,
 boolean checkPrefix)
                 throws IOException

Append the given key/value pair to the file info, optionally checking the key prefix.

Parameters:

k - key to add

v - value to add

checkPrefix - whether to check that the provided key does not start with the reserved prefix

Returns:

this file info object

Throws:

IOException - if the key or value is invalid

NullPointerException - if key or value is null

isReservedFileInfoKey

public static boolean isReservedFileInfoKey(byte[] key)

Return true if the given file info key is reserved for internal use.

clear

public void clear()

Specified by:

clear in interface Map<byte[],byte[]>

comparator

public Comparator<? super byte[]> comparator()

Specified by:

comparator in interface SortedMap<byte[],byte[]>

containsKey

public boolean containsKey(Object key)

Specified by:

containsKey in interface Map<byte[],byte[]>

containsValue

public boolean containsValue(Object value)

Specified by:

containsValue in interface Map<byte[],byte[]>

entrySet

public Set<Map.Entry<byte[],byte[]>> entrySet()

Specified by:

entrySet in interface Map<byte[],byte[]>

Specified by:

entrySet in interface SortedMap<byte[],byte[]>

equals

public boolean equals(Object o)

Specified by:

equals in interface Map<byte[],byte[]>

Overrides:

equals in class Object

firstKey

public byte[] firstKey()

Specified by:

firstKey in interface SortedMap<byte[],byte[]>

get

public byte[] get(Object key)

Specified by:

get in interface Map<byte[],byte[]>

hashCode

public int hashCode()

Specified by:

hashCode in interface Map<byte[],byte[]>

Overrides:

hashCode in class Object

headMap

public SortedMap<byte[],byte[]> headMap(byte[] toKey)

Specified by:

headMap in interface SortedMap<byte[],byte[]>

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in interface Map<byte[],byte[]>

keySet

public Set<byte[]> keySet()

Specified by:

keySet in interface Map<byte[],byte[]>

Specified by:

keySet in interface SortedMap<byte[],byte[]>

lastKey

public byte[] lastKey()

Specified by:

lastKey in interface SortedMap<byte[],byte[]>

put

public byte[] put(byte[] key,
 byte[] value)

Specified by:

put in interface Map<byte[],byte[]>

putAll

public void putAll(Map<? extends byte[],? extends byte[]> m)

Specified by:

putAll in interface Map<byte[],byte[]>

remove

public byte[] remove(Object key)

Specified by:

remove in interface Map<byte[],byte[]>

size

public int size()

Specified by:

size in interface Map<byte[],byte[]>

subMap

public SortedMap<byte[],byte[]> subMap(byte[] fromKey,
 byte[] toKey)

Specified by:

subMap in interface SortedMap<byte[],byte[]>

tailMap

public SortedMap<byte[],byte[]> tailMap(byte[] fromKey)

Specified by:

tailMap in interface SortedMap<byte[],byte[]>

values

public Collection<byte[]> values()

Specified by:

values in interface Map<byte[],byte[]>

Specified by:

values in interface SortedMap<byte[],byte[]>

write

void write(DataOutputStream out)
    throws IOException

Write out this instance on the passed in out stream. We write it as a protobuf.

Throws:

IOException

See Also:

• read(DataInputStream)

read

void read(DataInputStream in)
   throws IOException

Populate this instance with what we find on the passed in in stream. Can deserialize protobuf of old Writables format.

Throws:

IOException

See Also:

• write(DataOutputStream)

parseWritable

void parseWritable(DataInputStream in)
            throws IOException

Now parse the old Writable format. It was a list of Map entries. Each map entry was a key and a value of a byte []. The old map format had a byte before each entry that held a code which was short for the key or value type. We know it was a byte [] so in below we just read and dump it.

Throws:

IOException

parsePB

void parsePB(org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos.FileInfoProto fip)

Fill our map with content of the pb we read off disk

Parameters:

fip - protobuf message to read

initTrailerAndContext

public void initTrailerAndContext(ReaderContext context,
 org.apache.hadoop.conf.Configuration conf)
                           throws IOException

Throws:

IOException

initMetaAndIndex

public void initMetaAndIndex(HFile.Reader reader)
                      throws IOException

should be called after initTrailerAndContext

Throws:

IOException

createHFileContext

private HFileContext createHFileContext(org.apache.hadoop.fs.Path path,
 FixedFileTrailer trailer,
 org.apache.hadoop.conf.Configuration conf)
                                 throws IOException

Throws:

IOException

loadMetaInfo

private void loadMetaInfo(HFileBlock.BlockIterator blockIter,
 HFileContext hfileContext)
                   throws IOException

Throws:

IOException

checkFileVersion

private void checkFileVersion(org.apache.hadoop.fs.Path path)

File version check is a little sloppy. We read v3 files but can also read v2 files if their content has been pb'd; files written with 0.98.

close

public void close()

getMajorVersion

public int getMajorVersion()

setTrailer

public void setTrailer(FixedFileTrailer trailer)

getTrailer

public FixedFileTrailer getTrailer()

getDataBlockIndexReader

public HFileBlockIndex.CellBasedKeyBlockIndexReader getDataBlockIndexReader()

getMetaBlockIndexReader

public HFileBlockIndex.ByteArrayKeyBlockIndexReader getMetaBlockIndexReader()

getHFileContext

public HFileContext getHFileContext()

getLoadOnOpenBlocks

public List<HFileBlock> getLoadOnOpenBlocks()

getLastKeyCell

public ExtendedCell getLastKeyCell()

getAvgKeyLen

public int getAvgKeyLen()

getAvgValueLen

public int getAvgValueLen()

getKeyOfBiggestCell

public String getKeyOfBiggestCell()

getLenOfBiggestCell

public long getLenOfBiggestCell()

shouldIncludeMemStoreTS

public boolean shouldIncludeMemStoreTS()

isDecodeMemstoreTS

public boolean isDecodeMemstoreTS()