Package org.apache.hadoop.hbase.test

Class IntegrationTestLoadCommonCrawl

java.lang.Object
org.apache.hadoop.hbase.util.AbstractHBaseTool
org.apache.hadoop.hbase.IntegrationTestBase
org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool
public class IntegrationTestLoadCommonCrawl extends IntegrationTestBase
This integration test loads successful resource retrieval records from the Common Crawl (https://commoncrawl.org/) public dataset into an HBase table and writes records that can be used to later verify the presence and integrity of those records.

Run like:

./bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl \
   -Dfs.s3n.awsAccessKeyId=<AWS access key> \
   -Dfs.s3n.awsSecretAccessKey=<AWS secret key> \
   /path/to/test-CC-MAIN-2021-10-warc.paths.gz \
   /path/to/tmp/warc-loader-output

Access to the Common Crawl dataset in S3 is made available to anyone by Amazon AWS, but Hadoop's S3N filesystem still requires valid access credentials to initialize.

The input path can either specify a directory or a file. The file may optionally be compressed with gzip. If a directory, the loader expects the directory to contain one or more WARC files from the Common Crawl dataset. If a file, the loader expects a list of Hadoop S3N URIs which point to S3 locations for one or more WARC files from the Common Crawl dataset, one URI per line. Lines should be terminated with the UNIX line terminator.

Included in hbase-it/src/test/resources/CC-MAIN-2021-10-warc.paths.gz is a list of all WARC files comprising the Q1 2021 crawl archive. There are 64,000 WARC files in this data set, each containing ~1GB of gzipped data. The WARC files contain several record types, such as metadata, request, and response, but we only load the response record types. If the HBase table schema does not specify compression (by default) there is roughly a 10x expansion. Loading the full crawl archive results in a table approximately 640 TB in size.

The loader can optionally drive read load during ingest by incrementing counters for each URL discovered in content. Add -DIntegrationTestLoadCommonCrawl.increments=true to the command line to enable.

You can also split the Loader and Verify stages:

Load with:

./bin/hbase 'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Loader' \
   -files /path/to/hadoop-aws.jar \
   -Dfs.s3n.awsAccessKeyId=<AWS access key> \
   -Dfs.s3n.awsSecretAccessKey=<AWS secret key> \
   /path/to/test-CC-MAIN-2021-10-warc.paths.gz \
   /path/to/tmp/warc-loader-output

Note: The hadoop-aws jar will be needed at runtime to instantiate the S3N filesystem. Use the -files ToolRunner argument to add it.

Verify with:

./bin/hbase 'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Verify' \
   /path/to/tmp/warc-loader-output