Package | Description |
---|---|
org.apache.hadoop.contrib.index.lucene | |
org.apache.hadoop.contrib.index.mapred | |
org.apache.hadoop.examples.terasort |
This package consists of 3 map/reduce applications for Hadoop to
compete in the annual terabyte sort
competition.
|
org.apache.hadoop.fs |
An abstract file system API.
|
org.apache.hadoop.fs.ftp | |
org.apache.hadoop.fs.kfs |
A client for the Kosmos filesystem (KFS)
|
org.apache.hadoop.fs.s3 |
A distributed, block-based implementation of
FileSystem that uses Amazon S3
as a backing store. |
org.apache.hadoop.fs.s3native |
A distributed implementation of
FileSystem for reading and writing files on
Amazon S3. |
org.apache.hadoop.io |
Generic i/o code for use when reading and writing data to the network,
to databases, and to files.
|
org.apache.hadoop.mapred |
A software framework for easily writing applications which process vast
amounts of data (multi-terabyte data-sets) parallelly on large clusters
(thousands of nodes) built of commodity hardware in a reliable, fault-tolerant
manner.
|
org.apache.hadoop.mapred.lib |
Library of generally useful mappers, reducers, and partitioners.
|
org.apache.hadoop.mapred.lib.db |
org.apache.hadoop.mapred.lib.db Package
|
org.apache.hadoop.streaming |
Hadoop Streaming is a utility which allows users to create and run
Map-Reduce jobs with any executables (e.g.
|
Constructor and Description |
---|
FileSystemDirectory(FileSystem fs,
Path directory,
boolean create,
Configuration conf)
Constructor
|
ShardWriter(FileSystem fs,
Shard shard,
String tempDir,
IndexUpdateConfiguration iconf)
Constructor
|
Modifier and Type | Method and Description |
---|---|
RecordWriter<Shard,Text> |
IndexUpdateOutputFormat.getRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable progress) |
Modifier and Type | Method and Description |
---|---|
RecordWriter<Text,Text> |
TeraOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
Modifier and Type | Class and Description |
---|---|
class |
ChecksumFileSystem
Abstract Checksumed FileSystem.
|
class |
FilterFileSystem
A
FilterFileSystem contains
some other file system, which it uses as
its basic file system, possibly transforming
the data along the way or providing additional
functionality. |
class |
HarFileSystem
This is an implementation of the Hadoop Archive
Filesystem.
|
class |
InMemoryFileSystem
Deprecated.
|
class |
LocalFileSystem
Implement the FileSystem API for the checksumed local filesystem.
|
class |
RawLocalFileSystem
Implement the FileSystem API for the raw local filesystem.
|
Modifier and Type | Field and Description |
---|---|
protected FileSystem |
FilterFileSystem.fs |
protected FileSystem |
FsShell.fs |
Modifier and Type | Method and Description |
---|---|
static FileSystem |
FileSystem.get(Configuration conf)
Returns the configured filesystem implementation.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority.
|
FileSystem |
Path.getFileSystem(Configuration conf)
Return the FileSystem that owns this Path.
|
static FileSystem |
FileSystem.getNamed(String name,
Configuration conf)
Deprecated.
call #get(URI,Configuration) instead.
|
FileSystem |
LocalFileSystem.getRaw() |
FileSystem |
ChecksumFileSystem.getRawFileSystem()
get the raw file system
|
Modifier and Type | Method and Description |
---|---|
static boolean |
FileUtil.copy(File src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy local files to a FileSystem.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path[] srcs,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf) |
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
File dst,
boolean deleteSource,
Configuration conf)
Copy FileSystem files to local files.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copyMerge(FileSystem srcFS,
Path srcDir,
FileSystem dstFS,
Path dstFile,
boolean deleteSource,
Configuration conf,
String addString)
Copy all files in a directory to one output file (merge).
|
static FSDataOutputStream |
FileSystem.create(FileSystem fs,
Path file,
FsPermission permission)
create a file with the provided permission
The permission of the file is set to be the provided permission as in
setPermission, not permission&~umask
It is implemented using two RPCs.
|
static void |
FileUtil.fullyDelete(FileSystem fs,
Path dir)
Deprecated.
|
Path |
Path.makeQualified(FileSystem fs)
Returns a qualified path object.
|
static boolean |
FileSystem.mkdirs(FileSystem fs,
Path dir,
FsPermission permission)
create a directory with the provided permission
The permission of the directory is set to be the provided permission as in
setPermission, not permission&~umask
|
Modifier and Type | Method and Description |
---|---|
static FileSystem.Statistics |
FileSystem.getStatistics(String scheme,
Class<? extends FileSystem> cls)
Get the statistics for a particular file system
|
Constructor and Description |
---|
ChecksumFileSystem(FileSystem fs) |
FilterFileSystem(FileSystem fs) |
HarFileSystem(FileSystem fs)
Constructor to create a HarFileSystem with an
underlying filesystem.
|
LocalFileSystem(FileSystem rawLocalFileSystem) |
Trash(FileSystem fs,
Configuration conf)
Construct a trash can accessor for the FileSystem provided.
|
Modifier and Type | Class and Description |
---|---|
class |
FTPFileSystem
A
FileSystem backed by an FTP client provided by Apache Commons Net. |
Modifier and Type | Class and Description |
---|---|
class |
KosmosFileSystem
A FileSystem backed by KFS.
|
Modifier and Type | Class and Description |
---|---|
class |
S3FileSystem
A block-based
FileSystem backed by
Amazon S3. |
Modifier and Type | Class and Description |
---|---|
class |
NativeS3FileSystem
A
FileSystem for reading and writing files stored on
Amazon S3. |
Modifier and Type | Method and Description |
---|---|
protected SequenceFile.Reader |
MapFile.Reader.createDataFileReader(FileSystem fs,
Path dataFile,
Configuration conf)
Override this method to specialize the type of
SequenceFile.Reader returned. |
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
Progressable progress)
Construct the preferred type of SequenceFile Writer.
|
static void |
BloomMapFile.delete(FileSystem fs,
String name) |
static void |
MapFile.delete(FileSystem fs,
String name)
Deletes the named map file.
|
static long |
MapFile.fix(FileSystem fs,
Path dir,
Class<? extends Writable> keyClass,
Class<? extends Writable> valueClass,
boolean dryrun,
Configuration conf)
This method attempts to fix a corrupt MapFile by re-creating its index.
|
protected void |
MapFile.Reader.open(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf) |
protected FSDataInputStream |
SequenceFile.Reader.openFile(FileSystem fs,
Path file,
int bufferSize,
long length)
Override this method to specialize the type of
FSDataInputStream returned. |
static void |
MapFile.rename(FileSystem fs,
String oldName,
String newName)
Renames an existing map directory.
|
Constructor and Description |
---|
Reader(FileSystem fs,
Path file,
Configuration conf)
Open the named file.
|
Reader(FileSystem fs,
String dirName,
Configuration conf)
Construct a set reader for the named set.
|
Reader(FileSystem fs,
String dirName,
Configuration conf) |
Reader(FileSystem fs,
String dirName,
Configuration conf)
Construct a map reader for the named map.
|
Reader(FileSystem fs,
String file,
Configuration conf)
Construct an array reader for the named file.
|
Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf)
Construct a set reader for the named set using the named comparator.
|
Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf) |
Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf)
Construct a map reader for the named map using the named comparator.
|
Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf,
boolean open) |
Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf,
boolean open)
Hook to allow subclasses to defer opening streams until further
initialization is complete.
|
Sorter(FileSystem fs,
Class<? extends WritableComparable> keyClass,
Class valClass,
Configuration conf)
Sort and merge files containing the named classes.
|
Sorter(FileSystem fs,
RawComparator comparator,
Class keyClass,
Class valClass,
Configuration conf)
Sort and merge using an arbitrary
RawComparator . |
Writer(Configuration conf,
FileSystem fs,
String file,
Class<? extends Writable> valClass)
Create the named file for values of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String file,
Class<? extends Writable> valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Create the named file for values of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass)
Create the named map for keys of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class<? extends Writable> valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress)
Create the named map for keys of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Create the named map for keys of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Create the named map for keys of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
SequenceFile.CompressionType compress)
Create a set naming the element class and compression type.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass)
Create the named map using the named key comparator.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress)
Create the named map using the named key comparator.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Create the named map using the named key comparator.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Create the named map using the named key comparator.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
SequenceFile.CompressionType compress)
Create a set naming the element comparator and compression type.
|
Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Create the named file.
|
Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
Progressable progress,
SequenceFile.Metadata metadata)
Create the named file with write-progress reporter.
|
Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
Progressable progress,
SequenceFile.Metadata metadata)
Create the named file with write-progress reporter.
|
Writer(FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass)
Deprecated.
pass a Configuration too
|
Modifier and Type | Method and Description |
---|---|
FileSystem |
JobClient.getFs()
Get a filesystem handle.
|
Modifier and Type | Method and Description |
---|---|
void |
SequenceFileAsBinaryOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
void |
FileOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
void |
OutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job)
Deprecated.
Check for validity of the output-specification for the job.
|
static MapFile.Reader[] |
MapFileOutputFormat.getReaders(FileSystem ignored,
Path dir,
Configuration conf)
Open the output generated by this format.
|
RecordWriter<BytesWritable,BytesWritable> |
SequenceFileAsBinaryOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
SequenceFileOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress)
Deprecated.
|
RecordWriter<WritableComparable,Writable> |
MapFileOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
TextOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress)
Deprecated.
|
abstract RecordWriter<K,V> |
FileOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
OutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress)
Deprecated.
Get the
RecordWriter for the given job. |
static boolean |
JobClient.isJobDirValid(Path jobDirPath,
FileSystem fs)
Checks if the job directory is clean and has all the required components
for (re) starting the job
|
protected boolean |
TextInputFormat.isSplitable(FileSystem fs,
Path file)
Deprecated.
|
protected boolean |
FileInputFormat.isSplitable(FileSystem fs,
Path filename)
Deprecated.
Is the given filename splitable? Usually, true, but if the file is
stream compressed, it will not be.
|
protected boolean |
KeyValueTextInputFormat.isSplitable(FileSystem fs,
Path file) |
static void |
JobHistory.parseHistoryFromFS(String path,
JobHistory.Listener l,
FileSystem fs)
Parses history file and invokes Listener.handle() for
each line of history.
|
static void |
DefaultJobHistoryParser.parseJobTasks(String jobHistoryFile,
JobHistory.JobInfo job,
FileSystem fs)
Populates a JobInfo object from the job's history log file.
|
Modifier and Type | Field and Description |
---|---|
protected FileSystem |
CombineFileRecordReader.fs |
Modifier and Type | Method and Description |
---|---|
void |
NullOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job)
Deprecated.
|
protected abstract RecordWriter<K,V> |
MultipleOutputFormat.getBaseRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3) |
protected RecordWriter<K,V> |
MultipleTextOutputFormat.getBaseRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3) |
protected RecordWriter<K,V> |
MultipleSequenceFileOutputFormat.getBaseRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3) |
RecordWriter<K,V> |
MultipleOutputFormat.getRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3)
Create a composite record writer that can write key/value data to different
output files
|
RecordWriter<K,V> |
NullOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
void |
DBOutputFormat.checkOutputSpecs(FileSystem filesystem,
JobConf job)
Check for validity of the output-specification for the job.
|
RecordWriter<K,V> |
DBOutputFormat.getRecordWriter(FileSystem filesystem,
JobConf job,
String name,
Progressable progress)
Get the
RecordWriter for the given job. |
Constructor and Description |
---|
StreamBaseRecordReader(FSDataInputStream in,
FileSplit split,
Reporter reporter,
JobConf job,
FileSystem fs) |
StreamXmlRecordReader(FSDataInputStream in,
FileSplit split,
Reporter reporter,
JobConf job,
FileSystem fs) |
Copyright © 2010 The Apache Software Foundation