Package | Description |
---|---|
org.apache.hadoop.conf |
Configuration of system parameters.
|
org.apache.hadoop.contrib.failmon | |
org.apache.hadoop.contrib.index.example | |
org.apache.hadoop.contrib.index.lucene | |
org.apache.hadoop.contrib.index.mapred | |
org.apache.hadoop.examples |
Hadoop example code.
|
org.apache.hadoop.filecache | |
org.apache.hadoop.fs |
An abstract file system API.
|
org.apache.hadoop.fs.ftp | |
org.apache.hadoop.fs.kfs |
A client for the Kosmos filesystem (KFS)
|
org.apache.hadoop.fs.permission | |
org.apache.hadoop.fs.s3 |
A distributed, block-based implementation of
FileSystem that uses Amazon S3
as a backing store. |
org.apache.hadoop.fs.s3native |
A distributed implementation of
FileSystem for reading and writing files on
Amazon S3. |
org.apache.hadoop.fs.shell | |
org.apache.hadoop.http | |
org.apache.hadoop.io |
Generic i/o code for use when reading and writing data to the network,
to databases, and to files.
|
org.apache.hadoop.io.compress | |
org.apache.hadoop.io.compress.zlib | |
org.apache.hadoop.io.file.tfile | |
org.apache.hadoop.io.serializer |
This package provides a mechanism for using different serialization frameworks
in Hadoop.
|
org.apache.hadoop.ipc |
Tools to help define network clients and servers.
|
org.apache.hadoop.mapred |
A software framework for easily writing applications which process vast
amounts of data (multi-terabyte data-sets) parallelly on large clusters
(thousands of nodes) built of commodity hardware in a reliable, fault-tolerant
manner.
|
org.apache.hadoop.mapred.join |
Given a set of sorted datasets keyed with the same class and yielding equal
partitions, it is possible to effect a join of those datasets prior to the map.
|
org.apache.hadoop.mapred.lib |
Library of generally useful mappers, reducers, and partitioners.
|
org.apache.hadoop.mapred.pipes |
Hadoop Pipes allows C++ code to use Hadoop DFS and map/reduce.
|
org.apache.hadoop.mapred.tools | |
org.apache.hadoop.mapreduce | |
org.apache.hadoop.mapreduce.lib.input | |
org.apache.hadoop.net |
Network-related classes.
|
org.apache.hadoop.security | |
org.apache.hadoop.security.authorize | |
org.apache.hadoop.streaming |
Hadoop Streaming is a utility which allows users to create and run
Map-Reduce jobs with any executables (e.g.
|
org.apache.hadoop.util |
Common utilities.
|
org.apache.hadoop.util.hash |
Modifier and Type | Method and Description |
---|---|
Configuration |
Configurable.getConf()
Return the configuration used by this object.
|
Configuration |
Configured.getConf() |
Modifier and Type | Method and Description |
---|---|
void |
Configurable.setConf(Configuration conf)
Set the configuration to be used by this object.
|
void |
Configured.setConf(Configuration conf) |
Constructor and Description |
---|
Configuration(Configuration other)
A new configuration with the same settings cloned from another.
|
Configured(Configuration conf)
Construct a Configured.
|
Constructor and Description |
---|
Executor(Configuration conf)
Create an instance of the class and read the configuration
file to determine the set of jobs that will be run and the
maximum interval for which the thread can sleep before it
wakes up to execute a monitoring job on the node.
|
Constructor and Description |
---|
LineDocRecordReader(Configuration job,
FileSplit split)
Constructor
|
Constructor and Description |
---|
FileSystemDirectory(FileSystem fs,
Path directory,
boolean create,
Configuration conf)
Constructor
|
Modifier and Type | Method and Description |
---|---|
Configuration |
IndexUpdateConfiguration.getConfiguration()
Get the underlying configuration object.
|
Modifier and Type | Method and Description |
---|---|
void |
IndexUpdater.run(Configuration conf,
Path[] inputPaths,
Path outputPath,
int numMapTasks,
Shard[] shards) |
void |
IIndexUpdater.run(Configuration conf,
Path[] inputPaths,
Path outputPath,
int numMapTasks,
Shard[] shards)
Create a Map/Reduce job configuration and run the Map/Reduce job to
analyze documents and update Lucene instances in parallel.
|
Constructor and Description |
---|
IndexUpdateConfiguration(Configuration conf)
Constructor
|
Constructor and Description |
---|
MultiFileLineRecordReader(Configuration conf,
MultiFileSplit split) |
Modifier and Type | Method and Description |
---|---|
static void |
DistributedCache.addArchiveToClassPath(Path archive,
Configuration conf)
Add an archive path to the current set of classpath entries.
|
static void |
DistributedCache.addCacheArchive(URI uri,
Configuration conf)
Add a archives to be localized to the conf
|
static void |
DistributedCache.addCacheFile(URI uri,
Configuration conf)
Add a file to be localized to the conf
|
static void |
DistributedCache.addFileToClassPath(Path file,
Configuration conf)
Add an file path to the current set of classpath entries It adds the file
to cache as well.
|
static void |
DistributedCache.createAllSymlink(Configuration conf,
File jobCacheDir,
File workDir)
This method create symlinks for all files in a given dir in another directory
|
static void |
DistributedCache.createSymlink(Configuration conf)
This method allows you to create symlinks in the current working directory
of the task to all the cache files/archives
|
static Path[] |
DistributedCache.getArchiveClassPaths(Configuration conf)
Get the archive entries in classpath as an array of Path
|
static String[] |
DistributedCache.getArchiveTimestamps(Configuration conf)
Get the timestamps of the archives
|
static URI[] |
DistributedCache.getCacheArchives(Configuration conf)
Get cache archives set in the Configuration
|
static URI[] |
DistributedCache.getCacheFiles(Configuration conf)
Get cache files set in the Configuration
|
static Path[] |
DistributedCache.getFileClassPaths(Configuration conf)
Get the file entries in classpath as an array of Path
|
static String[] |
DistributedCache.getFileTimestamps(Configuration conf)
Get the timestamps of the files
|
static Path |
DistributedCache.getLocalCache(URI cache,
Configuration conf,
Path baseDir,
boolean isArchive,
long confFileStamp,
Path currentWorkDir)
Get the locally cached file or archive; it could either be
previously cached (and valid) or copy it from the
FileSystem now. |
static Path |
DistributedCache.getLocalCache(URI cache,
Configuration conf,
Path baseDir,
FileStatus fileStatus,
boolean isArchive,
long confFileStamp,
Path currentWorkDir)
Get the locally cached file or archive; it could either be
previously cached (and valid) or copy it from the
FileSystem now. |
static Path |
DistributedCache.getLocalCache(URI cache,
Configuration conf,
Path baseDir,
FileStatus fileStatus,
boolean isArchive,
long confFileStamp,
Path currentWorkDir,
boolean honorSymLinkConf)
Get the locally cached file or archive; it could either be
previously cached (and valid) or copy it from the
FileSystem now. |
static Path[] |
DistributedCache.getLocalCacheArchives(Configuration conf)
Return the path array of the localized caches
|
static Path[] |
DistributedCache.getLocalCacheFiles(Configuration conf)
Return the path array of the localized files
|
static boolean |
DistributedCache.getSymlink(Configuration conf)
This method checks to see if symlinks are to be create for the
localized cache files in the current working directory
|
static long |
DistributedCache.getTimestamp(Configuration conf,
URI cache)
Returns mtime of a given cache file on hdfs.
|
static String |
DistributedCache.makeRelative(URI cache,
Configuration conf) |
static void |
DistributedCache.purgeCache(Configuration conf)
Clear the entire contents of the cache and delete the backing files.
|
static void |
DistributedCache.releaseCache(URI cache,
Configuration conf)
This is the opposite of getlocalcache.
|
static void |
DistributedCache.setArchiveTimestamps(Configuration conf,
String timestamps)
This is to check the timestamp of the archives to be localized
|
static void |
DistributedCache.setCacheArchives(URI[] archives,
Configuration conf)
Set the configuration with the given set of archives
|
static void |
DistributedCache.setCacheFiles(URI[] files,
Configuration conf)
Set the configuration with the given set of files
|
static void |
DistributedCache.setFileTimestamps(Configuration conf,
String timestamps)
This is to check the timestamp of the files to be localized
|
static void |
DistributedCache.setLocalArchives(Configuration conf,
String str)
Set the conf to contain the location for localized archives
|
static void |
DistributedCache.setLocalFiles(Configuration conf,
String str)
Set the conf to contain the location for localized files
|
Modifier and Type | Method and Description |
---|---|
Configuration |
FilterFileSystem.getConf() |
Modifier and Type | Method and Description |
---|---|
static boolean |
FileUtil.copy(File src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy local files to a FileSystem.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path[] srcs,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf) |
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
File dst,
boolean deleteSource,
Configuration conf)
Copy FileSystem files to local files.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copyMerge(FileSystem srcFS,
Path srcDir,
FileSystem dstFS,
Path dstFile,
boolean deleteSource,
Configuration conf,
String addString)
Copy all files in a directory to one output file (merge).
|
File |
LocalDirAllocator.createTmpFileForWrite(String pathStr,
long size,
Configuration conf)
Creates a temporary file in the local FS.
|
static FileSystem |
FileSystem.get(Configuration conf)
Returns the configured filesystem implementation.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority.
|
static URI |
FileSystem.getDefaultUri(Configuration conf)
Get the default filesystem URI from a configuration.
|
FileSystem |
Path.getFileSystem(Configuration conf)
Return the FileSystem that owns this Path.
|
static LocalFileSystem |
FileSystem.getLocal(Configuration conf)
Get the local file syste
|
Path |
LocalDirAllocator.getLocalPathForWrite(String pathStr,
Configuration conf)
Get a path from the local FS.
|
Path |
LocalDirAllocator.getLocalPathForWrite(String pathStr,
long size,
Configuration conf)
Get a path from the local FS.
|
Path |
LocalDirAllocator.getLocalPathToRead(String pathStr,
Configuration conf)
Get a path from the local FS for reading.
|
static FileSystem |
FileSystem.getNamed(String name,
Configuration conf)
Deprecated.
call #get(URI,Configuration) instead.
|
boolean |
LocalDirAllocator.ifExists(String pathStr,
Configuration conf)
We search through all the configured dirs for the file's existence
and return true when we find
|
void |
FilterFileSystem.initialize(URI name,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
void |
RawLocalFileSystem.initialize(URI uri,
Configuration conf) |
void |
HarFileSystem.initialize(URI name,
Configuration conf)
Initialize a Har filesystem per har archive.
|
void |
FileSystem.initialize(URI name,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
void |
ChecksumFileSystem.setConf(Configuration conf) |
static void |
FileSystem.setDefaultUri(Configuration conf,
String uri)
Set the default filesystem URI in a configuration.
|
static void |
FileSystem.setDefaultUri(Configuration conf,
URI uri)
Set the default filesystem URI in a configuration.
|
Constructor and Description |
---|
DF(File path,
Configuration conf) |
DU(File path,
Configuration conf)
Keeps track of disk usage.
|
FsShell(Configuration conf) |
FsUrlStreamHandlerFactory(Configuration conf) |
InMemoryFileSystem(URI uri,
Configuration conf)
Deprecated.
|
Trash(Configuration conf)
Construct a trash can accessor.
|
Trash(FileSystem fs,
Configuration conf)
Construct a trash can accessor for the FileSystem provided.
|
Modifier and Type | Method and Description |
---|---|
void |
FTPFileSystem.initialize(URI uri,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
void |
KosmosFileSystem.initialize(URI uri,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static FsPermission |
FsPermission.getUMask(Configuration conf)
Get the user file creation mask (umask)
|
static void |
FsPermission.setUMask(Configuration conf,
FsPermission umask)
Set the user file creation mask (umask)
|
Modifier and Type | Method and Description |
---|---|
void |
S3FileSystem.initialize(URI uri,
Configuration conf) |
void |
S3Credentials.initialize(URI uri,
Configuration conf) |
void |
FileSystemStore.initialize(URI uri,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
void |
NativeS3FileSystem.initialize(URI uri,
Configuration conf) |
Constructor and Description |
---|
Command(Configuration conf)
Constructor
|
Count(String[] cmd,
int pos,
Configuration conf)
Constructor
|
Modifier and Type | Method and Description |
---|---|
void |
HttpServer.addSslListener(InetSocketAddress addr,
Configuration sslConf,
boolean needClientAuth)
Configure an ssl listener on the server.
|
protected org.mortbay.jetty.Connector |
HttpServer.createBaseListener(Configuration conf)
Create a required listener for the Jetty instance listening on the port
provided.
|
Constructor and Description |
---|
HttpServer(String name,
String bindAddress,
int port,
boolean findPort,
Configuration conf)
Create a status server on the given port.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
GenericWritable.getConf() |
Configuration |
ObjectWritable.getConf() |
Configuration |
AbstractMapWritable.getConf() |
Modifier and Type | Method and Description |
---|---|
static <T extends Writable> |
WritableUtils.clone(T orig,
Configuration conf)
Make a copy of a writable object using serialization to a buffer.
|
static void |
IOUtils.copyBytes(InputStream in,
OutputStream out,
Configuration conf)
Copies from one stream to another.
|
static void |
IOUtils.copyBytes(InputStream in,
OutputStream out,
Configuration conf,
boolean close)
Copies from one stream to another.
|
protected SequenceFile.Reader |
MapFile.Reader.createDataFileReader(FileSystem fs,
Path dataFile,
Configuration conf)
Override this method to specialize the type of
SequenceFile.Reader returned. |
static SequenceFile.Writer |
SequenceFile.createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Construct the preferred type of 'raw' SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
SequenceFile.Metadata metadata)
Construct the preferred type of 'raw' SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
Progressable progress)
Construct the preferred type of SequenceFile Writer.
|
static long |
MapFile.fix(FileSystem fs,
Path dir,
Class<? extends Writable> keyClass,
Class<? extends Writable> valueClass,
boolean dryrun,
Configuration conf)
This method attempts to fix a corrupt MapFile by re-creating its index.
|
static Class<?> |
WritableName.getClass(String name,
Configuration conf)
Return the class for a name.
|
static SequenceFile.CompressionType |
SequenceFile.getCompressionType(Configuration job)
Deprecated.
Use
SequenceFileOutputFormat.getOutputCompressionType(org.apache.hadoop.mapred.JobConf)
to get SequenceFile.CompressionType for job-outputs. |
static <K> K |
DefaultStringifier.load(Configuration conf,
String keyName,
Class<K> itemClass)
Restores the object from the configuration.
|
static <K> K[] |
DefaultStringifier.loadArray(Configuration conf,
String keyName,
Class<K> itemClass)
Restores the array of objects from the configuration.
|
static Writable |
WritableFactories.newInstance(Class<? extends Writable> c,
Configuration conf)
Create a new instance of a class with a defined factory.
|
protected void |
MapFile.Reader.open(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf) |
static Object |
ObjectWritable.readObject(DataInput in,
Configuration conf)
|
static Object |
ObjectWritable.readObject(DataInput in,
ObjectWritable objectWritable,
Configuration conf)
|
static void |
SequenceFile.setCompressionType(Configuration job,
SequenceFile.CompressionType val)
Deprecated.
Use the one of the many SequenceFile.createWriter methods to specify
the
SequenceFile.CompressionType while creating the SequenceFile or
SequenceFileOutputFormat.setOutputCompressionType(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.io.SequenceFile.CompressionType)
to specify the SequenceFile.CompressionType for job-outputs.
or |
void |
GenericWritable.setConf(Configuration conf) |
void |
ObjectWritable.setConf(Configuration conf) |
void |
AbstractMapWritable.setConf(Configuration conf) |
static void |
MapFile.Writer.setIndexInterval(Configuration conf,
int interval)
Sets the index interval and stores it in conf
|
static <K> void |
DefaultStringifier.store(Configuration conf,
K item,
String keyName)
Stores the item in the configuration with the given keyName.
|
static <K> void |
DefaultStringifier.storeArray(Configuration conf,
K[] items,
String keyName)
Stores the array of items in the configuration with the given keyName.
|
static void |
ObjectWritable.writeObject(DataOutput out,
Object instance,
Class declaredClass,
Configuration conf)
|
Constructor and Description |
---|
DefaultStringifier(Configuration conf,
Class<T> c) |
Reader(FileSystem fs,
Path file,
Configuration conf)
Open the named file.
|
Reader(FileSystem fs,
String dirName,
Configuration conf)
Construct a set reader for the named set.
|
Reader(FileSystem fs,
String dirName,
Configuration conf) |
Reader(FileSystem fs,
String dirName,
Configuration conf)
Construct a map reader for the named map.
|
Reader(FileSystem fs,
String file,
Configuration conf)
Construct an array reader for the named file.
|
Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf)
Construct a set reader for the named set using the named comparator.
|
Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf) |
Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf)
Construct a map reader for the named map using the named comparator.
|
Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf,
boolean open) |
Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf,
boolean open)
Hook to allow subclasses to defer opening streams until further
initialization is complete.
|
Sorter(FileSystem fs,
Class<? extends WritableComparable> keyClass,
Class valClass,
Configuration conf)
Sort and merge files containing the named classes.
|
Sorter(FileSystem fs,
RawComparator comparator,
Class keyClass,
Class valClass,
Configuration conf)
Sort and merge using an arbitrary
RawComparator . |
Writer(Configuration conf,
FileSystem fs,
String file,
Class<? extends Writable> valClass)
Create the named file for values of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String file,
Class<? extends Writable> valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Create the named file for values of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass)
Create the named map for keys of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class<? extends Writable> valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress)
Create the named map for keys of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Create the named map for keys of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Create the named map for keys of the named class.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
SequenceFile.CompressionType compress)
Create a set naming the element class and compression type.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass)
Create the named map using the named key comparator.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress)
Create the named map using the named key comparator.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Create the named map using the named key comparator.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress) |
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Create the named map using the named key comparator.
|
Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
SequenceFile.CompressionType compress)
Create a set naming the element comparator and compression type.
|
Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Create the named file.
|
Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
Progressable progress,
SequenceFile.Metadata metadata)
Create the named file with write-progress reporter.
|
Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
Progressable progress,
SequenceFile.Metadata metadata)
Create the named file with write-progress reporter.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
DefaultCodec.getConf() |
Modifier and Type | Method and Description |
---|---|
static List<Class<? extends CompressionCodec>> |
CompressionCodecFactory.getCodecClasses(Configuration conf)
Get the list of codecs listed in the configuration
|
static void |
CompressionCodecFactory.setCodecClasses(Configuration conf,
List<Class> classes)
Sets a list of codec classes in the configuration.
|
void |
DefaultCodec.setConf(Configuration conf) |
Constructor and Description |
---|
CompressionCodecFactory(Configuration conf)
Find the codecs specified in the config value io.compression.codecs
and register them.
|
Modifier and Type | Method and Description |
---|---|
static Compressor |
ZlibFactory.getZlibCompressor(Configuration conf)
Return the appropriate implementation of the zlib compressor.
|
static Class<? extends Compressor> |
ZlibFactory.getZlibCompressorType(Configuration conf)
Return the appropriate type of the zlib compressor.
|
static Decompressor |
ZlibFactory.getZlibDecompressor(Configuration conf)
Return the appropriate implementation of the zlib decompressor.
|
static Class<? extends Decompressor> |
ZlibFactory.getZlibDecompressorType(Configuration conf)
Return the appropriate type of the zlib decompressor.
|
static boolean |
ZlibFactory.isNativeZlibLoaded(Configuration conf)
Check if native-zlib code is loaded & initialized correctly and
can be loaded for this job.
|
Constructor and Description |
---|
Reader(FSDataInputStream fsdis,
long fileLength,
Configuration conf)
Constructor
|
Writer(FSDataOutputStream fsdos,
int minBlockSize,
String compressName,
String comparator,
Configuration conf)
Constructor
|
Constructor and Description |
---|
SerializationFactory(Configuration conf)
Serializations are found by reading the
io.serializations
property from conf , which is a comma-delimited list of
classnames. |
Modifier and Type | Method and Description |
---|---|
static Object[] |
RPC.call(Method method,
Object[][] params,
InetSocketAddress[] addrs,
Configuration conf)
Deprecated.
|
static Object[] |
RPC.call(Method method,
Object[][] params,
InetSocketAddress[] addrs,
UserGroupInformation ticket,
Configuration conf)
Expert: Make multiple, parallel calls to a set of servers.
|
static VersionedProtocol |
RPC.getProxy(Class<?> protocol,
long clientVersion,
InetSocketAddress addr,
Configuration conf)
Construct a client-side proxy object with the default SocketFactory
|
static VersionedProtocol |
RPC.getProxy(Class<?> protocol,
long clientVersion,
InetSocketAddress addr,
Configuration conf,
SocketFactory factory)
Construct a client-side proxy object that implements the named protocol,
talking to a server at the named address.
|
static VersionedProtocol |
RPC.getProxy(Class<?> protocol,
long clientVersion,
InetSocketAddress addr,
UserGroupInformation ticket,
Configuration conf,
SocketFactory factory)
Construct a client-side proxy object that implements the named protocol,
talking to a server at the named address.
|
static RPC.Server |
RPC.getServer(Object instance,
String bindAddress,
int port,
Configuration conf)
Construct a server for a protocol implementation instance listening on a
port and address.
|
static RPC.Server |
RPC.getServer(Object instance,
String bindAddress,
int port,
int numHandlers,
boolean verbose,
Configuration conf)
Construct a server for a protocol implementation instance listening on a
port and address.
|
static void |
Client.setPingInterval(Configuration conf,
int pingInterval)
set the ping interval value in configuration
|
static VersionedProtocol |
RPC.waitForProxy(Class protocol,
long clientVersion,
InetSocketAddress addr,
Configuration conf) |
Constructor and Description |
---|
Client(Class<? extends Writable> valueClass,
Configuration conf)
Construct an IPC client with the default SocketFactory
|
Client(Class<? extends Writable> valueClass,
Configuration conf,
SocketFactory factory)
Construct an IPC client whose values are of the given
Writable
class. |
Server(Object instance,
Configuration conf,
String bindAddress,
int port)
Construct an RPC server.
|
Server(Object instance,
Configuration conf,
String bindAddress,
int port,
int numHandlers,
boolean verbose)
Construct an RPC server.
|
Server(String bindAddress,
int port,
Class<? extends Writable> paramClass,
int handlerCount,
Configuration conf) |
Server(String bindAddress,
int port,
Class<? extends Writable> paramClass,
int handlerCount,
Configuration conf,
String serverName)
Constructs a server listening on the named port and address.
|
Modifier and Type | Class and Description |
---|---|
class |
JobConf
Deprecated.
Use
Configuration instead |
Modifier and Type | Field and Description |
---|---|
protected Configuration |
SequenceFileRecordReader.conf |
Modifier and Type | Method and Description |
---|---|
Configuration |
SequenceFileInputFilter.FilterBase.getConf() |
Modifier and Type | Method and Description |
---|---|
static InetSocketAddress |
JobTracker.getAddress(Configuration conf) |
static int |
SkipBadRecords.getAttemptsToStartSkipping(Configuration conf)
Get the number of Task attempts AFTER which skip mode
will be kicked off.
|
static boolean |
SkipBadRecords.getAutoIncrMapperProcCount(Configuration conf)
Get the flag which if set to true,
SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented
by MapRunner after invoking the map function. |
static boolean |
SkipBadRecords.getAutoIncrReducerProcCount(Configuration conf)
Get the flag which if set to true,
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented
by framework after invoking the reduce function. |
static Class<? extends org.apache.hadoop.mapred.JobTrackerInstrumentation> |
JobTracker.getInstrumentationClass(Configuration conf) |
static Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> |
TaskTracker.getInstrumentationClass(Configuration conf) |
static long |
SkipBadRecords.getMapperMaxSkipRecords(Configuration conf)
Get the number of acceptable skip records surrounding the bad record PER
bad record in mapper.
|
static SequenceFile.Reader[] |
SequenceFileOutputFormat.getReaders(Configuration conf,
Path dir)
Deprecated.
Open the output generated by this format.
|
static MapFile.Reader[] |
MapFileOutputFormat.getReaders(FileSystem ignored,
Path dir,
Configuration conf)
Open the output generated by this format.
|
static long |
SkipBadRecords.getReducerMaxSkipGroups(Configuration conf)
Get the number of acceptable skip groups surrounding the bad group PER
bad group in reducer.
|
static Path |
SkipBadRecords.getSkipOutputPath(Configuration conf)
Get the directory to which skipped records are written.
|
static void |
SkipBadRecords.setAttemptsToStartSkipping(Configuration conf,
int attemptsToStartSkipping)
Set the number of Task attempts AFTER which skip mode
will be kicked off.
|
static void |
SkipBadRecords.setAutoIncrMapperProcCount(Configuration conf,
boolean autoIncr)
Set the flag which if set to true,
SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented
by MapRunner after invoking the map function. |
static void |
SkipBadRecords.setAutoIncrReducerProcCount(Configuration conf,
boolean autoIncr)
Set the flag which if set to true,
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented
by framework after invoking the reduce function. |
void |
SequenceFileInputFilter.RegexFilter.setConf(Configuration conf)
configure the Filter by checking the configuration
|
void |
SequenceFileInputFilter.PercentFilter.setConf(Configuration conf)
configure the filter by checking the configuration
|
void |
SequenceFileInputFilter.MD5Filter.setConf(Configuration conf)
configure the filter according to configuration
|
static void |
SequenceFileInputFilter.setFilterClass(Configuration conf,
Class filterClass)
set the filter class
|
static void |
SequenceFileInputFilter.PercentFilter.setFrequency(Configuration conf,
int frequency)
set the frequency and stores it in conf
|
static void |
SequenceFileInputFilter.MD5Filter.setFrequency(Configuration conf,
int frequency)
set the filtering frequency in configuration
|
static void |
JobTracker.setInstrumentationClass(Configuration conf,
Class<? extends org.apache.hadoop.mapred.JobTrackerInstrumentation> t) |
static void |
TaskTracker.setInstrumentationClass(Configuration conf,
Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> t) |
static void |
SkipBadRecords.setMapperMaxSkipRecords(Configuration conf,
long maxSkipRecs)
Set the number of acceptable skip records surrounding the bad record PER
bad record in mapper.
|
static void |
SequenceFileInputFilter.RegexFilter.setPattern(Configuration conf,
String regex)
Define the filtering regex and stores it in conf
|
static void |
SkipBadRecords.setReducerMaxSkipGroups(Configuration conf,
long maxSkipGrps)
Set the number of acceptable skip groups surrounding the bad group PER
bad group in reducer.
|
Constructor and Description |
---|
JobClient(InetSocketAddress jobTrackAddr,
Configuration conf)
Build a job client, connect to the indicated job tracker.
|
JobConf(Configuration conf)
Deprecated.
Construct a map/reduce job configuration.
|
JobConf(Configuration conf,
Class exampleClass)
Deprecated.
Construct a map/reduce job configuration.
|
KeyValueLineRecordReader(Configuration job,
FileSplit split) |
LineReader(InputStream in,
Configuration conf)
Deprecated.
|
LineRecordReader(Configuration job,
FileSplit split)
Deprecated.
|
LineRecordReader(InputStream in,
long offset,
long endOffset,
Configuration job)
Deprecated.
|
SequenceFileAsBinaryRecordReader(Configuration conf,
FileSplit split) |
SequenceFileAsTextRecordReader(Configuration conf,
FileSplit split) |
SequenceFileRecordReader(Configuration conf,
FileSplit split) |
Modifier and Type | Method and Description |
---|---|
Configuration |
CompositeRecordReader.getConf()
Return the configuration used by this object.
|
Modifier and Type | Method and Description |
---|---|
void |
CompositeRecordReader.setConf(Configuration conf)
Set the configuration to be used by this object.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
InputSampler.getConf() |
Modifier and Type | Method and Description |
---|---|
void |
InputSampler.setConf(Configuration conf) |
Constructor and Description |
---|
Submitter(Configuration conf) |
Constructor and Description |
---|
MRAdmin(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Configuration |
JobContext.getConfiguration()
Return the configuration for the job.
|
Modifier and Type | Field and Description |
---|---|
protected Configuration |
SequenceFileRecordReader.conf |
Modifier and Type | Method and Description |
---|---|
Configuration |
ScriptBasedMapping.getConf() |
Configuration |
SocksSocketFactory.getConf() |
Modifier and Type | Method and Description |
---|---|
static SocketFactory |
NetUtils.getDefaultSocketFactory(Configuration conf)
Get the default socket factory as specified by the configuration
parameter hadoop.rpc.socket.factory.default
|
static String |
NetUtils.getServerAddress(Configuration conf,
String oldBindAddressName,
String oldPortName,
String newBindAddressName)
Deprecated.
|
static SocketFactory |
NetUtils.getSocketFactory(Configuration conf,
Class<?> clazz)
Get the socket factory for the given class according to its
configuration parameter
hadoop.rpc.socket.factory.class.<ClassName>.
|
static SocketFactory |
NetUtils.getSocketFactoryFromProperty(Configuration conf,
String propValue)
Get the socket factory corresponding to the given proxy URI.
|
void |
ScriptBasedMapping.setConf(Configuration conf) |
void |
SocksSocketFactory.setConf(Configuration conf) |
Constructor and Description |
---|
ScriptBasedMapping(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static UserGroupInformation |
UserGroupInformation.login(Configuration conf)
Login and return a UserGroupInformation object.
|
static UnixUserGroupInformation |
UnixUserGroupInformation.login(Configuration conf)
Equivalent to login(conf, false).
|
static UnixUserGroupInformation |
UnixUserGroupInformation.login(Configuration conf,
boolean save)
Get a user's name & its group names from the given configuration;
If it is not defined in the configuration, get the current user's
information from Unix.
|
static UserGroupInformation |
UserGroupInformation.readFrom(Configuration conf)
Read a
UserGroupInformation from conf |
static UnixUserGroupInformation |
UnixUserGroupInformation.readFromConf(Configuration conf,
String attr)
Read a UGI from the given
conf
The object is expected to store with the property name attr
as a comma separated string that starts
with the user name followed by group names. |
static void |
UnixUserGroupInformation.saveToConf(Configuration conf,
String attr,
UnixUserGroupInformation ugi)
Store the given
ugi as a comma separated string in
conf as a property attr
The String starts with the user name followed by the default group names,
and other group names. |
Modifier and Type | Method and Description |
---|---|
Configuration |
ConfiguredPolicy.getConf() |
Modifier and Type | Method and Description |
---|---|
void |
ConfiguredPolicy.setConf(Configuration conf) |
Constructor and Description |
---|
ConfiguredPolicy(Configuration conf,
PolicyProvider policyProvider) |
Modifier and Type | Field and Description |
---|---|
protected Configuration |
StreamJob.config_ |
Modifier and Type | Method and Description |
---|---|
Configuration |
StreamJob.getConf() |
Modifier and Type | Method and Description |
---|---|
static Class |
StreamUtil.goodClassOrNull(Configuration conf,
String className,
String defaultPackage)
It may seem strange to silently switch behaviour when a String
is not a classname; the reason is simplified Usage:
|
void |
StreamJob.setConf(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Configuration |
GenericOptionsParser.getConfiguration()
Get the modified configuration
|
Modifier and Type | Method and Description |
---|---|
static <T> T |
ReflectionUtils.copy(Configuration conf,
T src,
T dst)
Make a copy of the writable object using serialization to a buffer
|
static URL[] |
GenericOptionsParser.getLibJars(Configuration conf)
If libjars are set in the conf, parse the libjars.
|
boolean |
NativeCodeLoader.getLoadNativeLibraries(Configuration conf)
Return if native hadoop libraries, if present, can be used for this job.
|
static MemoryCalculatorPlugin |
MemoryCalculatorPlugin.getMemoryCalculatorPlugin(Class<? extends MemoryCalculatorPlugin> clazz,
Configuration conf)
Get the MemoryCalculatorPlugin from the class name and configure it.
|
static String[] |
Shell.getUlimitMemoryCommand(Configuration conf)
Get the Unix command for setting the maximum virtual memory available
to a given child process.
|
static <T> T |
ReflectionUtils.newInstance(Class<T> theClass,
Configuration conf)
Create an object for the given class and initialize it from conf
|
static int |
ToolRunner.run(Configuration conf,
Tool tool,
String[] args)
Runs the given
Tool by Tool.run(String[]) , after
parsing with the given generic arguments. |
static void |
ReflectionUtils.setConf(Object theObject,
Configuration conf)
Check and set 'configuration' if necessary.
|
void |
NativeCodeLoader.setLoadNativeLibraries(Configuration conf,
boolean loadNativeLibraries)
Set if native hadoop libraries, if present, can be used for this job.
|
Constructor and Description |
---|
GenericOptionsParser(Configuration conf,
org.apache.commons.cli.Options options,
String[] args)
Create a
GenericOptionsParser to parse given options as well
as generic Hadoop options. |
GenericOptionsParser(Configuration conf,
String[] args)
Create a
GenericOptionsParser |
LineReader(InputStream in,
Configuration conf)
Create a line reader that reads from the given stream using the
io.file.buffer.size specified in the given
Configuration . |
Modifier and Type | Method and Description |
---|---|
static int |
Hash.getHashType(Configuration conf)
This utility method converts the name of the configured
hash type to a symbolic constant.
|
static Hash |
Hash.getInstance(Configuration conf)
Get a singleton instance of hash function of a type
defined in the configuration.
|
Copyright © 2010 The Apache Software Foundation