Package | Description |
---|---|
org.apache.hadoop.conf |
Configuration of system parameters.
|
org.apache.hadoop.contrib.index.lucene | |
org.apache.hadoop.contrib.index.mapred | |
org.apache.hadoop.examples.terasort |
This package consists of 3 map/reduce applications for Hadoop to
compete in the annual terabyte sort
competition.
|
org.apache.hadoop.filecache | |
org.apache.hadoop.fs |
An abstract file system API.
|
org.apache.hadoop.fs.ftp | |
org.apache.hadoop.fs.kfs |
A client for the Kosmos filesystem (KFS)
|
org.apache.hadoop.fs.s3 |
A distributed, block-based implementation of
FileSystem that uses Amazon S3
as a backing store. |
org.apache.hadoop.fs.s3native |
A distributed implementation of
FileSystem for reading and writing files on
Amazon S3. |
org.apache.hadoop.fs.shell | |
org.apache.hadoop.io |
Generic i/o code for use when reading and writing data to the network,
to databases, and to files.
|
org.apache.hadoop.io.compress | |
org.apache.hadoop.mapred |
A software framework for easily writing applications which process vast
amounts of data (multi-terabyte data-sets) parallelly on large clusters
(thousands of nodes) built of commodity hardware in a reliable, fault-tolerant
manner.
|
org.apache.hadoop.mapred.join |
Given a set of sorted datasets keyed with the same class and yielding equal
partitions, it is possible to effect a join of those datasets prior to the map.
|
org.apache.hadoop.mapred.lib |
Library of generally useful mappers, reducers, and partitioners.
|
org.apache.hadoop.mapreduce | |
org.apache.hadoop.mapreduce.lib.input | |
org.apache.hadoop.mapreduce.lib.output | |
org.apache.hadoop.util |
Common utilities.
|
Modifier and Type | Method and Description |
---|---|
Path |
Configuration.getLocalPath(String dirsProp,
String path)
Get a local file under a directory named by dirsProp with
the given path.
|
Modifier and Type | Method and Description |
---|---|
void |
Configuration.addResource(Path file)
Add a configuration resource.
|
Constructor and Description |
---|
FileSystemDirectory(FileSystem fs,
Path directory,
boolean create,
Configuration conf)
Constructor
|
Modifier and Type | Method and Description |
---|---|
void |
IndexUpdater.run(Configuration conf,
Path[] inputPaths,
Path outputPath,
int numMapTasks,
Shard[] shards) |
void |
IndexUpdater.run(Configuration conf,
Path[] inputPaths,
Path outputPath,
int numMapTasks,
Shard[] shards) |
void |
IIndexUpdater.run(Configuration conf,
Path[] inputPaths,
Path outputPath,
int numMapTasks,
Shard[] shards)
Create a Map/Reduce job configuration and run the Map/Reduce job to
analyze documents and update Lucene instances in parallel.
|
void |
IIndexUpdater.run(Configuration conf,
Path[] inputPaths,
Path outputPath,
int numMapTasks,
Shard[] shards)
Create a Map/Reduce job configuration and run the Map/Reduce job to
analyze documents and update Lucene instances in parallel.
|
Modifier and Type | Method and Description |
---|---|
static void |
TeraInputFormat.writePartitionFile(JobConf conf,
Path partFile)
Use the input splits to take samples of the input and generate sample
keys.
|
Modifier and Type | Method and Description |
---|---|
static Path[] |
DistributedCache.getArchiveClassPaths(Configuration conf)
Get the archive entries in classpath as an array of Path
|
static Path[] |
DistributedCache.getFileClassPaths(Configuration conf)
Get the file entries in classpath as an array of Path
|
static Path |
DistributedCache.getLocalCache(URI cache,
Configuration conf,
Path baseDir,
boolean isArchive,
long confFileStamp,
Path currentWorkDir)
Get the locally cached file or archive; it could either be
previously cached (and valid) or copy it from the
FileSystem now. |
static Path |
DistributedCache.getLocalCache(URI cache,
Configuration conf,
Path baseDir,
FileStatus fileStatus,
boolean isArchive,
long confFileStamp,
Path currentWorkDir)
Get the locally cached file or archive; it could either be
previously cached (and valid) or copy it from the
FileSystem now. |
static Path |
DistributedCache.getLocalCache(URI cache,
Configuration conf,
Path baseDir,
FileStatus fileStatus,
boolean isArchive,
long confFileStamp,
Path currentWorkDir,
boolean honorSymLinkConf)
Get the locally cached file or archive; it could either be
previously cached (and valid) or copy it from the
FileSystem now. |
static Path[] |
DistributedCache.getLocalCacheArchives(Configuration conf)
Return the path array of the localized caches
|
static Path[] |
DistributedCache.getLocalCacheFiles(Configuration conf)
Return the path array of the localized files
|
Modifier and Type | Method and Description |
---|---|
static void |
DistributedCache.addArchiveToClassPath(Path archive,
Configuration conf)
Add an archive path to the current set of classpath entries.
|
static void |
DistributedCache.addFileToClassPath(Path file,
Configuration conf)
Add an file path to the current set of classpath entries It adds the file
to cache as well.
|
static Path |
DistributedCache.getLocalCache(URI cache,
Configuration conf,
Path baseDir,
boolean isArchive,
long confFileStamp,
Path currentWorkDir)
Get the locally cached file or archive; it could either be
previously cached (and valid) or copy it from the
FileSystem now. |
static Path |
DistributedCache.getLocalCache(URI cache,
Configuration conf,
Path baseDir,
FileStatus fileStatus,
boolean isArchive,
long confFileStamp,
Path currentWorkDir)
Get the locally cached file or archive; it could either be
previously cached (and valid) or copy it from the
FileSystem now. |
static Path |
DistributedCache.getLocalCache(URI cache,
Configuration conf,
Path baseDir,
FileStatus fileStatus,
boolean isArchive,
long confFileStamp,
Path currentWorkDir,
boolean honorSymLinkConf)
Get the locally cached file or archive; it could either be
previously cached (and valid) or copy it from the
FileSystem now. |
Modifier and Type | Field and Description |
---|---|
protected Path |
FSInputChecker.file
The file name from which data is read from
|
Modifier and Type | Method and Description |
---|---|
Path |
ChecksumFileSystem.getChecksumFile(Path file)
Return the name of the checksum file associated with a file.
|
Path |
FsShell.getCurrentTrashDir()
Returns the Trash object associated with this shell.
|
Path[] |
InMemoryFileSystem.getFiles(PathFilter filter)
Deprecated.
|
Path |
FilterFileSystem.getHomeDirectory() |
Path |
RawLocalFileSystem.getHomeDirectory() |
Path |
HarFileSystem.getHomeDirectory()
return the top level archive path.
|
Path |
FileSystem.getHomeDirectory()
Return the current user's home directory in this filesystem.
|
Path |
LocalDirAllocator.getLocalPathForWrite(String pathStr,
Configuration conf)
Get a path from the local FS.
|
Path |
LocalDirAllocator.getLocalPathForWrite(String pathStr,
long size,
Configuration conf)
Get a path from the local FS.
|
Path |
LocalDirAllocator.getLocalPathToRead(String pathStr,
Configuration conf)
Get a path from the local FS for reading.
|
Path |
Path.getParent()
Returns the parent of a path or null if at root.
|
Path |
FileStatus.getPath() |
Path |
FilterFileSystem.getWorkingDirectory()
Get the current working directory for the given file system
|
Path |
RawLocalFileSystem.getWorkingDirectory() |
Path |
HarFileSystem.getWorkingDirectory()
return the top level archive.
|
abstract Path |
FileSystem.getWorkingDirectory()
Get the current working directory for the given file system
|
Path |
Path.makeQualified(FileSystem fs)
Returns a qualified path object.
|
Path |
FilterFileSystem.makeQualified(Path path)
Make sure that a path specifies a FileSystem.
|
Path |
HarFileSystem.makeQualified(Path path) |
Path |
FileSystem.makeQualified(Path path)
Make sure that a path specifies a FileSystem.
|
Path |
FilterFileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to.
|
Path |
RawLocalFileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile) |
Path |
HarFileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
not implemented.
|
Path |
FileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to.
|
Path |
ChecksumFileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile) |
static Path[] |
FileUtil.stat2Paths(FileStatus[] stats)
convert an array of FileStatus to an array of Path
|
static Path[] |
FileUtil.stat2Paths(FileStatus[] stats,
Path path)
convert an array of FileStatus to an array of Path.
|
Path |
Path.suffix(String suffix)
Adds a suffix to the final name in the path.
|
Modifier and Type | Method and Description |
---|---|
boolean |
PathFilter.accept(Path path)
Tests whether or not the specified abstract pathname should be
included in a pathname list.
|
FSDataOutputStream |
FileSystem.append(Path f)
Append to an existing file (optional operation).
|
FSDataOutputStream |
FileSystem.append(Path f,
int bufferSize)
Append to an existing file (optional operation).
|
FSDataOutputStream |
FilterFileSystem.append(Path f,
int bufferSize,
Progressable progress)
Append to an existing file (optional operation).
|
FSDataOutputStream |
RawLocalFileSystem.append(Path f,
int bufferSize,
Progressable progress)
Append to an existing file (optional operation).
|
abstract FSDataOutputStream |
FileSystem.append(Path f,
int bufferSize,
Progressable progress)
Append to an existing file (optional operation).
|
FSDataOutputStream |
ChecksumFileSystem.append(Path f,
int bufferSize,
Progressable progress)
Append to an existing file (optional operation).
|
protected void |
FilterFileSystem.checkPath(Path path)
Check that a Path belongs to this FileSystem.
|
protected void |
FileSystem.checkPath(Path path)
Check that a Path belongs to this FileSystem.
|
void |
FilterFileSystem.completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Called when we're all done writing to the target.
|
void |
RawLocalFileSystem.completeLocalOutput(Path fsWorkingFile,
Path tmpLocalFile) |
void |
HarFileSystem.completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
not implemented.
|
void |
FileSystem.completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Called when we're all done writing to the target.
|
void |
ChecksumFileSystem.completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile) |
static boolean |
FileUtil.copy(File src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy local files to a FileSystem.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path[] srcs,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf) |
static boolean |
FileUtil.copy(FileSystem srcFS,
Path[] srcs,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf) |
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
File dst,
boolean deleteSource,
Configuration conf)
Copy FileSystem files to local files.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy files between FileSystems.
|
void |
FileSystem.copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path[] srcs,
Path dst)
The src files are on the local disk.
|
void |
FileSystem.copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path[] srcs,
Path dst)
The src files are on the local disk.
|
void |
FileSystem.copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path src,
Path dst)
The src file is on the local disk.
|
void |
FilterFileSystem.copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is on the local disk.
|
void |
HarFileSystem.copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
not implemented.
|
void |
FileSystem.copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is on the local disk.
|
void |
LocalFileSystem.copyFromLocalFile(boolean delSrc,
Path src,
Path dst) |
void |
ChecksumFileSystem.copyFromLocalFile(boolean delSrc,
Path src,
Path dst) |
void |
FileSystem.copyFromLocalFile(Path src,
Path dst)
The src file is on the local disk.
|
static boolean |
FileUtil.copyMerge(FileSystem srcFS,
Path srcDir,
FileSystem dstFS,
Path dstFile,
boolean deleteSource,
Configuration conf,
String addString)
Copy all files in a directory to one output file (merge).
|
void |
FilterFileSystem.copyToLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
void |
HarFileSystem.copyToLocalFile(boolean delSrc,
Path src,
Path dst)
copies the file in the har filesystem to a local file.
|
void |
FileSystem.copyToLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
void |
LocalFileSystem.copyToLocalFile(boolean delSrc,
Path src,
Path dst) |
void |
ChecksumFileSystem.copyToLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
void |
FileSystem.copyToLocalFile(Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
void |
ChecksumFileSystem.copyToLocalFile(Path src,
Path dst,
boolean copyCrc)
The src file is under FS, and the dst is on the local disk.
|
static FSDataOutputStream |
FileSystem.create(FileSystem fs,
Path file,
FsPermission permission)
create a file with the provided permission
The permission of the file is set to be the provided permission as in
setPermission, not permission&~umask
It is implemented using two RPCs.
|
FSDataOutputStream |
FileSystem.create(Path f)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite,
int bufferSize)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite,
int bufferSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
RawLocalFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
FilterFileSystem.create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
RawLocalFileSystem.create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
HarFileSystem.create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress) |
abstract FSDataOutputStream |
FileSystem.create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
ChecksumFileSystem.create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
HarFileSystem.create(Path f,
int bufferSize) |
FSDataOutputStream |
FileSystem.create(Path f,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
FileSystem.create(Path f,
short replication)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
FileSystem.create(Path f,
short replication,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
boolean |
FileSystem.createNewFile(Path f)
Creates the given Path as a brand-new zero-length file.
|
boolean |
FilterFileSystem.delete(Path f)
Deprecated.
|
boolean |
RawLocalFileSystem.delete(Path p)
Deprecated.
|
abstract boolean |
FileSystem.delete(Path f)
Deprecated.
Use delete(Path, boolean) instead
|
boolean |
FilterFileSystem.delete(Path f,
boolean recursive)
Delete a file
|
boolean |
RawLocalFileSystem.delete(Path p,
boolean recursive) |
boolean |
HarFileSystem.delete(Path f,
boolean recursive)
Not implemented.
|
abstract boolean |
FileSystem.delete(Path f,
boolean recursive)
Delete a file.
|
boolean |
ChecksumFileSystem.delete(Path f,
boolean recursive)
Implement the delete(Path, boolean) in checksum
file system.
|
boolean |
FileSystem.deleteOnExit(Path f)
Mark a path to be deleted when FileSystem is closed.
|
boolean |
FileSystem.exists(Path f)
Check if exists.
|
static void |
FileUtil.fullyDelete(FileSystem fs,
Path dir)
Deprecated.
|
long |
FileSystem.getBlockSize(Path f)
Deprecated.
Use getFileStatus() instead
|
Path |
ChecksumFileSystem.getChecksumFile(Path file)
Return the name of the checksum file associated with a file.
|
long |
ChecksumFileSystem.getChecksumFileLength(Path file,
long fileSize)
Return the length of the checksum file given the size of the
actual file.
|
ContentSummary |
FileSystem.getContentSummary(Path f)
Return the
ContentSummary of a given Path . |
FileChecksum |
FilterFileSystem.getFileChecksum(Path f)
Get the checksum of a file.
|
FileChecksum |
FileSystem.getFileChecksum(Path f)
Get the checksum of a file.
|
FileStatus |
FilterFileSystem.getFileStatus(Path f)
Get file status.
|
FileStatus |
RawLocalFileSystem.getFileStatus(Path f) |
FileStatus |
HarFileSystem.getFileStatus(Path f)
return the filestatus of files in har archive.
|
abstract FileStatus |
FileSystem.getFileStatus(Path f)
Return a file status object that represents the path.
|
static int |
HarFileSystem.getHarHash(Path p)
the hash of the path p inside iniside
the filesystem
|
long |
FileSystem.getLength(Path f)
Deprecated.
Use getFileStatus() instead
|
short |
FileSystem.getReplication(Path src)
Deprecated.
Use getFileStatus() instead
|
FileStatus[] |
FileSystem.globStatus(Path pathPattern)
Return all the files that match filePattern and are not checksum
files.
|
FileStatus[] |
FileSystem.globStatus(Path pathPattern,
PathFilter filter)
Return an array of FileStatus objects whose path names match pathPattern
and is accepted by the user-supplied path filter.
|
static boolean |
ChecksumFileSystem.isChecksumFile(Path file)
Return true iff file is a checksum file name.
|
boolean |
FileSystem.isDirectory(Path f)
Deprecated.
Use getFileStatus() instead
|
boolean |
FileSystem.isFile(Path f)
True iff the named path is a regular file.
|
FileStatus[] |
FilterFileSystem.listStatus(Path f)
List files in a directory.
|
FileStatus[] |
RawLocalFileSystem.listStatus(Path f) |
FileStatus[] |
HarFileSystem.listStatus(Path f)
liststatus returns the children of a directory
after looking up the index files.
|
abstract FileStatus[] |
FileSystem.listStatus(Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
FileStatus[] |
ChecksumFileSystem.listStatus(Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
FileStatus[] |
FileSystem.listStatus(Path[] files)
Filter files/directories in the given list of paths using default
path filter.
|
FileStatus[] |
FileSystem.listStatus(Path[] files,
PathFilter filter)
Filter files/directories in the given list of paths using user-supplied
path filter.
|
FileStatus[] |
FileSystem.listStatus(Path f,
PathFilter filter)
Filter files/directories in the given path using the user-supplied path
filter.
|
Path |
FilterFileSystem.makeQualified(Path path)
Make sure that a path specifies a FileSystem.
|
Path |
HarFileSystem.makeQualified(Path path) |
Path |
FileSystem.makeQualified(Path path)
Make sure that a path specifies a FileSystem.
|
static boolean |
FileSystem.mkdirs(FileSystem fs,
Path dir,
FsPermission permission)
create a directory with the provided permission
The permission of the directory is set to be the provided permission as in
setPermission, not permission&~umask
|
boolean |
RawLocalFileSystem.mkdirs(Path f)
Creates the specified directory hierarchy.
|
boolean |
FileSystem.mkdirs(Path f)
Call
FileSystem.mkdirs(Path, FsPermission) with default permission. |
boolean |
ChecksumFileSystem.mkdirs(Path f) |
boolean |
FilterFileSystem.mkdirs(Path f,
FsPermission permission)
Make the given file and all non-existent parents into
directories.
|
boolean |
RawLocalFileSystem.mkdirs(Path f,
FsPermission permission)
Make the given file and all non-existent parents into
directories.
|
boolean |
HarFileSystem.mkdirs(Path f,
FsPermission permission)
not implemented.
|
abstract boolean |
FileSystem.mkdirs(Path f,
FsPermission permission)
Make the given file and all non-existent parents into
directories.
|
void |
FileSystem.moveFromLocalFile(Path[] srcs,
Path dst)
The src files is on the local disk.
|
void |
FileSystem.moveFromLocalFile(Path[] srcs,
Path dst)
The src files is on the local disk.
|
void |
RawLocalFileSystem.moveFromLocalFile(Path src,
Path dst) |
void |
FileSystem.moveFromLocalFile(Path src,
Path dst)
The src file is on the local disk.
|
void |
FileSystem.moveToLocalFile(Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
boolean |
Trash.moveToTrash(Path path)
Move a file or directory to the current trash directory.
|
FSDataInputStream |
FileSystem.open(Path f)
Opens an FSDataInputStream at the indicated Path.
|
FSDataInputStream |
FilterFileSystem.open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
FSDataInputStream |
RawLocalFileSystem.open(Path f,
int bufferSize) |
FSDataInputStream |
HarFileSystem.open(Path f,
int bufferSize)
Returns a har input stream which fakes end of
file.
|
abstract FSDataInputStream |
FileSystem.open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
FSDataInputStream |
ChecksumFileSystem.open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
File |
RawLocalFileSystem.pathToFile(Path path)
Convert a path to a File.
|
File |
LocalFileSystem.pathToFile(Path path)
Convert a path to a File.
|
boolean |
FilterFileSystem.rename(Path src,
Path dst)
Renames Path src to Path dst.
|
boolean |
RawLocalFileSystem.rename(Path src,
Path dst) |
abstract boolean |
FileSystem.rename(Path src,
Path dst)
Renames Path src to Path dst.
|
boolean |
ChecksumFileSystem.rename(Path src,
Path dst)
Rename files/dirs
|
boolean |
LocalFileSystem.reportChecksumFailure(Path p,
FSDataInputStream in,
long inPos,
FSDataInputStream sums,
long sumsPos)
Moves files to a bad file directory on the same device, so that their
storage will not be reused.
|
boolean |
ChecksumFileSystem.reportChecksumFailure(Path f,
FSDataInputStream in,
long inPos,
FSDataInputStream sums,
long sumsPos)
Report a checksum error to the file system.
|
boolean |
InMemoryFileSystem.reserveSpaceWithCheckSum(Path f,
long size)
Deprecated.
Register a file with its size.
|
void |
FilterFileSystem.setOwner(Path p,
String username,
String groupname)
Set owner of a path (i.e.
|
void |
RawLocalFileSystem.setOwner(Path p,
String username,
String groupname)
Use the command chown to set owner.
|
void |
HarFileSystem.setOwner(Path p,
String username,
String groupname)
not implemented.
|
void |
FileSystem.setOwner(Path p,
String username,
String groupname)
Set owner of a path (i.e.
|
void |
FilterFileSystem.setPermission(Path p,
FsPermission permission)
Set permission of a path.
|
void |
RawLocalFileSystem.setPermission(Path p,
FsPermission permission)
Use the command chmod to set permission.
|
void |
HarFileSystem.setPermission(Path p,
FsPermission permisssion)
Not implemented.
|
void |
FileSystem.setPermission(Path p,
FsPermission permission)
Set permission of a path.
|
boolean |
FilterFileSystem.setReplication(Path src,
short replication)
Set replication for an existing file.
|
boolean |
HarFileSystem.setReplication(Path src,
short replication)
Not implemented.
|
boolean |
FileSystem.setReplication(Path src,
short replication)
Set replication for an existing file.
|
boolean |
ChecksumFileSystem.setReplication(Path src,
short replication)
Set replication for an existing file.
|
void |
FileSystem.setTimes(Path p,
long mtime,
long atime)
Set access time of a file
|
void |
FilterFileSystem.setWorkingDirectory(Path newDir)
Set the current working directory for the given file system.
|
void |
RawLocalFileSystem.setWorkingDirectory(Path newDir)
Set the working directory to the given directory.
|
void |
HarFileSystem.setWorkingDirectory(Path newDir) |
abstract void |
FileSystem.setWorkingDirectory(Path new_dir)
Set the current working directory for the given file system.
|
Path |
FilterFileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to.
|
Path |
RawLocalFileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile) |
Path |
HarFileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
not implemented.
|
Path |
FileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to.
|
Path |
ChecksumFileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile) |
static Path[] |
FileUtil.stat2Paths(FileStatus[] stats,
Path path)
convert an array of FileStatus to an array of Path.
|
Constructor and Description |
---|
FileStatus(long length,
boolean isdir,
int block_replication,
long blocksize,
long modification_time,
long access_time,
FsPermission permission,
String owner,
String group,
Path path) |
FileStatus(long length,
boolean isdir,
int block_replication,
long blocksize,
long modification_time,
Path path) |
FSInputChecker(Path file,
int numOfRetries)
Constructor
|
FSInputChecker(Path file,
int numOfRetries,
boolean verifyChecksum,
Checksum sum,
int chunkSize,
int checksumSize)
Constructor
|
Path(Path parent,
Path child)
Resolve a child path against a parent path.
|
Path(Path parent,
String child)
Resolve a child path against a parent path.
|
Path(String parent,
Path child)
Resolve a child path against a parent path.
|
Modifier and Type | Method and Description |
---|---|
Path |
FTPFileSystem.getHomeDirectory() |
Path |
FTPFileSystem.getWorkingDirectory() |
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
FTPFileSystem.append(Path f,
int bufferSize,
Progressable progress)
This optional operation is not yet supported.
|
FSDataOutputStream |
FTPFileSystem.create(Path file,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
A stream obtained via this call must be closed before using other APIs of
this class or else the invocation will block.
|
boolean |
FTPFileSystem.delete(Path file)
Deprecated.
Use delete(Path, boolean) instead
|
boolean |
FTPFileSystem.delete(Path file,
boolean recursive) |
FileStatus |
FTPFileSystem.getFileStatus(Path file) |
FileStatus[] |
FTPFileSystem.listStatus(Path file) |
boolean |
FTPFileSystem.mkdirs(Path file,
FsPermission permission) |
FSDataInputStream |
FTPFileSystem.open(Path file,
int bufferSize) |
boolean |
FTPFileSystem.rename(Path src,
Path dst) |
void |
FTPFileSystem.setWorkingDirectory(Path newDir) |
Modifier and Type | Method and Description |
---|---|
Path |
KosmosFileSystem.getWorkingDirectory() |
Path |
KosmosFileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile) |
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
KosmosFileSystem.append(Path f,
int bufferSize,
Progressable progress)
This optional operation is not yet supported.
|
void |
KosmosFileSystem.completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile) |
void |
KosmosFileSystem.copyFromLocalFile(boolean delSrc,
Path src,
Path dst) |
void |
KosmosFileSystem.copyToLocalFile(boolean delSrc,
Path src,
Path dst) |
FSDataOutputStream |
KosmosFileSystem.create(Path file,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress) |
boolean |
KosmosFileSystem.delete(Path path)
Deprecated.
|
boolean |
KosmosFileSystem.delete(Path path,
boolean recursive) |
FileStatus |
KosmosFileSystem.getFileStatus(Path path) |
long |
KosmosFileSystem.getLength(Path path)
Deprecated.
|
short |
KosmosFileSystem.getReplication(Path path)
Deprecated.
|
boolean |
KosmosFileSystem.isDirectory(Path path)
Deprecated.
|
boolean |
KosmosFileSystem.isFile(Path path)
Deprecated.
|
FileStatus[] |
KosmosFileSystem.listStatus(Path path) |
void |
KosmosFileSystem.lock(Path path,
boolean shared)
Deprecated.
|
boolean |
KosmosFileSystem.mkdirs(Path path,
FsPermission permission) |
FSDataInputStream |
KosmosFileSystem.open(Path path,
int bufferSize) |
void |
KosmosFileSystem.release(Path path)
Deprecated.
|
boolean |
KosmosFileSystem.rename(Path src,
Path dst) |
boolean |
KosmosFileSystem.setReplication(Path path,
short replication) |
void |
KosmosFileSystem.setWorkingDirectory(Path dir) |
Path |
KosmosFileSystem.startLocalOutput(Path fsOutputFile,
Path tmpLocalFile) |
Modifier and Type | Method and Description |
---|---|
Path |
S3FileSystem.getWorkingDirectory() |
Modifier and Type | Method and Description |
---|---|
Set<Path> |
FileSystemStore.listDeepSubPaths(Path path) |
Set<Path> |
FileSystemStore.listSubPaths(Path path) |
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
S3FileSystem.append(Path f,
int bufferSize,
Progressable progress)
This optional operation is not yet supported.
|
FSDataOutputStream |
S3FileSystem.create(Path file,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress) |
boolean |
S3FileSystem.delete(Path path)
Deprecated.
|
boolean |
S3FileSystem.delete(Path path,
boolean recursive) |
void |
FileSystemStore.deleteINode(Path path) |
FileStatus |
S3FileSystem.getFileStatus(Path f)
FileStatus for S3 file systems.
|
boolean |
FileSystemStore.inodeExists(Path path) |
boolean |
S3FileSystem.isFile(Path path) |
Set<Path> |
FileSystemStore.listDeepSubPaths(Path path) |
FileStatus[] |
S3FileSystem.listStatus(Path f) |
Set<Path> |
FileSystemStore.listSubPaths(Path path) |
boolean |
S3FileSystem.mkdirs(Path path,
FsPermission permission) |
FSDataInputStream |
S3FileSystem.open(Path path,
int bufferSize) |
boolean |
S3FileSystem.rename(Path src,
Path dst) |
INode |
FileSystemStore.retrieveINode(Path path) |
void |
S3FileSystem.setWorkingDirectory(Path dir) |
void |
FileSystemStore.storeINode(Path path,
INode inode) |
Modifier and Type | Method and Description |
---|---|
Path |
NativeS3FileSystem.getWorkingDirectory() |
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
NativeS3FileSystem.append(Path f,
int bufferSize,
Progressable progress)
This optional operation is not yet supported.
|
FSDataOutputStream |
NativeS3FileSystem.create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress) |
boolean |
NativeS3FileSystem.delete(Path path)
Deprecated.
|
boolean |
NativeS3FileSystem.delete(Path f,
boolean recursive) |
FileStatus |
NativeS3FileSystem.getFileStatus(Path f) |
FileStatus[] |
NativeS3FileSystem.listStatus(Path f)
If
f is a file, this method will make a single call to S3. |
boolean |
NativeS3FileSystem.mkdirs(Path f,
FsPermission permission) |
FSDataInputStream |
NativeS3FileSystem.open(Path f,
int bufferSize) |
boolean |
NativeS3FileSystem.rename(Path src,
Path dst) |
void |
NativeS3FileSystem.setWorkingDirectory(Path newDir)
Set the working directory to the given directory.
|
Modifier and Type | Method and Description |
---|---|
protected abstract void |
Command.run(Path path)
Execute the command on the input path
|
protected void |
Count.run(Path path) |
Modifier and Type | Method and Description |
---|---|
SequenceFile.Writer |
SequenceFile.Sorter.cloneFileAttributes(Path inputFile,
Path outputFile,
Progressable prog)
Clones the attributes (like compression of the input file and creates a
corresponding Writer
|
protected SequenceFile.Reader |
MapFile.Reader.createDataFileReader(FileSystem fs,
Path dataFile,
Configuration conf)
Override this method to specialize the type of
SequenceFile.Reader returned. |
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
Progressable progress)
Construct the preferred type of SequenceFile Writer.
|
static long |
MapFile.fix(FileSystem fs,
Path dir,
Class<? extends Writable> keyClass,
Class<? extends Writable> valueClass,
boolean dryrun,
Configuration conf)
This method attempts to fix a corrupt MapFile by re-creating its index.
|
SequenceFile.Sorter.RawKeyValueIterator |
SequenceFile.Sorter.merge(List<SequenceFile.Sorter.SegmentDescriptor> segments,
Path tmpDir)
Merges the list of segments of type
SegmentDescriptor |
SequenceFile.Sorter.RawKeyValueIterator |
SequenceFile.Sorter.merge(Path[] inNames,
boolean deleteInputs,
int factor,
Path tmpDir)
Merges the contents of files passed in Path[]
|
SequenceFile.Sorter.RawKeyValueIterator |
SequenceFile.Sorter.merge(Path[] inNames,
boolean deleteInputs,
int factor,
Path tmpDir)
Merges the contents of files passed in Path[]
|
SequenceFile.Sorter.RawKeyValueIterator |
SequenceFile.Sorter.merge(Path[] inNames,
boolean deleteInputs,
Path tmpDir)
Merges the contents of files passed in Path[] using a max factor value
that is already set
|
SequenceFile.Sorter.RawKeyValueIterator |
SequenceFile.Sorter.merge(Path[] inNames,
boolean deleteInputs,
Path tmpDir)
Merges the contents of files passed in Path[] using a max factor value
that is already set
|
void |
SequenceFile.Sorter.merge(Path[] inFiles,
Path outFile)
Merge the provided files.
|
void |
SequenceFile.Sorter.merge(Path[] inFiles,
Path outFile)
Merge the provided files.
|
SequenceFile.Sorter.RawKeyValueIterator |
SequenceFile.Sorter.merge(Path[] inNames,
Path tempDir,
boolean deleteInputs)
Merges the contents of files passed in Path[]
|
SequenceFile.Sorter.RawKeyValueIterator |
SequenceFile.Sorter.merge(Path[] inNames,
Path tempDir,
boolean deleteInputs)
Merges the contents of files passed in Path[]
|
protected FSDataInputStream |
SequenceFile.Reader.openFile(FileSystem fs,
Path file,
int bufferSize,
long length)
Override this method to specialize the type of
FSDataInputStream returned. |
void |
SequenceFile.Sorter.sort(Path[] inFiles,
Path outFile,
boolean deleteInput)
Perform a file sort from a set of input files into an output file.
|
void |
SequenceFile.Sorter.sort(Path[] inFiles,
Path outFile,
boolean deleteInput)
Perform a file sort from a set of input files into an output file.
|
void |
SequenceFile.Sorter.sort(Path inFile,
Path outFile)
The backwards compatible interface to sort.
|
SequenceFile.Sorter.RawKeyValueIterator |
SequenceFile.Sorter.sortAndIterate(Path[] inFiles,
Path tempDir,
boolean deleteInput)
Perform a file sort from a set of input files and return an iterator.
|
SequenceFile.Sorter.RawKeyValueIterator |
SequenceFile.Sorter.sortAndIterate(Path[] inFiles,
Path tempDir,
boolean deleteInput)
Perform a file sort from a set of input files and return an iterator.
|
Constructor and Description |
---|
Reader(FileSystem fs,
Path file,
Configuration conf)
Open the named file.
|
SegmentDescriptor(long segmentOffset,
long segmentLength,
Path segmentPathName)
Constructs a segment
|
Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Create the named file.
|
Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
Progressable progress,
SequenceFile.Metadata metadata)
Create the named file with write-progress reporter.
|
Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
Progressable progress,
SequenceFile.Metadata metadata)
Create the named file with write-progress reporter.
|
Modifier and Type | Method and Description |
---|---|
CompressionCodec |
CompressionCodecFactory.getCodec(Path file)
Find the relevant compression codec for the given file based on its
filename suffix.
|
Modifier and Type | Method and Description |
---|---|
static Path[] |
FileInputFormat.getInputPaths(JobConf conf)
Deprecated.
Get the list of input
Path s for the map-reduce job. |
static Path |
JobHistory.JobInfo.getJobHistoryLogLocation(String logFileName)
Get the job history file path given the history filename
|
static Path |
JobHistory.JobInfo.getJobHistoryLogLocationForUser(String logFileName,
JobConf jobConf)
Get the user job history file path
|
Path |
JobConf.getLocalPath(String pathString)
Deprecated.
Constructs a local file name.
|
static Path |
FileOutputFormat.getOutputPath(JobConf conf)
Get the
Path to the output directory for the map-reduce job. |
Path |
FileSplit.getPath()
Deprecated.
The file containing this split's data.
|
static Path |
FileOutputFormat.getPathForCustomFile(JobConf conf,
String name)
Helper function to generate a
Path for a file that is unique for
the task within the job output directory. |
static Path |
SkipBadRecords.getSkipOutputPath(Configuration conf)
Get the directory to which skipped records are written.
|
Path |
JobClient.getSystemDir()
Grab the jobtracker system directory path where job-specific files are to be placed.
|
static Path |
FileOutputFormat.getTaskOutputPath(JobConf conf,
String name)
Helper function to create the task's temporary output directory and
return the path to the task's output file.
|
Path |
JobConf.getWorkingDirectory()
Deprecated.
Get the current working directory for the default file system.
|
static Path |
FileOutputFormat.getWorkOutputPath(JobConf conf)
Get the
Path to the task's temporary output directory
for the map-reduce job
Tasks' Side-Effect Files |
static Path |
JobHistory.JobInfo.recoverJobHistoryFile(JobConf conf,
Path logFilePath)
Selects one of the two files generated as a part of recovery.
|
Modifier and Type | Method and Description |
---|---|
boolean |
OutputLogFilter.accept(Path path) |
static void |
FileInputFormat.addInputPath(JobConf conf,
Path path)
Deprecated.
Add a
Path to the list of inputs for the map-reduce job. |
static SequenceFile.Reader[] |
SequenceFileOutputFormat.getReaders(Configuration conf,
Path dir)
Deprecated.
Open the output generated by this format.
|
static MapFile.Reader[] |
MapFileOutputFormat.getReaders(FileSystem ignored,
Path dir,
Configuration conf)
Open the output generated by this format.
|
static boolean |
JobClient.isJobDirValid(Path jobDirPath,
FileSystem fs)
Checks if the job directory is clean and has all the required components
for (re) starting the job
|
protected boolean |
TextInputFormat.isSplitable(FileSystem fs,
Path file)
Deprecated.
|
protected boolean |
FileInputFormat.isSplitable(FileSystem fs,
Path filename)
Deprecated.
Is the given filename splitable? Usually, true, but if the file is
stream compressed, it will not be.
|
protected boolean |
KeyValueTextInputFormat.isSplitable(FileSystem fs,
Path file) |
static Path |
JobHistory.JobInfo.recoverJobHistoryFile(JobConf conf,
Path logFilePath)
Selects one of the two files generated as a part of recovery.
|
static void |
FileInputFormat.setInputPaths(JobConf conf,
Path... inputPaths)
Deprecated.
Set the array of
Path s as the list of inputs
for the map-reduce job. |
static void |
FileOutputFormat.setOutputPath(JobConf conf,
Path outputDir)
Set the
Path of the output directory for the map-reduce job. |
static void |
SkipBadRecords.setSkipOutputPath(JobConf conf,
Path path)
Set the directory to which skipped records are written.
|
void |
JobConf.setWorkingDirectory(Path dir)
Deprecated.
Set the current working directory for the default file system.
|
Constructor and Description |
---|
FileSplit(Path file,
long start,
long length,
JobConf conf)
Deprecated.
|
FileSplit(Path file,
long start,
long length,
String[] hosts)
Deprecated.
Constructs a split with host information
|
JobConf(Path config)
Deprecated.
Construct a map/reduce configuration.
|
MultiFileSplit(JobConf job,
Path[] files,
long[] lengths)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
static String |
CompositeInputFormat.compose(String op,
Class<? extends InputFormat> inf,
Path... path)
Convenience method for constructing composite formats.
|
Modifier and Type | Method and Description |
---|---|
Path |
CombineFileSplit.getPath(int i)
Returns the ith Path
|
Path[] |
CombineFileSplit.getPaths()
Returns all the Paths in the split
|
Modifier and Type | Method and Description |
---|---|
static void |
MultipleInputs.addInputPath(JobConf conf,
Path path,
Class<? extends InputFormat> inputFormatClass)
Add a
Path with a custom InputFormat to the list of
inputs for the map-reduce job. |
static void |
MultipleInputs.addInputPath(JobConf conf,
Path path,
Class<? extends InputFormat> inputFormatClass,
Class<? extends Mapper> mapperClass)
|
static void |
TotalOrderPartitioner.setPartitionFile(JobConf job,
Path p)
Set the path to the SequenceFile storing the sorted partition keyset.
|
Constructor and Description |
---|
CombineFileSplit(JobConf job,
Path[] files,
long[] lengths) |
CombineFileSplit(JobConf job,
Path[] files,
long[] start,
long[] lengths,
String[] locations) |
Modifier and Type | Method and Description |
---|---|
Path |
JobContext.getWorkingDirectory()
Get the current working directory for the default file system.
|
Modifier and Type | Method and Description |
---|---|
void |
Job.setWorkingDirectory(Path dir)
Set the current working directory for the default file system.
|
Modifier and Type | Method and Description |
---|---|
static Path[] |
FileInputFormat.getInputPaths(JobContext context)
Get the list of input
Path s for the map-reduce job. |
Path |
FileSplit.getPath()
The file containing this split's data.
|
Modifier and Type | Method and Description |
---|---|
static void |
FileInputFormat.addInputPath(Job job,
Path path)
Add a
Path to the list of inputs for the map-reduce job. |
protected boolean |
TextInputFormat.isSplitable(JobContext context,
Path file) |
protected boolean |
FileInputFormat.isSplitable(JobContext context,
Path filename)
Is the given filename splitable? Usually, true, but if the file is
stream compressed, it will not be.
|
static void |
FileInputFormat.setInputPaths(Job job,
Path... inputPaths)
Set the array of
Path s as the list of inputs
for the map-reduce job. |
Constructor and Description |
---|
FileSplit(Path file,
long start,
long length,
String[] hosts)
Constructs a split with host information
|
Modifier and Type | Method and Description |
---|---|
Path |
FileOutputFormat.getDefaultWorkFile(TaskAttemptContext context,
String extension)
Get the default path and filename for the output format.
|
static Path |
FileOutputFormat.getOutputPath(JobContext job)
Get the
Path to the output directory for the map-reduce job. |
static Path |
FileOutputFormat.getPathForWorkFile(TaskInputOutputContext<?,?,?,?> context,
String name,
String extension)
Helper function to generate a
Path for a file that is unique for
the task within the job output directory. |
static Path |
FileOutputFormat.getWorkOutputPath(TaskInputOutputContext<?,?,?,?> context)
Get the
Path to the task's temporary output directory
for the map-reduce job
Tasks' Side-Effect Files |
Path |
FileOutputCommitter.getWorkPath()
Get the directory that the task should write results into
|
Modifier and Type | Method and Description |
---|---|
static void |
FileOutputFormat.setOutputPath(Job job,
Path outputDir)
Set the
Path of the output directory for the map-reduce job. |
Constructor and Description |
---|
FileOutputCommitter(Path outputPath,
TaskAttemptContext context)
Create a file output committer
|
Modifier and Type | Method and Description |
---|---|
static Path[] |
StringUtils.stringToPath(String[] str) |
Copyright © 2010 The Apache Software Foundation