This documentation is for an unreleased version of Apache Flink. We recommend you use the latest stable version.

ファイルシステム #

Flink has its own file system abstraction via the org.apache.flink.core.fs.FileSystem class. This abstraction provides a common set of operations and minimal guarantees across various types of file system implementations.

The FileSystem’s set of available operations is quite limited, in order to support a wide range of file systems. 例えば、既存のファイルへの追加あるいは変更はサポートされません。

File systems are identified by a file system scheme, such as file://, hdfs://, etc.

実装 #

Flink は以下のファイルシステムスキーマを使って、ファイルシステムを直接に実装します:

• file, which represents the machine’s local file system.

Other file system types are accessed by an implementation that bridges to the suite of file systems supported by Apache Hadoop. 以下は例の完全ではないリストです:

• hdfs: Hadoop Distributed File System
• s3, s3n, and s3a: Amazon S3 file system
• gcs: Google Cloud Storage
• …

Flink loads Hadoop’s file systems transparently if it finds the Hadoop File System classes in the class path and finds a valid Hadoop configuration. デフォルトでは、クラスパス内でHadoopの設定を探します。Alternatively, one can specify a custom location via the configuration entry fs.hdfs.hadoopconf.

一貫性の保証 #

These FileSystem and its FsDataOutputStream instances are used to persistently store data, both for results of applications and for fault tolerance and recovery. 従ってこれらのストリームの永続性のセマンティクスが良く定義されていることが重要です。

一貫性の保証の定義 #

以下の2つの要求に合致する場合は、出力ストリームに書き込まれるデータは永続的なものとみなされます:

1. Visibility Requirement: It must be guaranteed that all other processes, machines, virtual machines, containers, etc. that are able to access the file see the data consistently when given the absolute file path. This requirement is similar to the close-to-open semantics defined by POSIX, but restricted to the file itself (by its absolute path).

2. Durability Requirement: The file system’s specific durability/persistence requirements must be met. これらは特定のファイルシステムに固有です。For example the {@link LocalFileSystem} does not provide any durability guarantees for crashes of both hardware and operating system, while replicated distributed file systems (like HDFS) guarantee typically durability in the presence of up n concurrent node failures, where n is the replication factor.

Updates to the file’s parent directory (such that the file shows up when listing the directory contents) are not required to be complete for the data in the file stream to be considered persistent. This relaxation is important for file systems where updates to directory contents are only eventually consistent.

The FSDataOutputStream has to guarantee data persistence for the written bytes once the call to FSDataOutputStream.close() returns.

例 #

• For fault-tolerant distributed file systems, data is considered persistent once it has been received and acknowledged by the file system, typically by having been replicated to a quorum of machines (durability requirement). In addition the absolute file path must be visible to all other machines that will potentially access the file (visibility requirement).

  Whether data has hit non-volatile storage on the storage nodes depends on the specific guarantees of the particular file system.

  The metadata updates to the file’s parent directory are not required to have reached a consistent state. It is permissible that some machines see the file when listing the parent directory’s contents while others do not, as long as access to the file by its absolute path is possible on all nodes.

• A local file system must support the POSIX close-to-open semantics. Because the local file system does not have any fault tolerance guarantees, no further requirements exist.

  The above implies specifically that data may still be in the OS cache when considered persistent from the local file system’s perspective. Crashes that cause the OS cache to lose data are considered fatal to the local machine and are not covered by the local file system’s guarantees as defined by Flink.

  That means that computed results, checkpoints, and savepoints that are written only to the local filesystem are not guaranteed to be recoverable from the local machine’s failure, making local file systems unsuitable for production setups.

ファイル内容の更新 #

Many file systems either do not support overwriting contents of existing files at all, or do not support consistent visibility of the updated contents in that case. For that reason, Flink’s FileSystem does not support appending to existing files, or seeking within output streams such that previously written data could be changed within the same file.

ファイルの上書き #

ファイルの上書きは一般的に可能です。ファイルを削除し新しいファイルを作成することで、ファイルは上書きされます。 しかし、あるファイルシステムはその変更をファイルにアクセスする全ての関係者へ同期的に見えるようにすることができません。 For example Amazon S3 guarantees only eventual consistency in the visibility of the file replacement: Some machines may see the old file, some machines may see the new file.

To avoid these consistency issues, the implementations of failure/recovery mechanisms in Flink strictly avoid writing to the same file path more than once.

スレッド セーフ #

Implementations of FileSystem must be thread-safe: The same instance of FileSystem is frequently shared across multiple threads in Flink and must be able to concurrently create input/output streams and list file metadata.

The FSDataOutputStream and FSDataOutputStream implementations are strictly not thread-safe. Instances of the streams should also not be passed between threads in between read or write operations, because there are no guarantees about the visibility of operations across threads (many operations do not create memory fences).

Back to top