Package org.elasticsearch.repositories.blobstore

This package exposes the blobstore repository used by Elasticsearch Snapshots.

Preliminaries

The BlobStoreRepository forms the basis of implementations of Repository on top of a blob store. A blobstore can be used as the basis for an implementation as long as it provides for GET, PUT, DELETE, and LIST operations. For a read-only repository, it suffices if the blobstore provides only GET operations. These operations are formally defined as specified by the BlobContainer interface that any BlobStoreRepository implementation must provide via its implementation of BlobStoreRepository.getBlobContainer().

The blob store is written to and read from by master-eligible nodes and data nodes. All metadata related to a snapshot's scope and health is written by the master node.

The data-nodes on the other hand, write the data for each individual shard but do not write any blobs outside of shard directories for shards that they hold the primary of. For each shard, the data-node holding the shard's primary writes the actual data in form of the shard's segment files to the repository as well as metadata about all the segment files that the repository stores for the shard.

For the specifics on how the operations on the repository documented below are invoked during the snapshot process please refer to the documentation of the org.elasticsearch.snapshots package.

BlobStoreRepository maintains the following structure of blobs containing data and metadata in the blob store. The exact operations executed on these blobs are explained below.

 
   STORE_ROOT
   |- index-N           - JSON serialized {@link org.elasticsearch.repositories.RepositoryData} containing a list of all snapshot ids
   |                      and the indices belonging to each snapshot, N is the generation of the file
   |- index.latest      - contains the numeric value of the latest generation of the index file (i.e. N from above)
   |- incompatible-snapshots - list of all snapshot ids that are no longer compatible with the current version of the cluster
   |- snap-20131010.dat - SMILE serialized {@link org.elasticsearch.snapshots.SnapshotInfo} for snapshot "20131010"
   |- meta-20131010.dat - SMILE serialized {@link org.elasticsearch.cluster.metadata.MetaData} for snapshot "20131010"
   |                      (includes only global metadata)
   |- snap-20131011.dat - SMILE serialized {@link org.elasticsearch.snapshots.SnapshotInfo} for snapshot "20131011"
   |- meta-20131011.dat - SMILE serialized {@link org.elasticsearch.cluster.metadata.MetaData} for snapshot "20131011"
   .....
   |- indices/ - data for all indices
      |- Ac1342-B_x/ - data for index "foo" which was assigned the unique id Ac1342-B_x (not to be confused with the actual index uuid)
      |  |             in the repository
      |  |- meta-20131010.dat - JSON Serialized {@link org.elasticsearch.cluster.metadata.IndexMetaData} for index "foo"
      |  |- 0/ - data for shard "0" of index "foo"
      |  |  |- __1                      \  (files with numeric names were created by older ES versions)
      |  |  |- __2                      |
      |  |  |- __VPO5oDMVT5y4Akv8T_AO_A |- files from different segments see snap-* for their mappings to real segment files
      |  |  |- __1gbJy18wS_2kv1qI7FgKuQ |
      |  |  |- __R8JvZAHlSMyMXyZc2SS8Zg /
      |  |  .....
      |  |  |- snap-20131010.dat - SMILE serialized {@link org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardSnapshot} for
      |  |  |                      snapshot "20131010"
      |  |  |- snap-20131011.dat - SMILE serialized {@link org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardSnapshot} for
      |  |  |                      snapshot "20131011"
      |  |  |- index-123         - SMILE serialized {@link org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardSnapshots} for
      |  |  |                      the shard
      |  |
      |  |- 1/ - data for shard "1" of index "foo"
      |  |  |- __1
      |  |  .....
      |  |
      |  |-2/
      |  ......
      |
      |- 1xB0D8_B3y/ - data for index "bar" which was assigned the unique id of 1xB0D8_B3y in the repository
      ......

Getting the Repository's RepositoryData

Loading the RepositoryData that holds the list of all snapshots as well as the mapping of indices' names to their repository IndexId is done by invoking BlobStoreRepository.getRepositoryData() and implemented as follows:

    1. The blobstore repository stores the RepositoryData in blobs named with incrementing suffix N at /index-N directly under the repository's root.
    2. The blobstore also stores the most recent N as a 64bit long in the blob /index.latest directly under the repository's root.
    1. First, find the most recent RepositoryData by getting a list of all index-N blobs through listing all blobs with prefix "index-" under the repository root and then selecting the one with the highest value for N.
    2. If this operation fails because the repository's BlobContainer does not support list operations (in the case of read-only repositories), read the highest value of N from the index.latest blob.
    1. Use the just determined value of N and get the /index-N blob and deserialize the RepositoryData from it.
    2. If no value of N could be found since neither an index.latest nor any index-N blobs exist in the repository, it is assumed to be empty and RepositoryData.EMPTY is returned.

Creating a Snapshot

Creating a snapshot in the repository happens in the three steps described in detail below.

Initializing a Snapshot in the Repository (Mixed Version Clusters only)

In mixed version clusters that contain a node older than SnapshotsService.NO_REPO_INITIALIZE_VERSION, creating a snapshot in the repository starts with a call to Repository.initializeSnapshot(org.elasticsearch.snapshots.SnapshotId, java.util.List<org.elasticsearch.repositories.IndexId>, org.elasticsearch.cluster.metadata.MetaData) which the blob store repository implements via the following actions:

  1. Verify that no snapshot by the requested name exists.
  2. Write a blob containing the cluster metadata to the root of the blob store repository at /meta-${snapshot-uuid}.dat
  3. Write the metadata for each index to a blob in that index's directory at /indices/${index-snapshot-uuid}/meta-${snapshot-uuid}.dat
TODO: Remove this section once BwC logic it references is removed

Writing Shard Data (Segments)

Once all the metadata has been written by the snapshot initialization, the snapshot process moves on to writing the actual shard data to the repository by invoking Repository.snapshotShard(org.elasticsearch.index.store.Store, org.elasticsearch.index.mapper.MapperService, org.elasticsearch.snapshots.SnapshotId, org.elasticsearch.repositories.IndexId, org.apache.lucene.index.IndexCommit, org.elasticsearch.index.snapshots.IndexShardSnapshotStatus, org.elasticsearch.action.ActionListener<java.lang.Void>) on the data-nodes that hold the primaries for the shards in the current snapshot. It is implemented as follows:

Note:

  • For each shard i in a given index, its path in the blob store is located at /indices/${index-snapshot-uuid}/${i}
  • All the following steps are executed exclusively on the shard's primary's data node.
  1. Create the IndexCommit for the shard to snapshot.
  2. List all blobs in the shard's path. Find the BlobStoreIndexShardSnapshots blob with name index-${N} for the highest possible value of N in the list to get the information of what segment files are already available in the blobstore.
  3. By comparing the files in the IndexCommit and the available file list from the previous step, determine the segment files that need to be written to the blob store. For each segment that needs to be added to the blob store, generate a unique name by combining the segment data blob prefix __ and a UUID and write the segment to the blobstore.
  4. After completing all segment writes, a blob containing a BlobStoreIndexShardSnapshot with name snap-${snapshot-uuid}.dat is written to the shard's path and contains a list of all the files referenced by the snapshot as well as some metadata about the snapshot. See the documentation of BlobStoreIndexShardSnapshot for details on its contents.
  5. Once all the segments and the BlobStoreIndexShardSnapshot blob have been written, an updated BlobStoreIndexShardSnapshots blob is written to the shard's path with name index-${N+1}.

Finalizing the Snapshot

After all primaries have finished writing the necessary segment files to the blob store in the previous step, the master node moves on to finalizing the snapshot by invoking Repository.finalizeSnapshot(org.elasticsearch.snapshots.SnapshotId, java.util.List<org.elasticsearch.repositories.IndexId>, long, java.lang.String, int, java.util.List<org.elasticsearch.snapshots.SnapshotShardFailure>, long, boolean, org.elasticsearch.cluster.metadata.MetaData, java.util.Map<java.lang.String, java.lang.Object>, org.elasticsearch.action.ActionListener<org.elasticsearch.snapshots.SnapshotInfo>). This method executes the following actions in order:

  1. Write a blob containing the cluster metadata to the root of the blob store repository at /meta-${snapshot-uuid}.dat
  2. Write the metadata for each index to a blob in that index's directory at /indices/${index-snapshot-uuid}/meta-${snapshot-uuid}.dat
  3. Write the SnapshotInfo blob for the given snapshot to the key /snap-${snapshot-uuid}.dat directly under the repository root.
  4. Write an updated RepositoryData blob to the key /index-${N+1} using the N determined when initializing the snapshot in the first step. When doing this, the implementation checks that the blob for generation N + 1 has not yet been written to prevent concurrent updates to the repository. If the blob for N + 1 already exists the execution of finalization stops under the assumption that a master failover occurred and the snapshot has already been finalized by the new master.
  5. Write the updated /index.latest blob containing the new repository generation N + 1.

Deleting a Snapshot

Deleting a snapshot is an operation that is exclusively executed on the master node that runs through the following sequence of action when BlobStoreRepository.deleteSnapshot(org.elasticsearch.snapshots.SnapshotId, long, org.elasticsearch.action.ActionListener<java.lang.Void>) is invoked:

  1. Get the current RepositoryData from the latest index-N blob at the repository root.
  2. Write an updated RepositoryData blob with the deleted snapshot removed to key /index-${N+1} directly under the repository root.
  3. Write an updated index.latest blob containing N + 1.
  4. Delete the global MetaData blob meta-${snapshot-uuid}.dat stored directly under the repository root for the snapshot as well as the SnapshotInfo blob at /snap-${snapshot-uuid}.dat.
  5. For each index referenced by the snapshot:
    1. Delete the snapshot's IndexMetaData at /indices/${index-snapshot-uuid}/meta-${snapshot-uuid}.
    2. Go through all shard directories /indices/${index-snapshot-uuid}/${i} and:
      1. Remove the BlobStoreIndexShardSnapshot blob at /indices/${index-snapshot-uuid}/${i}/snap-${snapshot-uuid}.dat.
      2. List all blobs in the shard path /indices/${index-snapshot-uuid} and build a new BlobStoreIndexShardSnapshots from the remaining BlobStoreIndexShardSnapshot blobs in the shard. Afterwards, write it to the next shard generation blob at /indices/${index-snapshot-uuid}/${i}/index-${N+1} (The shard's generation is determined from the list of index-N blobs in the shard directory).
      3. Delete all segment blobs (identified by having the data blob prefix __) in the shard directory which are not referenced by the new BlobStoreIndexShardSnapshots that has been written in the previous step.
TODO: The above sequence of actions can lead to leaking files when an index completely goes out of scope. Adjust this documentation once https://github.com/elastic/elasticsearch/issues/13159 is fixed.