# 修复 Elasticsearch 中损坏的索引

## 问题

"store_exception": {
"type": "corrupt_index_exception",
"reason": "failed engine (reason: [corrupt file (source: [force merge])]) (resource=preexisting_corruption)",
"caused_by": {
"type": "i_o_exception",
"reason": "failed engine (reason: [corrupt file (source: [force merge])])",
"caused_by": {
"type": "corrupt_index_exception",
"reason": "checksum failed (hardware problem?) : expected=70193aa8 actual=c48ab092 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path=\"/usr/share/elasticsearch/data/nodes/0/indices/w76sdoDFRm-xxxxxxxxxxx/0/index/_ke.fdt\")))"
}
}
}


## 情况分析

• 该集群没有使用 Elasticsearch Snapshot 进行备份
• 索引没有正在被写入新数据，数据都是两周之前的旧数据，基本属于只读状态。
• 试过重启相应的数据节点，没有好转

Thiago Souza 提供的潜在解决方案：

This is an indication of either hardware errors (check dmesg) or that it maybe ran out of disk space. At this point segments are corrupted and data is lost, meaning that you can’t recover the whole index anymore. Unless you have a snapshot (which is recommended for production). There are a couple of options to try to partially recover this index:

1. Try to partially recover the corrupted shard:
1. Close the index.
2. Set index.shard.check_on_startup: fix for this index.
3. Open the index. At this time index will start to be verified and may take a long time.
4. If it recovers, then you need to redo step 1 to 3 but set index.shard.check_on_startup: false otherwise it will always try to fix when it opens again.
2. If shard can’t be partially recovered then the only way is to completely drop it so at least the index can be recovered with the other healthy shards. For doing that you could try the allocate_empty_primary command of Cluster Reroute API 109.

None of these are guaranteed to work as it is highly dependent of the type of damage.

Yannick Welsch 提供的潜在解决方案：

Note that the corruption marker file corrupted_* will prevent the shard from being allocated as primary. This file is managed by Elasticsearch, and is unaware of the fact that you’ve fixed the index using Lucene’s CheckIndex. Removing this failure marker file should allow this shard to be allocated again.

## 解决步骤

### 第一步：尝试修复索引

1. Try to partially recover the corrupted shard:
1. Close the index.

1.1 使用 Close Index API (7.x) 关闭索引：

curl -X POST localhost:9200/MY_INDEX/_close


1. Set index.shard.check_on_startup: fix for this index.

1.2 添加配置索引层面的配置 index.shard.check_on_startup: fix 使得索引启动的时侯能够执行相应的检查。可是，在 Elasticsearch 7.0 以后，fix 这个选项被永久移除，已经不能使用。我们来看看官方文档 Index Modules (6.8) 关于 index.shard.check_on_startup 的说明：

Whether or not shards should be checked for corruption before opening. When corruption is detected, it will prevent the shard from being opened.

false (default) Don’t check for corruption when opening a shard. 默认，打开分片时不检查数据数据损坏。
checksum Check for physical corruption. 检查物理损坏。
true Check for both physical and logical corruption. This is much more expensive in terms of CPU and memory usage. 检查物理损坏和逻辑损坏。这是一个昂贵的操作，很耗 CPU 和内存。
fix The same as false. This option is deprecated and will be completely removed in 7.0. 跟 false 一样。这个选项已经被废弃，在 7.0 以后被永久移除（注：在 PR-32279 被废弃）

curl -X PUT localhost:9200/MY_INDEX/_settings \
-H 'Content-Type: application/json' -d'
{
"index.shard.check_on_startup": false
}
'


1. If it recovers, then you need to redo step 1 to 3 but set index.shard.check_on_startup: false otherwise it will always try to fix when it opens again.

1.4 跳过这一步，因为上一步都没有成功，没有继续下去的必要。再进行下一步之前，清除上文设置的配置，恢复默认配置。

### 第二步：接受数据丢失？

1. If shard can’t be partially recovered then the only way is to completely drop it so at least the index can be recovered with the other healthy shards. For doing that you could try the allocate_empty_primary command of Cluster Reroute API (7.x).

### 第三步：通过 Lucene 层面修复

1. 远程登录数据节点
2. 执行 Lucene 修复工具 CheckIndex
3. 移除数据损坏的标记文件 corrupted_*，使得 Elasticsearch 能正常启动

Command-line interface to check and exorcise corrupt segments from an index.

Run it like this:

java -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex pathToIndex [-exorcise] [-verbose] [-segment X] [-segment Y]

-exorcise: actually write a new segments_N file, removing any problematic segments. *LOSES DATA*
-segment X: only check the specified segment(s). This can be specified multiple times, to check more than one segment, eg -segment _2 -segment _a. You can't use this with the -exorcise option.


WARNING: -exorcise should only be used on an emergency basis as it will cause documents (perhaps many) to be permanently removed from the index. Always make a backup copy of your index before running this! Do not run this tool on an index that is actively being written to. You have been warned!

Run without -exorcise, this tool will open the index, report version information and report any exceptions it hits and what action it would take if -exorcise were specified. With -exorcise, this tool will remove any segments that have issues and write a new segments_N file. This means all documents contained in the affected segments will be removed.

This tool exits with exit code 1 if the index cannot be opened or has any corruption, else 0.

• Elasticsearch 分片的具体位置。如果一台机器只有一个 Elasticsearch 服务器运行的话，那应该是在 ${ES_HOME}/data/nodes/0/indices/... 下面的某个文件夹。具体路径可以在 Explain API 那边看到（见文章开头的 JSON 节选）。 • Elasticsearch library 的具体位置。应该是在 ${ES_HOME}/lib。我们需要这个位置，因为它是 Lucene Core 的 JAR 文件存放地点。
• 决定是否要先备份分片，再执行修复。

cd /usr/share/elasticsearch/lib
java -cp lucene-core*.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /usr/share/elasticsearch/data/nodes/0/indices/w76sdoDFRm-xxxxxxxxxxx/0/index -verbose -exorcise


No problems were detected with this index.

Took 118.609 sec total.