Run Issue17780 10 times

[serathius]

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: serathius

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [serathius]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.82%. Comparing base (9eb85ee) to head (5065b46).
Report is 11 commits behind head on main.

Additional details and impacted files

see 28 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #19191      +/-   ##
==========================================
+ Coverage   68.77%   68.82%   +0.05%     
==========================================
  Files         420      420              
  Lines       35641    35644       +3     
==========================================
+ Hits        24511    24532      +21     
+ Misses       9707     9694      -13     
+ Partials     1423     1418       -5     

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9eb85ee...5065b46. Read the comment docs.

[k8s-ci-robot]

@serathius: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-etcd-verify	5065b46	link	true	/test pull-etcd-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[serathius]

cc @fuweid any recommendation on how to reproduce the #19179

[fuweid]

Hi @serathius

I can't reproduce it in my local. But I used to etcd-dump-logs to get one thing in first three failure pipeline runs

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/directory/pull-etcd-robustness-amd64/1877585036438409216
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/directory/pull-etcd-robustness-arm64/1877364764741472256
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/directory/pull-etcd-robustness-arm64/1877466683589791744

There were only two kinds of requests before compaction panic. One is creation and other is deletion.
Even if Update type has high weight during random pick, there is no update type in first 400+ requests.
So, creation revision has been deleted in first compaction batch before panic. After restart, etcd key index ignores keys which has only one revision for tombstone. And then etcd resumes compaction, there are no such key in index, compaction will delete all the tombstones. However, it should keep these tombstones.

I am still investigating why there is no update before compaction panic. Hope that information helps