Document fields disappear in Elasticsearch after upgrading MongoDB from 3.6 to 7.0

[myrzan]

We are using Monstache v6.7.17 to sync data between MongoDB and Elasticsearch 8.3.2. Everything worked fine with MongoDB 3.6. However, after upgrading to MongoDB 7.0, we started experiencing issues: when updating some fields in a MongoDB document, the other fields seem to disappear from the document in Elasticsearch. The document itself still exists in Elasticsearch by _id, but I can't see indexed fields due to _source: disabled.

Here is my monstache configuration:

gzip = true
replay = false
resume = true
resume-name = 'models'
resume-write-unsafe = false
resume-from-earliest-timestamp = true
namespace-regex = "^cars\\.(models|\\$cmd)$"
delete-strategy = 1
dropped-databases = true
dropped-collections = true
verbose = false
enable-http-server = true
http-server-addr = ":8081"
stats = false
stats-duration = "60s"
fail-fast = false
enable-oplog = true

Could this be related to changes in the Oplog format or behavior in MongoDB 7.0?

• We tried updating a MongoDB document using standard update operations ($set) and expected Monstache to sync only the updated fields to Elasticsearch, preserving all other fields in the document.
• However, after the upgrade to MongoDB 7.0, Monstache seems to only send the updated fields to Elasticsearch, resulting in the removal of other fields in the document.
• Checked Monstache logs (verbose = true) and observed that the Oplog updates were being processed, but only partial updates (diff) were being sent to Elasticsearch.
• Verified the data in MongoDB, which was correct, but Elasticsearch documents were missing fields.

[myrzan]

switching to change-stream-namespaces works fine, but I want to know why we got such behavior with oplog