Research: What caused our dependency update issue?

[DerekCaelin]

Task to look into what the dependency update + main broken issue was (post-mortem?)

Slack: Ruxandra flags

[shrouxm]

as far as i can tell:

• the dependency update commits started failing because of a faulty iOS build cache (all the PRs with an iOS build failure share the same build cache key)
• the commit from ruxandra's PR failed due to a transient network error while downloading dependencies from npm
• i don't see a super strong pattern pointing to a dominant root cause for failed workflow runs on main. it's a bit hard to check though because logs are only stored for so long
• having better notifications & follow-ups when the main branch is broken seems like the best next step given that there isn't a single thing we can fix that would clearly make things way more reliable
• however:
  • i do know that i & others have encountered numerous CI issues due to the iOS build cache
  • i also see that this cache is not addressing the main bottleneck of the workflow. workflow runs with a cache miss seem to take ~19mins, and workflow runs with a cache hit take ~16 mins
  • to me, there's not much of a meaningful productivity difference between 16 and 19mins. in both cases, if i'm blocked on the workflow completing i will be context switching and coming back to it later anyway
  • i therefore suggest disabling the iOS build cache

[shrouxm]

waiting for reactions to suggestion of disabling iOS build cache before closing, that should be trivial work so i'll roll it into this issue if we decide to do it

[DerekCaelin]

I didn't see these findings shared in slack; flagging here so that @paulschreiber @knipec and @tm-ruxandra can respond to Garo's suggestion:

i therefore suggest disabling the iOS build cache

[knipec]

I don't have a ton of background on these issues, so would be happy to sync with Paul and garo to make sure I'm up to speed, but just from garo's description that sounds good to me