Racing the cache

There was a rather annoying bug discovered at work that we could only trigger in the live environment, but never reproduce it ourselves. The gist of it was that a group was created, but didn’t exist in the cached group tree leading to an uncaught exception when trying to go up the tree from the group.

Since the cache was being properly invalidated on group creation, and given our inability to reproduce the bug, we eventually narrowed it down to a race condition between two of the most annoying parts of a system to debug — the cross-request cache and database transactions.
Continue reading