And you can see in the SLOWLOG, there are 2 newer slow commands that were used after my delete command. It was also the first latency that occurred since I had deleted those 250k pointless keys. We had just experience a momentary latency right before I ran that SLOWLOG command. More specifically, it would loop over all the elements that were "updated" in our database, and run a bunch of "KEYS" commands, with a key pattern, in order to find the keys that contained information about these updated elements, in order to delete those keys. I looked through our code to see where we were using this command, and I found that it is being used heavily when clearing the cache. We would have random CPU spikes, without an increase in traffic. This is exactly what was happening to us. If it is high while the traffic is not, it is usually a sign that slow commands are used. ) to quickly check the CPU consumption of the main Redis process. Since Redis 2.8 a new commands were introduced in order to iterate the key space and other large collections incrementally, please check the SCAN, SSCAN, HSCAN and ZSCAN commands for more information.Īdditionally, you can use your favorite per-process monitoring program (top, htop, prstat, etc. KEYS, as documented in the Redis documentation, should only be used for debugging purposes. IMPORTANT NOTE: a VERY common source of latency generated by the execution of slow commands is the use of the KEYS command in production environments. I updated the code to add a TTL to these types of keys, then.Īpart from the "DEL" command I had just used the previous night to delete 250k+ keys, ALL the slow commands shown by the SLOWLOG were the "KEYS" command! I took a look at the Redis documentation regarding this command () and some other resources, and I saw this: Investigating the code, I found the key pattern needed for me to identify the, at least the most obvious, keys that served no more purpose. We had a ton of keys, especially a lot without a TTL (time to live), that will be sticking around forever, for no reason, if we didn't do something about them. Running the "INFO" command () using the redis-cli, I saw that we did not even exhaust half the available memory, so the spikes can't be due to memory swapping. I needed to investigate this, as simply scaling up this server might have bought us some time, but did not address the underlying issue, which was still an apparent problem even with the larger server.Īt first, I thought maybe Redis was doing a lot of memory swapping, which is a common culprit to a badly performing Redis server.
And the strange thing was, these CPU spikes did not correspond with any increase in traffic, or anything else for that matter.
This issue was causing latency across our entire app. The graph above shows the constant CPU spiking over the course of a week.
0 kommentar(er)
