Magic Keyboard sometimes does not work after waking up iMac from sleep

This happens randomly and it's super annoying, forces me to shut down the computer and power it back on every time it happens.

Anybody experiencing the same? iMac 27" 5K late 2015 running all the latest and Sequoia