I think it is because roughly each six seconds, all the defence towers (arrow tower, mortar etc) will show an animation to turn their barrel's facing direction. As animation in game is through switching a series of spirit images fast, a lot of animation means a lot of cpu running to handle the processing, which may cause other animation (animal running, farmer walking) laggy.
I think powerful device should have no issue, my iphone 5s and ipad pro have no this issue, but my ipad 3 has this issue.
also, if you are zooming out to view the whole map, this laggy will be more obvious because every animation is "visible" so need to be really drawn by CPU.