Especially in later ages, pyramids cause a terrible design flaw (if you're going to use it how it's supposed to be used, right by all or nearly all of your farms) You're going to be extremely open to those raider attacks, where someone just spams raiders - you can enclose it in walls, but they're just going to use a wall miner AND you're wasting room for defences to be placed inside those walls. Thus, you literally become a more attractive target for raiding, since for the farming boost, as many farms as possible need to be in the same spot. Combine this with logging off for several hours, or even days, and you have people hitting you for up to 250k+ food (GA/EA) just because they can drop a few troops nearby those farms.
Besides, if compared to raiding, farms are extremely unattractive and slow. Acropolis is the only viable Bronze Age wonder.