I don't ever bother with caravans, farms and oil wells. It is easier to raid others than farm your own resources. You obviously need markets and mills to hold enough resources but the key things to upgrade are attacking things so those buildings and troops. I know some people level up everything but the game is massively weighted in favour of the aggressor so attack is the best way forward.
If you war you should upgrade everything. The HP buff for the buildings is enough to justify it since you have no problem getting the resources like you mentioned and it's instant.
You should make the enemy work the most when clearing your base. A farm being attacked equals more time for your troops to spawn for example.
A baseline level 8 oil well has 15k HP, while the level 18 has 55k (mine has 100k HP after all the road buffs, etc.)
It absolutely matters.