In your point of view, yes it might not be worth it to upgrade walls because of mismatches and powerful opponents lurking around for resources. However, if you are viewing your own base from the attacker's perspective, it may look like your base is an easy target given that you have maxed out...