Firstly, let's be honest: You are unlikely to please everyone. The most important thing is that the destination and venue for your event are relevant to the event itself.
For this reason, plenty of diplomacy and persuasion are sometimes needed to convince others that a lovely holiday destination is not necessarily an appropriate location for a conference. Or why the venue should be changed, even though the last time was such a hit. Often, there should be a rule of thumb that no conference venue is used more than once.
Regardless of details, such as the requirements profile for the actual location, the presence of a large auditorium alone is generally not enough. The number of hotel rooms with a specific star rating is important, as are accessibility by coach or rail, travel in general and even sustainability. Not to mention the time of year and the local climate. These are issues that need to be considered because not all conference venues are always accessible and not all locations can be used in full all the time. And what about annual holidays, public holidays and other events?
If that were not enough, do political/social aspects need to be addressed? Are there security concerns? Is it possible to arrange supporting activities/entertainment? Finally, are there any grants and subsidies available that could take the pressure off the budget? Could sponsors be found?