I think the Party Crashers should be like the Golems in Halloween in the sense that the owning player should be the one that actually activates destruction, no matter how many buffs have been used. I try to team up with other players (specifically guild members and of course my wife) to ensure all the achievements are met but people have been jumping in when they feel like it without even asking me about my plans.
I feel like someone just removing mine without permission or notification is rather rude. It would be unfortunate if the only way to control it is to remove all your friends from your list except for those select few you have discussed things with.
~~~
Today I noticed that when another player used buffs I did get the Quest Completed notification but the camp didn't reset until I acknowledged that quest - so I'm wondering if I didn't click on it would the Party Crashers remain? (meaning other players could apply more buffs and get their achievements)? If this is the way it's working then it is actually manageable I suppose if the player is careful about acknowledging the Quest right away...