We've all been there, stuck in a lootspot or a co-op for whatever reason.
I would like to suggest say an 8 hour cool down period whereby the invitees are able to cancel their own presence on an invited adventure; perhaps a "Leave Adventure" button. The cost for doing this would be that the player leaving the adventure would only receive xp for the kill(s) they made, no loot reward.
This would allow the inviter to still play on when they are in a position to do so if desired but unable to invite more players.