It was hard to memorize which button controls which door. Maybe make an indicator, for example: the doorway is highlighted when you hover over its button. Other than that I think it's a cool concept.
This seems to be a really common suggestion. The disconnect between the clicks and the actions on the map was intentional, but I probably went too far with that one. Thank you for your comment!