It is a difficult position that you face and your willingness to be transparent in your reasoning is to be respected. As an elected representative in a democratic political system, your role is not merely to to act for all your constituents, whether they voted for you or not, whether they are members of the labour party or not. In this you must make decisions that will not always be in line with one group’s perceived interests. Some might confuse representative democracy some some form of direct democracy. This would be wrong. You need perhaps to take reactions in this light.