There might be a threat coming from them directly, or perhaps from their lifestyle, or the people they surround themselves with. Whatever the case, you know it’s a toxic environment if you actually feel afraid around them, and have a valid reason to feel that way.