You are free to disagree, but Jesus does not restrict our authority to forgive sins only to people who have sinned against us personally. The authority to forgive people their sins is, in this context, a direct result of being filled with the Holy Spirit. Anyone can forgive someone a person’s personal offenses against themselves. It doesn’t take the Holy Spirit to do that.