Ohms (no, i'm not meditating, though maybe that would help)

[DreadPirateRoberts]

Ohm's law: V=I*R V=voltage, I=current, R=resistance
Power: P=I*V I=current, V=voltage

put it together

P=I*(I*R)= I*I*R

for your current setup, P=165, R=3 ohms

If you replace the 3 ohm speaker with 8 ohms, there will less current used to generate the 165 watts, so the speakers should not sound as loud. It should not hurt anything.