Hmmm. Contradictory statement. Can't be both.
Best way to tell if this is the problem will require a second radio to use as a monitor. You won't need headphones to prevent feedback, because the first step is to turn down the radio's mike gain just above the wattage needed to key the relay. Should be under 3 to 4 Watts peak. Lower is better. If it has a "rough" sound that mostly goes away when the mike gain is turned back up, the amplifier transistors either have too little bias or none at all.
A sideband linear needs to amplify the lower-level parts of the voice waveform the same as the full-bore peaks. This is what the bias current on the RF transistors does. No need for this feature in AM mode. The AM carrier does this job with no need for a circuit to switch bias current on only when the relay is keyed. You don't want that bias current running on the receive side.
The other possibility is that the radio is overdriving the amplifier. The radio's internal sideband modulation limiter or "ALC" serves to hold the drive level to the radio's finals below their max peak limit. They really do sound better that way. But if the amplifier is pushed to the "flattopping" point where modulation peaks get sliced off you'll get distortion with the mike gain turned up, but clean audio with it turned down.
More than one way for an amplifier to sound bad on sideband, but those are the two most-popular methods.
73
