I haven't kept a running count of all the FT-101 radios we have serviced the last 43 years. Might be a thousand, probably less.
Just the same, this is not a common symptom. First fault that comes to mind is neutralizing the final tubes. This adjustment is tricky, in part because the metal screw shaft you turn has voltage on it. Also, it's a bit finicky to set in the first place.
More than once we have seen a radio with two finals that the owner thought were a "matched pair". Closer inspection showed that even though both tubes had the same brand name printed on them, each was made in a different country.
To match the finals in a FT-101 they must come from the same factory assembly line. Every manufacturer of that tube would build it a little differently. The small mechanical differences become big RF differences. The tube was not designed for RF, so these properties were not precisely the same from one factory to the next. Those small differences had no effect on how it performed in a color TV.
But to make that radio's neutralization work properly, both tubes must be mechanically the same on the inside. Not a big trick to determine, just by comparing the shape and position of parts visible inside each tube.
The correct setting of that neutralizing adjustment is determined in part by those small mechanical differences. A tube from one factory will need the screw at one position. The other tube from a different factory would need that same adjustment turned a bit farther.
The result is that there is never one setting of that screw that will be right for both tubes at the same time. If you have discovered a forum post by someone who got frustrated with this procedure, this was probably the root of the problem.
But wait a minute! This probably has nothing to do with what's causing your fault.
If it tunes up , and the transmit is stable on sideband, the final tubes are neutralized well enough. If they aren't you'll see odd things like a "carrier" on the wattmeter in SSB transmit, even when you aren't talking. Or the meter readings will jump erratically when you turn the Plate Tune and/or Load controls during tuneup.
If they show a smooth peak when each control is turned while tuning up, the final tubes probably are NOT the culprit.
One big difference between sideband and AM in this radio is the modulation limiter. Yaesu calls it "ALC", as in Automatic Level Control".
But it only works on sideband.
Only.
IN AM MODE THERE IS NO MODULATION LIMITING OF ANY KIND!
None!
This leads to a suspicion that some part of the radio is being overdriven in AM transmit.
But not on sideband. The radio will hold itself below the overdrive point in sideband.
This 'wide open' quality of AM transmit mode in this model has a lot do with its popularity for 11-meter AM.
This problem probably isn't so simple as just "Turn down the mike gain".
But a block diagram of this radio's transmitter will reveal that it has more than one circuit that can become overdriven, way upstream from the final tubes.
Just be sure that the Preselect, Plate Tune and Load controls are all set for max.
NEVER, NEVER, NEVER use those controls to 'turn down' the radio. They must be tuned for max power. This is what tunes them to the frequency you're using.
The Carrier knob is the only one that should serve to set the radio's power level for AM.
I do hope that's how you were tuning it. The instruction manual's procedure is a bit obscure, and was written for an operator who does NOT have an inline RF wattmeter connected.
Much better to peak those three knobs for max RF wattage. A different setting could cause odd things to happen.
73