Ike,
From how you describe what's happening, the amplifier seems to be at least part of the problem. Another part of the problem is that your antenna doesn't appear to be tuned as well as it could be. And although RG-8x coax doesn't have the best reputation in the world, I very much doubt it it's any part of the problem at all. Another part of the problem is that you are 'new' to all this and may not understand what the whole process of tuning an antenna system is all about.
The easy part first! None of us knew what all this stuff was about when we first started so it's certainly not something to be particularly worried about. (Except me, of course, I was talking to the doctor that delivered me about antennas when I was born. He got embarrassed about being so 'dumb' and actually spanked me and made me cry! Can you believe that?) So, a little clarification to start with.
To work well, an antenna has to be 'tuned'. That means that it has to be adjusted so that the radio 'likes' what it sees. Which means that it has to have a 50 ohm input impedance which will be the same as the feed line's (coax) impedance, 50 ohms, and the radio's output impedance of 50 ohms. When all of those are 'perfectly' matched, the SWR is then 1:1, or one to one. Like anything else that's man made, you are almost never going to get things perfectly matched. But that SWR (Standing Wave Ratio) is a measurement of -how- close things are matched, and that means that getting it as low as possible is good. The higher it gets, the less the radio likes it and if the radio doesn't like it much, it just isn't going to put out as much, so the worse the resulting signal. All kinds of analogies for that, pick the one you like best. Feeding your cars engine the wrong 'size' or amount of fuel and it not 'putting out' as much performance is a good one. That SWR reading can tell you how 'close' you are.
That's not an exact explanation by any means but it'll get you into the ball park of antenna tuning. Now for some numbers. An SWR of 1.0:1 is the ideal. Anything from about 1.5:1 (one point five to one) and lower is actually good enough. From about 2:1 (two to one) and up is not so good to very bad. Your described SWR being about 2:1 to 3.5:1, means your antenna isn't tuned well and especially when using an amplifier, falls into the very bad category. That's because a high SWR and high power typically means something is going be damaged. I would suggest NOT using the amplifier until you can make things better.
Now for some of the "but's" you'll always find in things dealing with SWR and antenna systems. Any and everything that's 'in-line', connected to the feed line/coax, will affect what the SWR will end up being, or can change it. Your SWR going from about a 2 something to a 4:1 something when you put the amplifier in line means that there is something 'not right' about that amplifier. No one can tell you exactly what that 'not right' thingy is without some additional checking (you can get a few good guesses about it, but that's all they are, guesses). You have two choices. Learn more than you ever wanted to know about electronics and fix it your self. Or, take it to someone who knows what they are doing and let them fix it. One takes time, the other takes money. Your choice (I'll bet I can tell you what most people in your circumstances do, and I'll bet you can too).
And now for the disclaimers.
I am not a good teacher. I do not know all there is to know about this stuff. What I do know I learned through experience, which is just another way of saying I have made the same mistakes as everyone else does (at least once), and is a terrible way to learn anything! All of this a generalization, which means it is never always completely correct in every circumstance. Your millage may vary. No one in their right mind would pay me to do this $#;+! Which is about the only 'absolute' in this whole mess.
Best advice I can give you is to find someone who knows what they are doing and have them look at ALL of your set up, not just one particular part of it. Which is a pretty good 'trick' all by it's self... sorry 'bout that.
Good luck!
- 'Doc
(and just to clarify a little, that "Doc" is just a nickname. The only thing I've got a 'doctorate' in is cheapness!)
... and you hear me humming... "Oh Lord, it's hard to be humble, when I'm perfect in every way"... as I walk off into the sunset...
...all puns intended.
