For a CB VERTICAL.... NO, can never be too high (performance wise), but possibly too high 'mechanically'....... (i.e.: if it doesn't come down in a storm, it wasn't high enough.... etc...)
However, for HORIZONTALLY POLARIZED ANTENNAS like dipoles, flat-top yagis, etc., where ground reflection ADDS gain at various wave (TOA) angles, height can play a factor. As a general rule of thumb (for HF freqs), at 5/8 of a wave above avg terrain, you begin to achieve a fairly low angle of radiation suitable for global DX propagation. Get up to 1 or 2 wavelengths above ground, and you get into the single digits (<5 deg TOA) radiation angle. Also, with multi-element yagis and multi-el quads, the more forward gain, the lower the TOA, because you are focusing power in 1 direction. Height above ground also affects TOA of W8JK's, ZL Specials, Collinear, Lazy H Arrays, etc...
With Half Squares & BobTail Curtain (vertically polarized driven arrays), and to some extent: Bruce Arrays and Sterba Curtains, height above ground is practically irrevelant AND, you may actually deter low angle performance or directivity if you make the antenna too high above ground.
Lastly, if you can reduce or eliminate ground losses with a suitable radial field under a 1/4 wave monopole, you can basically mount the thing at ground level and still get a low angle of radiation for DX performance. The only thing which suffers with a ground mounted antenna is LOS: Line of Sight. So, if you have say, an A99 type of antenna mounted at ground level, and you are working a station 600-1000 miles away on 10 meters, he's not gonna know the difference between the same antenna mounted at ground level, or up 30-40 feet on your roof; because you are working him via SKYWAVE. However, if you are checking into a local 10 meter net, and you are trying to work a station across town, say 10 - 20 miles away, you might be Q5 copy on the roof mounted A99, but completely uncopyable on the ground mounted A99, because you have NO line of sight (or clearance of obstructions like other buildings) with your ground mounted antenna.
The above is WHY many serious contesters run switchable STACKS on 200' towers, to have the ability to switch to a lower antenna for closer in stations and/or run a tribander at 40-60' as their run or bird-dog, or STATESIDE antenna.
So, general rules of thumb are: 1. The higher the better. 2. Optimal <compromise> height would be 5/8 of a wavelength above ground; for this is where you begin to achieve a low enough TOA for global DXing (some say 1/2 wavelength above ground, but this is really only for 40/80/160 meter antennas; where signals typically ARRIVE at higher angles of radiation to begin with).... 5/8' wave (which is only 21 or so feet at 10 meters) is a good compromise height 3. A good ground screen / radial field can reduce ground losses and help to achieve a lower angle of radiation for DXing 4. Formula for determining the LOS (Line of Sight) to the horizon (for UHF/VHF signals, but I've used it for 6M and 10M as well) is: 1.415 X the SQ Root of the Height above ground. Example. Antenna height at 25' vs at 100': 25' antenna sq root is 5 X 1.415 = 7.075 miles to the radio horizon. So, if the other station's antenna is at 25', and his radio horizon is also 7.075 miles, the two of you could be 14 or so miles apart and communicate with each other successfully MOST or all of the time. (even greater distances over flat & unobstructed ground <i.e.: midwest cornfields>, and MUCH greater distances over highly conductive saltwater.) Same antenna(s) at 100' = Square of 100 is 10 X 1.415 = 14.15 miles to horizon. If other station is also at 100', combined effective communicable range is over 28 miles.
Besides LOS, height also gives other benefits like: reduced RFI to adjacent buildings, reduction of near-field RF energy, reduction in ground losses (worm-burning), clearing obstructions (like trees, other buildings, poles, etc. <especially important at UHF where the wavelengths are smaller and even leaves on trees can attenuate signals>, lower TOA's (lobe-splitting notwithstanding), etc.
Finally, I would have to say that at frequencies of 14 MHz and UP, the general rule is: THE HIGHER THE BETTER under 99% of the circumstances.
At 7 MHz and below however, where wave angles tend to arrive at higher incidences, height can become somewhat irrevelant. PLUS, the waves are so LARGE, that 'obstructions' like trees, leaves, poles, and even buildings <sometimes> won't even obstruct or attenuate your signal. Many stations communicate effectively on these frequencies with NVIS (Near Vertical Incidence Skywave) antennas, which can even be laying on, or even slightly below, the ground.
10 MHz (30 meter band): the jury is out. It's a crap-shoot, as 50/50 for high vs. low angle of incidence for effective communications.
I'll give you an example: I ran HF mobile for many years, and have given out several hundred counties in over 20 states on OMISS / 3905 Century Club / and MARAC (County Hunters) Nets. While vacationing in New Hampshire, I decided to check out HF mobile from atop Mount Washington: over 6,000 feet elevation. I was on 20 meters (14 MHz). At about 1/2 to 3/4 of the way up the mountain, signals within the US and Europe were at their PEAK in signal strengths. BUT.... once I reached the summit... signals were even lower than they were before I had even driven up the mountain. Switching to 40 meters didn't improve performance (band hadn't 'opened' yet), and since it was the daytime, I didn't even bother to try 80 meters. (I didn't have 10/11/or 15 meter antennas with me at the time to experiment). Coming back down the mountain, with 40 meter antenna on the car, I couldn't even begin to hear the ECARS net control station on 7.255 MHz until I was lower than 1/2 way back down the mountain. And then he remained Q5 all the way to Boston.
For Local or Line of Sight work: HEIGHT is your friend. For global DX'ing however, Salt Water can be your friend. You can be a 1' above sea level, in your mobile, at the beach, and hear stations 1/2 way around the world you wouldn't stand a chance of hearing up on a hill or mountain.
Good luck!!!