What you are saying does make sense, to few segments can screw up a model. I have also modified a model's segment count in the past to get it's AGT closer to 1, and from this I have learned that there is always one auto segmentation setting that will get me as close as possible to that point.
To test this I took the linear loaded model I made in the video and did some testing. I made 5 copies of this model, one to be run with 50 segments per half wavelength, the others have 40, 30, 20, and 10 segments per half wavelength. As I did in the video, I will modify the s variable, which is a separator between the elements of the linear loading section, so, in short, I will attempt to use the distance between the linear loading sections as a control for AGT. I am hoping the differences in this variable are relatively minor, but we will see. This is my biggest concern, if I have to significantly change this variable to maintain my AGT requirements I will have a model that will naturally have a different pattern.
I have two requirements for each of these models, I want to be able to tune SWR to be as low as possible, I will use the optimizer for that, and I want to keep AGT as close to one as I can. preferably between .99 and 1.01. This range is to make sure we are comparing apples to apples.
Taking a look at said S variable:
50 segments, 0.3
40 segments, 0.36
30 segments, 0.5
20 segments, 0.6
10 segments, 1.0
This is what it took to maintain the AGT requirements I put on this project. The 50 segments is the baseline, 40 segments is near as makes not real difference. At 30 and 20 segments, we are beginning to push the limits, and the 10 segments model is beyond what I would consider acceptable in any way, it is over three times the difference in length, and this is applied to the antenna model three times. I will run through the results, but I won't consider the 10 segments per half wavelength model accurate, it simply required to much of a modification of the antenna to be achieved.
The SWR results for these models is as follows:
50 segments, 1.32 SWR
40 segments, 1.32 SWR
30 segments, 1.33 SWR
20 segments, 1.35 SWR
10 segments, 1.49 SWR
It seems the ability to tune SWR is pretty stable, at least to a point. All of these models are also close to resonance.
I'm not going to show them here, but the SWR curves are also very similar, except the 10 segments per half wavelength model the curve has shifted up to a higher SWR compared to the others.
Now for the pattern comparisons...

Most of the patterns are consistent, although again the 10 segment per half wavelength model, as expected, is the most different. The problem is, the big change I needed to maintain a reasonable AGT is more likely to be responsible for this than the segment count itself.
I am going to have to rethink how to do this type of testing to rule out said types of changes. However, based on my experiences with modeling, I am reasonably certain your low segment count causing problems idea has merit, there has to be a minimum reliable segment count. The data above shows that as you go up in segments, the results change less and less. It might me interesting to do the same thing with more segments as well, up to 100 segments per half wavelength, to see if everything stabilizes further, or if other problems are created. That will have to wait, however.
The DB