Your feed-point being at or less than a 1/4 wavelength off the ground (for 40M), will lower your antenna's input impedance. The way to raise input impedance in any capacitive antenna, is to increase the angle between the elements, and with an Inverted-V, that may mean making it a horizontal dipole to get to 50 Ohms. If changing the angle is not possible, then you will need to start moving your feed-point off-center, keeping the overall length of the antenna the same; but this may have consequences for being multi-banded, in which case you may have to add other elements specifically to get to 20M...in other words, make it a Fan-Dipole.
I'm guessing if you know the input impedance, you are using an antenna analyzer, or VNA, in which case you have the right gear to make a Fan-Dipole, and tune it.