@jgr The Ca in tap water is already available to plants. Any Ca in water that is dissolved in it will be present as Ca+2 ions, which plants can uptake. If you have water with high alkalinity, then that Ca+2 is present along with CO3-2 ions (carbonate anions). All of them randomly dispersed among the water molecules.
When you lower the pH, the acid reacts with the bases present (hydroxide ions and carbonate ions), but the acids do not react at all with the Ca2+. If the concentration of Ca+2 is not present at large concentrations (>300 ppm) then calcium sulfate will not precipitate, even if you added a molar equivalent to the Ca2+. For any precipitation to happen with phosphoric acid, there would need to be even more calcium present.
However, the conductivity of carbonate, sulfate and acid phosphate ions are all different, so the conductivity might change substantially. When you lower the pH you also turn carbonates into carbonic acid, which then turn into CO2, so that loss also causes a shift in conductivity.