Most of the Feta brine recipes I have seen use citric acid to acidify the brine, but you can use any other household acid :-)Â , well like vinegar.
The first time I used the vinegar I thought that it would give a taste to the feta, but it did not - maybe you are more sensitive to the taste.
Also remember to test the acidity of the brine - if it is too alkaline your feta will dissolve in time. I use methyl red indicator to test the acidity. If it turns pink it is acid enough and if it turns yellow, you need more acid.