Some 67 military properties are slated to be sold off by the Labor government to raise funds, including Sydney's Victoria Barracks - a complex established in 1840.
Between 450 and 650 personnel and their families who are based at the barracks will be relocated to other defence sites.
A directive that Defence Minister Richard Marles reportedly gave officials to vacate the facility by December 19 - a deadline since walked back - has sparked a political firestorm.
The federal government's NSW Labor counterparts have also backed plans to apply heritage protections to the site to prevent its development after a sale.
But on Sunday, Mr Marles was in no mood to appease his critics.
"It is not my priority for defence to be a heritage organisation," he told News24.
"It's not my priority to see defence manage the oldest wine cellar in Australia."
Every dollar spent on maintaining the barracks' heritage value was in competition with spending on current defence capability, Mr Marles added, before launching a broadside against ex-military brass.
"There are a few former generals who liked hanging out there, who are lamenting this," he said.
The mass sale of defence sites, announced in February, is expected to rake in $1.8 billion.
The RSL's NSW branch recently lambasted the imminent booting of veterans from the eastern Sydney site to enable defence to take the "irreplaceable national asset" off its books.
NSW Premier Chris Minns also jumped into the fray, saying he supported the idea of listing the military site on the state's heritage register.
The sale has been criticised by various figures from across the political spectrum, including independent MP Allegra Spender, whose seat covers the inner-city barracks, Greens MP David Shoebridge and former prime minister Tony Abbott.
A petition started by coalition defence spokesman James Paterson opposing the sale has garnered about 95,000 signatures.
Mr Paterson lambasted the defence minister for his "fairly extraordinary and contemptuous" comments on Sunday, saying veterans would be insulted by his language.
"The truth is that they are planning on relocating most of these military personnel to an office tower in Sydney, which itself will cost hundreds of millions of dollars to fit out and lease and maintain," he said.
Defence properties across the nation span about 3.8 million hectares.
