Social Development Minister Margaret Ritchie said 25 people have already left and a further 75 were going to leave as soon as they could.
Mrs Ritchie said just 14 people had decided to stay in the country.
The minister was speaking after City Church, which last week provided shelter for the ethnic Roma people, was damaged in an overnight attack.
The families have since been moved to temporary accommodation at a secret location.
Pastor Malcolm Morgan said the church was covered in broken glass.
"I arrived this morning to find windows smashed at the front of our church and our main glass doorway smashed as well," he said.
"Stones were lying scattered on the floor inside and outside and obviously broken glass was everywhere.
"It would be easy to conclude it was carried out by someone who didn't like our response to the Romanians, but that is only guess work.
"We were just so thrilled that we were able to respond to the Romanian situation and these broken windows wouldn't have stopped us anyway."
Mrs Ritchie said she was saddened, but not shocked at the incident.
"The action of these mindless thugs greatly contrasts with the outpouring of warmth and generosity demonstrated by the people of Belfast toward the plight of the Romanians," she said.
"This church community was the first to extend the hand of friendship and that makes it doubly disgraceful that it should be attacked."
Police have said they have yet to establish a motive for the attack.
A 15-year-old boy appeared in court on Monday charged with the intimidation of Romanians living on Belgravia Avenue in south Belfast
He was also accused, along with a 16-year-old boy, of provocative behaviour at an anti-racism rally in the city.
A 21-year-old man who was also arrested in connection with the intimidation is due to appear in court later on Tuesday.
Police do not believe paramilitaries were involved in last week's attacks, which were condemned by all political parties.