At stake were exploration and drilling rights in an area which Romania says may contain some 100bn cubic metres of natural gas and 10m tonnes of oil.
Both countries said the court's ruling was fair and promised to abide by it.
Romania submitted the boundary dispute to the UN court in 2004 after a decade of failed bilateral negotiations.
Bucharest had claimed a border extending into the northern part of the Black Sea, excluding an area surrounding Serpents' Island, which was owned by Romania until 1948, when it was ceded to the then USSR.
Kiev had meanwhile claimed a border closer to the Black Sea's western coast, saying Serpents' Island gave it territorial rights over the surrounding waters.
The unanimous ruling by the International Court of Justice's 15 judges, which both parties agreed in advance would be binding, gives Romania about four-fifths of the area it claimed.
The new border includes a 12-nautical-mile arc (22 km) around Serpents' Island, which then hits a line equidistant from Romania and Ukraine's adjacent coasts.
The court also determined that Serpents' Island could be considered an island, rather than just a rocky outcrop as Romania had claimed.
This will now permit Ukraine to create an Exclusive Economic Zone (EEZ), which according to UN Convention on the Law of the Sea extends up to 200 nautical miles from the coastline of the 17 hectare (42 acre) landmass.
"We consider this an equitable and correct solution by the court," Bogdan Aurescu, Romania's agent on the case, told reporters afterwards.
Ukraine's Deputy Foreign Minister Oleksandr Kupchyshyn described the ICJ's decision as a "wise compromise".