Aid workers are among the dead in Rann, with the Red Cross saying six of its employees were killed.
The MSF aid agency said that over 200 people had been injured and appealed for help with medical evacuations.
President Muhammadu Buhari, whose army is fighting Boko Haram militants, expressed dismay and urged calm.
The attack took place near the border with Cameroon, where the military is engaged in what it calls its final push against Boko Haram.
It is thought to be the first time Nigeria's military has admitted to making such a mistake.
Red Cross spokeswoman Aleksandra Matijevic said that along with the staff killed, more than a dozen of the group's volunteers, who arrived on Tuesday to deliver food to the thousands displaced people, were wounded.
"Our thoughts are now with the families of our colleagues who have lost their loved ones," Ms Matijevic said, adding that the Red Cross would continue bringing humanitarian aid to those affected by the conflict.
MSF told the BBC that its organisation also had medical teams working in the Rann camp when the bombs struck.
"More than 50 people have been killed," said Hugues Robert, head of MSF's emergency response.
"Our team was there and counted the bodies, and more than 200 people have been wounded following these two different blasts from aerial bombardment," he said.
MSF said it had teams in Cameroon and Chad ready to treat wounded patients.
Many of the casualties, it said, were believed to be displaced people who had fled from areas where Boko Haram had carried out attacks.
Gen Rabe Abubakar, a spokesman for the Nigerian military, said that some "remnants" of Boko Haram had been detected outside Rann and the military had acted to eliminate them.
After the military realised its mistake, they were "all in pain", he said.
"However, in a military operation such as this, from time to time these things do occur," he added.
"Even though it was highly regretful, it was never intended."
Gen Lucky Irabor, who commands counter-insurgency operations in the north-east, said there would be an investigation.
A spokesman for the Nigerian president said the administration would offer help to the government of Borno state "in attending to this regrettable operational mistake".
Boko Haram has caused havoc in Africa's most populous country through a wave of bombings, assassinations and abductions.
It is fighting to overthrow the government and create an Islamic state.
The group has stepped up attacks in the past few weeks as the end of the rainy season enabled its fighters to move more easily in the bush.
Last month, the UN launched a $1bn (£800m) appeal for those facing hunger and starvation in the region.
It said nearly 5.1 million people in three north-eastern states were expected to face serious food shortages as for a third year in a row farmers had been unable to plant, fearing unexploded improvised devices left behind by militants.
Urgent aid was needed for some 100,000 people, mostly children, at risk of dying of starvation.