NHS bosses in England say the way maternity care is organised is "not fit for purpose" with safety at risk and warn some smaller units may have to close.
The NHS Alliance, which represents health managers, warn difficult decisions lie ahead and maternity services need to be consolidated to support safer care.
It said a government taskforce into neonatal and maternity care must tackle key issues such as a lack of specialist staff and a rise in C-sections.
The Department of Health and Social Care said every woman deserves safe maternity care, and the taskforce "will deliver the reforms needed to ensure services meet the needs of women and families now and in the future".
The NHS Alliance said there are particular pressures on smaller units in rural and coastal areas and that those delivering 1,500 to 2,000 babies each year might need to close.
The rise in C-sections was a challenge, according to the NHS Alliance, and there were inadequate operating theatres and beds in outdated and decaying buildings, posing a safety risk.
The NHS leaders' organisation also said some smaller units struggled to maintain safe staffing numbers for 24-hour care, including midwives and consultants such as obstetricians.
It goes on to say the government's taskforce must "grasp the nettle" by tackling these strategic issues, including significantly worse outcomes in deprived communities and raised maternity mortality rates for Black and Asian women.
The director of the NHS Alliance's acute and ambulance network, Matthew Hopkins, said: "Difficult decisions lie ahead, consolidating maternity services to support safer, more sustainable care."
He said any local plans had to involve NHS leaders, clinical staff, patients and communities, and that politicians "must have the courage to make the case for patient safety".
"It is time now to take a step back and consider how we should best organise maternity services across England, and where those services should be located", he said.
In August, a Devon trust suspended standard birthing services at a hospital in Barnstaple leaving pregnant mothers having to make a 53 mile journey to Exeter to give birth.
The Royal Devon University Healthcare NHS Foundation Trust said it was the "only safe course of action" due to staffing issues.
The trust added that it had explored every option including locum cover and support from other health organisations but that a temporary suspension was the best solution.
The North Devon maternity facility remains suspended with the trust saying there would be a "readiness assessment" later in October with patient safety the "overriding priority".
Campaigners say the delay is "disappointing". One couple told the BBC the thought of the longer journey for the birth of their second son had left them anxious and angry.
Catherine Mchomvu, chief executive of the Devon Antenatal Family Centre, has been campaigning to secure the future of the Barnstaple facility, and warned that reductions nationally would undermine the quality of care.
"If we had closures of smaller units you are going to find that women are not going to receive any level of continuity of care - that's purely because they are not going to see the same healthcare team more than once throughout the whole antenatal and post-natal experience", she said.
In June, Baroness Valerie Amos's review found women and babies were not being "listened to, heard or believed", with racism and discrimination "embedded throughout the system".