ByGlenn CampbellScotland political editor and James DelaneyBBC Scotland
The prime minister has invited the Scottish government to work together to put "power back in the hands" of regions.
Andy Burnham said he wanted to start work on a new deal for regional economic development in the next few months, including an offer of devolution from the Department for Work and Pensions (DWP).
Speaking at an event in Motherwell, Burnham called for further powers to be devolved from Holyrood to local areas before that happened.
Before Burnham's speech, Swinney said he was interested in "working constructively" with Westminster, adding he was "all for the Scottish Parliament having as many powers" as possible.
Transport Secretary Stephen Flynn said the first minister had written to Burnham at the start of the week calling for regions to have more power.
Burnham acknowledged that the Scottish government was working on local government reform, but said this had to "move faster".
He said politicians had to "move beyond the constitutional questions" and "focus on solutions for people right now".
Burnham invites Swinney to work together on 'deeper Scottish devolution'
Burnham said devolution "was never supposed to stop at the doors of the Scottish Parliament" when Holyrood opened in 1999.
He said it was meant to be the "start of something deeper" where "communities had more of a say in their own future".
However, he described Scotland as the most devolved nation in the UK but "one of the most centralised in Europe".
"During my first week in office, I told the first minister that I wanted to work with him to spread opportunities to communities across Scotland's regions and cities," he said.
"Through No10 North, we are rewiring the state and I want to make sure regions in Scotland are fully part of this.
"This is me making an offer to the Scottish government. With us working together, there is no limit to what we can achieve for the Scottish people."
The Scottish government set out plans for a radical rethink of the country's regional authorities last month.
A new two-tiered system would see the 32 local authorities replaced with six new regional authorities.
They would have power over major issues such as economic growth and education.
Below that, between 120 and 160 community authorities would govern more local issues, such as licensing and town centre improvements.
It would be the most significant change to Scotland's regional makeup since the current system was introduced by the UK government in 1996.
John Swinney denied that too much power was centralised at Holyrood
The prime minister also called for an end to "tribal politics" between the Scottish and UK governments.
He said he and Swinney "agreed on some things" but "didn't have to agree on everything".
Swinney is expected to promote his renewed push for independence in his keynote speech at the SNP conference on Saturday.
He previously welcomed Burnham telling MPs that a border poll would be considered in both Scotland and Northern Ireland if there was a clear consensus and public opinion had changed.
But Burnham later pushed back on that by telling Swinney at a meeting in Glasgow that a second Scottish independence referendum remains "off limits".
Asked whether devolving power more locally in Scotland would make independence more or less likely, the prime minister said: "I don't want to be the last PM of the UK.
"I just feel we all have to move beyond the constitutional questions.
"In my view, people in Scotland are suffering from the cost of living crisis, like they are in England and everywhere else, and I think our job is to focus on the practicalities, to focus on solutions for people right now."
He added: "Too much time is wasted on those dramas, it consumes too much energy."
'Devolution is a process'
SNP transport secretary Stephen Flynn said the party had been calling for increased devolved powers since the Smith Commission report was published in 2014.
He said: "Interestingly the first minister sent a letter to the prime minister on Monday where he outlined a number of points he would like to see engagement on.
"This was number seven on that list, so we will of course engage with the prime minister fully in respect of his announcement today."
The Conservative leader, Kemi Badenoch, said Burnham should be focusing on creating new opportunities in the energy sector.
She said: "This regional devolution plan is more jobs for the boys. What Andy Burnham should be doing is granting new oil and gas licences in the north sea."
"Devolution is a process, not a solution."