The outcome was celebrated almost immediately by Prime Minister Mark Carney, who said in a statement that "Albertans made a clear choice".
But the movement's leaders say their battle is long from over and that the issue could dominate Alberta politics for years to come.
Jeffrey Rath, a Calgary-lawyer and one of the main spokespeople for Alberta separation, said he and others "plan to roll up our sleeves and work harder".
Keith Wilson, another separatist, told supporters in Calgary that while the vote is over, "the conversation about Alberta's future in Canada and our place in Canada and our role in Canada will continue".
Smith, who openly supported and campaigned for the unity vote, thanked voters for the result.
The Alberta premier has been criticised for taking a "dangerous" gamble that the independence push would likely face defeat while still giving her restive base what they wanted.
She lowered the petition threshold needed to get the citizen-led campaign the ballot in the first place.
By making Monday's result non-binding, she got around a court decision blocking the pro-separation petition because it failed to consult indigenous First Nations whose land would be affected if the province separated.
That gamble could test her leadership of the United Conservative Party, which she has led since 2022.
"She's lost the confidence of the (party) base," said Rath.
Albertans who voted against an independence referendum told the BBC throughout the campaign that they were motivated by a desire to keep the country united, especially as Canada grapples with tariff threats from the US, its longtime ally.
They also feared that an independent Alberta could be vulnerable to outside influence - or even annexation.
Thomas Lukaszuk, a prominent pro-unity campaigner, told a crowd of supporters on Monday night that the early results made him proud to be Albertan.
At a pro-unity results watch party in Edmonton, Lukaszuk appealled directly to separatists, saying that "some of the issues they have raised are legitimate", and that the province must do the hard work of healing the divisions laid bare by this debate.
"I don't want to see us doing another referendum like this ever again," Lukaszuk said.
Premier Smith told CBC earlier in the week that she would not hold an independence referendum if Albertans vote against it as long as she is premier.
Some of the rifts can't easily be smoothed over.
The separatist sentiment has its roots in "western alienation" which is a term that has been used for decades by some in Canada's western provinces to describe a feeling of being ignored or overlooked by the federal government in central Canada.
She also put nine other questions to Albertans on Monday, most that seek to give the province more autonomy from Ottawa, including on matters like immigration.
The (very) preliminary results suggest that many might pass.
Some who voted in favour of a second binding separation referendum have said they didn't want outright independence, but some leverage over Ottawa. Smith has said these other nine questions help give her just that.
Quebec, another province with similar sovereigntist sentiments, could offer clues on where Alberta's separation movement might end up.
They held two separation referendums, in 1980 and 1995, both of which were defeated by a close margin.
Alberta independence supporters don't hide the fact that they gain inspiration from the majority French-speaking province.
They view Quebec as having successfully used the threat of separation to gain significantly more autonomy over provincial affairs.
And despite support for Quebec sovereignty being at historic lows, a newly elected provincial government has pledged to hold a third referendum on independence by 2030.
Rath made a note of that as he pondered his side's defeat.
The sovereigntist Parti Quebecois "has just formed the government of Quebec with 28% of the vote," he said.
"It's not out of the realm of possibility", he added, that Alberta separatists could do the same in the future.