Members of United Patriots Front occupied a rotunda in the same park.
The rallies were relatively peaceful except for a scuffle after an anti-racism protester allegedly took a camera from a member of the rival group. Superintendent Mick West said four people were arrested -- two for carrying knives, one with a flare and one for minor assault.
He was not sure whether those arrested were supporting or opposing the mosque.
The UPF rally was the second time in two months that groups had demonstrated against the council's approval of a mosque.
Anti-racism activist Tashara Roberts called on those at the rally to respect those opposite them, "even the idiots". "We must lead by example, we must show tolerance, even to the idiots."
The federal Member for Bendigo, Lisa Chesters, said those protesting over the construction of a mosque were using Bendigo as a backdrop for their battles. She said the regional city was an inclusive one, and the protesters did not represent local views.
"It's the radical of the left and the radical of the right using Bendigo as a backdrop for the latest stage for their battles."
Bendigo Mayor Peter Cox said he did not understand why the UPF had targeted Bendigo.
He said the council had to better understand what was driving people's fears about Bendigo's future.
"I really don't understand how people can look at their TV sets in their lounge room about terrorism around the world and link it to building a mosque in Bendigo.
There was also a heightened police presence at other mosques across the state, including in Melbourne.
Bendigo approved plans to build the mosque in June last year but objectors took the case to the Victorian Civil and Administrative Tribunal, unsuccessfully arguing the development would cause traffic and social issues.
Cox said the mosque would go ahead despite the ongoing protests.
Police were called in to escort Bendigo councillors from their regular meeting last month after anti-mosque protesters began chanting and hurling abuse at them from the public gallery.
