They were held in cities including Paris, Orleans, Nice, Pau, Toulouse and Nantes, to commemorate the victims of this week's violence.
Seventeen people were killed in attacks on a satirical magazine, police officers and a kosher supermarket.
Police are hunting for accomplices of the three gunmen, killed by police.
The interior minister says France will stay on high alert in the coming weeks.
During the marches, held largely in silence, some protesters held banners that read "I am against racism", "unity", or "I am Charlie" - the latter a reference to Charlie Hebdo, the satirical magazine whose Paris offices were attacked by brothers Cherif and Said Kouachi on Wednesday.
Twelve people - including eight journalists and two police officers - were killed and 11 injured in that attack. Another gunman, Amedy Coulibaly, took several hostages at a kosher supermarket in eastern Paris on Friday. Four hostages were later found dead.
Coulibaly is also believed to be behind the killing of a policewoman in southern Paris on Thursday.
Addressing a large gathering outside the kosher supermarket that was targeted, Prime Minister Manuel Valls said: "Today, we are all Charlie, we are all police officers, we are all Jews of France."
The rally will depart from Paris's Place de la Republique at 15:00 local time (14:00 GMT).
More than a million people are expected to take part, the BBC's Chris Morris in Paris reports.
Relatives of the victims are expected to lead the demonstration, followed by the government and foreign leaders.
Prior to the march, Prime Minister Francois Hollande is expected to meet leaders of the Jewish community, while European interior ministers gather to discuss the implications of the attacks on the continent.
Rassemblement silencieux, Washington Square Park, NY. #JeSuisCharlie pic.twitter.com/m1K86gBfnt
— Elisabeth Guédel (@EGuedel) January 10, 2015
Toute la France est Charlie. Ici à Limoges, Nantes, Marseille - par @AFPphoto #AFP #JeSuisCharlie pic.twitter.com/0zLHFi1VNV
— Agence France-Presse (@afpfr) January 10, 2015
Même le Rialto #jesuischarlie #alcv15 pic.twitter.com/rvY9qVpcBo
— Jerick Develle (@jerickdevelle) January 10, 2015