Mandarin is the southernmost community in Duval County. When the first post office was opened in 1765, the community was called St. Anthony. In 1783 Florida was returned to Spain and St. Anthony was renamed San Antonio. After Florida was transferred to the United States in 1821, San Antonio was renamed "Monroe", honoring President James Monroe. The name Mandarin, was first registered in 1830. The named was selected because the Mandarin orange was a major crop in the immediate area.
Mandarin was incorporated as a town sometime around 1841. On December 20, 1841, four residents were killed when Indians raided the village and set many structures on fire. Mandarin would also suffer during the Civil War with its produce and livestock being taken by Union troops. In 1864, the Union steamship, the Maple Leaf, hit a Confederate mine and sank just off Mandarin Point. After the war, many northerners found the community as an ideal place to relocate to. In 1868, Harriet Beecher Stowe, the author of Uncle Tom's Cabin, purchased a 30-acre site to live on in Mandarin. Mrs. Stowe would become the state's most important publicist and devote much of her time to the development of the community. By 1881, Mandarin's population had grown to 1,200 residents. Eleven different steamboats shipped agricultural products stored in Mandarin's wharves to ports along the Atlantic Coast. Several freezes in the 1890's would bring an end to Mandarin's citrus production as the industry moved south into Central Florida. Mandarin fell into decline until after World War I, when it began to gradually grow as a residential community. The opening of the Buckman Bridge in 1971, would stimulate the growth of sprawl into the area. Despite being engulfed by sprawl development over years, Old Mandarin still retains its rural charm.