The Rolle Canal (completed in 1827) terminated at a complex of large lime kilns at Rosemoor (known then as "Rowe's Moor"). The lime kiln complex, designed by James Green, survives in a ruinous condition in a working compound at the gardens, inaccessible to the public. George Braginton, the manager and later a major leaseholder of the canal, moved into the Rowe's Moor estate some time before 1851.

On the death in 1931 of Robert Horace Walpole, the fifth Earl of Orford, the estate became the property of his daughter, Lady Anne Berry (then Palmer). She created the original garden of 8 acres (3.2 ha) in 1959, and developed it over a 30-year period. The garden developed in a naturalistic style, with sweeping lawns and curving borders set out as the plantings expanded. There was no masterplan, but designer John Codrington who later became a life member of the Royal Horticultural Society, provided drawings, in particular for the early development of warmer sheltered areas near the house.

The garden was first opened to the public in 1967, under the National Gardens Scheme. A small nursery was started in 1979. Both the garden and nursery were noted for rare and unusual plants. By the 1980s, the garden was attracting significant numbers of visitors.

In 1988 Lady Palmer gave the garden to the RHS, together with an additional 32 acres (13 ha) of land. In the mid-1990s 37.5 hectares of woodland surrounding the site, mainly coniferous forest, was added to the garden, securing the land bordering the garden from unwanted change, providing opportunities to blend the garden into its surrounding landscape and also providing it with a range of additional experiences for visitors.

Today Rosemoor Garden covers 65 acres (26 ha) and it includes a visitor centre, a plant centre, a shop, a restaurant and the Wisteria tearoom. There is also a reference library, located near the entrance to the garden, which provides a small collection of books on practical gardening, garden design, botanical art, garden history, wildlife gardening, plant hunting, as well as a selection of the major gardening magazines to browse through.

In 2019 the garden received 255,861 visitors.

A brand new venue with a seating capacity of up to 420 people and bi-fold doors opening out onto the garden was completed five years ago. A truly inspiring setting for our bonsai exhibition.

The main room will hold the bonsai from societies predominately from the south west: the smaller room will be used for a demonstration with seating to observe Roy Hutchison creating a bonsai from raw nursery material.