First established as a trade settlement, the castle later became one of the most important stops on the route of the Atlantic slave trade. The Dutch seized the fort from the Portuguese in 1637, after an unsuccessful attempt to the same extent in 1596, and took over all of the Portuguese Gold Coast in 1642. The slave trade continued under the Dutch until 1814. In 1872, the Dutch Gold Coast, including the fort, became a possession of Great Britain.

The Gold Coast, which is now Ghana, gained its independence in 1957 from Britain, and had control of the castle. Elmina Castle is a historical site, and was a major filming location for Werner Herzog's 1987 drama film Cobra Verde. The castle is recognized by UNESCO as a World Heritage Site.