Charlie's great-grandfather, John Edwards was born in Liverpool, Lancashire in 1814 and first came to New South Wales as a convict in 1832. In 1837, he was recorded at the penal settlement of Port Macquarie and in September 1842, he was granted permission as a convict to marry Agnes Thompson who was a free emigrant. John received his Certificate of Freedom on 29 June 1846.

Agnes Thompson was born in Glasgow, Lanarkshire, Scotland in 1819 and came to Australia as a bounty emigrant in 1841 in the care of an uncle and aunt.

John and Agnes' first child, Samuel Edward, was born in Port Macquarie. They then moved away to the South Coast where their second son, Edward was born at Moruya in 1847. The family moved to the Macleay in 1849 where John was recorded as a tenant farmer of John Verge at Austral Eden on the Lower Macleay. Other children born to John and Agnes were Archibald Peter (1850), Jessie May (1852), Florence Agnes (1854) and Lillie Annie (1857). By 1862 both parents were dead, Agnes having tragically drowned. The three boys raised themselves, whilst the girls were raised by other families

Edward Edwards and his family took up land at Shark Island where they were dairy farmers. Samuel Edwards owned land near the New Entrance, South West Rocks whilst Archibald was associated with the nearby Post Office at Goolundurri, long defunct. Edward Edwards married Louise Herwin at Gulgong in 1885 and the couple had eight children. After abandoning dairying in favour of fishing, Edward and his sons built fishing boats on Shark Island.

Charlie Edwards was born in 1928, the son of Edward's youngest son, Victor William Edwards. In 1947 Charlie won the two mile Macleay River Championship title. In 1950, aged 22, Charlie won the Lightweight World Championship at Picnic Point on the Georges River Sydney. He successfully defended his World Title in 1952 on the Macleay River before retiring undefeated.