Williams was born in Zachary, Louisiana, to a family of sharecroppers. He had no formal schooling, and spent his childhood picking cotton and cutting sugar cane. In 1928, he moved to Baton Rouge, Louisiana and worked in a lumberyard. At the age of 20, Williams fashioned a crude guitar by attaching five copper strings to a cigar box, and soon after bought a cheap, mass-produced one. Robert was taught by Frank and Robert Metty and began to play for small events such as Church gatherings, fish fries, suppers, and dances. From the 1930s-1950s, Williams played music and continued to work in the lumberyards of Baton Rouge.
He was discovered in Angola prison, by ethnomusicologists Dr Harry Oster and Richard Allen, where he was serving a life sentence for shooting a man dead in a local club in 1956, an act which he claimed was in self-defense. Oster and Allen recorded Williams performing several of his songs about life in prison and pleaded for him to be pardoned. Under pressure from Oster, the parole board issued a pardon and commuted his sentence to 12 years. In December 1958 he was released into 'servitude parole', which required 80 hours of labor per week on a Denham Springs farm without due compensation, and only room and board provided. This parole prevented him from working in music, though he was able to occasionally play with Willie B. Thomas and Butch Cage at Thomas's home in Zachary. By this time, Williams' music had achieved some favorable word-of-mouth reviews, and he played his first concert outside Louisiana at the 1964 Newport Folk Festival.
By 1965 he was able to tour the country, traveling to Los Angeles, Massachusetts, Chicago and Berkeley, California. In 1966 he also toured Europe. In 1968 he settled in Maringouin, west of Baton Rouge and began to work outside of music.
In 1970, Williams began to perform once again, touring blues and folk festivals throughout the United States and Europe. His music has appeared in several films notably, the Roots of American Music; Country and Urban Music (1971); Out of the Blues into the Blacks (1972) and Blues Under the Skin (1972) the last two being French-made films.
His most popular recordings included "Prisoner's Talking Blues" and "Pardon Denied Again". Williams has been inducted into the Louisiana Blues Hall of Fame.
Williams had slowed down his work schedule by the late 1970s, largely due to his old age and declining health. Williams died in Rosedale, Louisiana on December 31, 1980, at the age of 66.