He was married to actresses Doris Rankin and Irene Fenwick, a one-time lover of his brother John. Doris Rankin bore Lionel two daughters, Ethel Barrymore II (born 1909) and Mary Barrymore (born 1910). Unfortunately, neither baby girl survived infancy, though Mary lived a few months. Lionel never truly recovered from the deaths of his girls, and their loss undoubtedly strained his marriage to Doris Rankin which ended in 1923. Years later, Barrymore developed a fatherly affection for Jean Harlow, who was born around the same time as his two daughters and would have been around their age. When Jean died in 1937, Lionel and Clark Gable mourned her as though she had been family.
Perhaps his best known role, due to perennial Christmas time replays on television, was Mr. Potter, the miserly and mean-spirited banker in It's a Wonderful Life (1946). The role suggested that of the "unreformed" stage of Barrymore's "Scrooge" characterization.
Barrymore died on November 15, 1954 from a heart attack in Van Nuys, California, and was entombed in the Calvary Cemetery in East Los Angeles, California.