Strong in talent and business associations, industrious and honest, close in his attention to detail, he was one of Victoria's most prominent men of commerce. A good provider rather than a high liver, his life and the outstanding collection of records he left for historians form a fascinating reflection of colonial Victoria's commercial history.

On 24 September 1845 Graham had married Mary Alleyne, née Cobham; of their eighteen children, eight died young. He died at South Yarra on 31 July 1898, leaving an estate worth more than £189,000. (Frank Strahan, 'Graham, James (1819–1898)', Australian Dictionary of Biography).