It's a cliche that baseball is the most literary sport, but I think the real winner is mountain climbing, which in my youth supported a vast outpouring of books despite not being a spectator sport, and not even having many participants. (Here's somebody's list of Top 100 Mountaineering Books.)

Mountaineering is half sport / half exploration. Polar exploration is a related field that has generated numerous books.

Novelist Will Self notes:

For many years I liked nothing better than to lie down – preferably in low-lying country such as East Anglia – and lose myself in the halting, pained progress of mountain climbers being winched ever-upwards by their own deranged romanticism.

In October 1905, Mallory entered Magdalene College, Cambridge to study history. There, he became good friends with members of the Bloomsbury Group including James Strachey, Lytton Strachey, Rupert Brooke, John Maynard Keynes, and Duncan Grant, who painted several portraits of Mallory. Mallory was a keen oarsman and rowed in the college eight for his three years at Cambridge.

In case you were wondering (and, yes, you should), Mallory left a widow and three small children.