Then I made a mapping from vertices to their adjacent patches, and every time I iterate on radiosity I replace the color at a given vertex with a simple average.
Lots left to do, not the least of which is checking the radiometry. But first I'll continually run it while moving the light source to get a nice video, but then I need to modify the algorithm to allow me to load Sponza. Right now I'm taking the easy way out and computing the form factors for all pairs of patches in advance. This is n^2, so on scenes with a million triangles, this takes 4 TB of space... If I switch to only storing all the form factors for one patch at a time I can reign this in to be reasonable.
I'll start again after I wake up. Good night!