I finally figured it out. It wasn't the license but a rather silly mistake and a lot of consequent confusion on my part.
The first time it failed was because I still had leftover files from the old version, which probably broke the new version's trial license or something. But after that, the reason PRORES_KS worked and LIBSVTAV1 or LIBAOM-AV1 didn't work was because of the default muxer container. I forgot to change MOV to MATROSKA when I deleted everything and tried different installations. I'm guessing MOV still doesn't support AV1 or VP9? Whatever the reason, everything seems to be working after I switched to MATROSKA.
Just leaving it here in case anyone else encounters the same issue. You may also consider implementing a more specific error description, a log entry, or something like a red border around the Muxer node and an "incompatible container" popup on an encoder/container mismatch. Anyway, thanks for the help.