I don't think it's the audio filters. To me it seems your system is sometimes too busy / stalling.
Code
[09:32:49] Frame #1277: vRender: 496 us, vProcess: 2 us, vEncoding: 7666 us, aRender: 162 us, aEncoding: 9 us, Latency: 8345 us
[09:32:49] Frame #1278: vRender: 17 us, vProcess: 4 us, vEncoding: 8207 us, aRender: 161 us, aEncoding: 734 us, Latency: 9135 us
[09:32:53] Frame #1279: vRender: 3814635 us, vProcess: 4 us, vEncoding: 683 us, aRender: 210 us, aEncoding: 1013 us, Latency: 3816571 us
[09:32:53] Frame #1280: vRender: 10 us, vProcess: 3 us, vEncoding: 584 us, aRender: 188 us, aEncoding: 11 us, Latency: 807 us
[09:32:53] Frame #1281: vRender: 1026 us, vProcess: 2 us, vEncoding: 5962 us, aRender: 148 us, aEncoding: 576 us, Latency: 7723 us
[09:32:53] Frame #1282: vRender: 11 us, vProcess: 3 us, vEncoding: 7750 us, aRender: 147 us, aEncoding: 849 us, Latency: 8769 us
[09:32:53] Frame #1283: vRender: 15 us, vProcess: 4 us, vEncoding: 7507 us, aRender: 190 us, aEncoding: 687 us, Latency: 8413 us
Premiere caused a delay of almost 4s to render one single frame.