The thing is I need to specify these values when creating the muxer / format instance:
- matrix
- primaries
- transfer
They are "unspecified" by default. Which seems to be okay for all non-2020 color spaces.
Currently they will be only set when using the color space filter and this seems to be the issue. I need to think about a smart way to set these values even when not using any filter. But if a filter is used the filter settings will overwrite these values.