Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Color shift after first frame of output when using two composite transitions #623

Open
MaxPower15 opened this issue Nov 25, 2020 · 4 comments
Open

Color shift after first frame of output when using two composite transitions #623

MaxPower15 opened this issue Nov 25, 2020 · 4 comments
Labels
confirmed workaround

Comments

@MaxPower15
Copy link

MaxPower15 commented Nov 25, 2020 •
edited
Loading

Hi there! I'm diving into MLT to composite two videos next to each other at specific sizes and orientations. They may animate between different orientations, but for this issue, I kept it to just one to demonstrate the problem. The problem is that I often see a noticeable color shift right after the video starts. If there's no transition, I don't see a color shift in the output. (The color is slightly different from the original, but maybe close enough to get by. But what I mean is, it doesn't noticeably change from the first frame to the second.)

I can get rid of the shift if I switch the profile colorspace from 709 to 601, though I'm not sure if that means all but the first frame is output in 601 or if the smaller colorspace is just masking the color shift. The output does seem slightly darker than the original, at least in this example.

I was initially having this issue using the mlt library via swig, but I was also able to reproduce it with melt. The input XML in my example was created using the XML consumer from a ruby script, hence quite a bit of unnecessary attributes. But I included it all for completeness.

Ideally my output would be in the 709 colorspace and be as true to the original colors as possible while also having no discernible color shift in the output. Am I doing that correctly or is there a better way? What am I missing?

The input file: https://embed-ssl.wistia.com/deliveries/d2e5245e04b6bcdce7981a5bb550b602/pets000.mov?disposition=attachment

The output file (you can see the color shift here): https://embed-ssl.wistia.com/deliveries/0fa094f2a99c63b1253f01d7675082ef/output.mov?disposition=attachment

The command I'm using:

melt \
  xml:input.xml \
  -consumer \
    avformat:output.mov \
    vcodec=libx264 \
    f=mov \
    crf=23 \
    x264opts=keyint=89:min-keyint=89:scenecut=-1 \
    movflags=faststart \
    strict=strict \
    threads=0 \
    width=1280 \
    height=720

input.xml:

<?xml version="1.0"?>
<mlt LC_NUMERIC="en_US.UTF-8" version="6.23.0">
  <profile description="DV/DVD PAL" width="1280" height="720" progressive="1" sample_aspect_num="1" sample_aspect_den="1" display_aspect_num="1" display_aspect_den="1" frame_rate_num="30" frame_rate_den="1" colorspace="709"/>
  <producer id="producer0" in="0" out="114">
    <property name="length">15000</property>
    <property name="eof">continue</property>
    <property name="resource">0x000000ff</property>
    <property name="aspect_ratio">1</property>
    <property name="mlt_service">colour</property>
    <property name="colour">0x000000FF</property>
  </producer>
  <producer id="producer1" in="0" out="114">
    <property name="length">358</property>
    <property name="eof">continue</property>
    <property name="resource">pets000.mov</property>
    <property name="meta.media.nb_streams">2</property>
    <property name="meta.media.0.stream.type">video</property>
    <property name="meta.media.0.stream.frame_rate">23.976</property>
    <property name="meta.media.0.stream.sample_aspect_ratio">0</property>
    <property name="meta.media.0.codec.width">1280</property>
    <property name="meta.media.0.codec.height">720</property>
    <property name="meta.media.0.codec.rotate">0</property>
    <property name="meta.media.0.codec.frame_rate">24000</property>
    <property name="meta.media.0.codec.pix_fmt">yuv420p</property>
    <property name="meta.media.0.codec.sample_aspect_ratio">0</property>
    <property name="meta.media.0.codec.colorspace">709</property>
    <property name="meta.media.0.codec.color_trc">1</property>
    <property name="meta.media.0.codec.name">h264</property>
    <property name="meta.media.0.codec.long_name">H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10</property>
    <property name="meta.media.0.codec.bit_rate">8638147</property>
    <property name="meta.attr.0.stream.language.markup">eng</property>
    <property name="meta.attr.0.stream.handler_name.markup">Apple Video Media Handler</property>
    <property name="meta.media.1.stream.type">audio</property>
    <property name="meta.media.1.codec.sample_fmt">s16</property>
    <property name="meta.media.1.codec.sample_rate">44100</property>
    <property name="meta.media.1.codec.channels">2</property>
    <property name="meta.media.1.codec.name">pcm_s16le</property>
    <property name="meta.media.1.codec.long_name">PCM signed 16-bit little-endian</property>
    <property name="meta.media.1.codec.bit_rate">1411200</property>
    <property name="meta.attr.1.stream.language.markup">eng</property>
    <property name="meta.attr.1.stream.handler_name.markup">Apple Sound Media Handler</property>
    <property name="meta.attr.major_brand.markup">qt  </property>
    <property name="meta.attr.minor_version.markup">512</property>
    <property name="meta.attr.compatible_brands.markup">qt  </property>
    <property name="meta.attr.encoder.markup">Lavf58.20.100</property>
    <property name="seekable">1</property>
    <property name="meta.media.sample_aspect_num">1</property>
    <property name="meta.media.sample_aspect_den">1</property>
    <property name="aspect_ratio">1</property>
    <property name="audio_index">1</property>
    <property name="video_index">0</property>
    <property name="mute_on_pause">1</property>
    <property name="mlt_service">avformat</property>
  </producer>
  <producer id="producer2" in="0" out="114">
    <property name="length">358</property>
    <property name="eof">continue</property>
    <property name="resource">pets000.mov</property>
    <property name="meta.media.nb_streams">2</property>
    <property name="meta.media.0.stream.type">video</property>
    <property name="meta.media.0.stream.frame_rate">23.976</property>
    <property name="meta.media.0.stream.sample_aspect_ratio">0</property>
    <property name="meta.media.0.codec.width">1280</property>
    <property name="meta.media.0.codec.height">720</property>
    <property name="meta.media.0.codec.rotate">0</property>
    <property name="meta.media.0.codec.frame_rate">24000</property>
    <property name="meta.media.0.codec.pix_fmt">yuv420p</property>
    <property name="meta.media.0.codec.sample_aspect_ratio">0</property>
    <property name="meta.media.0.codec.colorspace">709</property>
    <property name="meta.media.0.codec.color_trc">1</property>
    <property name="meta.media.0.codec.name">h264</property>
    <property name="meta.media.0.codec.long_name">H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10</property>
    <property name="meta.media.0.codec.bit_rate">8638147</property>
    <property name="meta.attr.0.stream.language.markup">eng</property>
    <property name="meta.attr.0.stream.handler_name.markup">Apple Video Media Handler</property>
    <property name="meta.media.1.stream.type">audio</property>
    <property name="meta.media.1.codec.sample_fmt">s16</property>
    <property name="meta.media.1.codec.sample_rate">44100</property>
    <property name="meta.media.1.codec.channels">2</property>
    <property name="meta.media.1.codec.name">pcm_s16le</property>
    <property name="meta.media.1.codec.long_name">PCM signed 16-bit little-endian</property>
    <property name="meta.media.1.codec.bit_rate">1411200</property>
    <property name="meta.attr.1.stream.language.markup">eng</property>
    <property name="meta.attr.1.stream.handler_name.markup">Apple Sound Media Handler</property>
    <property name="meta.attr.major_brand.markup">qt  </property>
    <property name="meta.attr.minor_version.markup">512</property>
    <property name="meta.attr.compatible_brands.markup">qt  </property>
    <property name="meta.attr.encoder.markup">Lavf58.20.100</property>
    <property name="seekable">1</property>
    <property name="meta.media.sample_aspect_num">1</property>
    <property name="meta.media.sample_aspect_den">1</property>
    <property name="aspect_ratio">1</property>
    <property name="audio_index">1</property>
    <property name="video_index">0</property>
    <property name="mute_on_pause">1</property>
    <property name="mlt_service">avformat</property>
  </producer>
  <tractor id="tractor0" global_feed="1" in="" out="">
    <track producer="producer0"/>
    <track producer="producer1"/>
    <track producer="producer2"/>
    <transition id="transition0" out="114">
      <property name="a_track">0</property>
      <property name="b_track">1</property>
      <property name="start">-213.33333333333337/120.0:853.3333333333335x480.00000000000006</property>
      <property name="factory">loader</property>
      <property name="aligned">1</property>
      <property name="progressive">1</property>
      <property name="mlt_service">composite</property>
      <property name="end">-213.33333333333337/120.0:853.3333333333335x480.00000000000006</property>
      <property name="distort">1</property>
    </transition>
    <transition id="transition1" out="114">
      <property name="a_track">0</property>
      <property name="b_track">2</property>
      <property name="start">426.66666666666663/120.0:853.3333333333335x480.00000000000006</property>
      <property name="factory">loader</property>
      <property name="aligned">1</property>
      <property name="progressive">1</property>
      <property name="mlt_service">composite</property>
      <property name="end">426.66666666666663/120.0:853.3333333333335x480.00000000000006</property>
      <property name="distort">1</property>
    </transition>
  </tractor>
</mlt>

I compiled mlt from commit 0e756b6--I upgraded from 16.6.0 where I had the same issue--with this configuration:

./configure --disable-decklink --disable-dv --disable-feeds --disable-frei0r --disable-gtk2 --disable-jackrack --disable-kdevlive --disable-kino --disable-linsys --disable-lumas --disable-motion_est --disable-ndi --disable-normalize --disable-oldfilm --disable-opencv --disable-opengl --disable-qt --disable-resample --disable-rtaudio --disable-sox --disable-swfdec --disable-vid.stab --disable-videostab --disable-vmfx --disable-vorbis --disable-xine --swig-languages=ruby

And I'm on Mac OS Catalina 10.15.7.
@MaxPower15
Copy link
Author

MaxPower15 commented Nov 25, 2020 •
edited
Loading

If it's helpful at all, here's a shorter version of input.xml where I stripped out the pieces that didn't seem to be relevant. It should still reproduce the issue.

<mlt LC_NUMERIC="en_US.UTF-8" version="6.23.0">
  <profile description="DV/DVD PAL" width="1280" height="720" progressive="1" sample_aspect_num="1" sample_aspect_den="1" display_aspect_num="1" display_aspect_den="1" frame_rate_num="30" frame_rate_den="1" colorspace="709"/>
  <producer id="producer0" in="0" out="114">
    <property name="mlt_service">colour</property>
    <property name="colour">0x000000FF</property>
  </producer>
  <producer id="producer1" in="0" out="114">
    <property name="resource">pets000.mov</property>
  </producer>
  <producer id="producer2" in="0" out="114">
    <property name="resource">pets000.mov</property>
  </producer>
  <tractor id="tractor0" global_feed="1" in="" out="">
    <track producer="producer0"/>
    <track producer="producer1"/>
    <track producer="producer2"/>
    <transition id="transition0" out="114">
      <property name="a_track">0</property>
      <property name="b_track">1</property>
      <property name="start">-213.33333333333337/120.0:853.3333333333335x480.00000000000006</property>
      <property name="progressive">1</property>
      <property name="mlt_service">composite</property>
      <property name="end">-213.33333333333337/120.0:853.3333333333335x480.00000000000006</property>
      <property name="distort">1</property>
    </transition>
    <transition id="transition1" out="114">
      <property name="a_track">0</property>
      <property name="b_track">2</property>
      <property name="start">426.66666666666663/120.0:853.3333333333335x480.00000000000006</property>
      <property name="progressive">1</property>
      <property name="mlt_service">composite</property>
      <property name="end">426.66666666666663/120.0:853.3333333333335x480.00000000000006</property>
      <property name="distort">1</property>
    </transition>
  </tractor>
</mlt>

@bmatherly
Copy link
Member

I wonder if this problem still exists in MLT master?

I do not reproduce the problem, but I also do not have your source file pets000.mov

One possible explanation is if your source file might be incorrectly identifying the colorspace. It says it is 709, but maybe it is actually 601. If this is the case, then the first time it require conversion it will experience a subtle color shift. It is not uncommon for a transition to trigger a colorspace conversion since it operates in 4:2:2 while the source file is 4:2:0.

@MaxPower15
Copy link
Author

Hm, I just pulled and recompiled (SHA 817e4d8), but the output seems the same for me. The source file pets000.mov can be downloaded here if you decide to look any further: https://embed-ssl.wistia.com/deliveries/d2e5245e04b6bcdce7981a5bb550b602/pets000.mov?disposition=attachment.

To your point, relevant pieces of ffprobe on pets000.mov:

pix_fmt=yuv420p
color_range=unknown
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=left

And from mediainfo:

Color space                              : YUV
Chroma subsampling                       : 4:2:0

Still, if MLT is converting the output to a different colorspace, and the transition applies to the entire duration of the file, why wouldn't it be converted on the first frame, too? Then again, if it's not showing up for you when testing with the same input file, maybe there's something else entirely going on with my setup.

@bmatherly
Copy link
Member

A workaround for your problem is to replace the composite transitions with affine instead. It is as easy as changing:

<property name="mlt_service">composite</property>
to
<property name="mlt_service">affine</property>

Perhaps there is a bug in the composite transition. I did not investigate inside the composite transition because it has so many other deficiencies relative to the affine transition that I typically recommend people use the affine transition in all cases. The only drawback is that the affine transition is slightly slower.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
confirmed workaround
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@MaxPower15 @bmatherly