Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Colours discrepancy when applying overlay #1509

Open
1 task done
mikekudzin opened this issue Jun 28, 2024 · 12 comments
Open
1 task done

Colours discrepancy when applying overlay #1509

mikekudzin opened this issue Jun 28, 2024 · 12 comments

Comments

@mikekudzin
Copy link

mikekudzin commented Jun 28, 2024

Version

Media3 pre-release (alpha, beta or RC not in this list)

More version details

1.4.0-beta01

Devices that reproduce the issue

Pixel 4a Android 13

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Not tested

Reproduction steps

Overlaying video with a bitmap.

28 Jun 2024 11_18_34 GMT_overlays

no. 1: Create transparent media items with a BitmapOverlay
Code:

    val bitmapOverlayWithText = BitmapOverlay.createStaticBitmapOverlay(overlayBitmap)
    val overlayFrameEMI = EditedMediaItem.Builder(MediaItem.fromUri(frameFile.toUri()))
        .setFrameRate(D_F_RATE)
        .setDurationUs(D_DURATION)
        .setRemoveAudio(true)
        .setEffects(
            Effects(
                listOf(),
                listOf(
                    OverlayEffect(ImmutableList.of(bitmapOverlayWithText)),
                )
            )
        )
        .build()

    val backgroundMI = MediaItem.fromUri(backgroundBitmapFile.toUri())
    val backgroundEMI = EditedMediaItem.Builder(backgroundMI)
        .setFrameRate(D_F_RATE)
        .setDurationUs(D_DURATION)
        .setRemoveAudio(true)
        .build()

    val videoMI = MediaItem.Builder().setUri(baseMediaUri)
        .setClippingConfiguration(
            MediaItem.ClippingConfiguration.Builder()
                .setStartPositionUs(0)
                .setEndPositionUs(D_DURATION)
                .build()
        )
        .build()

    val videoEMI = EditedMediaItem.Builder(videoMI)
        .setEffects(
            Effects(
                listOf(),
                listOf(ScaleAndRotateTransformation.Builder().setRotationDegrees(45f).build())
            )
        )
        .build()

    val composition = Composition.Builder(
        EditedMediaItemSequence(overlayFrameEMI),
        EditedMediaItemSequence(videoEMI),
        EditedMediaItemSequence(backgroundEMI),            
    )
        .build()

    val transformer: Transformer =
        Transformer.Builder(context)
            .setVideoMimeType(MimeTypes.VIDEO_H264)
            .setAudioMimeType(MimeTypes.AUDIO_AAC)
            .build()
    transformer.start(composition, outputFilePath)

Result:

image

no. 2: create MediaItem directly from the file
Code diff
val overlayFrameEMI = EditedMediaItem.Builder(MediaItem.fromUri(overlayFile.toUri()))
.setFrameRate(D_F_RATE)
.setDurationUs(D_DURATION)
.build()

image

Although, the the last one looks close to origin, there is a significant colours discrepancy between the origin no. 1 and no. 2
I didn't observe no. 1 issue on 1.3.1

Expected result

Colours in the output are the same as in the origin.

Actual result

There is a difference between colours in the origin and BitmapOverlay

Media

Bug Report

  • You will email the zip file produced by adb bugreport to [email protected] after filing this issue.
@tof-tof
Copy link
Contributor

tof-tof commented Jul 3, 2024

Hi @mikekudzin, thanks for writing in.

just to make sure I understand the setup, are you saying in (1) you used overlays and in (2) you didn't use overlays at all?

Since you didn't see it in 1.3.1 I figure it could have some involvment to with the way we process color changing. Using the code in #1050 (comment) may help your case

In addition I see there are some artefacts around the borders of your image. @ychaparov does any of the recent work you've done around improving sampling help with this?

@ychaparov
Copy link
Contributor

Not really, we haven't made improvements to resampling of 45-degree rotated videos yet.

The original image bitmap overlay is not aligned with the no. 1 output -- the text is a bit scaled and rotated even though both images have resolution 1080x1920.
I don't think BitmapOverlay causes this image modification.

@mikekudzin can you share your bitmap-reading code?
How did you create overlayBitmap used in

    val bitmapOverlayWithText = BitmapOverlay.createStaticBitmapOverlay(overlayBitmap)

@yuriikonovalov
Copy link

yuriikonovalov commented Jul 3, 2024

@tof-tof @ychaparov I have the same problem. The version is 1.4.0-beta01. The device I'm using is Samsung S21. I've tried the suggestion in the #1050 comment but it does not help.

Here is the input and output resources:

  • This image is added as a MediaItem
  • This image is added as a BitmapOverlay
  • This is a frame in an output video

 
Here is the way the overlay bitmap is created:

   val stream = context.contentResolver.openInputStream(uri)
   val bitmap = BitmapFactory.decodeStream(stream)

@ychaparov
Copy link
Contributor

ychaparov commented Jul 3, 2024

you should probably set some bitmap factory options like this:

    if (Util.SDK_INT >= 26) {
      options = new BitmapFactory.Options();
      options.inPreferredColorSpace = ColorSpace.get(ColorSpace.Named.SRGB);
    }

Here's how the MediaItem image BitmapFactory options will be set

@Nullable BitmapFactory.Options options = null;
if (Util.SDK_INT >= 26) {
options = new BitmapFactory.Options();
options.inPreferredColorSpace = ColorSpace.get(ColorSpace.Named.SRGB);
}

@yuriikonovalov
Copy link

@ychaparov just tried that and it stays the same as I mentioned above.

@mikekudzin
Copy link
Author

mikekudzin commented Jul 9, 2024

just to make sure I understand the setup, are you saying in (1) you used overlays and in (2) you didn't use overlays at all?

@tof-tof Correct.

@ychaparov here is bitmap initialisation code.

    private fun createCanvasBitmap(width: Int = D_WIDTH, height: Int = D_HEIGHT): Bitmap {
        val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
        return bitmap
    }

    protected fun initCanvas(width: Int = D_WIDTH, height: Int = D_HEIGHT): Pair<Canvas, Bitmap> {
        val bitmap = createCanvasBitmap(width, height)
        val canvas = Canvas(bitmap)
        return canvas to bitmap
    }

I also tried approach with the BitmapFactory

        val options = BitmapFactory.Options()
        options.inPreferredColorSpace = ColorSpace.get(ColorSpace.Named.SRGB)
        val overlayBitmap = BitmapFactory.decodeFile(overlayFile.path, options)

Still observe the issue .

@droid-girl droid-girl assigned ychaparov and unassigned tof-tof Jul 22, 2024
@droid-girl
Copy link
Contributor

@yuriikonovalov and @mikekudzin how many EditedMediaItemSequences do you have in your composition?

@yuriikonovalov
Copy link

yuriikonovalov commented Jul 22, 2024

@droid-girl In my case, it's 3 EditedMediaItemSequences. One is for a video. Another one is a transparent image with those bitmap overlays. One more is just used to add the "a media item" image as a media item to the composition, not as a bitmap overlay so that the difference is visible.

@mikekudzin
Copy link
Author

mikekudzin commented Jul 23, 2024

@droid-girl I use 3 EditedMediaItemSequences: background, video with some transformations, overlay

@droid-girl
Copy link
Contributor

@ychaparov figured out the problem, we will have a fix soon in main. The issue was with Composition that contains more than 1 sequence containing video/image assets.

copybara-service bot pushed a commit that referenced this issue Jul 23, 2024
Build upon Transformer.videoFrameProcessorFactory in MultipleInputVideoGraph
Check that Transformer.videoFrameProcessorFactory is DefaultVideoFrameProcessor
for multi-input video

Fixes #1509

PiperOrigin-RevId: 655232381
@ychaparov
Copy link
Contributor

Could you please try verify if 7103f21 fixes the issue for your use cases?

@yuriikonovalov
Copy link

yuriikonovalov commented Jul 24, 2024

@ychaparov Just tried that, it's fixed for my case. Thanks.

Just need to set SdrWorkingColorSpace. Otherwise colors will still be a little bit off but not like it was before.

Transformer.Builder(this)
    .setVideoFrameProcessorFactory(
        DefaultVideoFrameProcessor.Factory.Builder()
            .setSdrWorkingColorSpace(
                DefaultVideoFrameProcessor.WORKING_COLOR_SPACE_ORIGINAL
            )
            .build()
    )

With setting SdrWorkingColorSpace

Without setting SdrWorkingColorSpace - here it's possible to see some difference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants