YUNPM get frame is black

JESTERRRRRR · June 28, 2015, 5:07pm

Hi, I am trying to use YUNPM to get frames of video onto a BufferedImage. My method works for the first 3 frames but then stops returning a bytebuffer, just null. Here is what I am doing:

File film = new File(path.getAbsolutePath()+"\\Assets\\Movies\\1.mp4");//This part works as it plays the sound from the sample video
player = new MoviePlayer(film);

Then to get a frame

player.tick();
ByteBuffer b = player.movie.videoStream().pollFrameData();

After the first 3 times it returns null. It is displaying correctly at least one frame, but I’m not sure the 2nd and 3rd are actually different from the first it displays. I have also tried the following after the above

player.movie.videoStream().freeFrameData(b);
player.movie.onUpdatedVideoFrame();

And while this does let me get more than 3 frames, they are bizzare colours and all over the place, with it eventually giving me


java.lang.IllegalArgumentException: Number of remaining buffer elements is 528384, must be at least 921600.

Has anyone else used this and knows what I am doing wrong?

Riven · June 28, 2015, 5:44pm

There is no need to bring in movie playback, which syncs with audio. As you’re not signalling you played audio samples, the video stream will eventually stall. I’ll give you some working code in 10 minutes. Gotta finish some cookin’ first.

Riven · June 28, 2015, 6:20pm


import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.io.DataInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import net.indiespot.media.impl.FFmpeg;
import net.indiespot.media.impl.VideoMetadata;

public class TestFrameExtraction {
	public static void main(String[] args) throws Exception {
		File movieFile = new File(args[0]);

		VideoMetadata meta = FFmpeg.extractMetadata(movieFile);
		InputStream rgb24Stream = FFmpeg.extractVideoAsRGB24(movieFile, 0);

		DataInputStream videoStream = new DataInputStream(rgb24Stream);

		BufferedImage buf = new BufferedImage(meta.width, meta.height, BufferedImage.TYPE_3BYTE_BGR);
		byte[] rgb = accessRasterByteArray(buf);
		for (int frame = 0; true; frame++) {
			readFrame(videoStream, rgb);
			swapChannels(rgb);
			System.out.println("frame: " + frame + " (" + meta.width + "x" + meta.height + ")");
		}
	}

	private static void readFrame(DataInputStream videoStream, byte[] dst) throws IOException {
		// videoStream.readFully(dst);
		int chunk = 64 * 1024;
		for (int off = 0; off != dst.length;) {
			int len = Math.min(chunk, dst.length - off);
			videoStream.readFully(dst, off, len);
			off += len;
		}
	}

	private static byte[] accessRasterByteArray(BufferedImage src) {
		return ((DataBufferByte) src.getRaster().getDataBuffer()).getData();
	}

	private static void swapChannels(byte[] rgb) {
		// BGR <-> RGB
		for(int i=0; i<rgb.length; i+=3) {
			byte tmp = rgb[i+0];
			rgb[i+0] = rgb[i+2];
			rgb[i+2] = tmp;
		}
	}
}

JESTERRRRRR · June 29, 2015, 4:23am

Thank you! This is almost perfect, I’m having 1 tiny problem though…

I am getting the colors on the left, instead of how it should be on the right. I’m assuming this is since its BGR. Do I needs to convert the BufferedImage, or is there a quicker way I can get it as RGB?
Either way, this is brilliant thanks.

Riven · June 29, 2015, 8:36am

BufferedImage sadly doesn’t have a 3BYTE_RGB type :


// BGR <-> RGB
for(int i=0; i<rgb.length; i+=3) {
   byte tmp = rgb[i+0];
   rgb[i+0] = rgb[i+2];
   rgb[i+2] = tmp;
}

Keep in mind YUNPM uses STDIN/STDOUT to ‘pipe’ raw video/audio data from a separate FFMPEG process. This means that for 1920x1080 @ 24Hz, over 140MB/sec is transfered over I/O channels not really meant for this bandwidth (on Windows, at least). The overhead is generally acceptable, but with Full HD and higher resolutions and/or framerates, you need to look for alternatives.

nsigma · June 29, 2015, 1:59pm

hmmm … possible to make one though (code not tested - based on this)


WritableRaster raster = Raster.createInterleavedRaster(
        DataBuffer.TYPE_BYTE, width, height, 
        width * 3, // scanlineStride
        3, // pixelStride
        new int[]{0, 1, 2}, // bandOffsets
        null);

ColorModel colorModel = new ComponentColorModel(
        ColorSpace.getInstance(ColorSpace.CS_sRGB), 
        new int[]{8, 8, 8}, // bits
        false, // hasAlpha
        false, // isPreMultiplied
        ComponentColorModel.OPAQUE, 
        DataBuffer.TYPE_BYTE);

BufferedImage img = new BufferedImage(colorModel, raster, false, null);

Riven · June 29, 2015, 3:04pm

I think that triggers an extraordinarily slow path when rendering. :persecutioncomplex:

nsigma · June 29, 2015, 3:15pm

Quite possibly, although at worst it’s probably* only doing the same byte rearranging you are.

probably implies logic in Java2D :persecutioncomplex:

Riven · June 29, 2015, 3:35pm

A decade ago I looked into the Java2D (software) backend, and for any non-standard type, it basically did pretty ‘impressive’ per pixel operations. It’s like doing bufimg.setRGB(x, y, value) for every pixel. That’s a fancy trip down the rabbit hole, btw. Back in the day every pixel op involved a few object creations and dozen layers of indirection. I doubt much has changed, especially considering that people report factor 1000 slowdowns for some paths in the link you mentioned.

nsigma · June 29, 2015, 4:37pm

Been there, done that. IIRC though it should in this case pass everything to the native conversion loops in one go rather than per pixel, hence blindly hoping that the OP might find it fast enough. It’s a quicker thing to test.

Does YUNPM give any way to specify what color model FFMPEG provides the data? Easy to modify to BGR?

That also mentions some versions of OSX, that makes me think it might refer to the old Quartz renderer - nice and fast in some circumstances, FUBAR in others.

Riven · June 29, 2015, 4:49pm

You could add a method that takes the format as a parameter.

package net.indiespot.media.impl;

...

public class FFmpeg
{
	...

	public static InputStream extractVideo(File srcMovieFile, int seconds, String pixfmt) throws IOException {
		return streamData(new ProcessBuilder().command(//
		   FFMPEG_PATH, //
		   "-ss", String.valueOf(seconds), //
		   "-i", srcMovieFile.getAbsolutePath(), //		   
		   "-f", "rawvideo", //
-		   "-pix_fmt", "rgb24", //
+		   "-pix_fmt", pixfmt, //
		   "-" //
		));
	}

	...
}

List of all supported formats:


FLAGS NAME            NB_COMPONENTS BITS_PER_PIXEL
-----
IO... yuv420p                3            12
IO... yuyv422                3            16
IO... rgb24                  3            24
@@IO... bgr24                  3            24 <--- there we go
IO... yuv422p                3            16
IO... yuv444p                3            24
IO... yuv410p                3             9
IO... yuv411p                3            12
IO... gray                   1             8
IO..B monow                  1             1
IO..B monob                  1             1
I..P. pal8                   1             8
IO... yuvj420p               3            12
IO... yuvj422p               3            16
IO... yuvj444p               3            24
..H.. xvmcmc                 0             0
..H.. xvmcidct               0             0
IO... uyvy422                3            16
..... uyyvyy411              3            12
IO... bgr8                   3             8
.O..B bgr4                   3             4
IO... bgr4_byte              3             4
IO... rgb8                   3             8
.O..B rgb4                   3             4
IO... rgb4_byte              3             4
IO... nv12                   3            12
IO... nv21                   3            12
IO... argb                   4            32
IO... rgba                   4            32
IO... abgr                   4            32
IO... bgra                   4            32
IO... gray16be               1            16
IO... gray16le               1            16
IO... yuv440p                3            16
IO... yuvj440p               3            16
IO... yuva420p               4            20
..H.. vdpau_h264             0             0
..H.. vdpau_mpeg1            0             0
..H.. vdpau_mpeg2            0             0
..H.. vdpau_wmv3             0             0
..H.. vdpau_vc1              0             0
IO... rgb48be                3            48
IO... rgb48le                3            48
IO... rgb565be               3            16
IO... rgb565le               3            16
IO... rgb555be               3            15
IO... rgb555le               3            15
IO... bgr565be               3            16
IO... bgr565le               3            16
IO... bgr555be               3            15
IO... bgr555le               3            15
..H.. vaapi_moco             0             0
..H.. vaapi_idct             0             0
..H.. vaapi_vld              0             0
IO... yuv420p16le            3            24
IO... yuv420p16be            3            24
IO... yuv422p16le            3            32
IO... yuv422p16be            3            32
IO... yuv444p16le            3            48
IO... yuv444p16be            3            48
..H.. vdpau_mpeg4            0             0
..H.. dxva2_vld              0             0
IO... rgb444le               3            12
IO... rgb444be               3            12
IO... bgr444le               3            12
IO... bgr444be               3            12
I.... gray8a                 2            16
IO... bgr48be                3            48
IO... bgr48le                3            48
IO... yuv420p9be             3            13
IO... yuv420p9le             3            13
IO... yuv420p10be            3            15
IO... yuv420p10le            3            15
IO... yuv422p10be            3            20
IO... yuv422p10le            3            20
IO... yuv444p9be             3            27
IO... yuv444p9le             3            27
IO... yuv444p10be            3            30
IO... yuv444p10le            3            30
IO... yuv422p9be             3            18
IO... yuv422p9le             3            18
..H.. vda_vld                0             0
I.... gbrp                   3            24
I.... gbrp9be                3            27
I.... gbrp9le                3            27
I.... gbrp10be               3            30
I.... gbrp10le               3            30
I.... gbrp16be               3            48
I.... gbrp16le               3            48
I.... rgba64be               4            64
I.... rgba64le               4            64
..... bgra64be               4            64
..... bgra64le               4            64
IO... 0rgb                   3            24
IO... rgb0                   3            24
IO... 0bgr                   3            24
IO... bgr0                   3            24
IO... yuva444p               4            32
IO... yuva422p               4            24
IO... yuv420p12be            3            18
IO... yuv420p12le            3            18
IO... yuv420p14be            3            21
IO... yuv420p14le            3            21
IO... yuv422p12be            3            24
IO... yuv422p12le            3            24
IO... yuv422p14be            3            28
IO... yuv422p14le            3            28
IO... yuv444p12be            3            36
IO... yuv444p12le            3            36
IO... yuv444p14be            3            42
IO... yuv444p14le            3            42
I.... gbrp12be               3            36
I.... gbrp12le               3            36
I.... gbrp14be               3            42
I.... gbrp14le               3            42

JESTERRRRRR · June 29, 2015, 8:04pm

Now solved, thanks for the posts guys.