Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Video stuttering when rendering to SFML texture #203

Open
oomek opened this issue May 13, 2024 · 21 comments
Open

Video stuttering when rendering to SFML texture #203

oomek opened this issue May 13, 2024 · 21 comments

Comments

@oomek
Copy link

oomek commented May 13, 2024

I'm getting a very uneven framerate when rendering the video to SFML texure.
When I play a 30fps video at 60Hz or 120Hz for example the callback function should fire every 2nd or 4th frame, shouldn't it?
I've also tried putting all the calls from if (new_frame == true) inside the callback, but the result is the same.
Am I using setRenderCallback() correctly?

Platform: Windows 11

#include <SFML/Window.hpp>
#include <SFML/Graphics.hpp>
#include "mdk/Player.h"
using namespace MDK_NS;

#include <chrono>
using namespace std::chrono;

#include <atomic>

int main(int argc, char** argv)
{
	sf::Context vid_context;
	vid_context.setActive(true);

	Player player;

	std::atomic<int> new_frame(0);
	auto t0 = steady_clock::now();

	player.setRenderCallback([&](void*)
	{
		new_frame++;
		const auto t = steady_clock::now();
		printf("elapsed: %lld new_frame: %d\n", duration_cast<milliseconds>(t-t0).count(), new_frame.load());
		t0 = t;
	});

	player.setMedia(argv[argc-1]);
	player.prepare();

	for(;;)
	{
		if (player.mediaStatus() > MediaStatus::Buffering) break;
		if (player.mediaStatus() == MediaStatus::Invalid) return 0;
	}

	auto video_info = player.mediaInfo().video[0];
	player.setVideoSurfaceSize(video_info.codec.width * video_info.codec.par, video_info.codec.height);


	sf::RenderWindow window(sf::VideoMode(800, 600), "Test");
	window.setVerticalSyncEnabled(true);

	sf::RenderTexture texture;

	if (!texture.create(video_info.codec.width * video_info.codec.par, video_info.codec.height))
		return -1;

	player.set(State::Playing);

	while (window.isOpen())
	{
		sf::Event event;
		while (window.pollEvent(event))
		{
			if (event.type == sf::Event::Closed)
				window.close();
			if (event.type == sf::Event::KeyPressed)
			{
				if (event.key.code == sf::Keyboard::Escape)
				{
					player.setVideoSurfaceSize(-1, -1);
					return 0;
				}
			}
		}

		if (new_frame > 0)
		{
			vid_context.setActive(true);
			texture.setActive(true);
			player.renderVideo();
			texture.setActive(false);
			texture.display();
			vid_context.setActive(false);
			new_frame = 0;
		}

		window.setActive(true);
		window.clear();
		sf::Sprite sprite(texture.getTexture());
		window.draw(sprite);
		window.display();
	}
	return 0;
}
@wang-bin
Copy link
Owner

you can print the draw interval

@oomek
Copy link
Author

oomek commented May 14, 2024

I would need a little more help please.

@wang-bin
Copy link
Owner

if (new_frame == true) {
        const auto t = steady_clock::now();
        printf("elapsed: %lld\n", duration_cast<milliseconds>(t-t0).count());
        t0 = t;

@oomek
Copy link
Author

oomek commented May 14, 2024

Playback of 30fps video at 60Hz

elapsed: 1
elapsed: 71
elapsed: 16
elapsed: 50
elapsed: 16
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 34
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 49
elapsed: 34
elapsed: 16
elapsed: 32
elapsed: 49
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 49
elapsed: 16
elapsed: 49
elapsed: 16
elapsed: 33
elapsed: 33
elapsed: 16
elapsed: 49
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 50
elapsed: 16
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 50
elapsed: 17
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 50
elapsed: 16
elapsed: 31
elapsed: 33

@oomek
Copy link
Author

oomek commented May 14, 2024

Playback of 60fps video at 60Hz

elapsed: 1
elapsed: 76
elapsed: 33
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 17
elapsed: 17
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 16
elapsed: 31
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 17
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 17
elapsed: 16
elapsed: 33
elapsed: 31
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 16

@oomek
Copy link
Author

oomek commented May 14, 2024

Elapsed print after window.display()

elapsed: 48
elapsed: 5
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16

@oomek
Copy link
Author

oomek commented May 14, 2024

Video1

Video: MPEG4 Video (H264) 854x480 30fps 2025kbps [V: VideoHandler [eng] (h264 main L4.1, yuv420p, 854x480, 2025 kb/s)]
Audio: AAC 44100Hz stereo 126kbps [A: Stereo [eng] (aac lc, 44100 Hz, stereo, 126 kb/s)]

Video2

Video: MPEG4 Video (H264) 640x480 59.94fps 375kbps [V: h264 high L3.1, yuv420p, 640x480, 375 kb/s]
Audio: AAC 48000Hz stereo 64kbps [A: aac lc, 48000 Hz, stereo, 64 kb/s]

@wang-bin
Copy link
Owner

what about the interval in render callback? what about changing new_frame type to atomic and check if (new_frame > 0)?

@oomek
Copy link
Author

oomek commented May 14, 2024

I've updated the code in my first post

elapsed: 30 new_frame: 1
elapsed: 156 new_frame: 2
elapsed: 2 new_frame: 3
elapsed: 1 new_frame: 4
elapsed: 41 new_frame: 1
elapsed: 2 new_frame: 2
elapsed: 43 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 62 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 15 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 48 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 15 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 15 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 45 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 32 new_frame: 1
elapsed: 32 new_frame: 1
elapsed: 15 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 45 new_frame: 1
elapsed: 15 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 45 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1

@oomek
Copy link
Author

oomek commented May 14, 2024

I've noticed that removing sf::Context and switching Threaded Optimization in nvidiaProfileInspector to ON has an effect on player.renderVideo() which is stalling the thread for 16ms. Unfortunately it has no effect on the stutter.

@oomek
Copy link
Author

oomek commented May 14, 2024

But with Threaded Optimization set to ON the log looks different:
Btw, In my project where I use pure ffmpeg to play videos I need to disable Threaded Optimization because I get stuttering.

elapsed: 29 new_frame: 1
elapsed: 164 new_frame: 2
elapsed: 29 new_frame: 3
elapsed: 46 new_frame: 1
elapsed: 2 new_frame: 2
elapsed: 28 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 2
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 45 new_frame: 1
elapsed: 15 new_frame: 2
elapsed: 46 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 32 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 2
elapsed: 31 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 15 new_frame: 2
elapsed: 46 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 2
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 32 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 15 new_frame: 2
elapsed: 46 new_frame: 1
elapsed: 30 new_frame: 2
elapsed: 31 new_frame: 1
elapsed: 45 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 2
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 15 new_frame: 2

@oomek
Copy link
Author

oomek commented May 14, 2024

With the following hardcoded framerate example I was able to render 30fps@60Hz smoothly.
One condition Threaded Optimization must be OFF, otherwise there is occasional stutter and renderVideo() takes 16ms

#include <SFML/Window.hpp>
#include <SFML/Graphics.hpp>
#include "mdk/Player.h"
using namespace MDK_NS;

#include <iostream>

int main(int argc, char** argv)
{
	int flipflop = 1;

	Player player;

	player.setMedia(argv[argc-1]);
	player.prepare();

	for(;;)
	{
		if (player.mediaStatus() > MediaStatus::Buffering) break;
		if (player.mediaStatus() == MediaStatus::Invalid) return 0;
	}

	auto video_info = player.mediaInfo().video[0];
	player.setVideoSurfaceSize(video_info.codec.width * video_info.codec.par, video_info.codec.height);

	sf::RenderWindow window(sf::VideoMode(800, 600), "Test");
	window.setVerticalSyncEnabled(true);

	sf::RenderTexture texture;

	if (!texture.create(video_info.codec.width * video_info.codec.par, video_info.codec.height))
		return -1;

	player.set(State::Playing);

	while (window.isOpen())
	{
		sf::Event event;
		while (window.pollEvent(event))
		{
			if (event.type == sf::Event::Closed)
				window.close();
			if (event.type == sf::Event::KeyPressed)
			{
				if (event.key.code == sf::Keyboard::Escape)
				{
					player.setVideoSurfaceSize(-1, -1);
					return 0;
				}
			}
		}

		if (flipflop == 1)
		{
			texture.setActive(true);
			sf::Clock clk;
			player.renderVideo();
			std::cout << clk.getElapsedTime().asMicroseconds() << std::endl;
			texture.display();
			texture.setActive(false);
		}

		flipflop *= -1;

		window.setActive(true);
		window.clear();
		sf::Sprite sprite(texture.getTexture());
		window.draw(sprite);
		window.display();
	}
	return 0;
}

@oomek
Copy link
Author

oomek commented May 14, 2024

The following code is stuttering very little with TO-OFF, Can this be achieved with callbacks?

#include <SFML/Window.hpp>
#include <SFML/Graphics.hpp>
#include "mdk/Player.h"
using namespace MDK_NS;

#include <iostream>

int main(int argc, char** argv)
{
	sf::Context ctx;
	Player player;

	player.setMedia(argv[argc-1]);
	player.prepare();

	for(;;)
	{
		if (player.mediaStatus() > MediaStatus::Buffering) break;
		if (player.mediaStatus() == MediaStatus::Invalid) return 0;
	}

	auto video_info = player.mediaInfo().video[0];
	player.setVideoSurfaceSize(video_info.codec.width * video_info.codec.par, video_info.codec.height);

	float fps = video_info.codec.frame_rate;
	std::cout << video_info.codec.frame_rate << std::endl;

	ctx.setActive(false);

	sf::RenderWindow window(sf::VideoMode(640, 480), "Test");
	window.setVerticalSyncEnabled(true);

	sf::RenderTexture texture;

	if (!texture.create(video_info.codec.width * video_info.codec.par, video_info.codec.height))
		return -1;

	player.set(State::Playing);

	sf::Clock clock;
	sf::Time timeSinceLastUpdate = sf::Time::Zero;
	sf::Time timePerFrame = sf::seconds(1.0f / fps);

	while (window.isOpen())
	{
		sf::Event event;
		while (window.pollEvent(event))
		{
			if (event.type == sf::Event::Closed)
				window.close();
			if (event.type == sf::Event::KeyPressed)
			{
				if (event.key.code == sf::Keyboard::Escape)
				{
					player.setVideoSurfaceSize(-1, -1);
					return 0;
				}
			}
		}

		timeSinceLastUpdate += clock.restart();

		if (timeSinceLastUpdate > timePerFrame)
		{
			ctx.setActive(true);
			texture.setActive(true);
			// sf::Clock c;
			player.renderVideo();
			// std::cout << c.getElapsedTime().asMicroseconds() << std::endl;
			texture.display();
			texture.setActive(false);
			ctx.setActive(false);

			timeSinceLastUpdate -= timePerFrame;
		}

		window.setActive(true);
		window.clear();
		sf::Sprite sprite(texture.getTexture());
		window.draw(sprite);
		window.display();
	}
	return 0;
}
30.mp4
60.mp4

I've attached test videos I'm using

@wang-bin
Copy link
Owner

player.setFrameRate(fps)

@oomek
Copy link
Author

oomek commented May 15, 2024

Thank you for sharing this undocumented function. This definitely helps in the situation when for example the framerate is 29.97 and the refresh rate of the monitor is 59.97. Unfortunately the hassle with calling renderVideo() at a certain moment persists. I'm still unable to call

texture.setActive(true);
player.renderVideo();
texture.display();
texture.setActive(false);

on each frame, because that is causing a massive judder.

Would it be possible to make renderVideo() call with some parameter that would just tell it internally to return without updating the texture if the new frame is not ready to be scheduled for display?

@oomek
Copy link
Author

oomek commented May 16, 2024

It appears that this juddering is not caused by my implementation.
I tried glfwplay.exe and it's even worse.
Would you be able to investigate and see if it can be fixed please?

Below is a screen capture of a smooth 60fps video posted above played using glfwplay.exe
https://github.com/wang-bin/mdk-sdk/assets/2974860/70ebb07c-4200-4589-9bb6-0a764065d18e

@oomek
Copy link
Author

oomek commented May 17, 2024

So, is this solvable?

@wang-bin
Copy link
Owner

Would it be possible to make renderVideo() call with some parameter that would just tell it internally to return without updating the texture if the new frame is not ready to be scheduled for display?

render callback is invoked when a frame is ready, then call renderVideo() by user

It appears that this juddering is not caused by my implementation.
I tried glfwplay.exe and it's even worse.
Would you be able to investigate and see if it can be fixed please?

depending on video frame rate and display frame rate

@oomek
Copy link
Author

oomek commented May 18, 2024

This judder is weird. It skips frames and repeats at the same time, check the video I posted frame by frame. For a framerate mismatch it should only do one or the other thing, not both. Seems like a timing issue to me.

@oomek
Copy link
Author

oomek commented May 18, 2024

This works almost perfect, a little frame skip once for a while, but no forward and back judder. I think the videoRender callback should be fixed.

#include <SFML/Window.hpp>
#include <SFML/Graphics.hpp>
#include "mdk/Player.h"
using namespace MDK_NS;
using namespace std;

#include <iostream>
#include <cmath>

int main(int argc, char** argv)
{
	Player player;

	player.setMedia(argv[argc-1]);
	player.prepare();

	for(;;)
	{
		if (player.mediaStatus() > MediaStatus::Buffering) break;
		if (player.mediaStatus() == MediaStatus::Invalid) return 0;
	}

	auto video_info = player.mediaInfo().video[0];
	player.setVideoSurfaceSize(video_info.codec.width * video_info.codec.par, video_info.codec.height);

	float fps = video_info.codec.frame_rate;

	sf::RenderWindow window(sf::VideoMode(640, 480), "Test");
	window.setVerticalSyncEnabled(true);

	sf::RenderTexture texture;

	if (!texture.create(video_info.codec.width * video_info.codec.par, video_info.codec.height))
		return -1;

	player.set(State::Playing);
	// player.setFrameRate(29);

	sf::Clock clock;
	sf::Time timeSinceLastFrame = sf::Time::Zero;
	sf::Time timePerFrame = sf::seconds(1.0 / fps);

	float framerate = 60.0f;

	sf::Font font;
	if (!font.loadFromFile("RobotoMono-Regular.ttf")){cout << "Font not found." << endl;}

	sf::Text fps_text;
	fps_text.setFont(font);
	fps_text.setString(to_string(fps));
	fps_text.setCharacterSize(24);

	sf::Text framerate_text;
	framerate_text.setFont(font);
	framerate_text.setString(to_string(framerate));
	framerate_text.setCharacterSize(24);
	framerate_text.setPosition(0, floor(fps_text.getLocalBounds().height * 1.25));

	sf::Clock framerate_clock;
	sf::Time next_timestamp = sf::seconds(0);

	sf::Clock video_timer;

	while (window.isOpen())
	{
		sf::Event event;
		while (window.pollEvent(event))
		{
			if (event.type == sf::Event::Closed)
				window.close();
			if (event.type == sf::Event::KeyPressed)
			{
				if (event.key.code == sf::Keyboard::Escape)
				{
					player.setVideoSurfaceSize(-1, -1);
					return 0;
				}
			}
		}

		if ( next_timestamp < video_timer.getElapsedTime() - timePerFrame )
		{
			texture.setActive(true);
			sf::Time last_timestamp = sf::seconds(player.renderVideo());
			next_timestamp += timePerFrame;
			texture.display();
			texture.setActive(false);
		}

		window.setActive(true);
		window.clear();
		sf::Sprite sprite(texture.getTexture());
		window.draw(sprite);
		window.draw(fps_text);
		window.draw(framerate_text);
		window.display();

		sf::Time elapsedTime = framerate_clock.restart();

		if(elapsedTime.asMilliseconds() > 1.0f)
			framerate = framerate * 0.99f + 0.01f / elapsedTime.asSeconds();
		framerate_text.setString(to_string(framerate));
	}
	return 0;
}

@oomek
Copy link
Author

oomek commented May 18, 2024

I was trying to add
if (next_timestamp < last_timestamp + timePerFrame) next_timestamp = last_timestamp + timePerFrame; after renderVideo(), but that caused a framerate drop to 1/4 of the actual video framerate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants