Using ffmpeg for Audio-Video Synchronization Playback in Javacv

Recently, I have synchronized the playback of captured audio frames and video frames using the FFmpegFrameGrabber frame grabber from the javaCV ffmpeg package. The synchronization method used is video-to-audio synchronization.

Program and source code

The specific approach is as follows:

)1Firstly, let's introduce how ffmpeg captures the images and sound from video files

FFmpegFrameGrabber fg = new FFmpegFrameGrabber("a video file path or a url"); 

After obtaining the frame grabber object, calling its grab() method will return the captured Frame object. This Frame can be a video frame or an audio frame, as the audio and video frames are arranged in the playback time sequence according to the timestamp. Of course, the captured frames are already decoded and stored in the java.nio.Buffer object. For video frames, the Buffer stores image pixel data such as RGB, and then through

BufferedImage bi = (new Java2DFrameConverter()).getBufferedImage(f); 

You can obtain an image, which can be processed or displayed directly in a Swing component without processing. For audio frames, the Buffer stores the PCM data of the audio, which can be of type float or short. Then, using the write method from java.sounds.sample's sourceDataLine, these audio PCM data can be written to the speaker.

)2)Next, let's introduce how to continuously play the obtained frames. First, let's play the video alone:

while(true) 
{ 
  Frame f = fg.grab();  
  if(f.image!=null) 
  label.setIcon(new ImageIcon((new Java2DFrameConverter()).getBufferedImage(f))); 
  Thread.sleep(1000/video frame rate);  
} 

Similarly, to play audio alone, the data is written to the sound card. Example

)3)Producer-consumer pattern.

The figure above shows the method of program implementation. It uses the producer-consumer pattern to judge the captured frames, and if it is a video frame, it is produced to the video FIFO, and if it is an audio frame, it is produced to the audio FIFO. Then, the audio playback thread and the video playback thread consume the frames from their respective frame repositories. The reason for using the producer-consumer pattern is that the speed of frame capture is greater than the consumption of frames, so we prioritize capturing frames to buffer, or further preprocess the captured frames. The video and audio playback threads only need to play and display the processed frames directly.

)4)A method to achieve audio-video synchronization: play all video frames in the two audio frames.

To achieve audio-video synchronization, frame timestamps must be available. Here, the captured frames only have playback timestamps PTS, not decoding timestamps DTS, so we only need to decide on playback based on the playback timestamps.

The implementation of the program is based on the above figure, when the audio thread starts to play the audio frame A1when, the setRun method of the video thread is called, and the current audio frame timestamp curTime and the next audio frame A are passed2timestamp nextTime is given to the video thread in the wait state, and then the video thread starts, beginning to take out video frame G from the video FIFO1, and then calculate G1and A1time difference is used as the playback delay, Thread.sleep(t1After that, the video thread displays the image on the swing component, such as JLabel.setIcon(image). Then the video thread takes out another frame of image G2, compare G2timestamp and A2timestamp, if G2timestamp is less than A2, then the video thread continues to delay t2After that, play this G2image, and then G3Similarly, until G is obtained4, and A2comparison finds G4timestamp is greater than A2, then the video thread enters the wait state, waiting for the next startup. Then the audio thread plays A1After the audio frame A is played, the audio frame A is taken out from the warehouse3, and then A is played2timestamp and A3timestamp is passed to the video thread, and then playback of A starts.2Then, the video thread is continued to play similarly.

)5)Dynamic adjustment of delay time

Since personal PCs are not real-time operating systems, that is, Thread.sleep is not accurate and is constrained by the sound card's playback of sound, so the basic implementation idea mentioned above needs to be improved. Firstly, the sourceDataLine method in Java is to take out audio thread data from the internal buffer at a certain speed, if the audio data written is taken out, then the audio playback will be stuck, but if too much audio data is written at a time, the audio and video may not be synchronized, so we need to ensure that the internal buffer of sourceDataLine is left with some data, otherwise it will cause the sound to be stuck, but the data volume cannot be too much, so we in G3to A2This period of time for sound playback adjustment, due to the inaccuracy of the delay, the A1Frame data may not be full t6It may be taken up by the sound card, so the data in G3After the image, the sound thread will judge according to the data volume returned by sourceDataLine.available(), if the data volume is about to run out, reduce G3to A2Delay time t4. This way, we can ensure that the data volume will not become 0 and cause the sound to be stuck.

)6)The following is the program in window64The following is the test and ubuntu14The following is the test result image: The playback is smooth, and synchronization is also good, but the playback will be stuck when coding in an IDE like IDEA, because IDEA is also developed in Java, so the running of IDEA will affect other Java programs, but other processes will not affect.

That's all for this article. I hope it will be helpful to everyone's learning, and I also hope everyone will support the Nahan Tutorial more.

Declaration: The content of this article is from the network, the copyright belongs to the original author. The content is contributed and uploaded by Internet users spontaneously. This website does not own the copyright, does not undergo manual editing, and does not assume relevant legal liabilities. If you find any content suspected of copyright infringement, please send an email to: notice#oldtoolbag.com (when sending an email, please replace # with @ for reporting, and provide relevant evidence. Once verified, this site will immediately delete the infringing content.)