FFmpeg

FFmpeg은 디지털 음성 스트림과 영상 스트림에 대해서 다양한 종류의 형태로 기록하고 변환하는 컴퓨터 프로그램이다. FFmpeg은 명령어를 직접 입력하는 방식으로 동작하며 여러가지 자유 소프트웨어와 오픈 소스 라이브러리로 구성되어 있다. 라이브러리 중에는 libavcodec도 들어있는데, 이 라이브러리는 음성/영상 코덱 라이브러리로 여러 프로젝트에서 쓰이고 있다. 또, libavformat 라는 음성/영상 다중화, 역다중화 라이브러리도 있다. 이 프로젝트의 이름은 MPEG 영상 표준화 그룹에서 유래했고, "mpeg" 앞에 붙은 "FF"는 "fast forward"를 의미한다.

Library 호출 순서.

Library 종속성 관계로 인해, 호출 순서가 컴파일에 영향을 줄 수 있다. 그 순서는 아래와 같다.

libavutil.so
libavcodec.so - encoding/decoding 관련 내용 포함.
libswscale.so
libswresample.so
libavformat.so
~~libpostproc.so~~
libavfilter.so
libavdevice.so

PTS vs DTS

How to synchronize audio and video using ffmpeg libraries?

개념에 대한 내용은 Presentation timestamp#PTS vs DTS 항목 참조.

Syncing decoded video using FFmpeg

Syncing decoded video using Ffmpeg

To convert pts or dts to floating-point seconds, use av_q2d() on proper time_base:

// You got the context from open_input:
AVFormatContext *pFormatCtx;
avformat_open_input(&pFormatCtx, inputfilename, NULL, &format_opts);

// Get a stream from the context
AVStream pStream= pFormatCtx->streams[i];

// Convert packet time (here, dts) to seconds with:  
double seconds= (dts - pStream->start_time) * av_q2d(pStream->time_base);

// Or convert frame number to seconds with the codec context
AVCodecContext *pCodecCtx= pStream->pVideoStream->codec;
double seconds= framenumber * av_q2d(pCodecCtx->time_base);

AV_NOPTS_VALUE

FFmpeg: Timestamp specific

#define AV_NOPTS_VALUE   ((int64_t)UINT64_C(0x8000000000000000))

정의되지 않은 타임스탬프 값입니다.

일반적으로 pts 또는 dts를 제공하지 않는 컨테이너에서 작동하는 디먹서에 의해 보고됩니다.

Time Base

분자(num; Numerator)와 분모(den; Denominator)로 이루어진 AVRational 구조로 되어있다. 시간 측정치의 기준이 된다.

AVStream.time_base: 이것은 프레임 타임 스탬프가 표시되는 기본 시간 단위(초)입니다. (This is the fundamental unit of time (in seconds) in terms of which frame timestamps are represented); 예컨데, duration 초 수 계산 공식은 (AVStream.time_base.num / AVStream.time_base.den) * AVStream.start_time 초가 된다.

주의할 점은 AVFormatContext의 경우 duration 값은 AV_TIME_BASE값을 참조하고 있다. ¹

AV_TIME_BASE는 다음과 같다:

AV_TIME_BASE: #define AV_TIME_BASE 1000000; Internal time base represented as integer

만약 AVFormatContext.duration의 값이 2166833 일 경우, 실제 초 수는 2166833 / 1000000초 가 된다.

여러 구조체들 (e.g. AVCodecContext) 모두 타임 스탬프 값들이 존재하며 각 구조체 마다 time_base 값의 기준이 다르다. 공식문서를 참조하는 것이 좋다.

동영상의 남은 시간 구하기

Stackoverflow - How to get the duration of a video in Python?

위의 TimeBase 를 참조하여 AVFormatContext.duration 의 값을 참조할 수 있다. 또는 아래와 같은 방법이 있다.

opencv 0.0065 sec
ffprobe 0.0998 sec
moviepy 2.8239 sec
PyAV ???

OpenCV Duration

def with_opencv(filename):
    import cv2
    video = cv2.VideoCapture(filename)

    duration = video.get(cv2.CAP_PROP_POS_MSEC)
    frame_count = video.get(cv2.CAP_PROP_FRAME_COUNT)

    return duration, frame_count

FFProbe Duration

def with_ffprobe(filename):
    import subprocess, json

    result = subprocess.check_output(
            f'ffprobe -v quiet -show_streams -select_streams v:0 -of json "{filename}"',
            shell=True).decode()
    fields = json.loads(result)['streams'][0]

    duration = fields['tags']['DURATION']
    fps      = eval(fields['r_frame_rate'])
    return duration, fps

MoviePy Duration

def with_moviepy(filename):
    from moviepy.editor import VideoFileClip
    clip = VideoFileClip(filename)
    duration       = clip.duration
    fps            = clip.fps
    width, height  = clip.size
    return duration, fps, (width, height)

PyAV Duration

def with_pyav(filename):
    import av
    c = av.open(filename)
    return c.duration / 1000000  # `1000000` is `AV_TIME_BASE`

FFmpeg 로그문자열 확인 방법

char error_buffer[256] = {0,};
av_strerror(ret, error_buffer, 256);
printf("STEP #5000-5-result: %d(%s)", ret, error_buffer);

Minimize delay

I found three commands that helped me reduce the delay of live streams. The first command its very basic and straight forward, the second one it's been combined with other options which might work differently on each environment and the last command it is a hacky version that I found in the documentation It was useful at the beginning but now the first option is more stable.

Basic using -fflags nobuffer

This format flag reduces the latency introduced by buffering during initial input streams analysis. This command will reduce noticeable the delay and will not introduce audio glitches.

$ ffplay -fflags nobuffer -rtsp_transport tcp rtsp://<host>:<port>

Advanced -flags low_delay and other options

We can combine the previous -fflags nobuffer format flag with other generic options and advanced options for a more elaborated command:

-flags low_delay this codec generic flag will force low delay.
-framedrop: to drop video frames if video is out of sync. Enabled by default if the master clock is not set to video. Use this option to enable frame dropping for all master clock sources
-strict experimental, finally -strict specifies how strictly to follow the standards and the experimental option allows non standardized experimental things, experimental (unfinished/work in progress/not well tested) decoders and encoders. This option is optional and remember that experimental decoders can pose a security risk, do not use this for decoding untrusted input.

$ ffplay -fflags nobuffer -flags low_delay -framedrop -strict experimental -rtsp_transport tcp rtsp://<host>:<port>

This command might introduce some audio glitches, but rarely.

Also you can try adding: * -avioflags direct to reduce buffering, and * -fflags discardcorrupt to discard corrupted packets, but I think is very aggressive approach. This might break the audio-video synchronization

$ ffplay -fflags nobuffer -fflags discardcorrupt -flags low_delay -framedrop -avioflags direct -rtsp_transport tcp rtsp://<host>:<port>

A hacky option (found on the old documentation)

This is an debugging solution based on setting -probesize and -analyzeduration to low values to help your stream start up more quickly.

-probesize 32 sets the probing size in bytes (i.e. the size of the data to analyze to get stream information). A higher value will enable detecting more information in case it is dispersed into the stream, but will increase latency. Must be an integer not lesser than 32. It is 5000000 by default.
analyzeduration 0 specifies how many microseconds are analyzed to probe the input. A higher value will enable detecting more accurate information, but will increase latency. It defaults to 5000000 microseconds (5 seconds).
-sync ext sets the master clock to an external source to try and stay realtime. Default is audio. The master clock is used to control audio-video synchronization. This means this options sets the audio-video synchronization to a type (i.e. type=audio/video/ext).

$ ffplay -probesize 32 -analyzeduration 0 -sync ext -rtsp_transport tcp rtsp://<host>:<port>

This command might introduce some audio glitches sometimes.

The -rtsp_transport can be setup as udp or tcp according to your streaming. For this example I'm using tcp.

C Code

Try set flags of AVFormatContext to AVFMT_FLAG_NOBUFFER | AVFMT_FLAG_FLUSH_PACKETS

AVFormatContext *ctx;
// ...
ctx->flags = AVFMT_FLAG_NOBUFFER | AVFMT_FLAG_FLUSH_PACKETS;

Then try to set decoder thread to 1. It seems like more thread will cause more latency.

AVCodecContext *ctx;
// ...
ctx->thread_count = 1;

FFmpeg Deprecated Functions and Symbols

https://github.com/chelyaev/ffmpeg-tutorial

This section describes the changes made to work around deprecated functions and symbols. In some cases, a simple rename sufficed (e.g. dump_format), but in others, more significant changes to the code were required (e.g. avcodec_decode_audio2). Consult the diffs for each respective tutorial to see exactly what has changed since the original version of the tutorial.

before	after
`av_open_input_file`	`avformat_open_input`
`av_find_stream_info`	`avformat_find_stream_info`
`dump_format`	`av_dump_format`
`CODEC_TYPE_VIDEO`	`AVMEDIA_TYPE_VIDEO`
`avcodec_open`	`avcodec_open2`
`avcodec_decode_video`	`avcodec_decode_video2`
`img_convert`	`sws_scale`
`av_close_input_file`	`avformat_close_input`
`avcodec_decode_audio2`	`avcodec_decode_audio4`
`CODEC_TYPE_AUDIO`	`AVMEDIA_TYPE_AUDIO`
`url_set_interrupt_cb`	`avio_open2`
`url_ferror`	`check attribute is->pFormatCtx->pb->error`
`pstrcpy`	`av_strlcpy`

Troubleshooting

FFmpeg사용중 발생할 수 있는 문제점 해결 방법에 대하여 정리한다.

AVPacket의 PTS Sync 문제

ffmpeg에서 영상 PTS를 구하는 방법

AVPacket의 PTS를 사용할 경우 영상의 Sync가 미묘하게 틀어질 수 있다. 이 경우 PTS는 av_frame_get_best_effort_timestamp(frame)를 사용하는 것이 좋다.

RTP: missed xxx packets

RTSP over SSL

Invalid data found when processing input

바로 다음, #Nonmatching transport in server reply 항목 참조.

스트리밍중 이라면 송출 측 에서도 -rtsp_transport tcp 를 추가하자.

송출 측:

ffmpeg -i video.mp4 -f rtsp -rtsp_transport tcp rtsp://localhost:8554/demo

수신 측:

ffmpeg -rtsp_transport tcp -i rtsp://localhost:8554/demo ...

Nonmatching transport in server reply

(FFmpeg) RSTP 영상 저장하기 — Enough is not enough

RTSP 스트리밍시 다음과 같은 에러 발생:

[rtsp @ 0x556150991900] Nonmatching transport in server reply
rtsp://address~: Invalid data found when processing input

다음과 같이 해결:

ffmpeg -rtsp_transport tcp -i "rtsp://address~" "001.mp4"

Frame rate very high for a muxer not efficiently supporting it

(FFmpeg) RSTP 영상 저장하기 — Enough is not enough

[mp4 @ 0x562101de46a0] Frame rate very high for a muxer not efficiently supporting it.
Please consider specifying a lower framerate, a different muxer or -vsync 2

메시지와 같이 -vsync 2옵션을 추가하면 된다:

ffmpeg -rtsp_transport tcp -i "rtsp://address~" -vsync 2 "001.mp4"

Local Download

FFmpeg 3.0.1 "Einstein": 3.0.1 was released on 2016-03-29. It is the latest stable FFmpeg release from the 3.0 release branch, which was cut from master on 2016-02-14.; Ffmpeg-3.0.1.tar.gz

FFmpeg 2.8.6 "Feynman": 2.8.6 was released on 2016-02-01. It is the latest stable FFmpeg release from the 2.8 release branch, which was cut from master on 2015-09-05. Amongst lots of other changes, it includes all changes from ffmpeg-mt, libav master of 2015-08-28, libav 11 as of 2015-08-28.; Ffmpeg-2.8.6.tar.gz

FFmpeg 2.8.5: Ffmpeg-2.8.5.tar.bz2

Favorite site

FFmpeg Wiki

How to use library

Commandline help

FFmpeg 몇가지 옵션

How to compile

Article

FFMpeg, 손으로 작성한 AVX-512 어셈블리 코드로 94x 성능 향상 | GeekNews - 논쟁의 요소가 있음.

Guide

Tutorials

Project

Decoding

References

원문: Duration of the stream, in AV_TIME_BASE fractional seconds ↩
Hello_world_»_FFmpeg_iOS_Video_Player.pdf ↩
How_to_Write_a_Video_Player_in_Less_Than_1000_Lines.pdf ↩
How_to_Write_a_Video_Player_in_Less_Than_1000_Lines-example.zip ↩
Demonstration_of_SDL2+FFmpeg_player_based_on.pdf ↩
FFmpeg-Mingw-compile.pdf ↩
Ffmpeg-pts-sync-error.pdf ↩
Ffmpeg_open_media_stream.pdf ↩
Ffmpeg_play.pdf ↩
Ffmpeg_transcoding.pdf ↩
Save_avframe_to_jpeg_file.pdf ↩
FFMPEG_-_H264_Decoding.pdf ↩