Fixes issue #2. Simplifies code. Improves chunk handling. Changes default to x265 for greater efficiency.
This commit is contained in:
Graeme Humphries
parent
6729e95ea1
commit
1fdbba0fed
79
README.md
79
README.md
@ -21,27 +21,25 @@ to encode with:
|
||||
|
||||
dve -l host1,host2,host3 media/test.mp4
|
||||
|
||||
If you're using a statically linked ffmpeg binary (recommended), then you'll also
|
||||
want to specify the path to that binary:
|
||||
|
||||
dve -e ~/bin/ffmpeg -l host1,host2,host3 media/test.mp4
|
||||
|
||||
After the encoding is completed and the chunks stitched back together, you
|
||||
should end up with an output file named "test.mp4_new.mkv" in your current working
|
||||
directory. You can adjust output naming, but note that the output container format
|
||||
will currently always be mkv:
|
||||
should end up with an output file named something like "original_new.mkv" in
|
||||
your current working directory. You can adjust output naming, but note that the
|
||||
output container format will currently always be mkv:
|
||||
|
||||
dve -s .encoded.mkv -e ~/bin/ffmpeg -l host1,host2,host3 media/test.mp4
|
||||
|
||||
dve automatically copies the local ffmpeg to remote systems for encoding, but this
|
||||
won't work if you want to mix architectures / OSes. If you ensure
|
||||
there's a valid ffmpeg install in the $PATH on each remote system, you can
|
||||
disable copying over the local encoder:
|
||||
Encoding currently breaks input videos into 1m (60s) chunks. This should give
|
||||
reasonable parallelism across a reasonable number of hosts. If you have
|
||||
many hosts you may need to adjust this down using -t. If you have a small number
|
||||
of hosts and a long video, you may wish to bump this up to encode larger chunks
|
||||
and get marginally better compression. Values larger than 300 (15m) are
|
||||
probably a waste of time.
|
||||
|
||||
dve -d -l win1,win2,lin1,lin2 media/test.mp4
|
||||
|
||||
This allows you to mix Windows/cygwin and Linux hosts on the same
|
||||
encoding job.
|
||||
Since the ffmpeg situation in Ubuntu has been resolved, dve no longer
|
||||
tries to copy over your local copy of ffmpeg for encoding, which greatly
|
||||
simplifies the script logic. This means you need to have an ffmpeg binary on
|
||||
every system used for encoding, and if you specify a custom path, that custom
|
||||
path should be the same on every system.
|
||||
|
||||
## Benchmarks
|
||||
|
||||
@ -115,30 +113,12 @@ setup key based authentication to all your remote hosts.
|
||||
|
||||
The following need to be installed on the host running this script:
|
||||
|
||||
- [ffmpeg](http://johnvansickle.com/ffmpeg/)
|
||||
- [ffmpeg](https://www.ffmpeg.org/download.html)
|
||||
- [GNU parallel](https://www.gnu.org/software/parallel/)
|
||||
|
||||
It will automatically copy the encoder binary to the target systems and run
|
||||
it there, but you will need to make sure any required shared libraries are installed
|
||||
on the target system.
|
||||
|
||||
It's recommended that you use the latest ffmpeg statically linked binaries (above),
|
||||
which will also improve your chances of transcoding new / strange video formats.
|
||||
|
||||
### bash
|
||||
|
||||
If you're using the statically linked ffmpeg binaries in ~/bin,
|
||||
you'll need to ensure bash adds ~/bin to your $PATH for non-interactive shells.
|
||||
Including the following at the top of your ~/.bashrc should be sufficient:
|
||||
|
||||
```bash
|
||||
# User dependent .bashrc file
|
||||
|
||||
# Set PATH so it includes user's private bin if it exists
|
||||
if [ -d "${HOME}/bin" ] ; then
|
||||
PATH="${HOME}/bin:${PATH}"
|
||||
fi
|
||||
```
|
||||
It's recommended that you use recent (>= 2.5.x) versions of ffmpeg to ensure
|
||||
they have all the required functionality for splitting and combining the
|
||||
video chunks.
|
||||
|
||||
### Windows
|
||||
|
||||
@ -161,29 +141,10 @@ You'll also need to do the following if you want to use the host to render with:
|
||||
|
||||
## ⚠ Known Issues
|
||||
|
||||
### chunk sizing
|
||||
|
||||
Sometimes the last chunk to be split out will be too small to be properly encoded,
|
||||
and this will cause the whole encode to fail when the pieces are joined at the end
|
||||
of the job.
|
||||
|
||||
Currently the only solution is to adjust the -m and -c options so that the final
|
||||
chunk of the file is reasonably large.
|
||||
|
||||
### ffmpeg / libav
|
||||
Ubuntu ships the libav fork version ffmpeg wrapper instead of the actual ffmpeg
|
||||
release. This version is missing many necessary features, including the
|
||||
[concat demuxer](https://www.ffmpeg.org/faq.html#How-can-I-concatenate-video-files_003f)
|
||||
used to stitch encoded chunks back together.
|
||||
|
||||
For now, you should download and use the statically linked ffmpeg
|
||||
binaries listed above.
|
||||
|
||||
For more background about this fork, see:
|
||||
[The FFmpeg/Libav situation](http://blog.pkh.me/p/13-the-ffmpeg-libav-situation.html)
|
||||
See the [GitHub issues page](https://github.com/nergdron/dve/issues)
|
||||
|
||||
## License
|
||||
dve is copyright 2013 by Graeme Humphries <graeme@sudo.ca>.
|
||||
dve is copyright 2013-2015 by Graeme Nordgren <graeme@sudo.ca>.
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
|
||||
80
dve
80
dve
@ -2,14 +2,12 @@
|
||||
|
||||
ENC="ffmpeg"
|
||||
ENCPATH=""
|
||||
OPTS="-c:v libx264 -crf 20.0 -preset veryslow -c:a libvorbis -aq 5 -c:s copy"
|
||||
OPTS="-c:v libx265 -preset fast -x265-params crf=24 -c:a libvorbis -aq 5 -c:s copy"
|
||||
SUFFIX="_new.mkv"
|
||||
SERVERS="localhost"
|
||||
CHUNKS=100
|
||||
MINLEN=240
|
||||
LEN=60
|
||||
OUTDIR="$HOME/.dve"
|
||||
VERBOSE="error"
|
||||
COPYENC=true
|
||||
|
||||
function usage() {
|
||||
cat << EOF
|
||||
@ -20,11 +18,9 @@ multiple hosts.
|
||||
|
||||
OPTIONS:
|
||||
-h this help message.
|
||||
-c number of chunks to split input file into. (default=${CHUNKS})
|
||||
-d do not copy encoder, assume it already exists remotely.
|
||||
-e ffmpeg binary to use. (default=${ENC})
|
||||
-l comma separated list of hosts to use to encode. (default=${SERVERS})
|
||||
-m min length of individual video chunks, in seconds. (default=${MINLEN})
|
||||
-t rough length of individual video chunks, in seconds. (default=${MINLEN})
|
||||
-o encoding options. (default=${OPTS})
|
||||
-s output file suffix. (default=${SUFFIX})
|
||||
-v verbose job output. (default=false)
|
||||
@ -33,7 +29,7 @@ EOF
|
||||
|
||||
# check all required helper utils
|
||||
function checkpaths() {
|
||||
for cmd in ffprobe parallel grep sed wc; do
|
||||
for cmd in parallel; do
|
||||
if ! CMD=`which $cmd`; then
|
||||
echo "$cmd not found."
|
||||
exit 1
|
||||
@ -52,50 +48,22 @@ function checkpaths() {
|
||||
fi
|
||||
}
|
||||
|
||||
# returns total video length and optimal chunk length in seconds
|
||||
function chunksize() {
|
||||
if ! DURATION=`ffprobe "$1" 2>&1 | grep Duration | grep -P -o "[0-9]+:[0-9]+:[0-9]+\.[0-9]+" | sed -r 's/\..+//g'`; then
|
||||
echo "Couldn't find duration of \"$1\" using avprobe."
|
||||
exit 1
|
||||
fi
|
||||
declare -a DUR
|
||||
DUR=(`echo ${DURATION} | sed -r 's/:/ /g'`)
|
||||
|
||||
# strip leading zeros if present because bash can't handle it.
|
||||
H=`echo ${DUR[0]} | sed 's/^0*//'`
|
||||
M=`echo ${DUR[1]} | sed 's/^0*//'`
|
||||
S=`echo ${DUR[2]} | sed 's/^0*//'`
|
||||
# add extra second to length to account for subsecond variances
|
||||
SECONDS=$((H*3600+M*60+S+1))
|
||||
LEN=$((SECONDS / CHUNKS))
|
||||
if [ $LEN -lt $MINLEN ]; then
|
||||
LEN=$MINLEN
|
||||
fi
|
||||
echo ${SECONDS} ${LEN}
|
||||
}
|
||||
|
||||
checkpaths
|
||||
|
||||
while getopts “hc:de:l:m:o:s:v” OPTION; do
|
||||
while getopts “he:l:t:o:s:v” OPTION; do
|
||||
case $OPTION in
|
||||
h)
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
c)
|
||||
CHUNKS="$OPTARG"
|
||||
;;
|
||||
d)
|
||||
COPYENC=false
|
||||
;;
|
||||
e)
|
||||
ENC="$OPTARG"
|
||||
;;
|
||||
l)
|
||||
SERVERS="$OPTARG"
|
||||
;;
|
||||
m)
|
||||
MINLEN="$OPTARG"
|
||||
t)
|
||||
LEN="$OPTARG"
|
||||
;;
|
||||
o)
|
||||
OPTS="$OPTARG"
|
||||
@ -124,47 +92,29 @@ if ! mkdir -p ${OUTDIR}; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
declare -a TIMING
|
||||
TIMING=( `chunksize "$1"` )
|
||||
|
||||
START=0
|
||||
CHUNKFILE="${OUTDIR}/chunkfile.txt"
|
||||
rm "${CHUNKFILE}" 2> /dev/null
|
||||
while [ $START -lt ${TIMING[0]} ]; do
|
||||
printf "%06d\n" ${START} >> "${CHUNKFILE}"
|
||||
START=$((START+TIMING[1]))
|
||||
done
|
||||
|
||||
NUMSERVERS=`echo ${SERVERS} | sed -r 's/,/ /g' | wc -w`
|
||||
|
||||
echo "Creating chunks to encode"
|
||||
cat "$CHUNKFILE" | parallel --gnu --eta -j 1 $ENC -y -v ${VERBOSE} -fflags +genpts -i \"$1\" -ss {} -t ${TIMING[1]} -c:v copy -c:a copy -c:s copy -f matroska ${OUTDIR}/chunk-{}.orig
|
||||
$ENCPATH -i "$1" -map 0 -codec copy -f segment -segment_time $LEN -segment_format matroska -v ${VERBOSE} "${OUTDIR}/chunk-%03d.orig"
|
||||
|
||||
CWD=`pwd`
|
||||
cd "$OUTDIR"
|
||||
|
||||
echo "Running parallel encoding jobs"
|
||||
PAR_OPTS="--gnu -j 1 -S ${SERVERS} --eta --retries 2 --nice 10"
|
||||
PAR_OPTS="$PAR_OPTS --workdir ... --trc {.}.enc"
|
||||
if $COPYENC; then
|
||||
PAR_OPTS="$PAR_OPTS --basefile ffmpeg"
|
||||
fi
|
||||
#PAR_OPTS="$PAR_OPTS --workdir ... --trc {.}.enc"
|
||||
PAR_OPTS="$PAR_OPTS --workdir ... --transfer --return {.}.enc"
|
||||
ENC_OPTS="-y -v ${VERBOSE} -i {} ${OPTS} -f matroska {.}.enc"
|
||||
|
||||
if $COPYENC; then
|
||||
cp "${ENCPATH}" ./ffmpeg
|
||||
REMOTEENC='${HOME}/ffmpeg'
|
||||
else
|
||||
REMOTEENC="$ENC"
|
||||
fi
|
||||
parallel ${PAR_OPTS} ${REMOTEENC} ${ENC_OPTS} ::: chunk-*.orig
|
||||
echo "parallel ${PAR_OPTS} ${ENCPATH} ${ENC_OPTS} ::: chunk-*.orig"
|
||||
parallel ${PAR_OPTS} ${ENCPATH} ${ENC_OPTS} ::: chunk-*.orig
|
||||
|
||||
echo "Combining chunks into final video file"
|
||||
echo "ffconcat version 1.0" > concat.txt
|
||||
for f in `ls chunk-*.enc | sort`; do
|
||||
echo "file $f" >> concat.txt
|
||||
done
|
||||
${ENC} -y -v ${VERBOSE} -f concat -i concat.txt -f matroska -c copy "${CWD}"/"`basename \"$1\"`"${SUFFIX}
|
||||
BASE=`basename "$1"`
|
||||
OUTFILE="${CWD}"/"${BASE%.*}${SUFFIX}"
|
||||
${ENCPATH} -y -v ${VERBOSE} -f concat -i concat.txt -f matroska -map 0 -c copy "${OUTFILE}"
|
||||
|
||||
echo "Cleaning up temporary working files"
|
||||
rm -f "${OUTDIR}"/*
|
||||
|
||||
Loading…
Reference in New Issue
Block a user