Fixes issue #2. Simplifies code. Improves chunk handling. Changes default to x265 for greater efficiency.

README.md

@@ -2,7 +2,7 @@
 
 This is a small script to do distributed, high quality video encoding.
 
-The script: 
+The script:
 
 - breaks input video into chunks.
 - distributes chunks to different servers via SSH.
@@ -21,27 +21,25 @@ to encode with:
 
     dve -l host1,host2,host3 media/test.mp4
 
-If you're using a statically linked ffmpeg binary (recommended), then you'll also
-want to specify the path to that binary:
-
-    dve -e ~/bin/ffmpeg -l host1,host2,host3 media/test.mp4
-
 After the encoding is completed and the chunks stitched back together, you
-should end up with an output file named "test.mp4_new.mkv" in your current working
+should end up with an output file named something like "original_new.mkv" in
-directory. You can adjust output naming, but note that the output container format
+your current working directory. You can adjust output naming, but note that the
-will currently always be mkv:
+output container format will currently always be mkv:
 
     dve -s .encoded.mkv -e ~/bin/ffmpeg -l host1,host2,host3 media/test.mp4
 
-dve automatically copies the local ffmpeg to remote systems for encoding, but this
+Encoding currently breaks input videos into 1m (60s) chunks. This should give
-won't work if you want to mix architectures / OSes. If you ensure
+reasonable parallelism across a reasonable number of hosts. If you have
-there's a valid ffmpeg install in the $PATH on each remote system, you can
+many hosts you may need to adjust this down using -t. If you have a small number
-disable copying over the local encoder:
+of hosts and a long video, you may wish to bump this up to encode larger chunks
+and get marginally better compression. Values larger than 300 (15m) are
+probably a waste of time.
 
-    dve -d -l win1,win2,lin1,lin2 media/test.mp4
+Since the ffmpeg situation in Ubuntu has been resolved, dve no longer
-
+tries to copy over your local copy of ffmpeg for encoding, which greatly
-This allows you to mix Windows/cygwin and Linux hosts on the same
+simplifies the script logic. This means you need to have an ffmpeg binary on
-encoding job.
+every system used for encoding, and if you specify a custom path, that custom
+path should be the same on every system.
 
 ## Benchmarks
 
@@ -53,7 +51,7 @@ clip, 3:47 in length.
 
 ```bash
 $ time nice -n 10 ./ffmpeg -y -v error -stats -i test.mp4 -c:v libx264 -crf 20.0 -preset medium -c:a libvorbis -aq 5 -f matroska test.mkv
-frame= 5459 fps=7.4 q=-1.0 Lsize=  530036kB time=00:03:47.43 bitrate=19091.2kbits/s    
+frame= 5459 fps=7.4 q=-1.0 Lsize=  530036kB time=00:03:47.43 bitrate=19091.2kbits/s
 real    12m17.177s
 user    182m57.340s
 sys     0m36.240s
@@ -115,30 +113,12 @@ setup key based authentication to all your remote hosts.
 
 The following need to be installed on the host running this script:
 
-- [ffmpeg](http://johnvansickle.com/ffmpeg/)
+- [ffmpeg](https://www.ffmpeg.org/download.html)
 - [GNU parallel](https://www.gnu.org/software/parallel/)
 
-It will automatically copy the encoder binary to the target systems and run
+It's recommended that you use recent (>= 2.5.x) versions of ffmpeg to ensure
-it there, but you will need to make sure any required shared libraries are installed
+they have all the required functionality for splitting and combining the
-on the target system.
+video chunks.
-
-It's recommended that you use the latest ffmpeg statically linked binaries (above),
-which will also improve your chances of transcoding new / strange video formats.
-
-### bash
-
-If you're using the statically linked ffmpeg binaries in ~/bin,
-you'll need to ensure bash adds ~/bin to your $PATH for non-interactive shells.
-Including the following at the top of your ~/.bashrc should be sufficient:
-
-```bash
-# User dependent .bashrc file
-
-# Set PATH so it includes user's private bin if it exists
-if [ -d "${HOME}/bin" ] ; then
-  PATH="${HOME}/bin:${PATH}"
-fi
-```
 
 ### Windows
 
@@ -161,29 +141,10 @@ You'll also need to do the following if you want to use the host to render with:
 
 ## ⚠ Known Issues
 
-### chunk sizing
+See the [GitHub issues page](https://github.com/nergdron/dve/issues)
-
-Sometimes the last chunk to be split out will be too small to be properly encoded,
-and this will cause the whole encode to fail when the pieces are joined at the end
-of the job.
-
-Currently the only solution is to adjust the -m and -c options so that the final
-chunk of the file is reasonably large.
-
-### ffmpeg / libav
-Ubuntu ships the libav fork version ffmpeg wrapper instead of the actual ffmpeg
-release. This version is missing many necessary features, including the
-[concat demuxer](https://www.ffmpeg.org/faq.html#How-can-I-concatenate-video-files_003f)
-used to stitch encoded chunks back together.
-
-For now, you should download and use the statically linked ffmpeg
-binaries listed above.
-
-For more background about this fork, see:
-[The FFmpeg/Libav situation](http://blog.pkh.me/p/13-the-ffmpeg-libav-situation.html)
 
 ## License
-dve is copyright 2013 by Graeme Humphries <graeme@sudo.ca>.
+dve is copyright 2013-2015 by Graeme Nordgren <graeme@sudo.ca>.
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by

dve

@@ -2,14 +2,12 @@
 
 ENC="ffmpeg"
 ENCPATH=""
-OPTS="-c:v libx264 -crf 20.0 -preset veryslow -c:a libvorbis -aq 5 -c:s copy"
+OPTS="-c:v libx265 -preset fast -x265-params crf=24 -c:a libvorbis -aq 5 -c:s copy"
 SUFFIX="_new.mkv"
 SERVERS="localhost"
-CHUNKS=100
+LEN=60
-MINLEN=240
 OUTDIR="$HOME/.dve"
 VERBOSE="error"
-COPYENC=true
 
 function usage() {
     cat << EOF
@@ -20,11 +18,9 @@ multiple hosts.
 
 OPTIONS:
     -h  this help message.
-    -c  number of chunks to split input file into. (default=${CHUNKS})
-    -d  do not copy encoder, assume it already exists remotely.
     -e  ffmpeg binary to use. (default=${ENC})
     -l  comma separated list of hosts to use to encode. (default=${SERVERS})
-    -m  min length of individual video chunks, in seconds. (default=${MINLEN})
+    -t  rough length of individual video chunks, in seconds. (default=${MINLEN})
     -o  encoding options. (default=${OPTS})
     -s  output file suffix. (default=${SUFFIX})
     -v  verbose job output. (default=false)
@@ -33,7 +29,7 @@ EOF
 
 # check all required helper utils
 function checkpaths() {
-    for cmd in ffprobe parallel grep sed wc; do
+    for cmd in parallel; do
         if ! CMD=`which $cmd`; then
             echo "$cmd not found."
             exit 1
@@ -52,50 +48,22 @@ function checkpaths() {
     fi
 }
 
-# returns total video length and optimal chunk length in seconds
-function chunksize() {
-    if ! DURATION=`ffprobe "$1" 2>&1 | grep Duration | grep -P -o "[0-9]+:[0-9]+:[0-9]+\.[0-9]+" | sed -r 's/\..+//g'`; then
-        echo "Couldn't find duration of \"$1\" using avprobe."
-        exit 1
-    fi
-    declare -a DUR
-    DUR=(`echo ${DURATION} | sed -r 's/:/ /g'`)
-
-    # strip leading zeros if present because bash can't handle it.
-    H=`echo ${DUR[0]} | sed 's/^0*//'`
-    M=`echo ${DUR[1]} | sed 's/^0*//'`
-    S=`echo ${DUR[2]} | sed 's/^0*//'`
-    # add extra second to length to account for subsecond variances
-    SECONDS=$((H*3600+M*60+S+1))
-    LEN=$((SECONDS / CHUNKS))
-    if [ $LEN -lt $MINLEN ]; then
-        LEN=$MINLEN
-    fi
-    echo ${SECONDS} ${LEN}
-}
-
 checkpaths
 
-while getopts “hc:de:l:m:o:s:v” OPTION; do
+while getopts “he:l:t:o:s:v” OPTION; do
     case $OPTION in
     h)
     usage
     exit 1
     ;;
-    c)
-    CHUNKS="$OPTARG"
-    ;;
-    d)
-    COPYENC=false
-    ;;
     e)
     ENC="$OPTARG"
     ;;
     l)
     SERVERS="$OPTARG"
     ;;
-    m)
+    t)
-    MINLEN="$OPTARG"
+    LEN="$OPTARG"
     ;;
     o)
     OPTS="$OPTARG"
@@ -124,47 +92,29 @@ if ! mkdir -p ${OUTDIR}; then
     exit 1
 fi
 
-declare -a TIMING
-TIMING=( `chunksize "$1"` )
-
-START=0
-CHUNKFILE="${OUTDIR}/chunkfile.txt"
-rm "${CHUNKFILE}" 2> /dev/null
-while [ $START -lt ${TIMING[0]} ]; do
-    printf "%06d\n" ${START} >> "${CHUNKFILE}"
-    START=$((START+TIMING[1]))
-done
-
-NUMSERVERS=`echo ${SERVERS} | sed -r 's/,/ /g' | wc -w`
-
 echo "Creating chunks to encode"
-cat "$CHUNKFILE" | parallel --gnu --eta -j 1 $ENC -y -v ${VERBOSE} -fflags +genpts -i \"$1\" -ss {} -t ${TIMING[1]} -c:v copy -c:a copy -c:s copy -f matroska ${OUTDIR}/chunk-{}.orig
+$ENCPATH -i "$1"  -map 0 -codec copy -f segment -segment_time $LEN -segment_format matroska -v ${VERBOSE} "${OUTDIR}/chunk-%03d.orig"
 
 CWD=`pwd`
 cd "$OUTDIR"
 
 echo "Running parallel encoding jobs"
 PAR_OPTS="--gnu -j 1 -S ${SERVERS} --eta --retries 2 --nice 10"
-PAR_OPTS="$PAR_OPTS --workdir ... --trc {.}.enc"
+#PAR_OPTS="$PAR_OPTS --workdir ... --trc {.}.enc"
-if $COPYENC; then
+PAR_OPTS="$PAR_OPTS --workdir ... --transfer --return {.}.enc"
-    PAR_OPTS="$PAR_OPTS --basefile ffmpeg"
-fi
 ENC_OPTS="-y -v ${VERBOSE} -i {} ${OPTS} -f matroska {.}.enc"
 
-if $COPYENC; then
+echo "parallel ${PAR_OPTS} ${ENCPATH} ${ENC_OPTS} ::: chunk-*.orig"
-    cp "${ENCPATH}" ./ffmpeg
+parallel ${PAR_OPTS} ${ENCPATH} ${ENC_OPTS} ::: chunk-*.orig
-    REMOTEENC='${HOME}/ffmpeg'
-else
-    REMOTEENC="$ENC"
-fi
-parallel ${PAR_OPTS} ${REMOTEENC} ${ENC_OPTS} ::: chunk-*.orig
 
 echo "Combining chunks into final video file"
 echo "ffconcat version 1.0" > concat.txt
 for f in `ls chunk-*.enc | sort`; do
     echo "file $f" >> concat.txt
 done
-${ENC} -y -v ${VERBOSE} -f concat -i concat.txt -f matroska -c copy "${CWD}"/"`basename \"$1\"`"${SUFFIX}
+BASE=`basename "$1"`
+OUTFILE="${CWD}"/"${BASE%.*}${SUFFIX}"
+${ENCPATH} -y -v ${VERBOSE} -f concat -i concat.txt -f matroska -map 0 -c copy "${OUTFILE}"
 
 echo "Cleaning up temporary working files"
 rm -f "${OUTDIR}"/*