Posted by: yegorich | April 9, 2011

Get total rsync progress using Python

Doing large backups with rsync can take really long time. So it is very important to know the exact progress. rsync provides such an option as –progress to view the progress of each file:
 
1238099 100%  146.38kB/s    0:00:08  (xfer#5, to-check=169/396)

The last number shows the whole number of files to proceed. This number is true for small amount of files. But if you have to transfer a big amount, rsync will manage them in chunks, so the last number will be increased each time rsync takes another chunk of files.

My solution was just to make a dry-run and extract the real total number of files and then calculate the whole progress using it.

import subprocess
import re
import sys

print('Dry run:')
cmd = 'rsync -az --stats --dry-run ' + sys.argv[1] + ' ' + sys.argv[2]
proc = subprocess.Popen(cmd,
                                   shell=True,
                                   stdin=subprocess.PIPE,
                                   stdout=subprocess.PIPE,
                                   )
remainder = proc.communicate()[0]
mn = re.findall(r'Number of files: (\d+)', remainder)
total_files = int(mn[0])
print('Number of files: ' + str(total_files))

print('Real rsync:')
cmd = 'rsync -avz  --progress ' + sys.argv[1] + ' ' + sys.argv[2]
proc = subprocess.Popen(cmd,
                                   shell=True,
                                   stdin=subprocess.PIPE,
                                   stdout=subprocess.PIPE,
)
while True:
             output = proc.stdout.readline()
if 'to-check' in output:
             m = re.findall(r'to-check=(\d+)/(\d+)', output)
             progress = (100 * (int(m[0][1]) - int(m[0][0]))) / total_files
             sys.stdout.write('\rDone: ' + str(progress) + '%')
             sys.stdout.flush()
             if int(m[0][0]) == 0:
                      break

print('\rFinished')
Advertisements
Posted by: deepcani | June 2, 2010

Calling PsExec from Ant Script

Part of the work I’m doing has to do with launching Amazon instances on EC2. Once the instance is running and is ready to be used we install various software on it remotely, via ant script. And, like many other folks out there we rely on PsExec, a tool formerly from Sysinternals (and now from Microsoft).

If you google for ‘psexec from ant’ you’ll find quite a few unresolved issues and disappointed forum posts. People complain that psExec hangs, and that they cannot see the output of the tool when it is called from ant… Below I describe a workaround that worked for me. The workaround wraps PsExec into a batch file, and starts the tool in a separate console window. This way the tool is launched from a console window environment, and not really from ant.

Sample batch file (runPsExec.bat):

start "" /wait %pstools.home%/psexec %*
exit %ERRORLEVEL%

And here is how to use it in ant:

<exec executable="cmd.exe" failonerror="true" >
  <arg value="/C"/>
  <arg value="${basedir}/runPsExec.bat"/>
  <arg line="\\remoteHost -u xxxx -p xxxx ipconfig /all"/>
</exec>

The second line in runPsExec.bat script is needed, because otherwise the error code is not propagated from the shell to your ant script.

Posted by: yegorich | May 17, 2010

VIM as IDE

I’m using VIM for some years. It is a great editor, but till now I only used some small subset of its advanced features like splitting windows, using ctags, vimdiff etc. But after having some problems concerning VIM usage via putty, I decided to learn some more features. Below you’ll find a list of useful plugins I found during my search:

taglist

This is very useful plugin showing you an overview of a current source file, delivering macro, variables and function definitions in a special window.

TTrCodeAssistor

This plugin enables auto-completion of function prototypes.

OmniCppComplete

This plugin enables auto-completion for structures/classes.

Note: all these plugins rely on ctags, I use following alias:

alias ctags='ctags -R --exclude=.pc --exclude=patches --exclude=.git --c++-kinds=+p --fields=+iaS --extra=+q'

the –exclude statements are very useful, if using quilt, otherwise you can spring into the backup version of the file instead of original one. Other statements are needed to correctly extract structure/classes information.

Posted by: yegorich | April 13, 2010

Automatically Convert Video Using Inotify

I just needed to convert some YouTube videos from flv into Xvid format to watch them on my old media player. At first I converted each video manually using ffmpeg. It was not very comfortable. Using WinFF made the stuff easier nevertheless it still required manual intervention. So after some searching I came across an article describing Inotify kernel feature. So here is my script:


#!/bin/sh

SRCDIR=/home/user/Videos/downloaded
DESTDIR=/home/user/Videos/converted/

inotifywait -mq --format="%w%f" -e close_write $SRCDIR | while read file; do
       ffmpeg -i "$file" -vcodec libxvid -aspect 4:3 -acodec libmp3lame -ab  192k $DESTDIR"`basename "$file" .flv`".avi
       rm "$file"
done

Now each downloaded video will be automatically converted and moved to the right folder. The script can be started on startup to make the process fully automatically.

Dependencies: inotify-tools

« Newer Posts

Categories