Friday, September 22, 2017

The python-vlc python module.

The python module for vlc is named python-vlc.
This python module let you to test libvlc API like the VLC video player.
You can install it easy with pip python tool.
C:\Python27\Scripts>pip2.7.exe install python-vlc
Collecting python-vlc
  Downloading python-vlc-1.1.2.tar.gz (201kB)
    100% |################################| 204kB 628kB/s
Installing collected packages: python-vlc
  Running setup.py install for python-vlc ... done
Successfully installed python-vlc-1.1.2
Let's see a simple example with this python module:
import os
import sys
import vlc
import pygame
 
def call_vlc(self, player):
 
    player.get_fps()
    player.get_time()
 
if len( sys.argv )< 2 or len( sys.argv )> 3:
        print 'Help: python vlc_001.py your_video.mp4'
else:
    pygame.init()
    screen = pygame.display.set_mode((800,600),pygame.RESIZABLE)
    pygame.display.get_wm_info()
    pygame.display.get_driver()

 
    # get path to movie specified as command line argument
    movie = os.path.expanduser(sys.argv[1])
    # see if movie is accessible
    if not os.access(movie, os.R_OK):
        print('Error: %s wrong read file: ' % movie)
        sys.exit(1)
 
    # make instane of VLC and create reference to movie.
    vlcInstance = vlc.Instance()
    media = vlcInstance.media_new(movie)
 
    # make new instance of vlc player
    player = vlcInstance.media_player_new()
 
    # start with a callback
    em = player.event_manager()
    em.event_attach(vlc.EventType.MediaPlayerTimeChanged, \
        call_vlc, player)
 
    # set pygame window id to vlc player
    win_id = pygame.display.get_wm_info()['window']
    if sys.platform == "win32": # for Windows
        player.set_hwnd(win_id)
 
    # load movie into vlc player instance
    player.set_media(media)
 
    # quit pygame mixer to allow vlc full access to audio device
    pygame.mixer.quit()
 
    # start movie play
    player.play()
 
    while player.get_state() != vlc.State.Ended:
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                sys.exit(2)
The base of this python script is to make a instance of vlc and put into pygame display.
Another simple example:
C:\Python27>python.exe
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vlc
>>> inst = vlc.Instance()
Warning: option --plugin-path no longer exists.
Warning: option --plugin-path no longer exists.
>>> med = inst.media_new('rain.mp4')
>>> p = med.player_new_from_media()
>>> p.play()
0
>>>

Tuesday, September 19, 2017

The numba python module - part 002 .

Today I tested how fast is jit from numba python and fibonacci math function.
You will see strange output I got for some values.
First example:
import numba
from numba import jit
from timeit import default_timer as timer

def fibonacci(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a
    return a
fibonacci_jit = jit(fibonacci)

start = timer()
fibonacci(100)
duration = timer() - start

startnext = timer()
fibonacci_jit(100)
durationnext = timer() - startnext

print(duration, durationnext)
The result of this run is:
C:\Python27>python numba_test_003.py
(0.00018731270733896962, 0.167499256682878)

C:\Python27>python numba_test_003.py
(1.6357787798437412e-05, 0.1683614083221368)

C:\Python27>python numba_test_003.py
(2.245186560569841e-05, 0.1758382003097716)

C:\Python27>python numba_test_003.py
(2.3093347480146938e-05, 0.16714964906130353)

C:\Python27>python numba_test_003.py
(1.5395564986764625e-05, 0.17471143739730277)

C:\Python27>python numba_test_003.py
(1.5074824049540363e-05, 0.1847134227837042)
As you can see the fibonacci function is not very fast.
The jit - just-in-time compile is very fast.
Let's see if the python source code may slow down.
Let's see the new source code with jit will not work well:
import numba
from numba import jit
from timeit import default_timer as timer

def fibonacci(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a
    return a
fibonacci_jit = jit(fibonacci)

start = timer()
print fibonacci(100)
duration = timer() - start

startnext = timer()
print fibonacci_jit(100)
durationnext = timer() - startnext

print(duration, durationnext)
The result is this:
C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0002334994022992635, 0.17628787910376)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0006886307922204926, 0.17579169287387408)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0008105123483657127, 0.18209553525407973)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.00025466830415606486, 0.17186550306131188)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0007348174871807866, 0.17523103771560608)
The result for value 100 is not the same: 927372692193078999176 and 1445263496.
First problem is:
The problem is that numba can't intuit the type of lookup. If you put a print nb.typeof(lookup) in your method, you'll see that numba is treating it as an object, which is slow.
The second problem is the output but can be from same reason.
I test with value 5 and the result is :
C:\Python27>python numba_test_003.py
13
13
13
13
(0.0007258367409385072, 0.17057997338491704)

C:\Python27>python numba_test_003.py
13
13
(0.00033709872502270044, 0.17213235952108247)

C:\Python27>python numba_test_003.py
13
13
(0.0004836773333341886, 0.17184433415945508)

C:\Python27>python numba_test_003.py
13
13
(0.0006854233828482501, 0.17381272129120037)

Monday, September 18, 2017

The numba python module - part 001 .

Today I tested the numba python module.
This python module allow us to speed up applications with high performance functions written directly in Python.
The numba python module works by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically.
The code can be just-in-time compiled to native machine instructions, similar in performance to C, C++ and Fortran.
For the installation I used pip tool:
C:\Python27>cd Scripts

C:\Python27\Scripts>pip install numba
Collecting numba
  Downloading numba-0.35.0-cp27-cp27m-win32.whl (1.4MB)
    100% |################################| 1.4MB 497kB/s
...
Installing collected packages: singledispatch, funcsigs, llvmlite, numba
Successfully installed funcsigs-1.0.2 llvmlite-0.20.0 numba-0.35.0 singledispatch-3.4.0.3

C:\Python27\Scripts>pip install numpy
Requirement already satisfied: numpy in c:\python27\lib\site-packages
The example test from official website working well:
The example source code is:
from numba import jit
from numpy import arange

# jit decorator tells Numba to compile this function.
# The argument types will be inferred by Numba when function is called.
@jit
def sum2d(arr):
    M, N = arr.shape
    result = 0.0
    for i in range(M):
        for j in range(N):
            result += arr[i,j]
    return result

a = arange(9).reshape(3,3)
print(sum2d(a))
The result of this run python script is:
C:\Python27>python.exe numba_test_001.py
36.0
Another example using just-in-time compile is used with Numba’s jit function:
import numba
from numba import jit

def fibonacci(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a
    return a

print fibonacci(10)

fibonacci_jit = jit(fibonacci)
print fibonacci_jit(14)
Also you can use jit is as a decorator:
@jit
def fibonacci_jit(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a

    return a
Numba is a complex python module because use compiling.
First, compiling takes time, but will work specially for small functions.
The Numba python module tries to do its best by caching compilation as much as possible though.
Another note: not all code is compiled equal.

YARA python module - part 002 .

This is another part of YARA python tutorial and the goal of this part is install the yara modules.
The YARA modules provides extending features to allow us to define data structures and functions which can be used in your rules to express more complex conditions.
You can also write your own modules.
Some known modules used by YARA are:
  • PE
  • ELF
  • Cuckoo
  • Magic
  • Hash
  • Math
First you need to install or reinstall YARA to the last version:
>>> yara.__version__
'3.6.3'
The Cuckoo module enables you to create YARA rules based on behavioral information generated by a Cuckoo sandbox.
C:\Python27\Scripts>pip install yara-python
Collecting yara-python
  Downloading yara_python-3.6.3-cp27-cp27m-win32.whl (606kB)
    100% |################################| 614kB 1.3MB/s
Installing collected packages: yara-python
Successfully installed yara-python-3.6.3
pip install cuckoo
Collecting cuckoo
  Downloading Cuckoo-2.0.4.4.tar.gz (3.1MB)
    100% |################################| 3.1MB 255kB/s
...
Successfully installed Mako-1.0.7 alembic-0.8.8 androguard-3.0.1 beautifulsoup4-4.5.3 
capstone-windows-3.0.4 chardet-2.3.0 click-6.6 colorama-0.3.7 cuckoo-2.0.4.4 django-1.8.4 
django-extensions-1.6.7 dpkt-1.8.7 ecdsa-0.13 egghatch-0.2.1 elasticsearch-5.3.0 
flask-sqlalchemy-2.1 httpreplay-0.2.1 jsbeautifier-1.6.2 jsonschema-2.6.0 olefile-0.43 
oletools-0.42 peepdf-0.3.6 pefile2-1.2.11 pillow-3.2.0 pyelftools-0.24 pymisp-2.4.54 
pymongo-3.0.3 python-dateutil-2.4.2 python-editor-1.0.3 python-magic-0.4.12 pythonaes-1.0 
requests-2.13.0 sflock-0.2.16 sqlalchemy-1.0.8 tlslite-ng-0.6.0 unicorn-1.0.1 wakeonlan-0.2.2
Let's test this python module:
>>> import cuckoo
>>> from cuckoo import *
>>> dir(cuckoo)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__',
 'auxiliary', 'common', 'compat', 'core', 'machinery', 'misc', 'plugins', 'processing', 
'reporting', 'signatures', 'web']
Let's test some yara modules:
>>> import yara
>>> rule = yara.compile(source='import \"pe\"')
>>> rule = yara.compile(source='import \"elf\"')
>>> rule = yara.compile(source='import \"cuckoo\"')
>>> rule = yara.compile(source='import \"math\"')
I could not use the YARA modules: hash and magic.
I will solve this problem in the future.
You can also write your own modules ( see this webpage ).

Friday, September 1, 2017

The beauty of Python: subprocess module - part 004 .

This series of python tutorials that we started at the beginning of this blog and called "The beauty of Python" is part of the series of tutorials aimed at the simplicity and beauty of the python programming language.
The main goal for us is how to use this programming language in everyday life with different tasks.
Today I will come up with examples to cover this goal and show you how to use the subprocess python module.
  • using the powershell with python :
  • >>> import subprocess
    >>> process=subprocess.Popen(["powershell","Get-Childitem C:\\Windows\\*.log"],stdout=subprocess.PIPE);
    >>> result=process.communicate()[0]
    >>> print result
  • get and print the hostname :
  • >>> print subprocess.check_output("hostname")
  • print the output of ping command :
  • >>> print subprocess.check_output("ping localhost", shell=True)
  • print the output of dir command :
  • >>> cmd = 'dir *'
    >>> supcmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    >>> print supcmd.communicate()[0]
    
  • run the python script like python shell :
  • >>> import sys
    >>> import subprocess
    >>> pid = subprocess.Popen([sys.executable, "calc.py"])