The numba python module - part 002 .

Today I tested how fast is jit from numba python and fibonacci math function.
You will see strange output I got for some values.
First example:

import numba
from numba import jit
from timeit import default_timer as timer

def fibonacci(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a
    return a
fibonacci_jit = jit(fibonacci)

start = timer()
fibonacci(100)
duration = timer() - start

startnext = timer()
fibonacci_jit(100)
durationnext = timer() - startnext

print(duration, durationnext)

The result of this run is:

C:\Python27>python numba_test_003.py
(0.00018731270733896962, 0.167499256682878)

C:\Python27>python numba_test_003.py
(1.6357787798437412e-05, 0.1683614083221368)

C:\Python27>python numba_test_003.py
(2.245186560569841e-05, 0.1758382003097716)

C:\Python27>python numba_test_003.py
(2.3093347480146938e-05, 0.16714964906130353)

C:\Python27>python numba_test_003.py
(1.5395564986764625e-05, 0.17471143739730277)

C:\Python27>python numba_test_003.py
(1.5074824049540363e-05, 0.1847134227837042)

As you can see the fibonacci function is not very fast.
The jit - just-in-time compile is very fast.
Let's see if the python source code may slow down.
Let's see the new source code with jit will not work well:

import numba
from numba import jit
from timeit import default_timer as timer

def fibonacci(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a
    return a
fibonacci_jit = jit(fibonacci)

start = timer()
print fibonacci(100)
duration = timer() - start

startnext = timer()
print fibonacci_jit(100)
durationnext = timer() - startnext

print(duration, durationnext)

The result is this:

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0002334994022992635, 0.17628787910376)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0006886307922204926, 0.17579169287387408)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0008105123483657127, 0.18209553525407973)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.00025466830415606486, 0.17186550306131188)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0007348174871807866, 0.17523103771560608)

The result for value 100 is not the same: 927372692193078999176 and 1445263496.
The first problem is:
The problem is that numba can't intuit the type of lookup. If you put a print nb.typeof(lookup) in your method, you'll see that numba is treating it as an object, which is slow.
The second problem is the output but can be from the same reason.
I test with value 5 and the result is :

C:\Python27>python numba_test_003.py
13
13
13
13
(0.0007258367409385072, 0.17057997338491704)

C:\Python27>python numba_test_003.py
13
13
(0.00033709872502270044, 0.17213235952108247)

C:\Python27>python numba_test_003.py
13
13
(0.0004836773333341886, 0.17184433415945508)

C:\Python27>python numba_test_003.py
13
13
(0.0006854233828482501, 0.17381272129120037)

The numba python module - part 001 .

Today I tested the numba python module.
This python module allows us to speed up applications with high-performance functions written directly in Python.
The numba python module works by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically.
The code can be just-in-time compiled to native machine instructions, similar in performance to C, C++ and Fortran.
For the installation I used the pip tool:

C:\Python27>cd Scripts

C:\Python27\Scripts>pip install numba
Collecting numba
  Downloading numba-0.35.0-cp27-cp27m-win32.whl (1.4MB)
    100% |################################| 1.4MB 497kB/s
...
Installing collected packages: singledispatch, funcsigs, llvmlite, numba
Successfully installed funcsigs-1.0.2 llvmlite-0.20.0 numba-0.35.0 singledispatch-3.4.0.3

C:\Python27\Scripts>pip install numpy
Requirement already satisfied: numpy in c:\python27\lib\site-packages

The example test from official website working well:
The example source code is:

from numba import jit
from numpy import arange

# jit decorator tells Numba to compile this function.
# The argument types will be inferred by Numba when function is called.
@jit
def sum2d(arr):
    M, N = arr.shape
    result = 0.0
    for i in range(M):
        for j in range(N):
            result += arr[i,j]
    return result

a = arange(9).reshape(3,3)
print(sum2d(a))

The result of this run python script is:

C:\Python27>python.exe numba_test_001.py
36.0

Another example using just-in-time compile is used with Numba’s jit function:

import numba
from numba import jit

def fibonacci(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a
    return a

print fibonacci(10)

fibonacci_jit = jit(fibonacci)
print fibonacci_jit(14)

Also, you can use jit is as a decorator:

@jit
def fibonacci_jit(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a

    return a

Numba is a complex python module because use compiling.
First, compiling takes time, but will work especially for small functions.
The Numba python module tries to do its best by caching compilation as much as possible though.
Another note: not all code is compiled equally.

YARA another python module - part 002 .

This is another part of YARA python tutorial and the goal of this part is to install the Yara modules.
This is another python module about Yara named yara-python from VirusTotal.
The last tutorial uses the Yara python module.
The YARA modules provide extending features to allow us to define data structures and functions which can be used in your rules to express more complex conditions.
You can also write your own modules.
Some known modules used by YARA are:

• PE
• ELF
• Cuckoo
• Magic
• Hash
• Math

First, you need to install or reinstall YARA to the last version:

>>> yara.__version__
'3.6.3'

The Cuckoo module enables you to create YARA rules based on behavioral information generated by a Cuckoo sandbox.

C:\Python27\Scripts>pip install yara-python
Collecting yara-python
  Downloading yara_python-3.6.3-cp27-cp27m-win32.whl (606kB)
    100% |################################| 614kB 1.3MB/s
Installing collected packages: yara-python
Successfully installed yara-python-3.6.3
pip install cuckoo
Collecting cuckoo
  Downloading Cuckoo-2.0.4.4.tar.gz (3.1MB)
    100% |################################| 3.1MB 255kB/s
...
Successfully installed Mako-1.0.7 alembic-0.8.8 androguard-3.0.1 beautifulsoup4-4.5.3 

capstone-windows-3.0.4 chardet-2.3.0 click-6.6 colorama-0.3.7 cuckoo-2.0.4.4 django-1.8.4 

django-extensions-1.6.7 dpkt-1.8.7 ecdsa-0.13 egghatch-0.2.1 elasticsearch-5.3.0 

flask-sqlalchemy-2.1 httpreplay-0.2.1 jsbeautifier-1.6.2 jsonschema-2.6.0 olefile-0.43 

oletools-0.42 peepdf-0.3.6 pefile2-1.2.11 pillow-3.2.0 pyelftools-0.24 pymisp-2.4.54 

pymongo-3.0.3 python-dateutil-2.4.2 python-editor-1.0.3 python-magic-0.4.12 pythonaes-1.0 

requests-2.13.0 sflock-0.2.16 sqlalchemy-1.0.8 tlslite-ng-0.6.0 unicorn-1.0.1 wakeonlan-0.2.2

Let's test this python module:

>>> import cuckoo
>>> from cuckoo import *
>>> dir(cuckoo)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__',

 'auxiliary', 'common', 'compat', 'core', 'machinery', 'misc', 'plugins', 'processing', 

'reporting', 'signatures', 'web']

Let's test some yara modules:

>>> import yara
>>> rule = yara.compile(source='import \"pe\"')
>>> rule = yara.compile(source='import \"elf\"')
>>> rule = yara.compile(source='import \"cuckoo\"')
>>> rule = yara.compile(source='import \"math\"')

I could not use the YARA modules: hash and magic.
I will solve this problem in the future.
You can also write your own modules ( see this webpage ).

The beauty of Python: subprocess module - part 004 .

This series of python tutorials that we started at the beginning of this blog and called "The beauty of Python" is part of the series of tutorials aimed at the simplicity and beauty of the Python programming language.
The main goal for us is how to use this programming language in everyday life with different tasks.
Today I will come up with examples to cover this goal and show you how to use the subprocess python module.

• using the PowerShell with python :

>>> import subprocess
>>> process=subprocess.Popen(["powershell","Get-Childitem C:\\Windows\\*.log"],stdout=subprocess.PIPE);
>>> result=process.communicate()[0]
>>> print result

• get and print the hostname :

>>> print subprocess.check_output("hostname")

• print the output of ping command :

>>> print subprocess.check_output("ping localhost", shell=True)

• print the output of dir command :

>>> cmd = 'dir *'
>>> supcmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> print supcmd.communicate()[0]

• run the python script like python shell :

>>> import sys
>>> import subprocess
>>> pid = subprocess.Popen([sys.executable, "calc.py"])

Using pip into shell to install and use pymunk.

The tutorial for today will show how to use pip into the python shell to install a python package.
The first step is shown in the next image:

The Google Cloud SDK - part 002 .

The next part of my tutorials about the Google Cloud SDK comes with some info about the project.
As you know I used the default sample app engine hello word standard application.
The goal is to understand how it works by working with Google's documentation and examples.
Into this project folder we have this files:

08/17/2017  11:12 PM                98 app.yaml
08/17/2017  11:12 PM               854 main.py
08/17/2017  11:12 PM               817 main_test.py

Let's see what these files contain:
First is app.yaml and come with:

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: main.app

The next is main.py file:

# Copyright 2016 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import webapp2


class MainPage(webapp2.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.write('Hello, World!')


app = webapp2.WSGIApplication([
    ('/', MainPage),
], debug=True)

The last from this folder is main_test.py :

# Copyright 2016 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import webtest

import main


def test_get():
    app = webtest.TestApp(main.app)

    response = app.get('/')

    assert response.status_int == 200
    assert response.body == 'Hello, World!'

The app.yaml file is used to configure your App Engine application's settings of the project.
You can have many application-level configuration files (dispatch.yaml, cron.yaml, index.yaml, and queue.yaml).
This all type of configuration files are included in the top level app directory ( in this case: hello_world).
Let's see some common gcloud commands:

• gcloud app deploy  --project XXXXXX - deploy your project;
• gcloud app browse - show your project running into your browser;
• gcloud components list - show all available components;
• gcloud components update - update all gcloud components;
• gcloud projects list --limit=10 - show all projects with a limit number;

Let's test some changes:
First, change the text from main.py file with something else:

self.response.write('Hello, World!')

Now use this commands:

C:\Python27\python-docs-samples\appengine\standard\hello_world>gcloud app deploy
C:\Python27\python-docs-samples\appengine\standard\hello_world>gcloud app browse

The result is shown in your browser.
You can read about this files into google documentation page - here.
Also some gcloud commands and reference you can read here.