analitics

Pages

Showing posts with label 2017. Show all posts
Showing posts with label 2017. Show all posts

Sunday, October 21, 2018

OpenGL and OpenCV with python 2.7 - part 006.

Today I deal with a simple example about how to use your webcam like a python module.
This will allow you to make your python module for your webcam.
My reason was to make a good webcam module to work with python modules like OpenCV and OpenGL and webcam devices.
The source code is simple and has just three functions: start, _update_frame and get_current_frame.
You can make more functions into this python module named webcam.
import cv2
from threading import Thread
  
class webcam:
  
    def __init__(self):
        self.video_capture = cv2.VideoCapture(0)
        self.current_frame = self.video_capture.read()[1]
          
    # create thread for capturing images
    def start(self):
        Thread(target=self._update_frame, args=()).start()
  
    def _update_frame(self):
        while(True):
            self.current_frame = self.video_capture.read()[1]
                  
    # get the current frame
    def get_current_frame(self):
        return self.current_frame
I make also a python script to test this python module:
from webcam import webcam
import cv2
 
dir(webcam)
cam = webcam()
cam.start()
 
while True:
     
    # get image from webcam
    image = cam.get_current_frame()

The shutil python module.

The shutil module helps you to accomplish tasks, such as: copying, moving, or removing directory trees.
This python script creates a zip file in the current directory containing all contents of dir and then clears dir.

import shutil
from os import makedirs

def zip(out_fileName, dir):
shutil.make_archive(str(out_fileName), 'zip', dir)
shutil.rmtree(dir)
makedirs(dir[:-1])

The scapy python module - part 002.

This is another python tutorial about scapy python module.
The last was made on Linux and now I used Windows 10 OS.
Let's install this python module with python version 2.7.13 and pip.
C:\>cd Python27

C:\Python27>cd Scripts

C:\Python27\Scripts>pip install scapy
Collecting scapy
  Downloading scapy-2.3.3.tgz (1.4MB)
    100% |################################| 1.4MB 736kB/s
  In the tar file c:\users\mythcat\appdata\local\temp\pip-26vi9x-unpack\scapy-2.3.3.tgz 
the member scapy-2.3.3/README is invalid: unable to resolve link inside archive
Installing collected packages: scapy
  Running setup.py install for scapy ... done
Successfully installed scapy-2.3.3
The next step is to deal with
C:\Python27\Scripts>python
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import scapy
>>> from scapy import *
>>> dir(scapy)
['VERSION', '_SCAPY_PKG_DIR', '__builtins__', '__doc__', '__file__', '__name__', '__package__',
 '__path__', '_version', '_version_from_git_describe', 'base_classes', 'config', 'dadict', 'data',
 'error', 'os', 'plist', 'pton_ntop', 're', 'subprocess', 'supersocket', 'themes', 'utils', 
'with_statement']

This is not working on WINDOWS

Using PyUSB with python 3.x .

C:\Python34\Scripts>pip3.4.exe install PyUSB
Downloading/unpacking PyUSB
  Running setup.py (path:C:\Users\mythcat\AppData\Local\Temp\pip_build_mythcat\PyUSB\setup.py) 
egg_info for package PyUSB

Installing collected packages: PyUSB
  Running setup.py install for PyUSB

Successfully installed PyUSB
Cleaning up...
Now you need to install this filter from here.
If not, you can get errors like this: raise NoBackendError('No backend available')
usb.core.NoBackendError: No backend available
Let's make a simple test example:
import usb.core
import usb.util
import sys

class find_class(object):
    def __init__(self, class_):
        self._class = class_
    def __call__(self, device):
        # first, let's check the device
        if device.bDeviceClass == self._class:
            return True
        # ok, transverse all devices to find an
        # interface that matches our class
        for cfg in device:
            # find_descriptor: what's it?
            intf = usb.util.find_descriptor(
                                        cfg,
                                        bInterfaceClass=self._class
                                )
            if intf is not None:
                return True

        return False

all = usb.core.find(find_all=1, custom_match=find_class(7))
print (all)
And show the result:
C:\Python34>python.exe usb_devs.py
< generator 0x0000000003d8d5e8="" at="" device_iter="" object="" >

The ansible python module - part 001.

Ansible is an IT automation tool.
It can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments or zero downtime rolling updates.
First, you need to install the VCForPython27.
Install now the pycrypto python module:
C:\Python27\Scripts>pip install  --upgrade  --trusted-host  pypi.python.org pycrypto
Collecting pycrypto
  Downloading pycrypto-2.6.1.tar.gz (446kB)
    100% |################################| 450kB 3.8MB/s
Installing collected packages: pycrypto
  Running setup.py install for pycrypto ... done
Successfully installed pycrypto-2.6.1

C:\Python27\Scripts>pip install --trusted-host pypi.python.org ansible
Collecting ansible
Downloading ansible-2.3.0.0.tar.gz (4.3MB)
100% |################################| 4.3MB 365kB/s
Requirement already satisfied: jinja2 in c:\python27\lib\site-packages (from ansible)
Collecting PyYAML (from ansible)
...
Installing collected packages: ansible
  Running setup.py install for ansible ... done
Successfully installed ansible-2.3.0.0

Wednesday, January 3, 2018

The ebooklib python module .

Happy new year 2018!
The official webpage of this python module comes with this intro:
EbookLib is a Python library for managing EPUB2/EPUB3 and Kindle files. It's capable of reading and writing EPUB files programmatically (Kindle support is under development).
First the installation of this python module named ebooklib.
C:\>cd Python27

C:\Python27>cd Script
The system cannot find the path specified.

C:\Python27>cd Scripts

C:\Python27\Scripts>pip install ebooklib
Collecting ebooklib
  Downloading EbookLib-0.16.tar.gz
Requirement already satisfied: lxml in c:\python27\lib\site-packages (from ebooklib)
Requirement already satisfied: six in c:\python27\lib\site-packages (from ebooklib)
Installing collected packages: ebooklib
  Running setup.py install for ebooklib ... done
Successfully installed ebooklib-0.16
If you don't see the Scripts folder into your Python27 folder you need to install pip tool.
Just download the get-pip.py script into your Python27 folder and run it with python.
Let's test some default example:
C:\Python27>python.exe get-pip.py
The next step is to test a simple example:
from ebooklib import epub

book = epub.EpubBook()

# set metadata
book.set_identifier('id123456')
book.set_title('Sample book')
book.set_language('en')

book.add_author('Author Python')
book.add_author('catafest', file_as='', role='writer', uid='author')

# create chapter
c1 = epub.EpubHtml(title='Intro', file_name='chap_01.xhtml', lang='hr')
c1.content=u'Intro heading.Python is a interpreted high-level programming language ...'

# add chapter
book.add_item(c1)

# define Table Of Contents
book.toc = (epub.Link('chap_01.xhtml', 'Introduction', 'intro'),
(epub.Section('Simple book'),
(c1, ))
)

# add default NCX and Nav file
book.add_item(epub.EpubNcx())
book.add_item(epub.EpubNav())

# define CSS style
style = 'BODY {color: white;}'
nav_css = epub.EpubItem(uid="style_nav", file_name="style/nav.css", media_type="text/css", content=style)

# add CSS file
book.add_item(nav_css)

# basic spine
book.spine = ['nav', c1]

# write to the file
epub.write_epub('test.epub', book, {})
You can update and make more good your epub book with HTML5 tags.
I used this example with headings and paragraph to change the text, see the result:

Tuesday, December 26, 2017

The development with python-instagram - python 3.6.3 .

Today I will show you how to deal with Instagram API using python-instagram python module.
The version of the python I used is this version: Python 3.6.3 |Anaconda custom (64-bit)| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)] on win32
This is the python install of PyCharm IDE - Miniconda3 on my Windows 10 account.
The first step is the install module with the pip tool:
pip install python-instagram
Collecting python-instagram
  Downloading python-instagram-1.3.2.tar.gz
Collecting simplejson (from python-instagram)
  Downloading simplejson-3.13.2-cp36-cp36m-win_amd64.whl (70kB)
    100% |████████████████████████████████| 71kB 816kB/s
Collecting httplib2 (from python-instagram)
  Downloading httplib2-0.10.3.tar.gz (204kB)
    100% |████████████████████████████████| 204kB 1.1MB/s
Requirement already satisfied: six in c:\users\catafest\miniconda3\lib\site-packages (from python-instagram)
Building wheels for collected packages: python-instagram, httplib2
...
Successfully built python-instagram httplib2
Installing collected packages: simplejson, httplib2, python-instagram
Successfully installed httplib2-0.10.3 python-instagram-1.3.2 simplejson-3.13.2
The next step is to take your Client Secret and Client ID from your Instagram account:
The next step is to set your Instagram API:

You can try some example from here.
I just got this error :...instagram.bind.InstagramAPIError: (400) OAuthAccessTokenException-The access_token provided is invalid.
I think the problem is Instagram because I search on the internet and many people come with this issue.
The team development of Instagram tell us to set some Permission:
All permissions require approval to be used out of Sandbox. Make sure to review our Platform Policies before submitting your app for review. To learn more about the review process, please read the Permissions Review documentation.
I try to use this but is not very clear for me.

Tuesday, December 5, 2017

Fix PyCharm error install python module from conda .

Today I fix an error about PyCharm and conda.
As you know :
Conda is an open source package management system and environmental management system that runs on Windows, macOS and Linux.
Also, Conda quickly installs, runs and updates packages dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN.
This error is from PyCharm install python modules using error check from PyCharm (Alt+Enter keys):

The result of this install come with this error from conda :

Close your PyCharm and use this command into your shell-like administrator:

C:\WINDOWS\system32>conda config --show
C:\WINDOWS\system32>conda config --set force True
C:\WINDOWS\system32>conda update conda
C:\WINDOWS\system32>conda install conda anaconda
Fetching package metadata .............
Solving package specifications: .

# All requested packages already installed.
# packages in environment at C:\Users\catafest\Miniconda3:
#
anaconda                  5.0.1            py36h8316230_2
conda                     4.3.30           py36h7e176b0_0
C:\WINDOWS\system32>conda update --prefix C:\Users\catafest\Miniconda3 anaconda
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment C:\Users\catafest\Miniconda3:

The following packages will be UPDATED:

    conda-env: 2.6.0-0 --> 2.6.0-h36134e3_1
Proceed ([y]/n)? y

conda-env-2.6. 100% |###############################| Time: 0:00:00 163.59 kB/s
This command installs anaconda and updates it using my account catafest .
Start the I.D.E. PyCharm and after indexing all you can try to fix the python install module (Alt+Enter keys).
If the python modules are not into conda repo from PyCharm then you can use this command:
C:\WINDOWS\system32>conda install -c conda-forge opencv
Fetching package metadata ...............
Solving package specifications: .

# All requested packages already installed.
# packages in environment at C:\Users\catafest\Miniconda3:
#
opencv                    3.3.0                  py36_202    conda-forge
In this example I used OpenCV python module named into python script like cv2, see the next image:






Saturday, November 11, 2017

Using kivy python module with PyCharm IDE.

First, you need to download the last version of PyCharm IDE.
My PyCharm IDE put the python version 3 into a folder named Miniconda3.
I use the command shell to go to Scripts and I used pip to install the kivy python module, see:
Scripts>pip install kivy
Collecting kivy
  Downloading Kivy-1.10.0-cp36-cp36m-win_amd64.whl (3.5MB)
    100% |████████████████████████████████| 3.5MB 380kB/s
Collecting pygments (from kivy)
  Downloading Pygments-2.2.0-py2.py3-none-any.whl (841kB)
    100% |████████████████████████████████| 849kB 1.3MB/s
Collecting Kivy-Garden>=0.1.4 (from kivy)
  Downloading kivy-garden-0.1.4.tar.gz
Collecting docutils (from kivy)
  Downloading docutils-0.14-py3-none-any.whl (543kB)
    100% |████████████████████████████████| 552kB 1.8MB/s
Requirement already satisfied: requests 
...
Building wheels for collected packages: Kivy-Garden
...
Successfully built Kivy-Garden
Installing collected packages: pygments, Kivy-Garden, docutils, kivy
Successfully installed Kivy-Garden-0.1.4 docutils-0.14 kivy-1.10.0 pygments-2.2.0
I got one error about SDL and I put the SDL 2.0 into my windows system 32 folders.
The next step is to try to use this steps from the official website into the Scripts folder from Miniconda3.
python -m pip install --upgrade pip wheel setuptools
python -m pip install docutils pygments pypiwin32 kivy.deps.sdl2 kivy.deps.glew
python -m pip install kivy.deps.gstreamer
python -m pip install kivy.deps.angle
python -m pip install kivy
I make one project into my PyCharm editor and I use the default script:
from kivy.app import App
from kivy.uix.button import Button

class TestApp(App):
    def build(self):
        return Button(text='Hello World')

TestApp().run()
The result can be see into the next image:

Sunday, October 22, 2017

The Google Cloud Pub/Sub python module.

This is a test of google feature from cloud.google.com/pubsub web page.
The Google development team tell us about this service:
The Google Cloud Pub/Sub service allows applications to exchange messages reliably, quickly, and asynchronously. To accomplish this, a producer of data publishes a message to a Cloud Pub/Subtopic. A subscriber client then creates a subscription to that topic and consumes messages from the subscription. Cloud Pub/Sub persists messages that could not be delivered reliably for up to seven days. This page shows you how to get started publishing messages with Cloud Pub/Sub using client libraries.
The simple idea about this is:
Publisher applications can send messages to a topic, and other applications can subscribe to that topic to receive the messages.
I start with the installation of the python module using python version 2.7 and pip tool.
C:\Python27>cd Scripts

C:\Python27\Scripts>pip install --upgrade google-cloud-pubsub
Collecting google-cloud-pubsub
  Downloading google_cloud_pubsub-0.28.4-py2.py3-none-any.whl (79kB)
    100% |################################| 81kB 300kB/s
...
Successfully installed google-cloud-pubsub-0.28.4 grpc-google-iam-v1-0.11.4 ply-3.8 
psutil-5.4.0 pyasn1-modules-0.1.5 setuptools-36.6.0
The next steps come with some settings on google console, see this google page.
The default settings can be started and set with this command: gcloud init .
You need to edit this settings and app.yaml at ~/src/.../appengine/flexible/pubsub$ nano app.yaml.
After that, you set all of this using the command gcloud app deploy you can see the output at https://[YOUR_PROJECT_ID].appspot.com.
The main goal of this tutorial was to start and run the Google Cloud Pub/Sub service with python and this has been achieved.

Tuesday, October 10, 2017

The online editor for python and google .

This is a good online editor for python and google.
Like any online editor, some python modules are not available for online security reasons.
I do not know what python modules are implemented in this online editor.
I tested just sys and math python modules.
The Google Apps come with this tool integration like application for Google drive:
Edit your python file directly in your browser:
- Save it to Google Drive integrated with Google Drive
- Test it in your browser with Skulpt
- Use autocompletion code (CTRL+SPACE)
- No registration required and totally free
- Export your file
- Work offline
New python libraries partially supported: numpy, matplotlib.

Sunday, October 1, 2017

The capstone python module - disassembly framework.

The official python module comes with this info about this python module:
Capstone is a disassembly framework with the target of becoming the ultimate
the disasm engine for binary analysis and reversing in the security community.

Created by Nguyen Anh Quynh, then developed and maintained by a small community,
Capstone offers some unparalleled features:

- Support multiple hardware architectures: ARM, ARM64 (ARMv8), Mips, PPC & X86.

- Having clean/simple/lightweight/intuitive architecture-neutral API.

- Provide details on disassembled instruction (called “decomposer” by others).

- Provide semantics of the disassembled instruction, such as list of implicit
registers read & written.

- Implemented in pure C language, with lightweight wrappers for C++, Python,
Ruby, OCaml, C#, Java and Go available.

- Native support for Windows & *nix platforms (with OSX, Linux, *BSD & Solaris
have been confirmed).

- Thread-safe by design.

- Distributed under the open source BSD license.

Today I tested this python module with python version 2.7.
First I need to use a build of this python module from the official website.
I used binaries 32 bits like my python 2.7 and I tested with pip 2.7:
C:\Python27\Scripts>pip install capstone
Requirement already satisfied: capstone in c:\python27\lib\site-packages
Let's make a simple test with this python module:

C:\Python27>python.exe
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from capstone import (
...     Cs,
...     CS_ARCH_X86,
...     CS_MODE_32,
...     CS_OPT_SYNTAX_ATT,
... )
>>> mode=Cs(CS_ARCH_X86, CS_MODE_32)
>>> mode.syntax = CS_OPT_SYNTAX_ATT
>>> def D_ASM(code):
...     for address, size, mnemonic, op_str in mode.disasm_lite(code, offset=0x08048060):
...         print("0x{0:x}\t{1:d}\t{2:s}\t{3:s}".format(address, size,mnemonic, op_str))
...
>>> D_ASM(b"\xe1\x0b\x40\xb9\x20\x04\x81\xda\x20\x08\x02\x8b")
0x8048060       2       loope   0x804806d
0x8048062       1       incl    %eax
0x8048063       5       movl    $0xda810420, %ecx
0x8048068       2       andb    %cl, (%eax)
It seems to work very well.


Friday, September 22, 2017

The python-vlc python module.

The python module for vlc is named python-vlc.
This python module let you test libvlc API like the VLC video player.
You can install it easily with pip python tool.
C:\Python27\Scripts>pip2.7.exe install python-vlc
Collecting python-vlc
  Downloading python-vlc-1.1.2.tar.gz (201kB)
    100% |################################| 204kB 628kB/s
Installing collected packages: python-vlc
  Running setup.py install for python-vlc ... done
Successfully installed python-vlc-1.1.2
Let's see a simple example with this python module:
import os
import sys
import vlc
import pygame
 
def call_vlc(self, player):
 
    player.get_fps()
    player.get_time()
 
if len( sys.argv )< 2 or len( sys.argv )> 3:
        print 'Help: python vlc_001.py your_video.mp4'
else:
    pygame.init()
    screen = pygame.display.set_mode((800,600),pygame.RESIZABLE)
    pygame.display.get_wm_info()
    pygame.display.get_driver()

 
    # get path to movie specified as command line argument
    movie = os.path.expanduser(sys.argv[1])
    # see if movie is accessible
    if not os.access(movie, os.R_OK):
        print('Error: %s wrong read file: ' % movie)
        sys.exit(1)
 
    # make instane of VLC and create reference to movie.
    vlcInstance = vlc.Instance()
    media = vlcInstance.media_new(movie)
 
    # make new instance of vlc player
    player = vlcInstance.media_player_new()
 
    # start with a callback
    em = player.event_manager()
    em.event_attach(vlc.EventType.MediaPlayerTimeChanged, \
        call_vlc, player)
 
    # set pygame window id to vlc player
    win_id = pygame.display.get_wm_info()['window']
    if sys.platform == "win32": # for Windows
        player.set_hwnd(win_id)
 
    # load movie into vlc player instance
    player.set_media(media)
 
    # quit pygame mixer to allow vlc full access to audio device
    pygame.mixer.quit()
 
    # start movie play
    player.play()
 
    while player.get_state() != vlc.State.Ended:
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                sys.exit(2)
The base of this python script is to make an instance of vlc and put into the pygame display.
Another simple example:
C:\Python27>python.exe
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vlc
>>> inst = vlc.Instance()
Warning: option --plugin-path no longer exists.
Warning: option --plugin-path no longer exists.
>>> med = inst.media_new('rain.mp4')
>>> p = med.player_new_from_media()
>>> p.play()
0
>>>

Tuesday, September 19, 2017

The numba python module - part 002 .

Today I tested how fast is jit from numba python and fibonacci math function.
You will see strange output I got for some values.
First example:
import numba
from numba import jit
from timeit import default_timer as timer

def fibonacci(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a
    return a
fibonacci_jit = jit(fibonacci)

start = timer()
fibonacci(100)
duration = timer() - start

startnext = timer()
fibonacci_jit(100)
durationnext = timer() - startnext

print(duration, durationnext)
The result of this run is:
C:\Python27>python numba_test_003.py
(0.00018731270733896962, 0.167499256682878)

C:\Python27>python numba_test_003.py
(1.6357787798437412e-05, 0.1683614083221368)

C:\Python27>python numba_test_003.py
(2.245186560569841e-05, 0.1758382003097716)

C:\Python27>python numba_test_003.py
(2.3093347480146938e-05, 0.16714964906130353)

C:\Python27>python numba_test_003.py
(1.5395564986764625e-05, 0.17471143739730277)

C:\Python27>python numba_test_003.py
(1.5074824049540363e-05, 0.1847134227837042)
As you can see the fibonacci function is not very fast.
The jit - just-in-time compile is very fast.
Let's see if the python source code may slow down.
Let's see the new source code with jit will not work well:
import numba
from numba import jit
from timeit import default_timer as timer

def fibonacci(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a
    return a
fibonacci_jit = jit(fibonacci)

start = timer()
print fibonacci(100)
duration = timer() - start

startnext = timer()
print fibonacci_jit(100)
durationnext = timer() - startnext

print(duration, durationnext)
The result is this:
C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0002334994022992635, 0.17628787910376)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0006886307922204926, 0.17579169287387408)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0008105123483657127, 0.18209553525407973)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.00025466830415606486, 0.17186550306131188)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0007348174871807866, 0.17523103771560608)
The result for value 100 is not the same: 927372692193078999176 and 1445263496.
The first problem is:
The problem is that numba can't intuit the type of lookup. If you put a print nb.typeof(lookup) in your method, you'll see that numba is treating it as an object, which is slow.
The second problem is the output but can be from the same reason.
I test with value 5 and the result is :
C:\Python27>python numba_test_003.py
13
13
13
13
(0.0007258367409385072, 0.17057997338491704)

C:\Python27>python numba_test_003.py
13
13
(0.00033709872502270044, 0.17213235952108247)

C:\Python27>python numba_test_003.py
13
13
(0.0004836773333341886, 0.17184433415945508)

C:\Python27>python numba_test_003.py
13
13
(0.0006854233828482501, 0.17381272129120037)

Monday, September 18, 2017

The numba python module - part 001 .

Today I tested the numba python module.
This python module allows us to speed up applications with high-performance functions written directly in Python.
The numba python module works by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically.
The code can be just-in-time compiled to native machine instructions, similar in performance to C, C++ and Fortran.
For the installation I used the pip tool:
C:\Python27>cd Scripts

C:\Python27\Scripts>pip install numba
Collecting numba
  Downloading numba-0.35.0-cp27-cp27m-win32.whl (1.4MB)
    100% |################################| 1.4MB 497kB/s
...
Installing collected packages: singledispatch, funcsigs, llvmlite, numba
Successfully installed funcsigs-1.0.2 llvmlite-0.20.0 numba-0.35.0 singledispatch-3.4.0.3

C:\Python27\Scripts>pip install numpy
Requirement already satisfied: numpy in c:\python27\lib\site-packages
The example test from official website working well:
The example source code is:
from numba import jit
from numpy import arange

# jit decorator tells Numba to compile this function.
# The argument types will be inferred by Numba when function is called.
@jit
def sum2d(arr):
    M, N = arr.shape
    result = 0.0
    for i in range(M):
        for j in range(N):
            result += arr[i,j]
    return result

a = arange(9).reshape(3,3)
print(sum2d(a))
The result of this run python script is:
C:\Python27>python.exe numba_test_001.py
36.0
Another example using just-in-time compile is used with Numba’s jit function:
import numba
from numba import jit

def fibonacci(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a
    return a

print fibonacci(10)

fibonacci_jit = jit(fibonacci)
print fibonacci_jit(14)
Also, you can use jit is as a decorator:
@jit
def fibonacci_jit(n):
    a, b = 1, 1
    for i in range(n):
        a, b = a+b, a

    return a
Numba is a complex python module because use compiling.
First, compiling takes time, but will work especially for small functions.
The Numba python module tries to do its best by caching compilation as much as possible though.
Another note: not all code is compiled equally.

YARA another python module - part 002 .

This is another part of YARA python tutorial and the goal of this part is to install the Yara modules.
This is another python module about Yara named yara-python from VirusTotal.
The last tutorial uses the Yara python module.
The YARA modules provide extending features to allow us to define data structures and functions which can be used in your rules to express more complex conditions.
You can also write your own modules.
Some known modules used by YARA are:
  • PE
  • ELF
  • Cuckoo
  • Magic
  • Hash
  • Math
First, you need to install or reinstall YARA to the last version:
>>> yara.__version__
'3.6.3'
The Cuckoo module enables you to create YARA rules based on behavioral information generated by a Cuckoo sandbox.
C:\Python27\Scripts>pip install yara-python
Collecting yara-python
  Downloading yara_python-3.6.3-cp27-cp27m-win32.whl (606kB)
    100% |################################| 614kB 1.3MB/s
Installing collected packages: yara-python
Successfully installed yara-python-3.6.3
pip install cuckoo
Collecting cuckoo
  Downloading Cuckoo-2.0.4.4.tar.gz (3.1MB)
    100% |################################| 3.1MB 255kB/s
...
Successfully installed Mako-1.0.7 alembic-0.8.8 androguard-3.0.1 beautifulsoup4-4.5.3 
capstone-windows-3.0.4 chardet-2.3.0 click-6.6 colorama-0.3.7 cuckoo-2.0.4.4 django-1.8.4 
django-extensions-1.6.7 dpkt-1.8.7 ecdsa-0.13 egghatch-0.2.1 elasticsearch-5.3.0 
flask-sqlalchemy-2.1 httpreplay-0.2.1 jsbeautifier-1.6.2 jsonschema-2.6.0 olefile-0.43 
oletools-0.42 peepdf-0.3.6 pefile2-1.2.11 pillow-3.2.0 pyelftools-0.24 pymisp-2.4.54 
pymongo-3.0.3 python-dateutil-2.4.2 python-editor-1.0.3 python-magic-0.4.12 pythonaes-1.0 
requests-2.13.0 sflock-0.2.16 sqlalchemy-1.0.8 tlslite-ng-0.6.0 unicorn-1.0.1 wakeonlan-0.2.2
Let's test this python module:
>>> import cuckoo
>>> from cuckoo import *
>>> dir(cuckoo)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__',
 'auxiliary', 'common', 'compat', 'core', 'machinery', 'misc', 'plugins', 'processing', 
'reporting', 'signatures', 'web']
Let's test some yara modules:
>>> import yara
>>> rule = yara.compile(source='import \"pe\"')
>>> rule = yara.compile(source='import \"elf\"')
>>> rule = yara.compile(source='import \"cuckoo\"')
>>> rule = yara.compile(source='import \"math\"')
I could not use the YARA modules: hash and magic.
I will solve this problem in the future.
You can also write your own modules ( see this webpage ).

Friday, September 1, 2017

The beauty of Python: subprocess module - part 004 .

This series of python tutorials that we started at the beginning of this blog and called "The beauty of Python" is part of the series of tutorials aimed at the simplicity and beauty of the Python programming language.
The main goal for us is how to use this programming language in everyday life with different tasks.
Today I will come up with examples to cover this goal and show you how to use the subprocess python module.
  • using the PowerShell with python :
  • >>> import subprocess
    >>> process=subprocess.Popen(["powershell","Get-Childitem C:\\Windows\\*.log"],stdout=subprocess.PIPE);
    >>> result=process.communicate()[0]
    >>> print result
  • get and print the hostname :
  • >>> print subprocess.check_output("hostname")
  • print the output of ping command :
  • >>> print subprocess.check_output("ping localhost", shell=True)
  • print the output of dir command :
  • >>> cmd = 'dir *'
    >>> supcmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    >>> print supcmd.communicate()[0]
    
  • run the python script like python shell :
  • >>> import sys
    >>> import subprocess
    >>> pid = subprocess.Popen([sys.executable, "calc.py"])

Monday, August 21, 2017

Using pip into shell to install and use pymunk.

The tutorial for today will show how to use pip into the python shell to install a python package.
The first step is shown in the next image:

Friday, August 18, 2017

The Google Cloud SDK - part 002 .

The next part of my tutorials about the Google Cloud SDK comes with some info about the project.
As you know I used the default sample app engine hello word standard application.
The goal is to understand how it works by working with Google's documentation and examples.
Into this project folder we have this files:
08/17/2017  11:12 PM                98 app.yaml
08/17/2017  11:12 PM               854 main.py
08/17/2017  11:12 PM               817 main_test.py
Let's see what these files contain:
First is app.yaml and come with:
runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: main.app
The next is main.py file:
# Copyright 2016 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import webapp2


class MainPage(webapp2.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.write('Hello, World!')


app = webapp2.WSGIApplication([
    ('/', MainPage),
], debug=True)
The last from this folder is main_test.py :
# Copyright 2016 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import webtest

import main


def test_get():
    app = webtest.TestApp(main.app)

    response = app.get('/')

    assert response.status_int == 200
    assert response.body == 'Hello, World!'
The app.yaml file is used to configure your App Engine application's settings of the project.
You can have many application-level configuration files (dispatch.yaml, cron.yaml, index.yaml, and queue.yaml).
This all type of configuration files are included in the top level app directory ( in this case: hello_world).
Let's see some common gcloud commands:
  • gcloud app deploy  --project XXXXXX - deploy your project;
  • gcloud app browse - show your project running into your browser;
  • gcloud components list - show all available components;
  • gcloud components update - update all gcloud components;
  • gcloud projects list --limit=10 - show all projects with a limit number;
Let's test some changes:
First, change the text from main.py file with something else:
self.response.write('Hello, World!')
Now use this commands:
C:\Python27\python-docs-samples\appengine\standard\hello_world>gcloud app deploy
C:\Python27\python-docs-samples\appengine\standard\hello_world>gcloud app browse
The result is shown in your browser.
You can read about this files into google documentation page - here.
Also some gcloud commands and reference you can read here.

Thursday, August 17, 2017

The Google Cloud SDK - part 001 .

This tutorial will cover this steps into development with Google Cloud SDK and Python version 2.7:

  • install the Google Cloud SDK on the computer;
  • make settings online for your Google project to use Google Cloud SDK;
  • run the online project of Google Cloud SDK;
  • make setting into your computer to run the local project ;

First, you need to download the Google Cloud SDK and run it.


After GUI install a window command will ask you to set the default project for your work.
Welcome to the Google Cloud SDK! Run "gcloud -h" to get the list of available commands.
---
Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to: [default]

You can skip diagnostics next time by using the following flag:
  gcloud init --skip-diagnostics

Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic (1/1 checks) passed.

You must log in to continue. Would you like to log in (Y/n)?  Y
...
The next step is to start online to deploying a Hello World app with: Deploy a Hello World app:

This will start an online tutorial into the right area of the screen with all commands and steps for your Google Cloud SDK online project.
Follow this steps and in the end will see how the online Google Cloud SDK project will show: Hello, World! into your browser.
The next step is to make a local project and run it.
You can use the python docs sample from GoogleCloudPlatform but is not the same with the online example.
To download the GoogleCloudPlatform sample use git command:
C:\Python27>git clone https://github.com/GoogleCloudPlatform/python-docs-samples
Cloning into 'python-docs-samples'...
remote: Counting objects: 12126, done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 12126 (delta 1), reused 10 (delta 1), pack-reused 12106
Receiving objects: 100% (12126/12126), 3.37 MiB | 359.00 KiB/s, done.
Resolving deltas: 100% (6408/6408), done.

C:\Python27>cd python-docs-samples/appengine/standard/hello_world
To start this sample into your google project you need to use this:
C:\Python27\python-docs-samples\appengine\standard\hello_world>gcloud app deploy app.yaml --project encoded-metrics-147522
Services to deploy:

descriptor:      [C:\Python27\python-docs-samples\appengine\standard\hello_world\app.yaml]
source:          [C:\Python27\python-docs-samples\appengine\standard\hello_world]
target project:  [encoded-metrics-147522]
target service:  [default]
target version:  [20170817t234925]
target url:      [https://encoded-metrics-147522.appspot.com]


Do you want to continue (Y/n)?  Y

Beginning deployment of service [default]...
#============================================================#
#= Uploading 5 files to Google Cloud Storage                =#
#============================================================#
File upload done.
Updating service [default]...done.
Waiting for operation [apps/encoded-metrics-147522/operations/XXXXXX] to complete...done.
Updating service [default]...done.
Deployed service [default] to [https://XXXXXX.appspot.com]

You can stream logs from the command line by running:
  $ gcloud app logs tail -s default

To view your application in the web browser run:
  $ gcloud app browse

C:\Python27\python-docs-samples\appengine\standard\hello_world>gcloud app browse
Opening [https://XXXXXX.appspot.com] in a new tab in your default browser.

C:\Python27\python-docs-samples\appengine\standard\hello_world>
This will start your application with the text - Hello, World! into your browser address bar with this web address: XXXXXX.appspot.com.