analitics

Pages

Thursday, March 16, 2017

The tensorflow python module - part 003.

This short tutorial come to fix your work and help you with examples and tutorials.
You can look to the internet and you can search many examples and tutorials but you can get errors.
How to fix that:
The first step is to know what tensorflow version of you use.
[mythcat@localhost ~]$ python -c "import tensorflow; print(tensorflow.__version__)"
1.0.1
Then yo need to know what parts from tensorflow old version is deprecated from the last version.
One superannuated method is to fix old source code with this:
import tensorflow as tf
tf.scalar_summary = tf.summary.scalar
tf.merge_all_summaries = tf.summary.merge_all
tf.train.SummaryWriter = tf.summary.FileWriter
A better method is to fix your source code and read the documentation.

Monday, March 13, 2017

The tensorflow python module - part 002.

Today I will show you how to install tensorflow python module on Windows OS with pip tool.
I used python version 3.5.3.
C:\Python35>cd Scripts

C:\Python35\Scripts>pip3 install --upgrade tensorflow
Collecting tensorflow
  Downloading tensorflow-1.0.1-cp35-cp35m-win_amd64.whl (14.7MB)
    100% |################################| 14.7MB 43kB/s
...
Successfully installed appdirs-1.4.3 numpy-1.12.0 packaging-16.8 
protobuf-3.2.0 pyparsing-2.2.0 setuptools-34.3.2 six-1.10.0 
tensorflow-1.0.1 wheel-0.29.0
Next step is to install the GPU version of TensorFlow:
C:\Python35\Scripts>pip3 install --upgrade tensorflow-gpu
Collecting tensorflow-gpu
  Downloading tensorflow_gpu-1.0.1-cp35-cp35m-win_amd64.whl (43.1MB)
    100% |################################| 43.1MB 11kB/
...
Installing collected packages: tensorflow-gpu
Successfully installed tensorflow-gpu-1.0.1
If you are installing TensorFlow with GPU support then the following NVIDIA software must be installed on your system:

CUDA Toolkit 8.0. , the NVIDIA drivers associated with CUDA Toolkit 8.0 and cuDNN v5.1.
The cuDNN is typically installed in a different location from the other CUDA DLLs.
Now you need to add the directory where you installed the cuDNN DLL to your %PATH% environment variable.
The result will be output under python result when you import this python module, see my output:
 Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 740M, pci bus id: 0000:01:00.0)

Friday, March 10, 2017

Strange code in python.

Try this:

>>> 2*2
4
>>> _*2
8
>>> print _*3
24
>>> print _*'a'
aaaaaaaa

The tensorflow python module - part 001.

TensorFlow™ is an open source software library for numerical computation using data flow graphs.
I used Fedora 25 distro and python version 2.7.
The base of this installation was the official website.
Fist step of the installation was the base python module: tensorflow.

[root@localhost build]# pip install tensorflow  
Collecting tensorflow
  Downloading tensorflow-1.0.1-cp27-cp27mu-manylinux1_x86_64.whl (44.1MB)
    100% |████████████████████████████████| 44.1MB 30kB/s 
Collecting mock>=2.0.0 (from tensorflow)
  Downloading mock-2.0.0-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 341kB/s 
Requirement already satisfied: six>=1.10.0 in /usr/lib/python2.7/site-packages (from tensorflow)
Requirement already satisfied: numpy>=1.11.0 in /usr/lib64/python2.7/site-packages (from tensorflow)
Collecting protobuf>=3.1.0 (from tensorflow)
  Downloading protobuf-3.2.0-cp27-cp27mu-manylinux1_x86_64.whl (5.6MB)
    100% |████████████████████████████████| 5.6MB 172kB/s 
Collecting wheel (from tensorflow)
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 532kB/s 
Collecting funcsigs>=1; python_version < "3.3" (from mock>=2.0.0->tensorflow)
  Downloading funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock>=2.0.0->tensorflow)
  Downloading pbr-2.0.0-py2.py3-none-any.whl (98kB)
    100% |████████████████████████████████| 102kB 518kB/s 
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from protobuf>=3.1.0->tensorflow)
Installing collected packages: funcsigs, pbr, mock, protobuf, wheel, tensorflow
Successfully installed funcsigs-1.0.2 mock-2.0.0 pbr-2.0.0 protobuf-3.2.0 tensorflow-1.0.1 wheel-0.29.0
The next step come with the installation of python module gpu: tensorflow-gpu.
[root@localhost build]# pip install --upgrade tensorflow-gpu
Collecting tensorflow-gpu
  Downloading tensorflow_gpu-1.0.1-cp27-cp27mu-manylinux1_x86_64.whl (94.8MB)
    100% |████████████████████████████████| 94.8MB 15kB/s 
Requirement already up-to-date: mock>=2.0.0 in /usr/lib/python2.7/site-packages (from tensorflow-gpu)
Requirement already up-to-date: six>=1.10.0 in /usr/lib/python2.7/site-packages (from tensorflow-gpu)
Collecting numpy>=1.11.0 (from tensorflow-gpu)
  Downloading numpy-1.12.0-cp27-cp27mu-manylinux1_x86_64.whl (16.5MB)
    100% |████████████████████████████████| 16.5MB 83kB/s 
Requirement already up-to-date: protobuf>=3.1.0 in /usr/lib64/python2.7/site-packages (from tensorflow-gpu)
Requirement already up-to-date: wheel in /usr/lib/python2.7/site-packages (from tensorflow-gpu)
Requirement already up-to-date: funcsigs>=1; python_version < "3.3" in /usr/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow-gpu)
Requirement already up-to-date: pbr>=0.11 in /usr/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow-gpu)
Collecting setuptools (from protobuf>=3.1.0->tensorflow-gpu)
  Downloading setuptools-34.3.1-py2.py3-none-any.whl (389kB)
    100% |████████████████████████████████| 399kB 637kB/s 
Collecting appdirs>=1.4.0 (from setuptools->protobuf>=3.1.0->tensorflow-gpu)
  Downloading appdirs-1.4.3-py2.py3-none-any.whl
Collecting packaging>=16.8 (from setuptools->protobuf>=3.1.0->tensorflow-gpu)
  Downloading packaging-16.8-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools->protobuf>=3.1.0->tensorflow-gpu)
  Downloading pyparsing-2.2.0-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 577kB/s 
Installing collected packages: numpy, tensorflow-gpu, appdirs, pyparsing, packaging, setuptools
  Found existing installation: numpy 1.11.2
    Uninstalling numpy-1.11.2:
      Successfully uninstalled numpy-1.11.2
  Found existing installation: setuptools 25.1.1
    Uninstalling setuptools-25.1.1:
      Successfully uninstalled setuptools-25.1.1
Successfully installed appdirs-1.4.3 numpy-1.12.0 packaging-16.8 pyparsing-2.2.0 setuptools-34.3.1 tensorflow-gpu-1.0.1
I got errors when I try to run this python module (libcudart.so.8.0).
I have a Intel I5 CPU with a video card without CUDA features.
    _mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/get_started/os_setup.md#import_error

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
So I used this command to fix with the pip upgrade:
[root@localhost ~]# export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.11.0rc0-cp27-none-linux_x86_64.whl
[root@localhost ~]# pip install --upgrade $TF_BINARY_URL
Collecting tensorflow==0.11.0rc0 from https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.11.0rc0-cp27-none-linux_x86_64.whl
  Downloading https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.11.0rc0-cp27-none-linux_x86_64.whl (39.7MB)
    100% |████████████████████████████████| 39.8MB 37kB/s 
Requirement already up-to-date: mock>=2.0.0 in /usr/lib/python2.7/site-packages (from tensorflow==0.11.0rc0)
Requirement already up-to-date: six>=1.10.0 in /usr/lib/python2.7/site-packages (from tensorflow==0.11.0rc0)
Requirement already up-to-date: numpy>=1.11.0 in /usr/lib64/python2.7/site-packages (from tensorflow==0.11.0rc0)
Collecting protobuf==3.0.0 (from tensorflow==0.11.0rc0)
  Downloading protobuf-3.0.0-cp27-cp27mu-manylinux1_x86_64.whl (5.2MB)
    100% |████████████████████████████████| 5.2MB 206kB/s 
Requirement already up-to-date: wheel in /usr/lib/python2.7/site-packages (from tensorflow==0.11.0rc0)
Requirement already up-to-date: funcsigs>=1; python_version < "3.3" in /usr/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow==0.11.0rc0)
Requirement already up-to-date: pbr>=0.11 in /usr/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow==0.11.0rc0)
Requirement already up-to-date: setuptools in /usr/lib/python2.7/site-packages (from protobuf==3.0.0->tensorflow==0.11.0rc0)
Requirement already up-to-date: appdirs>=1.4.0 in /usr/lib/python2.7/site-packages (from setuptools->protobuf==3.0.0->tensorflow==0.11.0rc0)
Requirement already up-to-date: packaging>=16.8 in /usr/lib/python2.7/site-packages (from setuptools->protobuf==3.0.0->tensorflow==0.11.0rc0)
Requirement already up-to-date: pyparsing in /usr/lib/python2.7/site-packages (from packaging>=16.8->setuptools->protobuf==3.0.0->tensorflow==0.11.0rc0)
Installing collected packages: protobuf, tensorflow
  Found existing installation: protobuf 3.2.0
    Uninstalling protobuf-3.2.0:
      Successfully uninstalled protobuf-3.2.0
  Found existing installation: tensorflow 1.0.1
    Uninstalling tensorflow-1.0.1:
      Successfully uninstalled tensorflow-1.0.1
Successfully installed protobuf-3.0.0 tensorflow-0.11.0rc0
The basic the python tensorflow works, so I need to test.

import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))

Hello, TensorFlow!

Monday, March 6, 2017

The pattern python module - part 001.

This is a very short presentation of pattern python module.
This python module is full of options and features.
I will try to show you some parts useful for most python users.
About pattern python module:
Pattern is a web mining module for the Python programming language.
It has tools for data mining (Google, Twitter and Wikipedia API, a web crawler, a HTML DOM parser), natural language processing (part-of-speech taggers, n-gram search, sentiment analysis, WordNet), machine learning (vector space model, clustering, SVM), network analysis and visualization.
Pattern developer documentation
ModuleFunctionality
pattern.web Asynchronous requests, web services, web crawler, HTML DOM parser.
pattern.db Wrappers for databases (MySQL, SQLite) and CSV-files.
pattern.text Base classes for parsers, parse trees and sentiment analysis.
pattern.search Pattern matching algorithm for parsed text (syntax & semantics).
pattern.vector Vector space model, clustering, classification.
pattern.graph Graph analysis & visualization.

I used with Fedora linux and you can see the instalation of this python module:
[root@localhost ~]# pip install pattern
Collecting pattern
  Downloading pattern-2.6.zip (24.6MB)
    100% |████████████████████████████████| 24.6MB 61kB/s 
Installing collected packages: pattern
  Running setup.py install for pattern ... done
Successfully installed pattern-2.6

Frequently used single character variable names:
Variable Meaning Example
a array, all a = [normalize(w) for w in words]
b boolean while b is False:
d distance, document d = distance(v1, v2)
e element e = html.find('#nav')
f file, filter, function f = open('data.csv', 'r')
i index for i in range(len(matrix)):
j index for j in range(len(matrix[i])):
k key for k in vector.keys():
n list length n = len(a)
p parser, pattern p = pattern.search.compile('NN')
q query for r in twitter.search(q):
r result, row for r in csv('data.csv):
s string s = s.decode('utf-8').strip()
t time t = time.time() - t0
v value, vector for k, v in vector.items():
w word for i, w in enumerate(sentence.words):
x horizontal position node.x = 0
y vertical position node.y = 0
Pattern contains part-of-speech taggers for a number of languages (including English, Spanish, German, French and Dutch). Part-of-speech tagging is useful in many data mining tasks. A part-of-speech tagger takes a string of text and identifies the sentences and the words in the text along with their word type. 


LanguageCode Speakers Example countries
Spanish es 350M Argentina (40), Colombia (40), Mexico (100), Spain (45)
English en 340M Canada (30), United Kingdom (60), United States (300)
German de 100M Austria (10), Germany (80), Switzerland (7)
French fr 70M France (65), Côte d'Ivoire (20)
Italian it 60M Italy (60)
Dutch nl 27M The Netherlands (25), Belgium (6), Suriname (1)
import pattern.en  
import pattern.es
import pattern.du  
import pattern.de
You can deal with many websites, see examples:
from pattern.web import Wikipedia
from pattern.web import Yahoo
from pattern.web import Twitter
from pattern.web import Facebook
from pattern.web import Flickr
from pattern.web import GMAIL
from pattern.web import GOOGLE
Now, about pattern.db.
The pattern.db module contains wrappers for databases (SQLite, MySQL), Unicode CSV files and Python's datetime. It offers a convenient way to work with tabular data, for example retrieved with the pattern.web module.
import pattern 
from pattern.db import Database, field, pk, STRING, BOOLEAN, DATE, NOW 
db = Database('people')
db.create('area_people',fields=(
pk(),
field('name', STRING(80), index=True),
field('type', STRING(20)),
field('date_birth', DATE, default=None),
field('date_created', DATE, default=NOW)
))
db.area_people.append(name=u'George', type='male')
1
print db.area_people.rows()[0]
(1, u'George', u'male', None, Date('2017-03-06 22:38:13'))

Saturday, March 4, 2017

Working with datetime python module.

This module is very good and I worked with this issue by using MySQL and python.
The version of python I used is 2.7 under Fedora distro.
You can take a look at the official page.
I use the pip and not the DNF fedora Linux tool.
 
[root@localhost lucru]# pip install datetime
Collecting datetime
Downloading DateTime-4.1.1.zip (66kB)
100% |████████████████████████████████| 71kB 703kB/s 
Collecting zope.interface (from datetime)
Downloading zope.interface-4.3.3.tar.gz (150kB)
100% |████████████████████████████████| 153kB 2.2MB/s 
Collecting pytz (from datetime)
Downloading pytz-2016.10-py2.py3-none-any.whl (483kB)
100% |████████████████████████████████| 491kB 2.4MB/s 
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from zope.interface->datetime)
Installing collected packages: zope.interface, pytz, datetime
Running setup.py install for zope.interface ... done
Running setup.py install for datetime ... done
Successfully installed datetime-4.1.1 pytz-2016.10 zope.interface-4.3.3

I solve this problem:
  • conversion using the lambda function
    parser.add_argument('date', type=lambda s: datetime.datetime.strptime(s, '%Y-%m-%d'))
  • solve last day
    datetime.datetime.strptime(new_value, '%Y-%m-%d %H:%M:%S')-timedelta(days=1)
  • print the today date
    print date.today()
  • show date using an explicit format string
    today=date.today()
    today.strftime("%A %d. %B %Y")
    'Sunday 05. March 2017'
    
  • using epoch issue [1]
    from datetime import datetime
    now_epoch = (datetime.utcnow() - datetime(1970, 1, 1)).total_seconds()
    datetime.utcfromtimestamp(now_epoch)
    datetime.datetime(2017, 3, 4, 22, 35, 13, 463409)
    datetime.fromtimestamp(now_epoch)
    datetime.datetime(2017, 3, 5, 0, 35, 13, 463409)
    import pytz
    datetime.fromtimestamp(now_epoch, pytz.utc)
    datetime.datetime(2017, 3, 4, 22, 35, 13, 463409, tzinfo=)
    
[1] The Unix epoch is the time 00:00:00 UTC on 1 January 1970. There is a problem with this definition, in that UTC did not exist in its current form until 1972;

Using pygeoip and maxmin database.


I try to locate one IP using the databases from maxmind website and is not good for me.
The database records show me the output from country area.
I read the docs from here.
This is the python script I used:
#wget -N -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
import pygeoip 
gip = pygeoip.GeoIP('GeoLiteCity.dat')
rec = gip.record_by_addr('___________________')
for key,val in rec.items():
    print "%s: %s" %(key,val)

Saturday, February 25, 2017

Linux: OpenCV and using Lucas-Kanade Optical Flow function.

Fist I install OpenCV python module and I try using with Fedora 25.
I used python 2.7 version.
[root@localhost mythcat]# dnf install opencv-python.x86_64 
Last metadata expiration check: 0:21:12 ago on Sat Feb 25 23:26:59 2017.
Dependencies resolved.
================================================================================
 Package              Arch          Version                Repository      Size
================================================================================
Installing:
 opencv               x86_64        3.1.0-8.fc25           fedora         1.8 M
 opencv-python        x86_64        3.1.0-8.fc25           fedora         376 k
 python2-nose         noarch        1.3.7-11.fc25          updates        266 k
 python2-numpy        x86_64        1:1.11.2-1.fc25        fedora         3.2 M

Transaction Summary
================================================================================
Install  4 Packages

Total download size: 5.6 M
Installed size: 29 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): opencv-python-3.1.0-8.fc25.x86_64.rpm    855 kB/s | 376 kB     00:00    
(2/4): opencv-3.1.0-8.fc25.x86_64.rpm           1.9 MB/s | 1.8 MB     00:00    
(3/4): python2-nose-1.3.7-11.fc25.noarch.rpm    543 kB/s | 266 kB     00:00    
(4/4): python2-numpy-1.11.2-1.fc25.x86_64.rpm   2.8 MB/s | 3.2 MB     00:01    
--------------------------------------------------------------------------------
Total                                           1.8 MB/s | 5.6 MB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Installing  : python2-nose-1.3.7-11.fc25.noarch                           1/4 
  Installing  : python2-numpy-1:1.11.2-1.fc25.x86_64                        2/4 
  Installing  : opencv-3.1.0-8.fc25.x86_64                                  3/4 
  Installing  : opencv-python-3.1.0-8.fc25.x86_64                           4/4 
  Verifying   : opencv-python-3.1.0-8.fc25.x86_64                           1/4 
  Verifying   : opencv-3.1.0-8.fc25.x86_64                                  2/4 
  Verifying   : python2-numpy-1:1.11.2-1.fc25.x86_64                        3/4 
  Verifying   : python2-nose-1.3.7-11.fc25.noarch                           4/4 

Installed:
  opencv.x86_64 3.1.0-8.fc25            opencv-python.x86_64 3.1.0-8.fc25       
  python2-nose.noarch 1.3.7-11.fc25     python2-numpy.x86_64 1:1.11.2-1.fc25    

Complete!
[root@localhost mythcat]# 
This is my test script with opencv to detect flow using Lucas-Kanade Optical Flow function.
This tracks some points in a black and white video.
First you need:
- one black and white video;
- not mp4 file type file;
- the color args need to be under 4 ( see is 3);
- I used this video:
I used cv2.goodFeaturesToTrack().
We take the first frame, detect some Shi-Tomasi corner points in it, then we iteratively track those points using Lucas-Kanade optical flow.
The function cv2.calcOpticalFlowPyrLK() we pass the previous frame, previous points and next frame.
The returns next points along with some status numbers which has a value of 1 if next point is found, else zero.
That iteratively pass these next points as previous points in next step.
See the code below:
import numpy as np
import cv2

cap = cv2.VideoCapture('candle')

# params for ShiTomasi corner detection
feature_params = dict( maxCorners = 77,
                       qualityLevel = 0.3,
                       minDistance = 7,
                       blockSize = 7 )

# Parameters for lucas kanade optical flow
lk_params = dict( winSize  = (17,17),
                  maxLevel = 1,
                  criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))

# Create some random colors
color = np.random.randint(0,255,(100,3))

# Take first frame and find corners in it
ret, old_frame = cap.read()
old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
p0 = cv2.goodFeaturesToTrack(old_gray, mask = None, **feature_params)

# Create a mask image for drawing purposes
mask = np.zeros_like(old_frame)

while(1):
    ret,frame = cap.read()
    frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # calculate optical flow
    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)

    # Select good points
    good_new = p1[st==1]
    good_old = p0[st==1]

    # draw the tracks
    for i,(new,old) in enumerate(zip(good_new,good_old)):
        a,b = new.ravel()
        c,d = old.ravel()
        mask = cv2.line(mask, (a,b),(c,d), color[i].tolist(), 2)
        frame = cv2.circle(frame,(a,b),5,color[i].tolist(),-1)
    img = cv2.add(frame,mask)

    cv2.imshow('frame',img)
    k = cv2.waitKey(30) & 0xff
    if k == 27:
        break

    # Now update the previous frame and previous points
    old_gray = frame_gray.copy()
    p0 = good_new.reshape(-1,1,2)

cv2.destroyAllWindows()
cap.release()
The output of this file is:

Thursday, February 23, 2017

The bad and good urllib.

This is a simple python script:
import urllib
opener = urllib.FancyURLopener({})
f = opener.open("http://www.ra___aer.ro/")
d=f.read()
fo = open('workfile.txt', 'w')
fo.write(d)
fo.close()
The really bad news comes from here:
http://blog.blindspotsecurity.com/2017/02/advisory-javapython-ftp-injections.html

Wednesday, February 22, 2017

The twill python module with Fedora 25.

Today I tested the twill python module with python 2.7 and Fedora 25.
This is: a scripting system for automating Web browsing. Useful for testing Web pages or grabbing data from password-protected sites automatically.
To install this python module I used pip command:
[root@localhost mythcat]# pip install twill
Collecting twill
Downloading twill-1.8.0.tar.gz (176kB)
100% |████████████████████████████████| 184kB 2.5MB/s
Installing collected packages: twill
Running setup.py install for twill ... done
Successfully installed twill-1.8.0

Let's try some tests:
[mythcat@localhost ~]$ python
Python 2.7.13 (default, Jan 12 2017, 17:59:37) 
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from twill import get_browser
>>> b = get_browser()
>>> 
>>> from twill.commands import *
>>> go("http://www.python.org/")
==> at https://www.python.org/
u'https://www.python.org/'
>>> b.showforms()

Form #1
## ## __Name__________________ __Type___ __ID________ __Value__________________
1     q                        search    id-searc ...   
To talk to the Web browser directly, call the get_browser function.
You can see most of the twill commands by using:
>>> import twill.shell
>>> twill.shell.main()

 -= Welcome to twill! =-

current page: https://www.python.org/widgets
>> ?

Undocumented commands:
======================
add_auth             fa           info             save_html           title
add_extra_header     find         load_cookies     setglobal           url  
agent                follow       notfind          setlocal          
back                 formaction   redirect_error   show              
clear_cookies        formclear    redirect_output  show_cookies      
clear_extra_headers  formfile     reload           show_extra_headers
code                 formvalue    reset_browser    showforms         
config               fv           reset_error      showhistory       
debug                get_browser  reset_output     showlinks         
echo                 getinput     run              sleep             
exit                 getpassword  runfile          submit            
extend_with          go           save_cookies     tidy_ok           

current page: https://www.python.org/widgets
>> 
Basic is used by setlocal to fill website forms and the go function.
Ban can be very good for some tasks.
The twill python module also provides a simple wrapper for mechanizing functionality with the API is still unstable.

Thursday, February 16, 2017

Compare two images: the histogram method.

This is a very simple example about how to compare the histograms of both images and print the inconsistencies are bound to arise.
The example come with alternative solution: Histogram method.
The script was run under Fedora 25.
If the images are the same the result will be 0.0.
For testing I change the image2.png by make a line into this with a coverage of 10%.
The result of the script was:
1116.63243729
The images come with this dimensions: 738 x 502 px.
import math
import operator
from math import *
import PIL

from PIL import Image
h1 = Image.open("image1.png").histogram()
h2 = Image.open("image2.png").histogram()

rms = math.sqrt(reduce(operator.add,
        map(lambda a,b: (a-b)**2, h1, h2))/len(h1))
print rms
About the operator module exports a set of efficient functions corresponding to the intrinsic operators of Python.
Example:
operator.lt(a, b)
operator.le(a, b)
operator.eq(a, b)
operator.ne(a, b)
operator.ge(a, b)
operator.gt(a, b)
operator.__lt__(a, b)
operator.__le__(a, b)
operator.__eq__(a, b)
operator.__ne__(a, b)
operator.__ge__(a, b)
operator.__gt__(a, b)

This is like math operators:
lt(a, b) is equivalent to a < b
le(a, b) is equivalent to a <= b
Another example:
>>> # Elementwise multiplication
>>> map(mul, [0, 1, 2, 3], [10, 20, 30, 40])
[0, 20, 60, 120]

>>> # Dot product
>>> sum(map(mul, [0, 1, 2, 3], [10, 20, 30, 40]))
200

Thursday, January 26, 2017

The kivy pyhon module for android.

First you need to install kivy python module.
The kivy python module
C:\>cd Python27

C:\Python27>cd Scripts

C:\Python27\Scripts>pip install kivy
Collecting kivy
Downloading Kivy-1.9.1-cp27-none-win32.whl (7.4MB)
100% |################################| 7.4MB 50kB/s
Collecting Kivy-Garden>=0.1.4 (from kivy)
Downloading kivy-garden-0.1.4.tar.gz
Requirement already satisfied: requests in c:\python27\lib\site-packages (from Kivy-Garden>=0.1.4->kivy)
Installing collected packages: Kivy-Garden, kivy
Running setup.py install for Kivy-Garden ... done
Successfully installed Kivy-Garden-0.1.4 kivy-1.9.1

Use this to add new pthon modules:
python -m pip install --upgrade docutils pygments pypiwin32 kivy.deps.sdl2 kivy.deps.glew kivy.deps.gstreamer --extra-index-url https://kivy.org/downloads/packages/simple/
If the python kivy.deps.gstreamer don't working then is not problem will try without this python module.
To see outdated python modules , use this commnds:
python -m pip install --upgrade pip wheel setuptools
pip list outdated
Let's upgrade all of python modules with this:
pip freeze > requirements.txt && pip install --upgrade -r requirements.txt && del requirements.txt
Another way to install this python module can be found here.
The demo can be star with this comman line:
C:\Python27>python share\kivy-examples\demo\showcase\main.py
You can also test one simple python script:
from kivy.app import App
from kivy.uix.button import Button

class TestApp(App):
def build(self):
return Button(text='Hello World')

TestApp().run()

To use with android operating system then test Buildozer.
This tool supports packaging for Android via the python-for-android project, and for iOS via the kivy-ios project. Support for other operating systems is intended in the future.

Friday, December 30, 2016

The python arch module for financial econometrics.

This python module arch: is a work-in-progress for ARCH and other tools for financial econometrics, written in Python (and Cython).
The arch python module come with tools for:
  •    Univariate volatility models
  •    Bootstrapping
  •    Multiple comparison procedures
  •    Unit root tests
You can read and see many examples here.
Let's start with instalation into my python 2.7.12 version.
First you need to install this python module with pip tool:
C:\Python27\Scripts>pip install Arch
Collecting Arch
Downloading arch-4.0.tar.gz (107kB)
100% |################################| 112kB 390kB/s
Requirement already satisfied: matplotlib>=1.4 in c:\python27\lib\site-packages (from Arch)
Requirement already satisfied: scipy>=0.15 in c:\python27\lib\site-packages (from Arch)
Collecting patsy>=0.2 (from Arch)
Downloading patsy-0.4.1-py2.py3-none-any.whl (233kB)
100% |################################| 235kB 906kB/s
Collecting statsmodels>=0.6 (from Arch)
Downloading statsmodels-0.6.1.tar.gz (7.0MB)
100% |################################| 7.0MB 85kB/s
Collecting pandas>=0.16 (from Arch)
Downloading pandas-0.19.2-cp27-cp27m-win32.whl (6.8MB)
100% |################################| 6.8MB 81kB/s
Requirement already satisfied: numpy>=1.6 in c:\python27\lib\site-packages (from matplotlib>=1.4->Arch)
Requirement already satisfied: python-dateutil in c:\python27\lib\site-packages (from matplotlib>=1.4->Arch)
Requirement already satisfied: cycler in c:\python27\lib\site-packages (from matplotlib>=1.4->Arch)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,>=1.5.6 in c:\python27\lib\site-packages (from matplotlib>=1.4->Arch)
Requirement already satisfied: pytz in c:\python27\lib\site-packages (from matplotlib>=1.4->Arch)
Requirement already satisfied: six in c:\python27\lib\site-packages (from patsy>=0.2->Arch)
Installing collected packages: patsy, pandas, statsmodels, Arch
Running setup.py install for statsmodels ... done
Running setup.py install for Arch ... done
Successfully installed Arch-4.0 pandas-0.19.2 patsy-0.4.1 statsmodels-0.6.1

This python module is very mathematical and technical so I'll refer you to a few examples.
Can be used in statistical research and econometrics, or the application of mathematics, statistics, and computer science to economic data.