analitics

Pages

Saturday, November 2, 2019

Python 3.7.5 : Intro about scikit-learn python module.

This python module named scikit-learn used like sklearn is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy and comes with various classification, regression and clustering algorithms including support vector machines, random forests, gradient boosting, k-means and DBSCAN.
The official webpage can be found here
Let't install this on my Fedora 30 distro:
[mythcat@desk proiecte_github]$ mkdir sklearn_examples
[mythcat@desk proiecte_github]$ cd sklearn_examples/
[mythcat@desk sklearn_examples]$ pip3 install scikit-learn --user
Python 3.7.5 (default, Oct 17 2019, 12:09:47) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sklearn
>>> print('sklearn: %s' % sklearn.__version__)
sklearn: 0.21.3 

First, this is a complex python module with many examples on web.
You can learn much about how can use simple and efficient all data mining and data analysis.
You can learn a lot about how all data exploitation and data analysis can be used simply and efficiently.
Mathematical functions are simple and complex. How to use python programming and existing examples can be used in several learning points.
I would start with discovering the input and output data sets and then continue with clear examples used daily by us.
I tested today with SVC and sklearn python module.
The SVMs were introduced initially in the 1960s and were later refined in the 1990s.
The base of this algorithm is the decision boundary that maximizes the distance from the nearest data points of all the classes.
The wikipedia article show all informations about support-vector machines (named SVM).
As applications we can use this function in: medical field for cell counting or similar cell quantification, astronomy, etc.
This simple example use multiple kernels and gammas parameters to group the input data.
import numpy as np
from sklearn.datasets import make_blobs
from sklearn import svm
from sklearn.svm import SVC
# importing scikit learn with make_blobs 
from sklearn.datasets.samples_generator import make_blobs 
# 
import matplotlib.pyplot as plt

import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets

# import some data to play with
iris = datasets.load_iris()
x = iris.data[:, :2]
y = iris.target

def plotSVC(title):
  # create a mesh to plot with dataset x and y
  x_min, x_max = x[:, 0].min() - 1, x[:, 0].max() + 1
  y_min, y_max = x[:, 1].min() - 1, x[:, 1].max() + 1
  # set the resolution by 100 
  h = (x_max / x_min)/100
  # create the meshgrid 
  xx, yy = np.meshgrid(np.arange(x_min, x_max, h),np.arange(y_min, y_max, h))
  # divides the current figure into an m-by-n grid and creates axes in the position specified by p
  plt.subplot(1, 1, 1)
  # the model can then be used to predict new values
  Z = svc.predict(np.c_[xx.ravel(), yy.ravel()])
  # reshape your test data because prediction needs an array that looks like your training data
  Z = Z.reshape(xx.shape)
  # use plt to show result 
  plt.contourf(xx, yy, Z, cmap=plt.cm.Paired, alpha=0.8) 
  plt.scatter(x[:, 0], x[:, 1], c=y, cmap=plt.cm.Paired)
  plt.xlabel('x length')
  plt.ylabel('y width')
  plt.xlim(xx.min(), xx.max())
  plt.title("Plot SVC")
  plt.show()

# create kernels for svg 
kernels = ['linear', 'rbf', 'poly']
# for each kernel show graphs
for kernel in kernels:
  svc = svm.SVC(kernel=kernel).fit(x, y)
  plotSVC('kernel=' + str(kernel))
# create gammas values
# the gamma parameter defines how far the influence of a single training example reaches
gammas = [0.1, 1, 10, 100, 1000]
# for each gammas and kernel rbf - fast processing,  show graphs
for gamma in gammas:
   svc = svm.SVC(kernel='rbf', gamma=gamma).fit(x, y)
   plotSVC('gamma=' + str(gamma))

See the last result for kernel rbf and gamma 1000.

Python 3.7.5 : The ani script with ascii.

ASCII, abbreviated from American Standard Code for Information Interchange, is a character encoding standard for electronic communication. ASCII codes represent text in computers, telecommunications equipment, and other devices. see Wikipedia.
This is a simple script named ani.py created by me to show an animation with ASCII ...
import os, time
os.system('cls')
filenames = ["0.txt","1.txt","2.txt","3.txt"]
frames = []
for name in filenames:
    with open (name, "r", encoding="utf8") as f:
        frames.append(f.readlines())
"""
for frame in frames:
    print("".join(frame))
    time.sleep(1)
    os.system('clear')
"""
for i in range (4):
    os.system('clear')
    for frame in frames:
        print("".join(frame))
        time.sleep(1)
        os.system('clear')
You need four text files with an 8X8 character matrix format: 0.txt , 1.txt , 2.txt and 3.txt.
The content of these files:
$ cat *.txt
        
 ###### 
        
        
        
        
 ###### 
                 
        
 ###### 
        
        
 ###### 
        
                 
        
        
  ####  
  ####  
        
        
                 
        
        
   ##   
   ##   
        
        
The end result is a square that shrinks to 4 characters #.

Saturday, October 26, 2019

Python 3.7.4 : About with the PyOpenCL python module.

PyOpenCL lets you access GPUs and other massively parallel compute devices from Python.
It is important to note that OpenCL is not restricted to GPUs.
In fact, no special hardware is required to use OpenCL for computation–your existing CPU is enough.
The documentation of this project can be found at this website.
Let's install the python module for python 3 version:
[mythcat@desk ~]$ pip3 install pyopencl --user
Collecting pyopencl
...
Successfully built pytools
Installing collected packages: pytools, pyopencl
Successfully installed pyopencl-2019.1.1 pytools-2019.1.1
The install of OpenCL driver can be done with these commands:
# get OpenCL driver automated installer (installs kernel 4.7)
curl https://software.intel.com/sites/default/files/managed/f6/77/install_OCL_driver.sh_.txt > install_OCL\
_driver.sh
chmod +x install_OCL_driver.sh
# install OpenCL driver
sudo ./install_OCL_driver.sh install
# check
ls /boot/vmlinuz-*intel*
This is a simple python script to test the opencl context:
import pyopencl as cl
import numpy as np
ctx = cl.create_some_context()
# cet platforms, both CPU and GPU
my_plat= cl.get_platforms()
CPU = my_plat[0].get_devices()
try:
    GPU = my_plat[1].get_devices()
except IndexError:
    GPU = "none"
# create context for GPU/CPU
if GPU != "none":
    ctx = cl.Context(GPU)
else:
    ctx = cl.Context(CPU)
# create queue for each kernel execution
queue = cl.CommandQueue(ctx)
mf = cl.mem_flags
This is another simple python script:
# -*- coding: utf-8 -*-
import pyopencl as cl 
import numpy
a = numpy.random.rand(50000).astype(numpy.float32)
ctx = cl.create_some_context()
queue = cl.CommandQueue(ctx)
a_buf = cl.Buffer(ctx ,cl.mem_flags.READ_WRITE,size=a.nbytes)
cl.enqueue_write_buffer(queue, a_buf , a)

prg= cl.Program(ctx,
"""
__kernel void twice(__global float ∗a)
{
 int gid=get_global_id(0);
 a[gid] ∗= 2;
}"""
). build()

prg.twice(queue, a.shape, None,a_buf ).wait()

Sunday, October 20, 2019

Python 3.7.4 : Usinge pytesseract for text recognition.

About this python module named tesseract, you can read here.
I tested with the tesseract tool install on my Fedora 30 distro and python module pytesseract version 0.3.0.
[root@desk mythcat]# dnf install tesseract
Last metadata expiration check: 0:24:18 ago on Sun 20 Oct 2019 10:56:23 AM EEST.
Package tesseract-4.1.0-1.fc30.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[root@desk mythcat]# whereis tesseract
tesseract: /usr/bin/tesseract /usr/share/tesseract
[mythcat@desk ~]$ pip3 install pytesseract --user
Collecting pytesseract
...
Installing collected packages: pytesseract
Successfully installed pytesseract-0.3.0
I test with many images and texts and works very well.
Text images with a printed font are very well recognized.
This test with this image does not have very good accuracy.

The result of the handwriting image.
[mythcat@desk ~]$ python3 ocr_image.py 001.png 
rake Yous mnislakes,
take you chances,
look silby,

bul hep. mv going
dont freeze up

Wednesday, October 16, 2019

Python 3.7.4 : Test the DHCP handshakes.

First, the DHCP is based on the earlier BOOTP protocol which uses well-known port numbers for both server and client instead of an ephemeral port. The server and the client communicate via broadcast and the server broadcasts the offered IP address to the client on UDP port 68.
This python example has a learning purpose and does not harm anyone.
import subprocess as sub
import re

def find_word(w):
    return re.compile(r'\b({0})\b'.format(w), flags=re.IGNORECASE).search

p = sub.Popen(('sudo', 'tcpdump', '-l', '-s 0', '-vvv', '-n', '((udp port 67) and (udp[8:1] = 0x1))'),
 stdout=sub.PIPE)
for row in iter(p.stdout.readline, b''):
    if find_word(row):
        print (row.split(' ')[-1])
    elif find_word(row):
        print (row.split(' ')[-1])
The result of my script ( I don't have inputs on this port).
[mythcat@desk scripts]$ python3 dhcpreq.py 
tcpdump: listening on ___, link-type EN10MB (Ethernet), capture size 262144 bytes
^CTraceback (most recent call last):
  File "dhcpreq.py", line 10, in 
    for row in iter(p.stdout.readline, b''):
KeyboardInterrupt
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[mythcat@desk scripts]$ vim dhcpreq.py 

Tuesday, October 15, 2019

Python 3.8.0 : New release of python development.

Good news from the python development area with the new release of python development:
Python 3.7.5 Oct. 15, 2019 and Python 3.8.0 Oct. 14, 2019

Now you can use the new python version 3.8.0 from the official webpage.

Major new features of the 3.8 series, compared to 3.7 - release Date: Oct. 14, 2019:
  • PEP 572, Assignment expressions
  • PEP 570, Positional-only arguments
  • PEP 587, Python Initialization Configuration (improved embedding)
  • PEP 590, Vectorcall: a fast calling protocol for CPython
  • PEP 578, Runtime audit hooks
  • PEP 574, Pickle protocol 5 with out-of-band data
  • Typing-related: PEP 591 (Final qualifier), PEP 586 (Literal types), and PEP 589 (TypedDict)
  • Parallel filesystem cache for compiled bytecode
  • Debug builds share ABI as release builds
  • f-strings support a handy = specifier for debugging
  • continue is now legal in finally: blocks
  • on Windows, the default asyncio event loop is now ProactorEventLoop
  • on macOS, the spawn start method is now used by default in multiprocessing
  • multiprocessing can now use shared memory segments to avoid pickling costs between processes
  • typed_ast is merged back to CPython
  • LOAD_GLOBAL is now 40% faster
  • pickle now uses Protocol 4 by default, improving performance

Let's install on Fedora 30 Linux distro:
[mythcat@desk ~]$ cd Python-3.8.0/
[mythcat@desk Python-3.8.0]$ ls
aclocal.m4          Doc         m4               Parser         README.rst
CODE_OF_CONDUCT.md  Grammar     Mac              PC             setup.py
config.guess        Include     Makefile.pre.in  PCbuild        Tools
config.sub          install-sh  Misc             Programs
configure           Lib         Modules          pyconfig.h.in
configure.ac        LICENSE     Objects          Python
[mythcat@desk Python-3.8.0]$ ./configure 
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for python3.8... no
...
creating Makefile

If you want a release build with all stable optimizations active (PGO, etc),
please run ./configure --enable-optimizations
If you want then you can run the tool to prepare the build with optimizations:
[mythcat@desk Python-3.8.0]$ ./configure --enable-optimizations --with-ensurepip=install
...
creating Modules/Setup.local
creating Makefile
Use the make with the -j option to use building into parallel steps to speed up the compilation.
[mythcat@desk Python-3.8.0]$ make -j 2
...
make[1]: Leaving directory '/home/mythcat/Python-3.8.0'
Since you’re installing Python into /usr/bin, you’ll need to run as root:
[mythcat@desk Python-3.8.0]$ sudo make altinstall
...
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-19.2.3 setuptools-41.2.0
Let's test it in this folder and with new python3.8 :
[mythcat@desk Python-3.8.0]$ ./python 
Python 3.8.0 (default, Oct 15 2019, 23:45:20) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
[mythcat@desk Python-3.8.0]$ python3.8
Python 3.8.0 (default, Oct 15 2019, 23:45:20) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Python 3.7.4 : Testing python source code with streamlit tool.

The official webpage for this python package can be found at streamlit.io.
Let's install it with pip3 tool:
[mythcat@desk proiecte_github]$ mkdir streamlit_examples
[mythcat@desk proiecte_github]$ cd streamlit_examples/
[mythcat@desk streamlit_examples]$ pip3 install streamlit --user
Let's try some examples.
Create a file named 001.py
This simple example will show a map with randoms spots:
import pandas as pd
import numpy as np
import streamlit as st    
df = pd.DataFrame(
    np.random.randn(100, 2) / [50, 50] + [47.45, 26.3],
    columns=['lat', 'lon'])
st.map(df)
Let's run it with this command:
[mythcat@desk streamlit_examples]$ streamlit run 001.py 

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
...
The next source code will show just the map because the df variable is empty:
import streamlit as st
df = []
st.deck_gl_chart(
    viewport={
        'latitude': 47.45,
        'longitude': 26.3,
        'zoom': 13,
        'pitch': 50,
    },
    layers=[{
        'type': 'HexagonLayer',
        'data': df,
        'radius': 200,
        'elevationScale': 4,
        'elevationRange': [0, 1000],
        'pickable': True,
        'extruded': True,
    }, {
        'type': 'ScatterplotLayer',
        'data': df,
    }])
The source code is added into another file named 002.py and can be run with this command:
[mythcat@desk streamlit_examples]$ streamlit run 002.py 

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
...
You can see more about this tool at the official youtube channel:




Thursday, October 10, 2019

Python 3.7.4 : Testing the PyUSB python module.

This python module named PyUSB can be found at pypi website.
[mythcat@desk scripts]$ pip3 install pyusb --user
Collecting pyusb
...
Successfully installed pyusb-1.0.2
Let' see some usb device with lsusb command:
[mythcat@desk scripts]$ lsusb
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 004: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 001 Device 003: ID 093a:2510 Pixart Imaging, Inc. Optical Mouse
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
First you need to set this to avoid the error: Access denied (insufficient permissions.
[mythcat@desk scripts]$ ll  /dev/bus/usb/001/004
crw-rw-r--. 1 root root 189, 3 Oct 10 20:34 /dev/bus/usb/001/004
[mythcat@desk scripts]$ chmod a+rw /dev/bus/usb/001/004
chmod: changing permissions of '/dev/bus/usb/001/004': Operation not permitted
[mythcat@desk scripts]$ sudo chmod a+rw /dev/bus/usb/001/004
[sudo] password for mythcat: 
[mythcat@desk scripts]$ ll  /dev/bus/usb/001/004
crw-rw-rw-. 1 root root 189, 3 Oct 10 20:34 /dev/bus/usb/001/004
The script is simple:
import sys
  
import usb.core
import usb.util
print(usb.__version__)
busses = usb.busses()
for bus in busses:
    devices = bus.devices
    for dev in devices:
        if dev != None:
            try:
                usd_dev = usb.core.find(idVendor=dev.idVendor, idProduct=dev.idProduct)
                print(usb_dev)
            except:
                pass
# 1a40:0101
dev  = usb.core.find(idVendor=0x1a40, idProduct=0x0101)
print ("The 8087:0024 is : ", dev)
if dev is None:
    raise ValueError("Device not found!")
else:

    if dev.is_kernel_driver_active(0):
        try:
                dev.detach_kernel_driver(0)
                print ("kernel driver detached")
        except usb.core.USBError as e:
                sys.exit("Could not detach kernel driver: %s" % str(e))
    else:
        print ("no kernel driver attached")
    try:
        usb.util.claim_interface(dev, 0)
        print ("claimed device")
    except:
        sys.exit("Could not claim the device: %s" % str(e))
    try:
        dev.set_configuration()
        dev.reset()
    except usb.core.USBError as e:
        sys.exit("Could not set configuration: %s" % str(e))

usb.util.release_interface(dev,interface)
dev.attach_kernel(interface
The result of this python script is this:
[mythcat@desk scripts]$ python3 usb_test.py 
1.0.2
The 8087:0024 is :  DEVICE ID 1a40:0101 on Bus 001 Address 004 =================
 bLength                :   0x12 (18 bytes)
 bDescriptorType        :    0x1 Device
 bcdUSB                 :  0x200 USB 2.0
 bDeviceClass           :    0x9 Hub
 bDeviceSubClass        :    0x0
 bDeviceProtocol        :    0x1
 bMaxPacketSize0        :   0x40 (64 bytes)
 idVendor               : 0x1a40
 idProduct              : 0x0101
 bcdDevice              :  0x111 Device 1.11
 iManufacturer          :    0x0 
 iProduct               :    0x1 USB 2.0 Hub
 iSerialNumber          :    0x0 
 bNumConfigurations     :    0x1
  CONFIGURATION 1: 100 mA ==================================
   bLength              :    0x9 (9 bytes)
   bDescriptorType      :    0x2 Configuration
   wTotalLength         :   0x19 (25 bytes)
   bNumInterfaces       :    0x1
   bConfigurationValue  :    0x1
   iConfiguration       :    0x0 
   bmAttributes         :   0xe0 Self Powered, Remote Wakeup
   bMaxPower            :   0x32 (100 mA)
    INTERFACE 0: Hub =======================================
     bLength            :    0x9 (9 bytes)
     bDescriptorType    :    0x4 Interface
     bInterfaceNumber   :    0x0
     bAlternateSetting  :    0x0
     bNumEndpoints      :    0x1
     bInterfaceClass    :    0x9 Hub
     bInterfaceSubClass :    0x0
     bInterfaceProtocol :    0x0
     iInterface         :    0x0 
      ENDPOINT 0x81: Interrupt IN ==========================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :   0x81 IN
       bmAttributes     :    0x3 Interrupt
       wMaxPacketSize   :    0x1 (1 bytes)
       bInterval        :    0xc
kernel driver detached
claimed device
Could not set configuration: [Errno 16] Resource busy
Set permisions for next usb:
[mythcat@desk scripts]$ ll /dev/bus/usb/001/003
crw-rw-rw-. 1 root root 189, 2 Oct 10 20:34 /dev/bus/usb/001/003
Te next source code will read the mouse device:
#!/usr/bin/python
import sys
import usb.core
import usb.util
# decimal vendor and product values
#dev = usb.core.find(idVendor=1118, idProduct=1917)
# or, uncomment the next line to search instead by the hexidecimal equivalent
# 093a:2510
dev = usb.core.find(idVendor=0x093a, idProduct=0x2510)
# first endpoint
interface = 0
endpoint = dev[0][(0,0)][0]
# if the OS kernel already claimed the device, which is most likely true
# thanks to http://stackoverflow.com/questions/8218683/pyusb-cannot-set-configuration
if dev.is_kernel_driver_active(interface) is True:
  # tell the kernel to detach
  dev.detach_kernel_driver(interface)
  # claim the device
  usb.util.claim_interface(dev, interface)
collected = 0
attempts = 50
while collected < attempts :
    try:
        data = dev.read(endpoint.bEndpointAddress,endpoint.wMaxPacketSize)
        collected += 1
        print (data)
    except usb.core.USBError as e:
        data = None
        if e.args == ('Operation timed out',):
            continue
# release the device
usb.util.release_interface(dev, interface)
# reattach the device to the OS kernel
dev.attach_kernel_driver(interface)
The output of mouse moves is this:
[mythcat@desk scripts]$ python3 usb_mouse.py 
[mythcat@desk scripts]$ python3 usb_mouse.py 
array('B', [0, 254, 255, 0])
array('B', [0, 253, 2, 0])
array('B', [0, 252, 3, 0])
array('B', [0, 251, 3, 0])
array('B', [0, 252, 3, 0])
array('B', [0, 254, 1, 0])
array('B', [0, 253, 2, 0])
array('B', [0, 255, 1, 0])
array('B', [0, 255, 4, 0])
array('B', [0, 0, 3, 0])
array('B', [0, 0, 3, 0])
array('B', [0, 0, 2, 0])
array('B', [0, 0, 2, 0])
array('B', [0, 2, 1, 0])
array('B', [0, 4, 1, 0])
array('B', [0, 3, 0, 0])
array('B', [0, 3, 0, 0])
array('B', [0, 1, 0, 0])

Monday, October 7, 2019

Python 3.7.4 : Example with subprocess - part 001.

This is a simple example with the python 3 subprocess package.
The source code is simple to understand.
The execute_proceess_with_communicate let run the ls command with the sudo user permissions:
import os
import sys
import string
import subprocess
import codecs

inp = ''
cmd = 'ls'
password = ''

def execute_proceess_with_communicate(inp):
    """Return a list of hops from traceroute command."""
    p = subprocess.Popen(
            ['sudo', cmd, inp],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            shell=False)
    text, _ = p.communicate(password)
    #print(type(text))
    outp = codecs.decode(text,'utf8')
    out_split=outp.split('\n')
    return out_split

def normalize_out(list_outp):
    """Extract information from traceroute line per line."""

    normalized_out = []
    for op in list_outp:
        # filer out if an empty line
        if len(op) is 0:
            continue
        op_split = op.split()
        normalized_out.append(op_split)
    return normalized_out

if __name__ == '__main__':
    inp = sys.argv[1]

    out = execute_proceess_with_communicate(inp)
    n_out = normalize_out(out)
    print(n_out)
The result is this:
[mythcat@desk scripts]$ python3 subprocess_001.py '/' 
[['bin'], ['boot'], ['dev'], ['etc'], ['home'], ['lib'], ['lib64'], ['media'], ['mnt'], ['opt'], ['proc'],
 ['root'], ['run'], ['sbin'], ['srv'], ['sys'], ['tmp'], ['usr'], ['var']]

Sunday, October 6, 2019

Python Qt5 : the drag and drop feature.

Today I tested drag and drop feature with PyQt5.
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
This is a simple example using setAcceptDrops and setDragEnabled:
import sys
from PyQt5.QtWidgets import QApplication, QWidget, QListWidget, QHBoxLayout,QListWidgetItem
from PyQt5.QtGui import QIcon

class Window(QWidget):
    def __init__(self):
        super().__init__()

        self.myListWidget1 = QListWidget()
        self.myListWidget2 = QListWidget()
        self.myListWidget2.setViewMode(QListWidget.IconMode)
        self.myListWidget1.setAcceptDrops(True)
        self.myListWidget1.setDragEnabled(True)
        self.myListWidget2.setAcceptDrops(True)
        self.myListWidget2.setDragEnabled(True)
        self.setGeometry(480, 400, 640, 480)
        self.myLayout = QHBoxLayout()
        self.myLayout.addWidget(self.myListWidget1)
        self.myLayout.addWidget(self.myListWidget2)

        l1 = QListWidgetItem(QIcon('house.png'), "House")
        l2 = QListWidgetItem(QIcon('cloud.png'), "Clouds ")
        l3 = QListWidgetItem(QIcon('user.png'), "User")
        l4 = QListWidgetItem(QIcon('save.png'), "Save")

        self.myListWidget1.insertItem(1, l1)
        self.myListWidget1.insertItem(2, l2)
        self.myListWidget1.insertItem(3, l3)
        self.myListWidget1.insertItem(4, l4)

        QListWidgetItem(QIcon('house.png'), "House", self.
                        myListWidget2)
        QListWidgetItem(QIcon('cloud.png'), "Clouds", self.
                        myListWidget2)
        QListWidgetItem(QIcon('save.png'), "Save", self.
                        myListWidget2)

        self.setWindowTitle('Example: Drag and Drop');
        self.setLayout(self.myLayout)

        self.show()

App = QApplication(sys.argv)
window = Window()
sys.exit(App.exec())
The result is a windows with two QListWidget with an drag and drop feature.

Wednesday, October 2, 2019

Python 3.7.4 : Using the paramiko pakage.

Today I tested the paramiko package.
First, I install and check the version of this package.
[mythcat@desk my_network_tools]$ pip3 install paramiko --user
Collecting paramiko
...
  Running setup.py install for pycparser ... done
Successfully installed asn1crypto-0.24.0 bcrypt-3.1.7 cffi-1.12.3 cryptography-2.7 paramiko-2.6.0
 pycparser-2.19 pynacl-1.3.0
[mythcat@desk my_network_tools]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
>>> dir(paramiko)
['AUTH_FAILED', 'AUTH_PARTIALLY_SUCCESSFUL', 'AUTH_SUCCESSFUL', 'Agent', 'AgentKey', 'AuthHandler',
 'AuthenticationException', 'AutoAddPolicy', 'BadAuthenticationType', 'BadHostKeyException', 'BaseSFTP',
 'BufferedFile', 'Channel', 'ChannelException', 'ChannelFile', 'ChannelStderrFile', 'ChannelStdinFile',
 'DSSKey', 'ECDSAKey', 'Ed25519Key', 'GSSAuth', 'GSS_AUTH_AVAILABLE', 'GSS_EXCEPTIONS', 'HostKeys',
 'InteractiveQuery', 'Message', 'MissingHostKeyPolicy', 'OPEN_FAILED_ADMINISTRATIVELY_PROHIBITED',
 'OPEN_FAILED_CONNECT_FAILED', 'OPEN_FAILED_RESOURCE_SHORTAGE', 'OPEN_FAILED_UNKNOWN_CHANNEL_TYPE',
 'OPEN_SUCCEEDED', 'PKey', 'Packetizer', 'PasswordRequiredException', 'ProxyCommand', 'ProxyCommandFailure',
 'PublicBlob', 'RSAKey', 'RejectPolicy', 'SFTP', 'SFTPAttributes', 'SFTPClient', 'SFTPError', 'SFTPFile',
 'SFTPHandle', 'SFTPServer', 'SFTPServerInterface', 'SFTP_BAD_MESSAGE', 'SFTP_CONNECTION_LOST', 'SFTP_EOF',
 'SFTP_FAILURE', 'SFTP_NO_CONNECTION', 'SFTP_NO_SUCH_FILE', 'SFTP_OK', 'SFTP_OP_UNSUPPORTED', 
'SFTP_PERMISSION_DENIED', 'SSHClient', 'SSHConfig', 'SSHException', 'SecurityOptions', 'ServerInterface',
 'SubsystemHandler', 'Transport', 'WarningPolicy', '__all__', '__author__', '__builtins__', '__cached__', 
'__doc__', '__file__', '__license__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 
'__version__', '__version_info__', '_version', 'agent', 'auth_handler', 'ber', 'buffered_pipe', 'channel',
 'client', 'common', 'compress', 'config', 'dsskey', 'ecdsakey', 'ed25519key', 'file', 'hostkeys', 
'io_sleep', 'kex_curve25519', 'kex_ecdh_nist', 'kex_gex', 'kex_group1', 'kex_group14', 'kex_group16',
 'kex_gss', 'message', 'packet', 'pipe', 'pkey', 'primes', 'proxy', 'py3compat', 'rsakey', 'server', 
'sftp', 'sftp_attr', 'sftp_client', 'sftp_file', 'sftp_handle', 'sftp_server', 'sftp_si', 'ssh_exception',
 'ssh_gss', 'sys', 'transport', 'util']
>>> paramiko.__version__
'2.6.0'
The documentation for this version can be found at this webpage.
A simple example with this python package and ssh connection to 192.168.0.143 on port 22 can be see on the next source code
#!/usr/bin/env python
"""Return the ssh output with password connection and Linux commands"""
import paramiko

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
conn= client.connect('192.168.0.143', port='22', username='mythcat', password='the_pass')
# Obtain session
session = client.get_transport().open_session()
print("| Retcode: "+str(session)+"|")

stdin, stdout, stderr=client.exec_command('sudo hostname;w')
save_stdout = stdout.readlines()
retcode = stdout.channel.recv_exit_status()

stdin, stdout, stderr=client.exec_command('ss -nap;')
save_stdout2 = stdout.readlines()
print(save_stdout,save_stdout2)
#for line in stdout:
#    print (line)

Tuesday, October 1, 2019

Python 3.7.4 : Install the protobuf from sources on Fedora distro.

Today I will show you how to build protobuf from sources using the Fedora distro.
The google team comes with this intro:
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler...
This google project comes with these tutorials.
The GitHub project can be found here.
To install the compiler, download the package.
[mythcat@desk ~]$ cd Downloads/
[mythcat@desk Downloads]$ cp protobuf-python-3.9.2.tar.gz ~/
[mythcat@desk Downloads]$ cd ..
[mythcat@desk ~]$ tar xvzf protobuf-python-3.9.2.tar.gz 
...
protobuf-3.9.2/aclocal.m4
protobuf-3.9.2/install-sh
protobuf-3.9.2/generate_descriptor_proto.sh
protobuf-3.9.2/CHANGES.txt
protobuf-3.9.2/configure.ac
protobuf-3.9.2/configure
Let's see the content:
[mythcat@desk ~]$ cd protobuf-3.9.2/
[mythcat@desk protobuf-3.9.2]$ ls
aclocal.m4                   config.sub                    ltmain.sh            README.md
ar-lib                       configure                     m4                   six.BUILD
autogen.sh                   configure.ac                  Makefile.am          src
benchmarks                   conformance                   Makefile.in          test-driver
BUILD                        CONTRIBUTORS.txt              missing              third_party
CHANGES.txt                  depcomp                       objectivec           update_file_lists.sh
cmake                        editors                       protobuf.bzl         util
compile                      examples                      protobuf_deps.bzl    WORKSPACE
compiler_config_setting.bzl  generate_descriptor_proto.sh  protobuf-lite.pc.in
config.guess                 install-sh                    protobuf.pc.in
config.h.in                  LICENSE                       python
[mythcat@desk protobuf-3.9.2]$ ./configure
...
checking how to run the C preprocessor... gcc -E
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/mythcat/protobuf-3.9.2':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details
...
[root@desk protobuf-3.9.2]# dnf install g++
...
Installed:
  gcc-c++-9.2.1-1.fc30.x86_64                                                                               

Complete![root@desk protobuf-3.9.2]# exit
exit
Let's build again:
[mythcat@desk protobuf-3.9.2]$ ./configure
checking whether to enable maintainer-specific portions of Makefiles... yes
checking build system type... x86_64-pc-linux-gnu
...
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/gmock-config
config.status: creating build-aux/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
[mythcat@desk protobuf-3.9.2]$ nproc --all
2
[mythcat@desk protobuf-3.9.2]$ make -j2
...
  CXXLD    protoc
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2'
[mythcat@desk protobuf-3.9.2]$ make check -j2
...PASS: protobuf-lazy-descriptor-test
PASS: protobuf-lite-test
PASS: google/protobuf/compiler/zip_output_unittest.sh
PASS: google/protobuf/io/gzip_stream_unittest.sh
PASS: protobuf-lite-arena-test
PASS: no-warning-test
PASS: protobuf-test
============================================================================
Testsuite summary for Protocol Buffers 3.9.2
============================================================================
# TOTAL: 7
# PASS:  7
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
The last steps is for install:
[mythcat@desk protobuf-3.9.2]$ sudo make install
[sudo] password for mythcat: 
...
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
[mythcat@desk protobuf-3.9.2]$ sudo ldconfig 
Let's test it:
[mythcat@desk protobuf-3.9.2]$ protoc --version
libprotoc 3.9.2
Now the next step comes for python module:
[mythcat@desk protobuf-3.9.2]$ cd python/
[mythcat@desk python]$ ls
google  MANIFEST.in  mox.py  README.md  release  release.sh  setup.cfg  setup.py  stubout.py  tox.ini
[mythcat@desk python]$ python setup.py build
...
testSerialize (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeMessageSetWireFormatUnknownExtension 
(google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeProto3 (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testByteSizeFunctions (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testPackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testUnpackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagDecode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagEncode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok

----------------------------------------------------------------------
Ran 818 tests in 5.343s

OK (skipped=10)
Let's test the python module:
[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import google
>>> from google.protobuf import descriptor as _descriptor
>>> from google.protobuf import message as _message
>>> from google.protobuf import reflection as _reflection
>>> from google.protobuf import symbol_database as _symbol_database
>>> from google.protobuf import descriptor_pb2
>>> from google.protobuf import text_format
>>> dir(_descriptor)
['Descriptor', 'DescriptorBase', 'DescriptorMetaclass', 'EnumDescriptor', 'EnumValueDescriptor', 
'Error', 'FieldDescriptor', 'FileDescriptor', 'MakeDescriptor', 'MethodDescriptor', 'OneofDescriptor', 
'ServiceDescriptor', 'TypeTransformationError', '_Lock', '_NestedDescriptorBase', '_OptionsOrNone', 
'_ParseOptions', '_ToCamelCase', '_ToJsonName', '_USE_C_DESCRIPTORS', '__author__', '__builtins__', 
'__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_lock', 
'api_implementation', 'six', 'threading']
Now I can to define the structure for the data structured as messages
These message is a small logical record of information containing a series of name-value pairs called fields.
The protobuffers compiler (protoc) to generate the source code in the language you need (from the .proto file).
To generate a Python file, you need to execute:
protoc -I=$SRC_DIR --python_out=$DST_DIR $SRC_DIR/my_example.proto
Let's test with am example (you can use any proto file) in default folder (use . for default folder):
[mythcat@desk python]$ protoc -I=. --python_out=. my_example.proto
[mythcat@desk python]$ ls my_example*
my_example_pb2.py  my_example.proto
[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_example_pb2
>>> dir(my_example_pb2)
['AddressBook', 'DESCRIPTOR', 'Person', '_ADDRESSBOOK', '_PERSON', '_PERSON_PHONENUMBER', '_PERSON_PHONETYPE',
 '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__',
 '_b', '_descriptor', '_message', '_reflection', '_sym_db', '_symbol_database', 'sys']

Saturday, September 28, 2019

The tensorflow python module - part 004.

If you using the tensorflow then you can get some warnings.
You can use warnings python package to manage all of this:
[mythcat@desk ~]$ $ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> import tensorflow as tf
/home/mythcat/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: 
Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be 
understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/mythcat/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: 
Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be 
understood as (type, (1,)) / '(1,)type'.
...

/home/mythcat/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: 
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of 
numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
>>>
>>> warnings.filterwarnings('ignore')
>>>
[7]+  Stopped                 python3
Use it in this way to filter these warnings:
[mythcat@desk ~]$ $ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> warnings.filterwarnings('ignore',category=FutureWarning)
>>> import tensorflow as tf
>>> 

Wednesday, September 25, 2019

Python 3.7.4 : Print with random colors.

This is a simple example for custom print output.
The script detect the platform for color settings and then use print.
The first print will print with blue color the name of the script.
I used random to select a random color from colors array and used to print the -=RANDOM COLOR=- text.
The print (W+'') is used to set default white color for terminal
import sys
import random
if sys.platform == "linux" or sys.platform == "linux2":
        BB = "\033[34;1m" # Blue light
        YY = "\033[33;1m" # Yellow light
        GG = "\033[32;1m" # Green light
        WW = "\033[0;1m"  # White light
        RR = "\033[31;1m" # Red light
        CC = "\033[36;1m" # Cyan light
        B = "\033[34m"    # Blue
        Y = "\033[33m"    # Yellow
        G = "\033[32m"    # Green
        W = "\033[0m"     # White
        R = "\033[31m"    # Red
        C = "\033[36m"    # Cyan
colors = [BB,YY,GG,WW,RR,CC,B,Y,G,W,R,C]
print (B+"\033[2;2m "+sys.argv[0]+"\n"+B)

color=random.choice(colors)
print (color+"-=RANDOM COLOR=-"+color)
print (W+'')
For winodws platform you need to add this:
elif sys.platform == "win32":

 BB = '' # Blue light
 YY = '' # Yellow light
 GG = '' # Green light
 WW = '' # White light
 RR = '' # Red light
 CC = '' # Cyan light
 B = ''  # Blue
 Y = ''  # Yellow
 G = ''  # Green
 W = ''  # White
 R = ''  # Red
 C = ''  # Cyan
 P = ''  # Random color

Wednesday, September 11, 2019

Python 3.7.4 : Using the theano pakage.

If you want to test theano then you need to see this webpage.
[root@desk mythcat]# dnf search theano
======================== Name & Summary Matched: theano ========================
python-theano-doc.noarch : Theano documentation
============================= Name Matched: theano =============================
python3-theano.noarch : Mathematical expressions involving multidimensional
                      : arrays
=========================== Summary Matched: theano ============================
python3-lasagne.noarch : Lightweight library to build and train neural networks
                       : in Theano
[root@desk mythcat]# pip3 install Theano --user
WARNING: Running pip install with root privileges is generally not a good idea. 
Try `pip3 install --user` instead.
Collecting Theano
...
  Running setup.py install for Theano ... done
Successfully installed Theano-1.0.4 scipy-1.3.1
Let's see first example:
[mythcat@desk ~]$ python3 
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
/home/mythcat/.local/lib/python3.7/site-packages/theano/configdefaults.py:560: 
UserWarning: DeprecationWarning: there is no c++ compiler.This is deprecated and with 
Theano 0.11 a c++ compiler will be mandatory
  warnings.warn("DeprecationWarning: there is no c++ compiler."
WARNING (theano.configdefaults): g++ not detected ! Theano will be unable to execute 
optimized C-implementations (for both CPU and GPU) and will default to Python implementations.
 Performance will be severely degraded. To remove this warning, set Theano flags cxx to an empty
 string.
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
>>> import theano
>>> import theano.tensor as T
>>> x = T.dmatrix('x')
>>> s = 1 / (1 + T.exp(-x))
>>> logistic = theano.function([x], s)
>>> logistic([[0, 1], [-1, -2]])
array([[0.5       , 0.73105858],
       [0.26894142, 0.11920292]])
>>> ... 

Monday, September 9, 2019

Python 3.7.4 : Using the sunpy - part 001.

I wrote about sunpy in the past on this website.
Now this package comes with new features, see the official webpage.
Let's install it.
[mythcat@desk ~]$ pip3 install sunpy --user
Collecting sunpy
...
Successfully installed aioftp-0.13.0 aiohttp-3.6.0 astropy-3.2.1 async-timeout-3.0.1 
multidict-4.5.2 parfive-1.0.0 scipy-1.3.1 sunpy-1.0.3 tqdm-4.35.0 yarl-1.3.0
If you search on web you can find example with spectra , but in the new package is not supported.
ModuleNotFoundError: No module named 'sunpy.spectra'
Let's test it:
[mythcat@desk ~]$ python3 
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sunpy
>>> dir(sunpy)
['SunPyTestRunner', 'UnsupportedPythonError', '__all__', '__builtins__', '__cached__', '__doc__',
 '__file__', '__loader__', '__minimum_python_version__', '__name__', '__package__', '__path__',
 '__spec__', '__version__', '_init_log', 'builtins', 'config', 'extern', 'load_config', 'log',
 'logging', 'os', 'print_config', 'self_test', 'sys', 'system_info', 'tests', 'util', 'version']
>>> sunpy.system_info()
==============================
SunPy Installation Information
==============================

#######
General
#######
Time : Monday, 09. September 2019 07:14PM UT
System : Linux
Processor : x86_64
Arch : 64bit
SunPy : 1.0.3
OS: Fedora 30 Thirty (Linux 5.2.11-200.fc30.x86_64 x86_64)


##################
Required Libraries
##################
Python: 3.7.4
NumPy: 1.16.4
SciPy: 1.3.1
matplotlib: 3.0.3
Astropy: 3.2.1
Pandas: 0.25.1
parfive: 1.0.0


#####################
Recommended Libraries
#####################
beautifulsoup: NOT INSTALLED
PyQt4: NOT INSTALLED
PyQt5: 5.12.2
Zeep: NOT INSTALLED
Sqlalchemy: 1.3.7
drms: NOT INSTALLED
>>> help(sunpy)

Help on package sunpy:

NAME
    sunpy

DESCRIPTION
    SunPy
    =====
    
    An open-source Python library for Solar Physics data analysis.
    
    Web Links
    ---------
    Homepage: https://sunpy.org
    Documentation: https://docs.sunpy.org/en/stable/

PACKAGE CONTENTS
    cm (package)
    compiler_version
    conftest
    coordinates (package)
    data (package)
    database (package)
    extern (package)
    image (package)
    instr (package)
    io (package)
    map (package)
    net (package)
    physics (package)
    roi (package)
    sun (package)
    tests (package)
    time (package)
    timeseries (package)
    util (package)
    version
    visualization (package)
... 
Let's install zeep, beautifulsoup4 and drms python packages:
[mythcat@desk ~]$ pip3 install zeep --user
Collecting zeep
...
Successfully installed appdirs-1.4.3 cached-property-1.5.1 isodate-0.6.0 lxml-4.4.1
 requests-toolbelt-0.9.1 zeep-3.4.0
[mythcat@desk ~]$ pip3 install --upgrade beautifulsoup4 --user
Collecting beautifulsoup4
...
Successfully installed beautifulsoup4-4.8.0 soupsieve-1.9.3
[mythcat@desk ~]$ pip3 install --upgrade drms --user
Collecting drms
...
Successfully installed drms-0.5.7
Now, we can see examples with this python package.
First, I will use the AIA_171_IMAGE image:
>>> from sunpy.data.sample import AIA_171_IMAGE 
Files Downloaded: 100%|███████████████████████| 26/26 [00:04<00:00 5.50file="" s="">>> import sunpy.map                                                            
>>> aiamap = sunpy.map.Map(AIA_171_IMAGE)
>>> aiamap.peek()                                                               
Attribute Qt::AA_EnableHighDpiScaling must be set before QCoreApplication is created.                                                                           

** (python3:4152): WARNING **: 22:53:15.329: AT-SPI: Could not obtain desktop path or name


** (python3:4152): WARNING **: 22:53:15.377: atk-bridge: GetRegisteredEvents returned message with unknown signature



Python 3.7.3 : Using the flask - part 018.

In this tutorial, I will show you how to fix auto increment in Flask SQLAlchemy.
The old source code for the user model from the server.py was this:
class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    password = db.Column(db.String(120), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)
The server.sqlite will be this:
[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE user (
 id INTEGER NOT NULL, 
 username VARCHAR(80), 
 password VARCHAR(120), 
 email VARCHAR(120), 
 gender VARCHAR(5), 
 work VARCHAR(33), 
 city VARCHAR(15), 
 PRIMARY KEY (id), 
 UNIQUE (username), 
 UNIQUE (password), 
 UNIQUE (email), 
 UNIQUE (gender), 
 UNIQUE (work), 
 UNIQUE (city)
);
CREATE TABLE alembic_version (
 version_num VARCHAR(32) NOT NULL, 
 CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num)
);
CREATE TABLE sqlite_stat1(tbl,idx,stat);
sqlite> ^Z 
If you want to change the id into auto increment then you need to follow this steps:
class User(db.Model):
    __tablename__ = 'user'
    __table_args__ = {'sqlite_autoincrement': True}
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    password = db.Column(db.String(120), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)
Delete the server.sqlite file or rename it.
Open python3 and create a new server.sqlite file:
[mythcat@desk my_flask]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
>>> db.create_all()
>>> db.engine.table_names()
['sqlite_sequence', 'user']
>>> 
[5]+  Stopat                  python3 
Open the new file to see the changes:
[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE user (
 id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, 
 username VARCHAR(80), 
 password VARCHAR(120), 
 email VARCHAR(120), 
 gender VARCHAR(5), 
 work VARCHAR(33), 
 city VARCHAR(15), 
 UNIQUE (username), 
 UNIQUE (password), 
 UNIQUE (email), 
 UNIQUE (gender), 
 UNIQUE (work), 
 UNIQUE (city)
);
CREATE TABLE sqlite_sequence(name,seq);
sqlite>  

Wednesday, September 4, 2019

Python 3.7.4 : Create an Stand Alone Executable on Fedora distro.

In this tutorial I will show you how to create an Stand Alone Executable with Python version 3.7.4 and Fedora 30 distro.
First you need to install using the dnf tool the python3 package.
You can test it easy with this command:
[mythcat@desk dist]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information. 
I created a simple python script named test.py for testing:
import os
import sys
print("Hello!")
You need to install the pyinstaller python module.
Let's see this step with the output:
[mythcat@desk ~]$ pip3 install pyinstaller --user
Now you can use this python module to create Stand Alone Executable.
[mythcat@desk ~]$ pip3 install pyinstaller --user
Requirement already satisfied: pyinstaller in ./.local/lib/python3.7/site-packages (3.5)
Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from pyinstaller) (40.8.0)
Requirement already satisfied: altgraph in ./.local/lib/python3.7/site-packages (from pyinstaller) (0.16.1)
[mythcat@desk ~]$ pyinstaller --onefile test.py
561 INFO: PyInstaller: 3.5
561 INFO: Python: 3.7.4
571 INFO: Platform: Linux-5.2.9-200.fc30.x86_64-x86_64-with-fedora-30-Thirty
573 INFO: wrote /home/mythcat/test.spec
596 INFO: UPX is not available.
598 INFO: Extending PYTHONPATH with paths
['/home/mythcat', '/home/mythcat']
598 INFO: checking Analysis
624 INFO: Building because /home/mythcat/test.py changed
624 INFO: Initializing module dependency graph...
640 INFO: Initializing module graph hooks...
678 INFO: Analyzing base_library.zip ...
5247 INFO: running Analysis Analysis-00.toc
5352 INFO: Caching module hooks...
5376 INFO: Analyzing /home/mythcat/test.py
5396 INFO: Loading module hooks...
5397 INFO: Loading module hook "hook-xml.py"...
5775 INFO: Loading module hook "hook-pydoc.py"...
5800 INFO: Loading module hook "hook-encodings.py"...
5892 INFO: Looking for ctypes DLLs
5893 INFO: Analyzing run-time hooks ...
5901 INFO: Looking for dynamic libraries
6507 INFO: Looking for eggs
6507 INFO: Using Python library /lib64/libpython3.7m.so.1.0
6514 INFO: Warnings written to /home/mythcat/build/test/warn-test.txt
6547 INFO: Graph cross-reference written to /home/mythcat/build/test/xref-test.html
6604 INFO: checking PYZ
6607 INFO: Building because toc changed
6607 INFO: Building PYZ (ZlibArchive) /home/mythcat/build/test/PYZ-00.pyz
7030 INFO: Building PYZ (ZlibArchive) /home/mythcat/build/test/PYZ-00.pyz completed successfully.
7034 INFO: checking PKG
7035 INFO: Building because toc changed
7035 INFO: Building PKG (CArchive) PKG-00.pkg
10033 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
10036 INFO: Bootloader 
/home/mythcat/.local/lib/python3.7/site-packages/PyInstaller/bootloader/Linux-64bit/run
10036 INFO: checking EXE
10123 INFO: Building because toc changed
10124 INFO: Building EXE from EXE-00.toc
10163 INFO: Appending archive to ELF section in EXE /home/mythcat/dist/test
10337 INFO: Building EXE from EXE-00.toc completed successfully.
This command will create some folders: dist, build, ...
The dist folder will have the Stand Alone Executable named test:
[mythcat@desk ~]$ cd dist/
[mythcat@desk dist]$ ls
test
[mythcat@desk dist]$ ./test
Hello!

Tuesday, August 27, 2019

Python 3.7.3 : Using the flask - part 017.

Today I make some changes with my server.py and database and solve some issues, see old version at my old tutorial.
Firt issue was start script.
I create a linux script named start_server.sh to run the flask run command:
[mythcat@desk my_flask]$ ./start_server.sh 
I update the User with new fiels:
class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    password = db.Column(db.String(120), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)
Let's see how I deal with this versus database and migrate process.
[mythcat@desk my_flask]$ rm server.sqlite 
[mythcat@desk my_flask]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
>>> db.create_all()
>>> db.engine.table_names()
['user']
>>> exit()
[mythcat@desk my_flask]$ ls server.sqlite 
server.sqlite
[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .tables
user
sqlite> .schema user
CREATE TABLE user (
        id INTEGER NOT NULL, 
        username VARCHAR(80), 
        password VARCHAR(120), 
        email VARCHAR(120), 
        gender VARCHAR(5), 
        work VARCHAR(33), 
        city VARCHAR(15), 
        PRIMARY KEY (id), 
        UNIQUE (username), 
        UNIQUE (password), 
        UNIQUE (email), 
        UNIQUE (gender), 
        UNIQUE (work), 
        UNIQUE (city)
);
I got a strange error:
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: city
I fix it with this , but I'm not sure if the right way:
[mythcat@desk my_flask]$ rm migrations/ -r -f 
[mythcat@desk my_flask]$ python3 server.py db init 
  Creating directory /home/mythcat/project_github/my_flask/migrations ... done
  Creating directory /home/mythcat/project_github/my_flask/migrations/versions ... done
  Generating /home/mythcat/project_github/my_flask/migrations/script.py.mako ... done
  Generating /home/mythcat/project_github/my_flask/migrations/env.py ... done
  Generating /home/mythcat/project_github/my_flask/migrations/alembic.ini ... done
  Generating /home/mythcat/project_github/my_flask/migrations/README ... done
  Please edit configuration/connection/logging settings in '/home/mythcat/project_github/my_flask/migrations/alembic.ini'
  before proceeding.
[mythcat@desk my_flask]$ python3 server.py db migrate
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.env] No changes in schema detected.
[mythcat@desk my_flask]$ python3 server.py db upgrade
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.

Sunday, August 25, 2019

Python 3.7.3 : Using the flask - part 016.

Today I tested a new feature of Flask version 1.1.1.
[mythcat@desk my_flask]$ pip list | grep Flask
Flask                    1.1.1            
Flask-Login              0.4.1            
Flask-Mail               0.9.1            
Flask-Migrate            2.5.2            
Flask-Script             2.0.6            
Flask-SQLAlchemy         2.4.0            
Flask-WTF                0.14.2 
This feature will remove the jsonify python module.
Let's start the blueprint blue_test:
[mythcat@desk my_flask]$ export FLASK_APP=blue_test
[mythcat@desk my_flask]$ flask run 
 * Serving Flask app "blue_test"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
The output is this:
 {"result":"This is main page!"}
Into my blue_test folder I will remove this jsonify python module:
[
mythcat@desk my_flask]$ cd blue_test/
[mythcat@desk blue_test]$ ll
total 40
-rw-rw-r--. 1 mythcat mythcat  192 Aug 23 11:02 extensions.py
-rw-rw-r--. 1 mythcat mythcat 1826 Aug 23 11:02 forms.py
-rw-rw-r--. 1 mythcat mythcat  701 Aug 23 12:03 __init__.py
-rw-rw-r--. 1 mythcat mythcat 1491 Aug 23 11:02 models.py
drwxrwxr-x. 2 mythcat mythcat  184 Aug 25 14:36 __pycache__
-rw-rw-r--. 1 mythcat mythcat 1050 Aug 23 11:02 routes.py
-rw-rw-r--. 1 mythcat mythcat  142 Aug 23 11:02 settings.py
-rw-rw-r--. 1 mythcat mythcat   34 Aug 23 11:02 start.bat
drwxrwxr-x. 2 mythcat mythcat   81 Aug 23 11:02 templates
-rw-rw-r--. 1 mythcat mythcat 8192 Aug 23 11:02 texts.sqlite
-rw-rw-r--. 1 mythcat mythcat  677 Aug 23 11:02 views.py 
The routes.py will have these changes:
#from flask import Blueprint, jsonify, request
# new flask
from flask import Blueprint, request
...
def home():
    #return jsonify({'result' : 'This is main page!'})
    #this is the new source code:
    return {'result' : 'This is main page!'} 
The next step is to find all jsonify word into my source code using the grep tool and make changes:
[mythcat@desk my_flask]$ grep -nr jsonify *
Binary file blue_test/__pycache__/views.cpython-37.pyc matches
blue_test/views.py:3:from flask import Blueprint, jsonify, request
blue_test/views.py:20:    return jsonify ({'texts':texts})
blue_test/routes.py:1:#from flask import Blueprint, jsonify, request
crud.py:1:from flask import Flask, request, jsonify
crud.py:57:    return jsonify(new_user)
crud.py:64:    return jsonify(result.data)
crud.py:70:    return user_schema.jsonify(user)
crud.py:83:    return user_schema.jsonify(user)
crud.py:92:    return user_schema.jsonify(user)
Binary file __pycache__/crud.cpython-37.pyc matches
Binary file __pycache__/server.cpython-37.pyc matches
server.py:9:from flask import jsonify
server.py:101:    #return jsonify(new_user)
server.py:109:    #return users_schema.jsonify(users)
server.py:111:    return jsonify(all_users.data)
server.py:117:    return jsonify(result.data)
server.py:122:        return jsonify([d['keyword'] for d in result.data])
server.py:132:    return user_schema.jsonify(new_user)
tserv.py:13:from flask import jsonify
tserv.py:56:    #return users_schema.jsonify(users)
tserv.py:58:    return jsonify(all_users.data)
tserv.py:69:    return user_schema.jsonify(user_post)
tserv.py:80:    return user_schema.jsonify(user_put)
tserv.py:88:    return user_schema.jsonify(user_delete)
[mythcat@desk my_flask]$  
After I make changes the result of grep is this:
[mythcat@desk my_flask]$ grep -nr jsonify *
Binary file blue_test/__pycache__/views.cpython-37.pyc matches
blue_test/routes.py:1:#from flask import Blueprint, jsonify, request
blue_test/views.py:3:#from flask import Blueprint, jsonify, request
crud.py:1:#from flask import Flask, request, jsonify
Binary file __pycache__/crud.cpython-37.pyc matches
Binary file __pycache__/server.cpython-37.pyc matches
server.py:9:#from flask import jsonify
server.py:109:    #return users_schema.jsonify(users)
tserv.py:13:#from flask import jsonify
tserv.py:56:    #return users_schema.jsonify(users) 
You can see the last version of my project here.







Saturday, August 24, 2019

Python 3.7.3 : Using the flask - part 015.

In this tutorial, I will show you how to migrate using the Database Migrations in flask project.
Because my laptop is gone I use my old Linux.
First you need to install these python modules with --user argument for Linux:
[mythcat@desk my_flask]$ pip3 install flask-migrate --user
...
[mythcat@desk my_flask]$ pip3 install flask-script --user
Let's test this new issue with server.py file by adding these python modules:
#migrate 
from flask_script import Manager
from flask_migrate import Migrate, MigrateCommand
...
# create migrate object with db 
migrate = Migrate(app, db)
# create manager 
manager = Manager(app)
# create db command for manager 
manager.add_command('db', MigrateCommand)
...
# add new columns into database 
class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)
...
# the default name main
if __name__ == '__main__':
    manager.run()
    app.run(debug=True)
Let's fix this migrate issue with the new command:
[mythcat@desk my_flask]$ python3 server.py db init 
  Creating directory /home/mythcat/project_github/my_flask/migrations ... done
  Creating directory /home/mythcat/project_github/my_flask/migrations/versions ... done
  Generating /home/mythcat/project_github/my_flask/migrations/script.py.mako ... done
  Generating /home/mythcat/project_github/my_flask/migrations/env.py ... done
  Generating /home/mythcat/project_github/my_flask/migrations/alembic.ini ... done
  Generating /home/mythcat/project_github/my_flask/migrations/README ... done
  Please edit configuration/connection/logging settings in '/home/mythcat/project_github/my_flask/migrations
/alembic.ini'
  before proceeding.

[mythcat@desk my_flask]$ python3 server.py db migrate
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.autogenerate.compare] Detected added column 'user.city'
INFO  [alembic.autogenerate.compare] Detected added column 'user.gender'
INFO  [alembic.autogenerate.compare] Detected added column 'user.work'
INFO  [alembic.autogenerate.compare] Detected added unique constraint 'None' on '['city']'
INFO  [alembic.autogenerate.compare] Detected added unique constraint 'None' on '['gender']'
INFO  [alembic.autogenerate.compare] Detected added unique constraint 'None' on '['work']'
  Generating /home/mythcat/project_github/my_flask/migrations/versions/ca70c42b5b7a_.py ... done

[mythcat@desk my_flask]$ python3 server.py db upgrade 
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> ca70c42b5b7a, empty message
ERROR [root] Error: No support for ALTER of constraints in SQLite dialect
The database fiels is changed by this command.
Let's see with sqlite3 tool:
[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .tables
alembic_version  user           
sqlite> .schema user
CREATE TABLE user (
        id INTEGER NOT NULL, 
        username VARCHAR(80), 
        email VARCHAR(120), city VARCHAR(15), gender VARCHAR(5), work VARCHAR(33), 
        PRIMARY KEY (id), 
        UNIQUE (username), 
        UNIQUE (email)
);
You can see my source code here.

Thursday, August 22, 2019

Python 3.7.3 : Using the inotify.

About this tool you can read here:
inotify functionality is available from the Linux kernel and allows you to register one or more directories for watching, and to simply block and wait for notification events.
[mythcat@desk ~]$ pip3 install inotify --user
...
Successfully installed inotify-0.2.10 nose-1.3.7
Let's test it and see how this can be load it:
[mythcat@desk ~]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import inotify
>>> from inotify import *
>>> print(dir(inotify))
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', 
'__path__', '__spec__', '__version__']
>>> import inotify.adapters
>>> print(dir(inotify))
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', 
'__path__', '__spec__', '__version__', 'adapters', 'calls', 'constants', 'library']
>>> print(dir(inotify.adapters))
['EINTR', 'EventTimeoutException', 'Inotify', 'InotifyTree', 'InotifyTrees', 'TerminalEventException', 
'_BaseTree', '_DEFAULT_EPOLL_BLOCK_DURATION_S', '_DEFAULT_TERMINAL_EVENTS', '_HEADER_STRUCT_FORMAT', 
'_INOTIFY_EVENT', '_IS_DEBUG', '_LOGGER', '_STRUCT_HEADER_LENGTH', '__builtins__', '__cached__', '__doc__',
 '__file__', '__loader__', '__name__', '__package__', '__spec__', 'collections', 'inotify', 'logging', 'os',
 'select', 'struct', 'time']
>>> import inotify.calls
>>> print(dir(inotify.calls))
['InotifyError', '_LIB', '_LOGGER', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', 
'__name__', '__package__', '__spec__', '_check_nonnegative', '_check_nonzero', '_check_zero', 'ctypes', 
'errno', 'inotify', 'inotify_add_watch', 'inotify_init', 'inotify_rm_watch', 'logging']
>>> import inotify.constants
>>> print(dir(inotify.constants))
['IN_ACCESS', 'IN_ALL_EVENTS', 'IN_ATTRIB', 'IN_CLOEXEC', 'IN_CLOSE', 'IN_CLOSE_NOWRITE', 'IN_CLOSE_WRITE', 
'IN_CREATE', 'IN_DELETE', 'IN_DELETE_SELF', 'IN_DONT_FOLLOW', 'IN_IGNORED', 'IN_ISDIR', 'IN_MASK_ADD', 
'IN_MODIFY', 'IN_MOVE', 'IN_MOVED_FROM', 'IN_MOVED_TO', 'IN_MOVE_SELF', 'IN_NONBLOCK', 'IN_ONESHOT', 
'IN_ONLYDIR', 'IN_OPEN', 'IN_Q_OVERFLOW', 'IN_UNMOUNT', 'MASK_LOOKUP', '__builtins__', '__cached__', 
'__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']
>>> import inotify.library
>>> print(dir(inotify.library))
['_FILEPATH', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', 
'__spec__',
 'ctypes', 'instance']
Let's test it with a simple watch issue for a file named test_file.
First, let create this file:
[mythcat@desk ~]$ touch /tmp/test_file
[mythcat@desk ~]$ ll /tmp/test_file 
-rw-rw-r--. 1 mythcat mythcat 0 Aug 22 13:54 /tmp/test_file
Now, I can use the next script to watch on it:
import inotify.adapters
def notif_tmp():
    adapter_tmp = inotify.adapters.Inotify()

    adapter_tmp.add_watch('/tmp')

    with open('/tmp/test_file', 'w'):
        pass

    events = adapter_tmp.event_gen(yield_nones=False, timeout_s=1)
    events = list(events)

    print(events)
if __name__ == '__main__':
    notif_tmp()
The result is this:
$ python3 notif.py 
[(_INOTIFY_EVENT(wd=1, mask=2, cookie=0, len=16), ['IN_MODIFY'], '/tmp', 'test_file'), 
(_INOTIFY_EVENT(wd=1, mask=32, cookie=0, len=16), ['IN_OPEN'], '/tmp', 'test_file'), 
(_INOTIFY_EVENT(wd=1, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], '/tmp', 'test_file')]
If remove this file the result will be this:
$ python3 notif.py 
[(_INOTIFY_EVENT(wd=1, mask=256, cookie=0, len=16), ['IN_CREATE'], '/tmp', 'test_file'), 
(_INOTIFY_EVENT(wd=1, mask=32, cookie=0, len=16), ['IN_OPEN'], '/tmp', 'test_file'), 
(_INOTIFY_EVENT(wd=1, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], '/tmp', 'test_file')]
Let's test another example:
import inotify.adapters
import logging
_LOGGER = logging.getLogger(__name__)
def test():
    i = inotify.adapters.Inotify()
    i.add_watch('/tmp')

    try:
        for event in i.event_gen():
            if event is not None:
                (header, type_names, path, filename) = event
                _LOGGER.info("WD=(%d) MASK=(%d) COOKIE=(%d) LEN=(%d) MASK->NAMES=%s "
                             "FILENAME=[%s]",
                             header.wd, header.mask, header.cookie, header.len, type_names,
                             filename)
                print(header, type_names, path, filename)
    finally:
        i.remove_watch('/tmp')

if __name__ == '__main__':
    test()
If you run this and will try to edit with vim editor the /tmp/text.txt file the output will be this:
$ python3 notif_002.py
_INOTIFY_EVENT(wd=1, mask=32, cookie=0, len=16) ['IN_OPEN'] /tmp text.txt
_INOTIFY_EVENT(wd=1, mask=256, cookie=0, len=16) ['IN_CREATE'] /tmp .text.txt.swp
_INOTIFY_EVENT(wd=1, mask=32, cookie=0, len=16) ['IN_OPEN'] /tmp .text.txt.swp
...
If you want to use many paths then use this:
    paths = [
        '/tmp',
    ]
    i = Inotify(paths=paths)
This module is not well documented but can be successfully used for certain tasks.

Wednesday, August 21, 2019

Python Qt5 : contextMenu example.

A context menu is a menu in a graphical user interface (GUI) that appears upon user interaction, such as a right-click mouse operation.
I create the default application and I use QMenu to create this context menu with New, Open and Quit.
from PyQt5 import QtGui
from PyQt5.QtWidgets import QApplication, QMainWindow, QMenu
import sys


class Window(QMainWindow):
    def __init__(self):
        super().__init__()
        self.title = "PyQt5 Context Menu"
        self.top = 100
        self.left = 100
        self.width = 640
        self.height = 480
        self.InitWindow()


    def InitWindow(self):
        self.setWindowIcon(QtGui.QIcon("icon.png"))
        self.setWindowTitle(self.title)
        self.setGeometry(self.left, self.top, self.width, self.height)
        self.show()

    def contextMenuEvent(self, event):
        contextMenu = QMenu(self)
        new_Act = contextMenu.addAction("New")
        open_Act = contextMenu.addAction("Open")
        quit_Act = contextMenu.addAction("Quit")
        action = contextMenu.exec_(self.mapToGlobal(event.pos()))
        if action == quit_Act:
            self.close()


App = QApplication(sys.argv)
window = Window()
sys.exit(App.exec())

Tuesday, August 20, 2019

Python Qt5 : the QTimer class.

I haven't written about PyQt5 in a while and today I decided to add a short tutorial on this python module.
The QTimer class is a high-level programming interface for timers and provides repetitive and single-shot timers.
I this example I call a method every second with these lines:
        self.timer = QTimer()
        self.timer.timeout.connect(self.handleTimer)
        self.timer.start(1000)
The timer is stop when the value check by handleTimer has value 100 else the ProgressBar increment the default value with 1.
The full source of code is this:
import sys
from PyQt5.QtCore import QTimer
from PyQt5.QtWidgets import QApplication, QMainWindow, QProgressBar
from PyQt5.QtCore import Qt

class QTimer_ProgressBar(QMainWindow):

    def __init__(self):
        super().__init__()

        self.pbar = QProgressBar(self)
        self.pbar.setGeometry(30, 70,400, 50)
        self.pbar.setValue(0)

        self.setWindowTitle("QTimer Progressbar")
        self.setGeometry(64,64,640,480)
        self.show()

        self.timer = QTimer()
        self.timer.timeout.connect(self.handleTimer)
        self.timer.start(1000)

    def handleTimer(self):
        value = self.pbar.value()
        if value < 100:
            value = value + 1
            self.pbar.setValue(value)
        else:
            self.timer.stop()


if __name__ == '__main__':
    app = QApplication(sys.argv)
    ex = QTimer_ProgressBar()
    sys.exit(app.exec_())
You can also use it like this:
def Qt():
    try:
        # Do things
    finally:
        QTimer.singleShot(5000, Qt)

Qt()

Wednesday, August 14, 2019

Python 3.7.3 : Using the flask - part 014.

Today I worked on YouTube search with flask and Google A.P.I. project.
The source code is simple to understand and you can test any A.P.I. from google using this way.
I created a new Google project with YouTube A.P.I. version 3 and with the A.P.I. key.
I use this key to connect with flask python module.
I used the isodate python module.
You can see the source code on my GitHub repo named flask_yt.
The result is this:

Friday, August 9, 2019

Python 3.7.3 : Using the flask - part 013.

Flask uses Jinga2 template engine.
The Jinga2 template engine uses the following delimiters for escaping from HTML.
We can use this:
  • {% ... %} for Statements
  • {{ ... }} for Expressions to print to the template output
  • {# ... #} for Comments not included in the template output
  • # ... ## for Line Statements
The documentation webpage comes with all information about how can be used.
I create a new HTML5 file named layout.html and I will use this like an example:
{% if current_user.is_authenticated %}
...
{% else %}
...
{% endif %}
This file can be add into another file like this:
{% extends "layout.html" %}
{% block content %}
...
{% endblock content %}
For example, my about.html webpage comes with this source code:
{% extends "layout.html" %}
{% block content %}
    

About Page

{% endblock content %}
The routes.py this webpage will have this call:
@app.route("/about")
def about():
    return render_template('about.html', title='About')
In this way, I will add more HTML5 files to the project.


Thursday, August 8, 2019

Python 3.7.3 : Using the flask - part 012.

The goal of this tutorial step is to understand how the project can use the new features and implementation of the project.
Because in the last tutorial I used the flask_mail python module, now I will add into my project structure.
One good issue is registration issue for users.
First, you need to see the full project and changes at my GitHub project.
I used the itsdangerous python module to use tokens.
Let's install this with the pip tool:
C:\Python373\Scripts>pip install itsdangerous
For registration is need to have login issues and we can use LoginManager, see the extensions.py
#use SQLAlchemy
from flask_sqlalchemy import SQLAlchemy
#use LoginManager
from flask_login import LoginManager
#create login_manager
login_manager = LoginManager()
#create db
db = SQLAlchemy()
Let's see the new User model, see the models.py:
#imports for user model
from datetime import datetime
from .extensions import db
from itsdangerous import TimedJSONWebSignatureSerializer as Serializer
from blue_test import login_manager
from flask_login import UserMixin

# import db from base folder, see dot
from .extensions import db 

#load the user by id
@login_manager.user_loader
def load_user(user_id):
    return User.query.get(int(user_id))

# create the Texts mode
class Texts(db.Model):
    # primary key 
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(50))
    txt_content = db.Column(db.String(1000))
    
# create the User model
'''
id = id key for user
username = name
email = user email
password = user password

'''
class User(db.Model, UserMixin):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(20), unique=True, nullable=False)
    email = db.Column(db.String(120), unique=True, nullable=False)
    password = db.Column(db.String(60), nullable=False)

    def get_reset_token(self, expires_sec=1800):
        s = Serializer(app.config['SECRET_KEY'], expires_sec)
        return s.dumps({'user_id': self.id}).decode('utf-8')

    @staticmethod
    def verify_reset_token(token):
        s = Serializer(app.config['SECRET_KEY'])
        try:
            user_id = s.loads(token)['user_id']
        except:
            return None
        return User.query.get(user_id)

    def __repr__(self):
        return f"User('{self.username}', '{self.email}')"
In the blue_test folder project I created the templates folder with two HTML5 files: home.html and register.html.
I created also the forms.py file and I update with my forms: RegistrationForm,RequestResetForm, ResetPasswordForm.
#import python for create forms 
from flask_wtf import FlaskForm
from wtforms import StringField, PasswordField, SubmitField
# import python for validate forms field
from wtforms.validators import DataRequired, Length, Email, EqualTo, ValidationError
#import python files project
from .extensions import db, login_manager

class RegistrationForm(FlaskForm):
    username = StringField('Username',
                           validators=[DataRequired(), Length(min=2, max=25)])
    email = StringField('Email',
                        validators=[DataRequired(), Email()])
    password = PasswordField('Password', validators=[DataRequired()])
    confirm_password = PasswordField('Confirm Password',
                                     validators=[DataRequired(), EqualTo('password')])
    submit = SubmitField('Sign Up')

    def validate_username(self, username):
        user = User.query.filter_by(username=username.data).first()
        if user:
            raise ValidationError('That username is taken.')

    def validate_email(self, email):
        user = User.query.filter_by(email=email.data).first()
        if user:
            raise ValidationError('That email is taken.')

class RequestResetForm(FlaskForm):
    email = StringField('Email',
                        validators=[DataRequired(), Email()])
    submit = SubmitField('Password Reset')

    def validate_email(self, email):
        #get user by email
        
        if user is None:
            raise ValidationError('There is no account with that email.')

class ResetPasswordForm(FlaskForm):
    password = PasswordField('Password', validators=[DataRequired()])
    confirm_password = PasswordField('Confirm Password',
                                     validators=[DataRequired(), EqualTo('password')])
    submit = SubmitField('Reset Password')
In the __init__.py file I add this:
#import python files project
from .extensions import db, login_manager
...
    login_manager.init_app(app)
The routers.py will have the new RegistrationForm:
@api.route("/register", methods=['GET', 'POST'])
def register():
    # check if the user is authenticated
    if current_user.is_authenticated:
        return redirect(url_for('home'))
    form = RegistrationForm()
    if form.validate_on_submit():
        flash('Your account has been created!, 'success')
        return redirect(url_for('login'))
    return render_template('register.html', title='Register', form=form)
Depending on your needs, we will send and redirect these forms to the HTML files.
Depending on your needs (users, page access, ...) we will be able to write the source code further.

Wednesday, August 7, 2019

Python 3.7.3 : Using the flask - part 011.

The tutorial for today is focused on the email issue.
I will start with the new python module for flask named flask_mail.
Let's install it:
C:\Python373>cd Scripts

C:\Python373\Scripts>pip3 install flask_mail
Collecting flask_mail
...
Installing collected packages: blinker, flask-mail
Successfully installed blinker-1.4 flask-mail-0.9.1
The next source code let show you how can use this python module.
from flask import Flask 
from flask_mail import Mail, Message

app = Flask(__name__)
app.config['DEBUG'] = True
app.config['TESTING'] = False
app.config['MAIL_SERVER'] = 'smtp...'
app.config['MAIL_PORT'] = 25
app.config['MAIL_USE_TLS'] = False
app.config['MAIL_USE_SSL'] = False
#app.config['MAIL_DEBUG'] = True # debug 
app.config['MAIL_USERNAME'] = None
app.config['MAIL_PASSWORD'] = None
# this will send with the name of another@mail.com
#app.config['MAIL_DEFALT_SENDER'] = ('Another mail', 'another@mail.com')
app.config['MAIL_DEFALT_SENDER'] = None
app.config['MAIL_MAX_EMAILS'] = None # limit the messages send
#app.config['MAIL_SUPRESS_SEND'] = False # testing
app.config['MAIL_ASCII_ATTACHMENTS'] = False

# send an email with the flask application
mail = Mail(app)

@app.route('/mail')
def mail():
    #msg = Message('Hey', sender='another@mail.com')
    #msg = Message('Hey', recipients=['catafest@yahoo.com', 'another@mail.com'])
    #msg.add_recipient = ('another@mail.com')
    msg = Message('Hey', recipients=['catafest@yahoo.com'])

    # you can use body or HTML, using both wills receive the HTML first
    #msg.body = 'This is a body text message!'
    msg.html = 'This is a body text message with HTML5 tags!'

    #add a file attachment to message 
    with app.open_resource('photo.jpg') as add_file_res:
         msg.attach('photo.jpg', 'image/jpeg', add_file_res.read())
    '''
    #create a template message 
    msg = Message(
    subject ='',
    recipients=[],
    body = '',
    html = '',
    sender = '',
    cc = [],
    bcc = [],
    attachments = [],
    reply_to = [],
    date = '',
    charset = [],
    extra_headers = {'':''},
    mail_options = [],
    rcpt_options = []
    )
    '''
    #send mail
    mail.send(msg)
    return 'Message sent!'

@app.route('/bulk')
def bulk():
    users = [{'name':'Me', 'email':'another@mail.com'}]
    # open an connection 
    with mail.connect() as con:
         for user in users:
             msg = Message('Bulk message!', recipients=[user['email']]
             msg.body = 'Body message!'
             con.send(msg)
'''
# another example 
@app.route('/bulk')
def bulk():
    users = [{'name':'Me', 'email':'another@mail.com'}]
    # open an connection 
    with mail.connect() as con:
         for user in users:
             msg = Message('Bulk message!', recipients=[user.email]
             msg.body = 'Body message!'
             con.send(msg)
'''
if __name__ == '__main__':
    app.run()
You can see you need to set the settings for your mail server and then use it into flaks application.
The source code is easy to understand if you follow the commented rows.
I add also put into comments alternative examples for the template message and the bulk function.
This is the first step into sending emails with flask.
We can have a complete implementation on the project but all depends on project structure.

Tuesday, August 6, 2019

Python 3.7.3 : Using the flask - part 010.

If you read my last tutorial about flask then you understand how to use the structure flask project with views.py and models.py.
If you run it and open the browser with http://127.0.0.1:5000/texts/ the result will be this:
{"texts":[{"title":"first title","txt_content":"this is first content"},{"title":null,"txt_content":null}]}
Let's create a file .env into the base folder named my_flask and add this source code:
SECRET_KEY='secret key'
DEBUG=True
SQLALCHEMY_TRACK_MODIFICATIONS = False
SQLALCHEMY_DATABASE_URI = sqlite:///texts.sqlite
Let's create a settings.py file into blue_test folder to get these settings:
import os
from os import environ
SECRET_KEY = os.environ.get('SECRET_KEY')
SQLALCHEMY_DATABASE_URI = os.environ.get('SQLALCHEMY_DATABASE_URI')
My blue_test project comes with the old views.py and a new routes.py file with this source code:
from flask import Blueprint, jsonify, request

api = Blueprint('api', __name__)

@api.route('/')
def home():
    return jsonify({'result' : 'You are in main page!'}) 
I create an extensions.py python script to deal with the database, see the source code:
from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy()
With this new python file will solve the avoid circular importing of circular dependency of importing db.
This changes can be made on models.py and views.py, like this:
# import db from base folder, see dot
from .extensions import db 
This two files views.py and routes.py come with two Blueprint's: main and api.
Into the __init__.py will need to import and register both :
from .extensions import db
from .models import Texts
from .routes import api
from .views import main
...
    app.register_blueprint(main)
    app.register_blueprint(api)
If you run it into my_flask folder with:
C:\Python373\my_flask>set FLASK_APP=blue_test

C:\Python373\my_flask>flask run
 * Serving Flask app "blue_test"
 * Environment: production
...
The result into the browser area with http://127.0.0.1:5000/ for the routes.py blueprint api will be:
{"result":"You are in main page!"}
The result into the browser area with http://127.0.0.1:5000/texts/ for the views.py blueprint main will be:
{"texts":[{"title":"first title","txt_content":"this is first content"},{"title":null,"txt_content":null}]}
This shows you how to link multiple blueprints into one project.
You can see the full project at my GitHub project.

Monday, August 5, 2019

Python 3.7.3 : Using the flask - part 009.

In this tutorial, I will show you how to use blueprints with a new type of flask project using multiple python files.
I show you just how to create the A.P.I. not the REACT front end.
Flask uses a concept of blueprints for making application components and supporting common patterns within an application or across applications. Blueprints can greatly simplify how large applications work and provide a central means for Flask extensions to register operations on applications. A Blueprint object works similarly to a Flask application object, but it is not actually an application. Rather it is a blueprint of how to construct or extend an application., see the official webpage.
In my my_flask folder, I create a blue_test folder with three python files: __init__.py, views.py, and models.py.
The __init__.py file has this source code:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy

# create path for SQLAlchemy
import os
basedir = os.path.abspath(os.path.dirname(__file__))

db = SQLAlchemy()

def create_app():
    app = Flask(__name__)
    # instantiate config for SQLAlchemy
    app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
    app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'texts.sqlite')
    # init db 
    db.init_app(app)
    # avoid circular importing of circular dependency 
    from .views import main
    app.register_blueprint(main)
    return app
The views.py come with this source code:
from flask import Blueprint
from flask import jsonify, request
from . import db
from .models import Texts

main = Blueprint('main', __name__)
@main.route('/add_text', methods = ['POST'])
def add_text():
    txt_data = request.get_json()
    new_txt = Texts(title=txt_data['title'], txt_content=txt_data['txt_content'])
    db.session(new_txt)
    db.session.commit()
    # 201 status code for create successfully
    return 'Done', 201
@main.route('/texts/')
def texts():
    txt_list=Texts.query.all()
    texts=[]
    for txt in txt_list:
        texts.append({'title': txt.title, 'txt_content':txt.txt_content})
    return jsonify ({'texts':texts})
The models.py has this source code:
# import db from base folder, see dot
from . import db 

class Texts(db.Model):
    # primary key 
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(50))
    txt_content = db.Column(db.String(1000))
Use the base folder of the blue_test folder to instantiate the database named texts.sqlite with a table texts.
C:\Python373>cd my_flask

C:\Python373\my_flask>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD6
4)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from blue_test.models import Texts
>>> from blue_test import db, create_app
>>> db.create_all(app=create_app())
>>> exit()
If not run into the base folder then you get this error:
    from . import db
ImportError: attempted relative import with no known parent package
Now you will have a texts.sqlite file for the database.
Use this command to set FLASK_APP:
C:\Python373\my_flask>set FLASK_APP=blue_test
C:\Python373\my_flask>flask run
Now, using the postman you can test it by running the python script and call these methods.



Sunday, August 4, 2019

Python 3.7.3 : Using the flask - part 008.

The tutorial for today will show you how to understand the flash method and fix exceptions.
First, the Flask module contains a flash method which passes a message to the next request, which generally is a template.
This lets you create feedback to users of a web application is critical, from notifications and error messages to warnings and progress alerts.
This system allows us to record a message at any point within a request, then display it at the start of the next request (and only the next request), see the documentation.
You need to import the flash with:
from flask import flash
The flash function takes up to 2 arguments, a message and a category like this: flash("message", "category").
In the next example, the random range output named out will send a flash message by category.
The flash warning category will send the number get by the random and will be into a range of 1 and 3.
Flashed messages are stored in the session until they are read it.
The HTML5 page named home.html can be found at my GitHub project.
This source code will send one message by flash and show two messages into a webpage.
@app.route('/',methods = ['GET','POST'])
def home():
    # test flash message
    out = random.randint(1,10)
    if out in range(1,3):
        flash(str(out),"warning" )
    if out in range(4,6):
        flash("This is a flash test for home.html with result:","success")
    if out in range(7,10):
        flash("This is a flash test for home.html with result:","danger")  
    return render_template("home.html")
The next example shows you how to use flash and exceptions to create an output error with render_template_string.
The output exception sends by flash without an HTML5 page request.
from flask import render_template_string
...
# fix Exception error , like 404
@app.errorhandler(Exception)
def page_not_found(e):
    flash(e, type(e))  
    return render_template_string('''
      {% with messages = get_flashed_messages(with_categories=true) %}
        {% if messages %}
          {% set printed_messages = dict() %}
          {% for category, message in messages %}
            {% if message not in printed_messages %}
              
{{message}}
{% set x = printed_messages.__setitem__(message, "value") %} {% endif %} {% endfor %} {% endif %} {% endwith %} ''')
If you put a bad path URL intro the server ( http://127.0.0.1:5000/bad ) then you get the result of the exception 404:
404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
Using the flash you can create your own notification system.

Saturday, August 3, 2019

Python 3.7.3 : Using the flask - part 007.

This will be a long tutorial because will try to link some information's from the last tutorials.
First, the structure of the project can see into my GitHub project.
I create new templates and another python script named tserv.py for testing.
You can see easier how the POST method works and how to deal with a python database issue.
This script comes with an upload file feature and upload database with sqlite3 python module.
First, for upload a file we need the HTML5 file from templates folder named upload.html.
I update the base.html file to use the bootstrap framework.
import sqlite3
class UploadForm(FlaskForm):
    file = FileField()
    submit = SubmitField("submit")
    
@app.route('/upload',methods = ['GET','POST'])
def upload():
    form = UploadForm()
    
    if request.method == "POST" and form.validate():
        if form.validate_on_submit():
            file_name = form.file.data
            file_database(name = file_name.filename,data = file_name.read())
            print("File {}".format(file_name.filename))
            return render_template("upload.html", form = form)
    return render_template("upload.html", form = form)
def file_database(name,data):
    con=sqlite3.connect("file_upload.db")
    cursor = con.cursor()
    cursor.execute("""CREATE TABLE IF NOT EXISTS my_table (name TEXT, data BLOP) """)
    cursor.execute("""INSERT INTO my_table (name , data ) VALUES (?,?) """, (name, data))
    con.commit()
    cursor.close()
    con.close()
Let's run it:
python tserv.py
Into the main folder a database will be create and will fill with the file uploaded.
Now, if you want to add more we can create a download button into our UploadForm, is no need to have another python class.
First create your form tag into upload.html file and link with the python tserv.py code show bellow:
from flask import send_file
from io import BytesIO
...
@app.route('/download', methods=['GET','POST'])
def download():
    form = UploadForm()
    if request.method == "POST":
        con = sqlite3.connect("file_upload.db")
        cursor = con.cursor()
        cur_ex = cursor.execute(""" SELECT * FROM my_table """)
        for i in cur_ex.fetchall():
            name=i[0]
            data=i[1]
            break
        con.commit()
        cursor.close()
        con.close()
        return send_file(BytesIO(data), attachment_filename='test', as_attachment=True)
    return render_template("home.html", form = form)
When you run it the download button will download the first file upload into database.
This example tutorial can be more complex.
For example, you can show the database content into a new HTML file and then save with an open dialog to disk.

Friday, August 2, 2019

Python 3.7.3 : Using the flask - part 006.

Today I will show you how to use the RESTful API application with flask python module that uses HTTP requests to GET, PUT, POST and DELETE data.
When HTTP is used, as is most common, the operations (HTTP methods) available are GET, HEAD, POST, PUT, PATCH, DELETE, CONNECT, OPTIONS and TRACE.[2], see the Wikipedia article.
All of these HTTP methods will be tested with postman software.
You need to have an account and use the downloaded software in order to interrogate with these methods.
Let's see the source new code first:
@app.route("/users/", methods=['GET'])
def users():
    users = User.query.all()
    #return users_schema.jsonify(users)
    all_users = users_schema.dump(users)
    return jsonify(all_users.data)

@app.route("/users/", methods=['POST'])
def user_post(id):
    user_post = User.query.get(id)
    print(user_post)
    username = request.json['username']
    email = request.json['email']
    user_post.username = username
    user_post.email = email
    db.session.commit()
    return user_schema.jsonify(user_post)

@app.route("/users/", methods=['PUT'])
def user_put(id):
    user_put = User.query.get(id)
    print(user_put)
    username = request.json['username']
    email = request.json['email']
    user_put.username = username
    user_put.email = email
    db.session.commit()
    return user_schema.jsonify(user_put)

@app.route("/users/", methods=['DELETE'])
def user_delete(id):
    user_delete = User.query.get(id)
    print(user_delete)
    db.session.delete(user_delete)
    db.session.commit()
    return user_schema.jsonify(user_delete)
Using the postman you can test it by running the python script and call these methods at http://127.0.0.1:5000/users/.