Python 3.8.5 : The hashlib python package - part 001.

The tutorial for today is about hashlib python module.
The official webpage comes for this python package has this intro:
This module implements a common interface to many different secure hash and message digest algorithms. Included are the FIPS secure hash algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 (defined in FIPS 180-2) as well as RSA’s MD5 algorithm (defined in Internet RFC 1321).
The example source code to test a simple hash is this:

import hashlib
import os

def file_sha1(filename):
    BUF_SIZE = 65536  # read stuff in 64kb chunks!
    get_sha1 = hashlib.sha1()
    with open(filename, 'rb') as f:
        while True:
            data = f.read(BUF_SIZE)
            if not data:
                break
            get_sha1.update(data)
    return get_sha1.hexdigest()

# I add this comment after first to see the hash difference.
files = [f for f in os.listdir('.') if os.path.isfile(f)]
for f in files:
    h = file_sha1(f)
    print(h) 

Let's test the source code with the default directory and two files.
I run it first with default source code and then I add a comment to test_hash_file.py file.
You can see the hash is changed from b222523567a8a806382b86578717ddbd00e0f4b4 to 2134660551cc67812413a3a75fd12efb05d591ef.

[mythcat@desk Projects_Python]$ ls
test_hash_file.py  test_numpy_001.py
[mythcat@desk Projects_Python]$ python test_hash_file.py 
98b2833527ad3d9fe263542c6aa06c04182d3dfb
b222523567a8a806382b86578717ddbd00e0f4b4
[mythcat@desk Projects_Python]$ python test_hash_file.py 
98b2833527ad3d9fe263542c6aa06c04182d3dfb
2134660551cc67812413a3a75fd12efb05d591ef

Python 3.8.5 : Pearson Product Moment Correlation with corrcoef from numpy.

The python package named numpy come with corrcoef function to return Pearson product-moment correlation coefficients.
This method has a limitation in that it can compute the correlation matrix between two variables only.
The full name is the Pearson Product Moment Correlation (PPMC).
The PPMC is not able to tell the difference between dependent variables and independent variables.
The documentation about this function can be found here.
More examples of Pearson Correlation can be found on this website.
My example presented in this tutorial, use the random packet to randomly generate integers and then calculate the correlation coefficients.
All of these are calculated five times in a for a cycle and each time the seed parameters are changed randomly.
Each time the correlation matrices are printed and then the random number graphs are displayed.
Let's see the source code:

import random

import numpy as np

nr_integers = 100
size_integers = 100

import matplotlib
import matplotlib.pyplot as plt

# set from 0 to 4 seed for random and show result 
for e in range(5):
    # change random seed
    np.random.seed(e)
    # nr_integers random integers between 0 and size_integers
    x = np.random.randint(0, size_integers, nr_integers)
    # Positive Correlation with some noise created with
    # nr_integers random integers between 0 and size_integers
    positive_y = x + np.random.normal(0, size_integers, nr_integers)
    correlation_positive = np.corrcoef(x, positive_y)
    # show matrix for correlation_positive
    print(correlation_positive)
    # Negative Correlation with same noise created with 
    # nr_integers random integers between 0 and size_integers
    negative_y = 100 - x + np.random.normal(0, size_integers, nr_integers)
    correlation_negative = np.corrcoef(x, negative_y)
    # show matrix for output with plt
    print(correlation_negative)
    # set graphic for plt with two graphics for each output with subplot
    plt.subplot(1, 2, 1)
    plt.scatter(x,positive_y)
    plt.subplot(1, 2, 2)
    plt.scatter(x,negative_y)
    # show the graph 
    plt.show()

Python 3.6.9 : My colab tutorials - part 008.

Today I deal with these two python packages named selenium and chromium-chromedriver.
I used selenium to get pieces of information from webpages.
These examples can be found at my GitHub project colab on the notebook named catafest_008.

Python 3.8.5 : PyEphem astronomy library for Python - part 001.

About this python package, you can find it from the official website.
PyEphem provides an ephem Python package for performing high-precision astronomy computations. The underlying numeric routines are coded in C and are the same ones that drive the popular XEphem astronomy application, whose author, Elwood Charles Downey, generously gave permission for their use in PyEphem. The name ephem is short for the word ephemeris, which is the traditional term for a table giving the position of a planet, asteroid, or comet for a series of dates.
Because I like astronomy and lately a lot has happened in this field, I thought I would come up with some simple tutorials for those who are interested.
This tutorial is tested on a Linux distribution called Fedora 32 with Python version 3.8.5.
I installed this python packet with the pip3 tool.

[mythcat@desk ~]$ sudo pip3 install ephem --user
WARNING: Running pip install with root privileges is generally not a good idea. 
Try `pip3 install --user` instead.
Collecting ephem
...
Installing collected packages: ephem
Successfully installed ephem-3.7.7.1

You know that NASA has launched the Mars Perseverance rover, so let's play with this topic a bit.
Let's see current azimuth and elevation for planet Mars.
For this is need the position of the observer and then is compute the position of the planet Mars.
I used the position of the Greenwich for the observer, but you can create your oun observer and use it.

[mythcat@desk ~]$ python3
Python 3.8.5 (default, Jul 20 2020, 00:00:00) 
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ephem
>>> import math
>>> convert = math.pi / 180.
>>> mars = ephem.Mars()
>>> greenwich = ephem.Observer()
>>> greenwich.lat = "51.477928"
>>> greenwich.lon = "-0.001545"
>>> mars.compute(greenwich)
>>> az_deg, alt_deg = mars.az*convert, mars.alt*convert
>>> print(f"Mars' current azimuth and elevation: {az_deg:.2f} {alt_deg:.2f}")
Mars' current azimuth and elevation: 0.11 -0.01

Let's see another example with Mars to take ascension and declination for the epoch specified:

...
>>> import datetime
>>> now = datetime.datetime.now()
>>> print(now)
2020-07-31 19:11:42.312027
>>> mars.compute(now)
>>> print(mars.ra)
1:12:43.64
>>> print(mars.dec)
3:33:22.6

You can get the magnitude, size (diameter in arcseconds) and size (radius as an angle):

>>> print(mars.mag)
-1.11
>>> print(mars.size)
14.555475234985352
>>> print(mars.radius)
0:00:07.3

You can easily see which constellation Mars is on.

>>> ephem.constellation(mars)
('Psc', 'Pisces')

Python 3.6.9 : My colab tutorials - parts 006 - 007.

This tutorial is called: My colab tutorials - parts 006 - 007.
The only reason for synchronization with the source code from my GitHub account on the Colab project.
I like collab more and more because I can quickly test the source code.
The example is taken from here and adapted to work on Colab and the new version of numba
Here is a simple example with the python numba package to creat that Mandelbrot fractal set.

import numba
from numba import jit

@jit
def mandel(x, y, max_iters):
  """
    Given the real and imaginary parts of a complex number,
    determine if it is a candidate for membership in the Mandelbrot
    set given a fixed number of iterations.
  """
  c = complex(x, y)
  z = 0.0j
  for i in range(max_iters):
    z = z*z + c
    if (z.real*z.real + z.imag*z.imag) >= 4:
      return i

  return max_iters

@jit
def create_fractal(min_x, max_x, min_y, max_y, image, iters):
  height = image.shape[0]
  width = image.shape[1]

  pixel_size_x = (max_x - min_x) / width
  pixel_size_y = (max_y - min_y) / height
    
  for x in range(width):
    real = min_x + x * pixel_size_x
    for y in range(height):
      imag = min_y + y * pixel_size_y
      color = mandel(real, imag, iters)
      image[y, x] = color

image = np.zeros((1024, 1536), dtype = np.uint8)
start = timer()
create_fractal(-2.0, 1.0, -1.0, 1.0, image, 20) 
dt = timer() - start

print ("Mandelbrot created in %f s" % dt)
imshow(image)
show() 

Python 3.8.2 : The numba python package - part 001 .

The development of this python package comes with this short intro:
Numba is a just-in-time compiler for Python that works best on code that uses NumPy arrays and functions and loops. The most common way to use Numba is through its collection of decorators that can be applied to your functions to instruct Numba to compile them. When a call is made to a Numba decorated function it is compiled to machine code “just-in-time” for execution and all or part of your code can subsequently run at native machine code speed!
I installed this python package on my folder Python38:

D:\Python38>pip3 install numba
Collecting numba
...
Successfully installed numba-0.50.1
D:\Python38>python.exe
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 22:45:29) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
>>> numba.__version__
'0.50.1'

This package did not work with python install on the folder Python38_64:

D:\Python38_64>pip3 install numba
Collecting numba
...
Installing collected packages: numpy, llvmlite, numba
Successfully installed llvmlite-0.33.0 numba-0.50.1 numpy-1.19.1
WARNING: You are using pip version 20.1; however, version 20.1.1 is available.
...
D:\Python38_64>python.exe
Python 3.8.4 (tags/v3.8.4:dfa645a, Jul 13 2020, 16:46:45) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
Traceback (most recent call last):
...
ModuleNotFoundError: No module named 'numba'

You can write standard Python functions and run them on a CUDA-capable GPU.
First, I need to enable this feature:

D:\Python38>SET NUMBA_ENABLE_CUDASIM=1

D:\Python38>python
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 22:45:29) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from numba import cuda
>>> print(cuda.gpus)
...Managed Device 0...

Let's test with a simple example to create a data and use it:

D:\Python38>python
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 22:45:29) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> import numpy as np
>>> from numba import cuda
>>>
>>> @cuda.jit
... def create(data):
...     data[cuda.blockIdx.x, cuda.threadIdx.x] = cuda.blockIdx.x
...
>>> numBlocks = 4
>>> threadsPerBlock = 6
>>>
>>> data = np.ones((numBlocks, threadsPerBlock), dtype=np.uint8)
>>> create[numBlocks, threadsPerBlock](data)
>>> print(data)
[[0 0 0 0 0 0]
 [1 1 1 1 1 1]
 [2 2 2 2 2 2]
 [3 3 3 3 3 3]]

Python Qt5 : Create a simple web browser.

This python package named PyQtWebEngine, see the official webpage for more infos:
The team development comes with this intro:
PyQtWebEngine is a set of Python bindings for The Qt Company’s Qt WebEngine framework. The framework provides the ability to embed web content in applications and is based on the Chrome browser. The bindings sit on top of PyQt5 and are implemented as three separate modules corresponding to the different libraries that make up the framework.
I used my python version: Python 3.8.3rc1 (tags/v3.8.3rc1:802eb67, Apr 29 2020, 21:39:14) [MSC v.1924 64 bit (AMD64)] on win32
... and PyQt5 version PyQt version: 5.15.0:

>>> from PyQt5.Qt import PYQT_VERSION_STR        
>>> print("PyQt version:", PYQT_VERSION_STR)  
PyQt version: 5.15.0

First, let's install this python package:

pip3 install PyQtWebEngine --user
Collecting PyQtWebEngine
  Using cached PyQtWebEngine-5.15.0-5.15.0-cp35.cp36.cp37.cp38-none-win_amd64.whl (57.9 MB)
Collecting PyQt5-sip<13>=12.8
  Using cached PyQt5_sip-12.8.0-cp38-cp38-win_amd64.whl (63 kB)
Collecting PyQt5>=5.15
  Using cached PyQt5-5.15.0-5.15.0-cp35.cp36.cp37.cp38-none-win_amd64.whl (64.5 MB)
Installing collected packages: PyQt5-sip, PyQt5, PyQtWebEngine
  WARNING: The scripts pylupdate5.exe, pyrcc5.exe and pyuic5.exe are installed in 
'C:\Users\catal\AppData\Roaming\Python\Python38\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, 
use --no-warn-script-location.
Successfully installed PyQt5-5.15.0 PyQt5-sip-12.8.0 PyQtWebEngine-5.15.0

The WARNING reflects the path for this tool, but you can find on user folder, see:

C:\Users\catal>pylupdate5.exe
Usage:
    pylupdate5 [options] project-file
    pylupdate5 [options] source-files -ts ts-files

Options:
    -help  Display this information and exit
    -version
           Display the version of pylupdate5 and exit
    -verbose
           Explain what is being done
    -noobsolete
           Drop all obsolete strings
    -tr-function name
           name() may be used instead of tr()
    -translate-function name
           name() may be used instead of translate()

C:\Users\catal>

This is a simple source code with an browser example:

import sys
from PyQt5.Qt import *
from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWidgets import QMainWindow, QWidget, QLabel, QLineEdit
from PyQt5.QtWidgets import QPushButton
from PyQt5.QtCore import QSize, QUrl
from PyQt5.QtWebEngineWidgets import *
from PyQt5.QtWebEngineWidgets import QWebEnginePage, QWebEngineView
class MainWindow(QMainWindow):
    def __init__(self):
        super(MainWindow, self).__init__()

        self.setWindowTitle('catafest browser')

        self.browser_toolbar = QToolBar()
        self.addToolBar(self.browser_toolbar)
        self.back_button = QPushButton()
        #self.back_button.setIcon(QIcon('left.png'))
        self.back_button.clicked.connect(self.back_page)
        self.browser_toolbar.addWidget(self.back_button)
        self.forward_button = QPushButton()
        #self.forward_button.setIcon(QIcon('right.png'))
        self.forward_button.clicked.connect(self.forward_page)
        self.browser_toolbar.addWidget(self.forward_button)

        self.web_address = QLineEdit()
        self.web_address.returnPressed.connect(self.load_page)
        self.browser_toolbar.addWidget(self.web_address)

        self.web_browser = QWebEngineView()
        self.setCentralWidget(self.web_browser)
        first_url = "https://pypi.org/project/PyQt5/"

        self.web_address.setText(first_url)
        self.web_browser.load(QUrl(first_url))
        self.web_browser.page().titleChanged.connect(self.setWindowTitle)
        self.web_browser.page().urlChanged.connect(self.changed_page)
        
    def load_page(self):
        url = QUrl.fromUserInput(self.web_address.text())
        if url.isValid():
            self.web_browser.load(url)

    def back_page(self):
        self.web_browser.page().triggerAction(QWebEnginePage.Back)

    def forward_page(self):
        self.web_browser.page().triggerAction(QWebEnginePage.Forward)

    def changed_page(self, url):
        self.web_address.setText(url.toString())
        
if __name__ == "__main__":
    app = QtWidgets.QApplication(sys.argv)
    mainWin = MainWindow()
    availableGeometry = app.desktop().availableGeometry(mainWin)
    mainWin.resize(availableGeometry.width(), availableGeometry.height())
    mainWin.show()
    sys.exit( app.exec_() )

Python 3.8.3 : Lists in Python 3 - part 001.

I am currently working on a project that involves the use of complex data structures and lists and my time is limited.
This led me to start a new series of python tutorials on python lists.
I realized that the lists had no substantial changes in the evolution of the python programming language, see the official documentation.
You will find on the internet a lot of questions related to lists, algorithms, and problems involving lists.
If you are not a beginner then it will seem boring at first but over time I will try to draw attention to really significant elements in python programming with lists.
Let's start taking significant steps in using lists.
Before using the methods of list objects, let's create some lists:

# create a empty list 
my_list_001 = []
# fill the list with 4 consecutive numbers from 0 to 3 
my_list_001 = list(range(0,4))
# show the list 
my_list_001
[0, 1, 2, 3]
# set the variable n with value 4
n = 4
# create a list with 4 zeroes 
list_of_zeros = [0] * n
# show the list
listofzeros
[0, 0, 0, 0]
# import string python package 
import string
# create a string with all ascii letters
my_letters = string.ascii_letters
# use list to create a list with all letters
list(my_letters)
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't',
 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O',
 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z']
# create a nested list is a list can also have another list as an item
my_list = ["catafest", [1,9,7,6], ["!"]]
# show the list
my_list
['catafest', [1, 9, 7, 6], ['!']]
# access elements from a list with index operator pozitive and negative [] 
my_list[0]
'catafest'
my_list[-1]
['!']
my_list[-2]
[1, 9, 7, 6]
my_list[-3]
'catafest'
# test conditions boolean
my_list[-3] == my_list[0]
True
my_list[-3] != my_list[0]
False
my_list_001[1] > listofzeros[1]
True
# sum an list element with another
sum_of_list_elemenet = my_list_001[1] + listofzeros[1]
# show result
sum_of_list_elemenet
1
# iterating through a list
for list_element in my_list:
...     print(list_element)
...
catafest
[1, 9, 7, 6]
['!']
# iterating each list  
for list_element in my_list:
...     for e in list_element:
...             print(e)
...
c
a
t
a
f
e
s
t
1
9
7
6
!
# create a list with all letters using chr and ord built-in functions
[chr(i) for i in range(ord('a'),ord('z')+1)]
# the result is
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't',
 'u', 'v', 'w', 'x', 'y', 'z']
# create a binary list from a integer
output = [int(x) for x in '{:08b}'.format(1976)]
# show the result of the list
output
[1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0]
# show the result just for on element of the list
output[3]
1
# convert a binary list to integer named result and set first with value 0
# using the bitwise left shift operator
result = 0
for digits in output:
    result = (result << 1) | digits
# show result of conversion from binary to integer
result
1976
# create a hexadecimal list from a list
hex_list = [hex(x) for x in my_list_001]
# show the hexadecimal list using the hex built-in function
hex_list
['0x0', '0x1', '0x2', '0x3']
# convert hexadecimal list to int list using the int 
int_list = [int(x,0) for x in hex_list]
# show the result
int_list
[0, 1, 2, 3]
# convert hexadecimal list to float list using the float.fromhex
float_list = [float.fromhex(x) for x in hex_list]
# show the result
float_list
[0.0, 1.0, 2.0, 3.0]
# create list using math python module 
import math
[2*x for x in my_list_001]
[0, 2, 4, 6]
[math.sin(x) for x in my_list_001]
[0.0, 0.8414709848078965, 0.9092974268256817, 0.1411200080598672]
# create sized list
sized_list = [1] * 4
# show the result
sized_list
[1, 1, 1, 1]
# list from 
>>> chars = ''.join(map(chr, range(32, 1048)))
>>> list(chars)
[' ', '!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5',
 '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K',
 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '[', '\\', ']', '^', '_', '`', 'a',
 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w',
 'x', 'y', 'z', '{', '|', '}', '~', '\x7f', '\x80', '\x81', '\x82', '\x83', '\x84', '\x85', '\x86', '\x87', 
'\x88', '\x89', '\x8a', '\x8b', '\x8c', '\x8d', '\x8e', '\x8f', '\x90', '\x91', '\x92', '\x93', '\x94', '\x95',
 '\x96', '\x97', '\x98', '\x99', '\x9a', '\x9b', '\x9c', '\x9d', '\x9e', '\x9f', '\xa0', '¡', '¢', '£', '¤', '¥',
 '¦', '§', '¨', '©', 'ª', '«', '¬', '\xad', '®', '¯', '°', '±', '²', '³', '´', 'µ', '¶', '·', '¸', '¹', 'º', '»',
 '¼', '½', '¾', '¿', 'À', 'Á', 'Â', 'Ã', 'Ä', 'Å', 'Æ', 'Ç', 'È', 'É', 'Ê', 'Ë', 'Ì', 'Í', 'Î', 'Ï', 'Ð', 'Ñ', 'Ò',
 'Ó', 'Ô', 'Õ', 'Ö', '×', 'Ø', 'Ù', 'Ú', 'Û', 'Ü', 'Ý', 'Þ', 'ß', 'à', 'á', 'â', 'ã', 'ä', 'å', 'æ', 'ç', 'è', 'é',
 'ê', 'ë', 'ì', 'í', 'î', 'ï', 'ð', 'ñ', 'ò', 'ó', 'ô', 'õ', 'ö', '÷', 'ø', 'ù', 'ú', 'û', 'ü', 'ý', 'þ', 'ÿ', 'Ā',
 'ā', 'Ă', 'ă', 'Ą', 'ą', 'Ć', 'ć', 'Ĉ', 'ĉ', 'Ċ', 'ċ', 'Č', 'č', 'Ď', 'ď', 'Đ', 'đ', 'Ē', 'ē', 'Ĕ', 'ĕ', 'Ė', 'ė',
 'Ę', 'ę', 'Ě', 'ě', 'Ĝ', 'ĝ', 'Ğ', 'ğ', 'Ġ', 'ġ', 'Ģ', 'ģ', 'Ĥ', 'ĥ', 'Ħ', 'ħ', 'Ĩ', 'ĩ', 'Ī', 'ī', 'Ĭ', 'ĭ', 'Į',
 'į', 'İ', 'ı', 'IJ', 'ij', 'Ĵ', 'ĵ', 'Ķ', 'ķ', 'ĸ', 'Ĺ', 'ĺ', 'Ļ', 'ļ', 'Ľ', 'ľ', 'Ŀ', 'ŀ', 'Ł', 'ł', 'Ń', 'ń', 'Ņ',
 'ņ', 'Ň', 'ň', 'ʼn', 'Ŋ', 'ŋ', 'Ō', 'ō', 'Ŏ', 'ŏ', 'Ő', 'ő', 'Œ', 'œ', 'Ŕ', 'ŕ', 'Ŗ', 'ŗ', 'Ř', 'ř', 'Ś', 'ś', 'Ŝ',
 'ŝ', 'Ş', 'ş', 'Š', 'š', 'Ţ', 'ţ', 'Ť', 'ť', 'Ŧ', 'ŧ', 'Ũ', 'ũ', 'Ū', 'ū', 'Ŭ', 'ŭ', 'Ů', 'ů', 'Ű', 'ű', 'Ų', 'ų', 
'Ŵ', 'ŵ', 'Ŷ', 'ŷ', 'Ÿ', 'Ź', 'ź', 'Ż', 'ż', 'Ž', 'ž', 'ſ', 'ƀ', 'Ɓ', 'Ƃ', 'ƃ', 'Ƅ', 'ƅ', 'Ɔ', 'Ƈ', 'ƈ', 'Ɖ', 'Ɗ', 'Ƌ', 'ƌ',
 'ƍ', 'Ǝ', 'Ə', 'Ɛ', 'Ƒ', 'ƒ', 'Ɠ', 'Ɣ', 'ƕ', 'Ɩ', 'Ɨ', 'Ƙ', 'ƙ', 'ƚ', 'ƛ', 'Ɯ', 'Ɲ', 'ƞ', 'Ɵ', 'Ơ', 'ơ', 'Ƣ', 'ƣ',
...
...
 'ϗ', 'Ϙ', 'ϙ', 'Ϛ', 'ϛ', 'Ϝ', 'ϝ', 'Ϟ', 'ϟ', 'Ϡ', 'ϡ', 'Ϣ', 'ϣ', 'Ϥ', 'ϥ', 'Ϧ', 'ϧ', 'Ϩ', 'ϩ', 'Ϫ', 'ϫ', 'Ϭ', 'ϭ',
 'Ϯ', 'ϯ', 'ϰ', 'ϱ', 'ϲ', 'ϳ', 'ϴ', 'ϵ', '϶', 'Ϸ', 'ϸ', 'Ϲ', 'Ϻ', 'ϻ', 'ϼ', 'Ͻ', 'Ͼ', 'Ͽ', 'Ѐ', 'Ё', 'Ђ', 'Ѓ', 'Є',
 'Ѕ', 'І', 'Ї', 'Ј', 'Љ', 'Њ', 'Ћ', 'Ќ', 'Ѝ', 'Ў', 'Џ', 'А', 'Б', 'В', 'Г', 'Д', 'Е', 'Ж', 'З']

This is not all about create and convert list, but it is quite close to the reality of the lists.
This tutorial remains open due to the complex issue and maybe I will complete it in the future.

Python 3.8.3 : Short intro to Appium-Python-Client python package.

This is a short intro of the Appium-Python-Client python package and Appium based on Client-Server Architecture.
The Appium Server can be installed using two ways: using NPM or using Appium Desktop.
I download and run the desktop version from here.

Appium-windows-1.18.0-beta.1>Appium.exe

The latest version of Java, needed for Android Studio ( you can use the installation of Android Studio with SDK) and mobile phone set on USB debugging.
The next step is to set all settings for android into Appium interface:

Using appium server, you can send commands to the Appium Server which translates it to platform-specific commands and executes on the devices.
The Appium-Python-Client python package is an extension library for use with the mobile testing framework Appium, see the official webpage.
Install Appium-Python-Client python package with pip3 tool.

pip3 install Appium-Python-Client
Collecting Appium-Python-Client
...
Successfully built Appium-Python-Client
Installing collected packages: Appium-Python-Client
Successfully installed Appium-Python-Client-1.0.1

Now you can test this python package with Appium and simple examples.

Python 3.8.3 : About aiohttp python package.

This python package can help you to writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives, see the official documentation.
In this simple tutorial, I will show you in a few simple steps how to use it.
It is a complex module and there are multiple ways to use it.
First, on the Windows operating system users can install easily with:

pip3 install aiohttp
Collecting aiohttp
...
Installing collected packages: attrs, multidict, yarl, async-timeout, aiohttp
Successfully installed aiohttp-3.6.2 async-timeout-3.0.1 attrs-19.3.0 multidict-4.7.6 yarl-1.4.2

If you use a Linus operating system then you can use this command:

[mythcat@desk ~]$ pip3 install aiohttp --user 
...
Successfully installed aiohttp-3.6.2 async-timeout-3.0.1 multidict-4.7.5 yarl-1.4.2

This python package can be used as client or server.
The aiohttp.web implements a basic CLI for quickly serving an Application in development over TCP/IP.
You can find some example on this webpage.
These simple examples show how you can use handlers for web and servers and a request handler with a coroutine.

[mythcat@desk ~]$ python3 
Python 3.7.6 (default, Jan 30 2020, 09:44:41) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from aiohttp import web 
>>> def handler_req(request):
...     return web.Response(text='Handler')
... 
>>> handler_req
<function 0x7fc8a76aab00="" at="" handler_req="">
>>> async def handler_req_async(request):
...     return web.Response(text='async Handler')
... 
>>> handler_req_async
<function+0x7fc8a574edd0...
>>> import aiohttp
>>> from aiohttp import web
>>> from aiohttp.client import _RequestContextManager
>>> async def test_await(test_server, loop):
...  
...     async def handler(request):
...         return web.HTTPOk()
...
...     app = web.Application(loop=loop)
...     app.router.add_route('GET', '/', handler)
...     server = await test_server(app)
...     resp = await aiohttp.get(server.make_url('/'), loop=loop)
...     assert resp.status == 200
...     assert resp.connection is not None
...     await resp.release()
...     assert resp.connection is None
>>> test_await('htthttp://localhost:8080/',1000)   
<coroutine object test_await at 0x00000261FB96EA40>

Let's see the output of the dir:

dir(aiohttp)
['AsyncIterablePayload', 'AsyncResolver', 'BadContentDispositionHeader', 'BadContentDispositionParam', 
'BaseConnector', 'BasicAuth', 'BodyPartReader', 'BufferedReaderPayload', 'BytesIOPayload', 'BytesPayload',
 'ChainMapProxy', 'ClientConnectionError', 'ClientConnectorCertificateError', 'ClientConnectorError', 
'ClientConnectorSSLError', 'ClientError', 'ClientHttpProxyError', 'ClientOSError', 'ClientPayloadError', 
'ClientProxyConnectionError', 'ClientRequest', 'ClientResponse', 'ClientResponseError', 'ClientSSLError', 
'ClientSession', 'ClientTimeout', 'ClientWebSocketResponse', 'ContentTypeError', 'CookieJar', 'DataQueue', 
'DefaultResolver', 'DummyCookieJar', 'EMPTY_PAYLOAD', 'EofStream', 'Fingerprint', 'FlowControlDataQueue', 
'FormData', 'HttpVersion', 'HttpVersion10', 'HttpVersion11', 'IOBasePayload', 'InvalidURL', 'JsonPayload', 
'MultipartReader', 'MultipartWriter', 'NamedPipeConnector', 'PAYLOAD_REGISTRY', 'Payload', 'RequestInfo', 
'ServerConnectionError', 'ServerDisconnectedError', 'ServerFingerprintMismatch', 'ServerTimeoutError', 
'Signal', 'StreamReader', 'StringIOPayload', 'StringPayload', 'TCPConnector', 'TextIOPayload', 
'ThreadedResolver', 'TooManyRedirects', 'TraceConfig', 'TraceConnectionCreateEndParams', 
'TraceConnectionCreateStartParams', 'TraceConnectionQueuedEndParams', 'TraceConnectionQueuedStartParams', 
'TraceConnectionReuseconnParams', 'TraceDnsCacheHitParams', 'TraceDnsCacheMissParams', 
'TraceDnsResolveHostEndParams', 'TraceDnsResolveHostStartParams', 'TraceRequestChunkSentParams', 
'TraceRequestEndParams', 'TraceRequestExceptionParams', 'TraceRequestRedirectParams', 'TraceRequestStartParams',
 'TraceResponseChunkReceivedParams', 'Tuple', 'UnixConnector', 'WSCloseCode', 'WSMessage', 'WSMsgType', 
'WSServerHandshakeError', 'WebSocketError', '__all__', '__builtins__', '__cached__', '__doc__', '__file__',
 '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'abc', 'base_protocol', 
'client', 'client_exceptions', 'client_proto', 'client_reqrep', 'client_ws', 'connector', 
'content_disposition_filename', 'cookiejar', 'formdata', 'frozenlist', 'get_payload', 'hdrs', 'helpers', 
'http', 'http_exceptions', 'http_parser', 'http_websocket', 'http_writer', 'locks', 'log', 'multipart', 
'parse_content_disposition', 'payload', 'payload_streamer', 'payload_type', 'request', 'resolver', 'signals', 
'streamer', 'streams', 'tcp_helpers', 'tracing', 'typedefs']

Python 3.8.3 : PyCryptodome python package - part 001.

In the last tutorial, I wrote on Sunday, June 16, 2019, you can see a simple example of this python package with KDF with PBKDF2 function.

I guess it should be interesting for visitors to this blog to read more about this package because it is very useful and interesting.

Today I come up with another tutorial covering how to use A.E.S. standard encryption and decrypting text in a binary file.

The A.E.S. is a standard?

The Federal Information Processing Standards Publications (FIPS PUBS) announcing the A.E.S. on November 26, 2001, on the Federal Information Processing Standards Publication 197.

A.E.S. known as Advanced Encryption Standard is a symmetric block cipher standardized by NIST.

The N.I.S.T is an abbreviation National Institute of Standards and Technology.

This python package named PyCryptodome is a self-contained Python package of low-level cryptographic primitives, see the readthedocs.io webpage.

First, you need to see if this python package is not on conflict with another one named PyCrypto.

Then use pip3 tool to install.

pip3 uninstall PyCrypto
WARNING: Skipping PyCrypto as it is not installed.
...
pip3 install pycryptodome
Collecting pycryptodome
...
Installing collected packages: pycryptodome
Successfully installed pycryptodome-3.9.8

Here is the source code commented for a better understanding of the encryption and decryption steps.

from Crypto.Cipher import AES
from Crypto.Random import get_random_bytes

# the data input needs to be encoded, else you will receive this error
# python TypeError: Object type  cannot be passed to C code

data = "Hello World! by catafest!".encode("utf8")
# first step - prepare for encrypt
key = get_random_bytes(16)
print("print key is: ", key)
cipher = AES.new(key, AES.MODE_EAX)
print("print cipher is: ", cipher)
ciphertext, tag = cipher.encrypt_and_digest(data)
print("print ciphertext is: ", ciphertext)

# open the binary file 
file_out = open("AES_encrypted.bin", "wb")
# write to binary file 
[ file_out.write(x) for x in (cipher.nonce, tag, ciphertext) ]

# the file is close 
file_out.close()

# next step - prepare for decrypt
new_cipher = AES.new(key, AES.MODE_EAX, cipher.nonce)

# open again the binary file 
file_in = open("AES_encrypted.bin", "rb")
# read all data from file
nonce, tag, ciphertext = [file_in.read(x) for x in (16, 16, -1) ]
# the file is close 
file_out.close()

# show data from file
print("print nonce is: ", nonce)
print("print tag is: ", tag)
print("print ciphertext is: ", ciphertext)

# create a new cipher
cipher = AES.new(key, AES.MODE_EAX, nonce)

# use this to decrypt
data = cipher.decrypt_and_verify(ciphertext, tag)
# show the result 
print(data)

The result of running the python script is this:

python.exe .\pycryptodome_AES_001.py
print key is:  b'\xbdEX\xf8\x1d!\xc5\xceI\x87\x81\xf1\xd5\xba\x8c\r'
print cipher is:  <Crypto.Cipher._mode_eax.EaxMode object at 0x000001D5F32DE100>
print cipher is:  <Crypto.Cipher._mode_eax.EaxMode object at 0x000001E062AFE100>
print ciphertext is:  b'ON\x1d\xb9\xb7\xa8\xf5\xd6\x0c\x91\xc5`B\xf4\x95u\xe1D\xb5\x88&I\x15\xc5\xc5'
print nonce is:  b'3\xa6R8\xbb\n \x9cimp$\xe4\xee\xf5-'
print tag is:  b'\xb8)\xe4\xe7\x08uE~\x84s]\xedX\xf5\xf9\xea'
print ciphertext is:  b'ON\x1d\xb9\xb7\xa8\xf5\xd6\x0c\x91\xc5`B\xf4\x95u\xe1D\xb5\x88&I\x15\xc5\xc5'
b'Hello World! by catafest!'

You can read about this process on this website.

Python 2.7.10 : IronPython and C# with Dynamic Language Runtime.

This is a simple tutorial about python and C# using the Dynamic Language Runtime with IronPython.

I use Visual Studio 2019 and .NET Framework 4.7.2 with my Console C# project named DynamicLanguageRuntime_001.

Let's install the package with Visual Studio by open the console using the main menu: Tools - NuGet Package Manager - Package Manager Console command.

PM> Install-Package DynamicLanguageRuntime
Package 'DynamicLanguageRuntime.1.2.3' already exists in project 'DynamicLanguageRuntime_001'
Time Elapsed: 00:00:01.2208674

Use Solution Explorer use right-click on References item from your project and use Add Reference ...

Into the new window dialog named Reference Manager on the Assemblies - Framework uses the edit box to search IronPython.

Then use the checkbox to select these two options: IronPython and IronPython.Modules.

See the screenshot from Visual Studio I.D.E.:

This is the source code I used:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using IronPython.Hosting;
using IronPython.Runtime;
using IronPython;
using Microsoft.Scripting.Hosting;
using Microsoft.Scripting;

namespace DynamicLanguageRuntime_001
{
    class Program
    {
        static void Main(string[] args)
        {
            // create python engine 
            ScriptEngine engine = Python.CreateEngine();
            // get and add paths to enfine
            var python_search_paths = engine.GetSearchPaths();
            python_search_paths.Add(@"C:\Program Files\IronPython 2.7\Lib");
            engine.SetSearchPaths(python_search_paths);
            // create a scope 
            ScriptScope scope = engine.CreateScope();
            // using CreateScriptSourceFromString
            engine.CreateScriptSourceFromString("print '... simple example with ironpython and C#'").Execute();
            // using Execute
            engine.Execute("print '                             by catafest!'", scope);
            // using ExecuteFile
            engine.ExecuteFile(@"D:\Projects\Python\testing\test_webpage_001.py", scope);
            dynamic testFunction = scope.GetVariable("GetFriends");
            var result = testFunction();
        }
    }
} 

Python 3.6.9 : My colab tutorials - part 005.

Today I tested google colab python features with google authentification and google.colab drive and files.
The first part of google colab code comes with authentification and you need to add the verification code for google account.
The google colab use the same version of python:

3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]

You can see all source code on my GitHub account.
The notebook can be found here.