Python 3.8.2 : The numba python package - part 001 .

The development of this python package comes with this short intro:
Numba is a just-in-time compiler for Python that works best on code that uses NumPy arrays and functions and loops. The most common way to use Numba is through its collection of decorators that can be applied to your functions to instruct Numba to compile them. When a call is made to a Numba decorated function it is compiled to machine code “just-in-time” for execution and all or part of your code can subsequently run at native machine code speed!
I installed this python package on my folder Python38:

D:\Python38>pip3 install numba
Collecting numba
...
Successfully installed numba-0.50.1
D:\Python38>python.exe
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 22:45:29) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
>>> numba.__version__
'0.50.1'

This package did not work with python install on the folder Python38_64:

D:\Python38_64>pip3 install numba
Collecting numba
...
Installing collected packages: numpy, llvmlite, numba
Successfully installed llvmlite-0.33.0 numba-0.50.1 numpy-1.19.1
WARNING: You are using pip version 20.1; however, version 20.1.1 is available.
...
D:\Python38_64>python.exe
Python 3.8.4 (tags/v3.8.4:dfa645a, Jul 13 2020, 16:46:45) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
Traceback (most recent call last):
...
ModuleNotFoundError: No module named 'numba'

You can write standard Python functions and run them on a CUDA-capable GPU.
First, I need to enable this feature:

D:\Python38>SET NUMBA_ENABLE_CUDASIM=1

D:\Python38>python
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 22:45:29) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from numba import cuda
>>> print(cuda.gpus)
...Managed Device 0...

Let's test with a simple example to create a data and use it:

D:\Python38>python
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 22:45:29) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> import numpy as np
>>> from numba import cuda
>>>
>>> @cuda.jit
... def create(data):
...     data[cuda.blockIdx.x, cuda.threadIdx.x] = cuda.blockIdx.x
...
>>> numBlocks = 4
>>> threadsPerBlock = 6
>>>
>>> data = np.ones((numBlocks, threadsPerBlock), dtype=np.uint8)
>>> create[numBlocks, threadsPerBlock](data)
>>> print(data)
[[0 0 0 0 0 0]
 [1 1 1 1 1 1]
 [2 2 2 2 2 2]
 [3 3 3 3 3 3]]

Python Qt5 : Create a simple web browser.

This python package named PyQtWebEngine, see the official webpage for more infos:
The team development comes with this intro:
PyQtWebEngine is a set of Python bindings for The Qt Company’s Qt WebEngine framework. The framework provides the ability to embed web content in applications and is based on the Chrome browser. The bindings sit on top of PyQt5 and are implemented as three separate modules corresponding to the different libraries that make up the framework.
I used my python version: Python 3.8.3rc1 (tags/v3.8.3rc1:802eb67, Apr 29 2020, 21:39:14) [MSC v.1924 64 bit (AMD64)] on win32
... and PyQt5 version PyQt version: 5.15.0:

>>> from PyQt5.Qt import PYQT_VERSION_STR        
>>> print("PyQt version:", PYQT_VERSION_STR)  
PyQt version: 5.15.0

First, let's install this python package:

pip3 install PyQtWebEngine --user
Collecting PyQtWebEngine
  Using cached PyQtWebEngine-5.15.0-5.15.0-cp35.cp36.cp37.cp38-none-win_amd64.whl (57.9 MB)
Collecting PyQt5-sip<13>=12.8
  Using cached PyQt5_sip-12.8.0-cp38-cp38-win_amd64.whl (63 kB)
Collecting PyQt5>=5.15
  Using cached PyQt5-5.15.0-5.15.0-cp35.cp36.cp37.cp38-none-win_amd64.whl (64.5 MB)
Installing collected packages: PyQt5-sip, PyQt5, PyQtWebEngine
  WARNING: The scripts pylupdate5.exe, pyrcc5.exe and pyuic5.exe are installed in 
'C:\Users\catal\AppData\Roaming\Python\Python38\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, 
use --no-warn-script-location.
Successfully installed PyQt5-5.15.0 PyQt5-sip-12.8.0 PyQtWebEngine-5.15.0

The WARNING reflects the path for this tool, but you can find on user folder, see:

C:\Users\catal>pylupdate5.exe
Usage:
    pylupdate5 [options] project-file
    pylupdate5 [options] source-files -ts ts-files

Options:
    -help  Display this information and exit
    -version
           Display the version of pylupdate5 and exit
    -verbose
           Explain what is being done
    -noobsolete
           Drop all obsolete strings
    -tr-function name
           name() may be used instead of tr()
    -translate-function name
           name() may be used instead of translate()

C:\Users\catal>

This is a simple source code with an browser example:

import sys
from PyQt5.Qt import *
from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWidgets import QMainWindow, QWidget, QLabel, QLineEdit
from PyQt5.QtWidgets import QPushButton
from PyQt5.QtCore import QSize, QUrl
from PyQt5.QtWebEngineWidgets import *
from PyQt5.QtWebEngineWidgets import QWebEnginePage, QWebEngineView
class MainWindow(QMainWindow):
    def __init__(self):
        super(MainWindow, self).__init__()

        self.setWindowTitle('catafest browser')

        self.browser_toolbar = QToolBar()
        self.addToolBar(self.browser_toolbar)
        self.back_button = QPushButton()
        #self.back_button.setIcon(QIcon('left.png'))
        self.back_button.clicked.connect(self.back_page)
        self.browser_toolbar.addWidget(self.back_button)
        self.forward_button = QPushButton()
        #self.forward_button.setIcon(QIcon('right.png'))
        self.forward_button.clicked.connect(self.forward_page)
        self.browser_toolbar.addWidget(self.forward_button)

        self.web_address = QLineEdit()
        self.web_address.returnPressed.connect(self.load_page)
        self.browser_toolbar.addWidget(self.web_address)

        self.web_browser = QWebEngineView()
        self.setCentralWidget(self.web_browser)
        first_url = "https://pypi.org/project/PyQt5/"

        self.web_address.setText(first_url)
        self.web_browser.load(QUrl(first_url))
        self.web_browser.page().titleChanged.connect(self.setWindowTitle)
        self.web_browser.page().urlChanged.connect(self.changed_page)
        
    def load_page(self):
        url = QUrl.fromUserInput(self.web_address.text())
        if url.isValid():
            self.web_browser.load(url)

    def back_page(self):
        self.web_browser.page().triggerAction(QWebEnginePage.Back)

    def forward_page(self):
        self.web_browser.page().triggerAction(QWebEnginePage.Forward)

    def changed_page(self, url):
        self.web_address.setText(url.toString())
        
if __name__ == "__main__":
    app = QtWidgets.QApplication(sys.argv)
    mainWin = MainWindow()
    availableGeometry = app.desktop().availableGeometry(mainWin)
    mainWin.resize(availableGeometry.width(), availableGeometry.height())
    mainWin.show()
    sys.exit( app.exec_() )

Python 3.8.3 : Simple example to fix maximum recursion depth exceeded.

This short tutorial try to solve simple and easy the stack limit for recursion without using advanced programming techniques.

sys.setrecursionlimit(limit)
    Set the maximum depth of the Python interpreter stack to limit. 
This limit prevents infinite recursion from causing an overflow of the C stack 
and crashing Python.

This can be done in most cases with:

import sys
sys.setrecursionlimit(1976)

Let's test with an simple example for a given number.

input_number = 1234567890987654321

I create a script for the next function to print the result:

def split_input(n):
    return split_input(n // 10) + [n % 10]

The result of this function is:

python flow_001.py
Traceback (most recent call last):
  File "flow_001.py", line 12, in 
    input_list = split_input(input_number)
  File "flow_001.py", line 10, in split_input
    return split_input(n // 10) + [n % 10]
  File "flow_001.py", line 10, in split_input
    return split_input(n // 10) + [n % 10]
  File "flow_001.py", line 10, in split_input
    return split_input(n // 10) + [n % 10]
  [Previous line repeated 996 more times]
RecursionError: maximum recursion depth exceeded

You can see I got an error. Let fix this function with a proper source code:

def split_nr(n):
    if n < 1:
        return [n]
    else:
        return split_nr(n // 10) + [n % 10] 

This will solve the recursion split function.

python flow_001.py
[1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 9, 8, 7, 6, 5, 4, 3, 2, 1]

IronPython 2.7.7 : Intro on RevitPythonShell.

This is one of the several ways to use RevitPythonWrapper:

• pyRevit;
• RevitPythonShell;
• Dynamo;

This tool help you to use python with the Revit tool and can be found at the GitHub webpage.

They say:
The RevitPythonShell adds an IronPython interpreter to Autodesk Revit and Vasari. The RevitPythonShell (RPS) lets you to write plugins for Revit in Python, my favourite scripting language! But even better, it provides you with an interactive shell that lets you see the results of your code as you type it. This is great for exploring the Revit API while writing your Revit Addins - use this in combination with the RevitLookup database exploration tool to become a Revit API Ninja :)
You need to install your instaler according with the Revit version.
You can start it from Main Menu - Add-Ins and you see it.
Is very easy to use it, let's see one simple example:

import clr
clr.AddReference("RevitAPI")
from Autodesk.Revit.DB import BuiltInCategory as Bic
from Autodesk.Revit.DB import FilteredElementCollector as Fec
from Autodesk.Revit.DB import Transaction
# reference to the current open revit model is
doc = __revit__.ActiveUIDocument.Document

Let's see a simple example on Revit with an wall and a simple door.

IronPython 2.7.7 (2.7.7.0) on .NET 4.0.30319.42000 (64-bit)
Type "help", "copyright", "credits" or "license" for more information.
>>> a = b = 1 
>>> c = a + b + 1
>>> print(c)
3
>>> from Autodesk.Revit.DB import FilteredElementCollector as Fec
>>> from Autodesk.Revit.DB import BuiltInCategory as Bic
>>> doors = Fec(doc).OfCategory(Bic.OST_Doors).WhereElementIsNotElementType().ToElements()
>>> print(dir(doors))
['Add', 'AddRange', 'AsReadOnly', 'BinarySearch', 'Capacity', 'Clear', 'Contains', 'ConvertAll', 'CopyTo',
'Count', 'Enumerator', 'Equals', 'Exists', 'Find', 'FindAll', 'FindIndex', 'FindLast', 'FindLastIndex', 
'ForEach', 'GetEnumerator', 'GetHashCode', 'GetRange', 'GetType', 'IndexOf', 'Insert', 'InsertRange', 
'IsReadOnly', 'IsSynchronized', 'Item', 'LastIndexOf', 'MemberwiseClone', 'ReferenceEquals', 'Remove', 
'RemoveAll', 'RemoveAt', 'RemoveRange', 'Reverse', 'Sort', 'SyncRoot', 'ToArray', 'ToString', 'TrimExcess', 
'TrueForAll', '__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__format__', 
'__getattribute__', '__getitem__', '__hash__', '__init__', '__iter__', '__len__', '__new__', 
'__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__', 
'__str__', '__subclasshook__']
>>> for each_door in doors:
... 	print(each_door.Id)
... 	print(each_door.Mirrored)
... 	print(each_door.Symbol.LookupParameter("Type Name").ToString())
... 
840590
False
Autodesk.Revit.DB.Parameter
>>> 

Let's see the output of this code on Revit.

Python 3.8.3 : Pyxel free game engine.

Pyxel is a free game engine is build for create old fashioned pixel art style games easily.
Pyxel is published under MIT License.
This allow you to use 2D sprites, sound and interactions.
The project can be found at the GitHub webpage.
The basic features are:

• Run on Windows, Mac, and Linux
• Code writing with Python3
• Fixed 16 color palette
• 256x256 sized 3 image banks
• 256x256 sized 8 tilemaps
• 4 channels with 64 definable sounds
• 8 musics which can combine arbitrary sounds
• Keyboard, mouse, and gamepad inputs
• Image and sound editor

The instalation is easy.

pip3 install pyxel
Collecting pyxel
...
Successfully installed altgraph-0.17 pefile-2019.4.18 pyinstaller-3.6 pywin32-ctypes-0.2.0 pyxel-1.3.7

Use this command to start the tool for create sprites and sounds.

pyxeleditor

You can test many examples on GitHub.
The basic example from the web is simple:

import pyxel

class App:
    def __init__(self):
        pyxel.init(160, 120, caption="Hello Pyxel")
        pyxel.image(0).load(0, 0, "assets/pyxel_logo_38x16.png")
        pyxel.run(self.update, self.draw)

    def update(self):
        if pyxel.btnp(pyxel.KEY_Q):
            pyxel.quit()

    def draw(self):
        pyxel.cls(0)
        pyxel.text(55, 41, "Hello, Pyxel!", pyxel.frame_count % 16)
        pyxel.blt(61, 66, 0, 0, 0, 38, 16)

App()

I start with a simple example. I don't find a collision system on Pyxel. Let's see the source code:

import pyxel
from pyxel import circ, cls, flip, init
from random import randint

# the position of the ball
x = y = 30  
# the speed of the ball
v = w = 3  
# create the screen as 160x120 size
pyxel.init(160, 112)  
#
data = [70, 60, 30, 70]


 # draw a line below the bar chart

while True:
    # erase the screen with color number 1 (blue)
    pyxel.cls(1)  

    # process the movement of the ball
    x += v
    y += w
    
    r = randint(0, 160)  
    a = randint(0, 112)
    rr = randint(0, 160) 
    aa = randint(0, 112)
    # create random lines on screen
    pyxel.line(a, aa, r, rr, 5)

    # set the border
    if x <= 7 or x >= 160:
        x = min(max(x, 7), 160)
        v = -v

    if y <= 7 or y >= 112:
        y = min(max(y, 7), 112)
        w = -w
  
    # draw the ball with different colors
    pyxel.circ(x, y, 4, pyxel.frame_count % 8) 
 
    # create a simple chart
    for i, d in enumerate(data):
        pyxel.rect(i * 33 + 10, 120 - d, 10, d, 8 + i) 

    # draw the game
    pyxel.flip()

Python 3.8.3 : Create shortcuts and add python to the context menu.

The tutorial for today is a simple script for python user.
If you run this python script in your folder you will get an python shortcut to the python and the python to the Context Menu in Windows 10.
The Context Menu can be used with right click from your mouse, see the screenshot.

This script can also be used with any executable if you make some changes.

import os
import pythoncom
from win32com.shell import shell

dirpath = os.getcwd()
if not os.path.exists(dirpath):
    os.makedirs(dirpath)

win_user = os.getenv('username')
shortcut_path = dirpath + r"\python.lnk"

shortcut_path = str(r"C:\\Users\\")+win_user+str("\\AppData\\Roaming\\Microsoft\\Windows\\SendTo\\python.lnk")

target_path = r"D:\\Python38_64\\python.exe"

shortcut_instance = pythoncom.CoCreateInstance(
    shell.CLSID_ShellLink, None, pythoncom.CLSCTX_INPROC_SERVER, shell.IID_IShellLink)
persist_file = shortcut_instance.QueryInterface(pythoncom.IID_IPersistFile)
shortcut_instance.SetPath(target_path)
persist_file.Save (shortcut_path, 0)

Python Qt5 : Simple text editor with QPlainTextEdit.

I haven't played python in a long time. It can be seen after the last article.
Today I installed Python 3.8.3 and my favorite PyQt5 and started to see how much I forgot from what I knew.
Installing PyQt5 python mode was simple.

pip3 install PyQt5
Collecting PyQt5
  Downloading PyQt5-5.14.2-5.14.2-cp35.cp36.cp37.cp38-none-win_amd64.whl (52.9 MB)
     |████████████████████████████████| 52.9 MB 80 kB/s
Collecting PyQt5-sip<13>=12.7
  Downloading PyQt5_sip-12.7.2-cp38-cp38-win_amd64.whl (59 kB)
     |████████████████████████████████| 59 kB 4.1 MB/s
Installing collected packages: PyQt5-sip, PyQt5
Successfully installed PyQt5-5.14.2 PyQt5-sip-12.7.2

The next step was to use QPlainTextEdit so the task was simple: an editor.
Because I created another article about a publisher, see this tutorial, I decided to attach a main menu with submenus and customize it.
The source code is simple to understand and is commented for a better understanding.

# -*- coding: utf-8 -*-
"""
@author: catafest
"""
import sys
import sys, os
from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWidgets import QMainWindow, QAction, qApp, QApplication, QDesktopWidget
from PyQt5.QtWidgets import QMenu, QPlainTextEdit, QSizePolicy
from PyQt5.QtGui import QIcon
from PyQt5.QtCore import Qt,QSize

class Example(QMainWindow):
    #init the example class to draw the window application    
    def __init__(self, parent = None):
        super(Example,self).__init__(parent)  
        size_policy = self.sizePolicy()
        size_policy.setHeightForWidth(True)
        size_policy.setVerticalPolicy(QSizePolicy.Preferred)
        self.setSizePolicy(size_policy)  
        self.initUI()

    #create the def center to select the center of the screen         
    def center(self):
        # geometry of the main window
        qr = self.frameGeometry()
        
        # center point of screen
        cp = QDesktopWidget().availableGeometry().center()
        # move rectangle's center point to screen's center point
        qr.moveCenter(cp)
        # top left of rectangle becomes top left of window centering it
        self.move(qr.topLeft())

    #create the init UI to draw the application
    def initUI(self):               
        #create the action for the exit application with shortcut and icon
        #you can add new action for File menu and any actions you need
        #self.setMinimumSize(QSize(440, 240))    
        self.setWindowTitle("parse sensors output - catafest")
        self.editor_text = QPlainTextEdit(self)
        # font
        default_font = self.editor_text.font()
        default_font.setPointSize(9)
        self.editor_text.setFont(default_font)

        self.setCentralWidget(self.editor_text)
        #self.editor_text.setMinimumSize(400,600)

        self.editor_text.setVerticalScrollBarPolicy(QtCore.Qt.ScrollBarAsNeeded)
        self.editor_text.setHorizontalScrollBarPolicy(QtCore.Qt.ScrollBarAsNeeded)
        
        self.editor_text.setObjectName("editor_text")
        self.editor_text.setEnabled(True)
        
        openAct = QAction(QIcon('open.png'), '&Open', self)        
        openAct.setShortcut('Ctrl+O')
        openAct.setStatusTip('Open file')
        openAct.triggered.connect(self.my_OpenDialog)

        exitAct = QAction(QIcon('exit.png'), '&Exit', self)        
        exitAct.setShortcut('Ctrl+Q')
        exitAct.setStatusTip('Exit application')
        exitAct.triggered.connect(qApp.quit)
        #create the status bar for menu 
        self.statusBar()
        #create the menu with the text File , add the exit action 
        #you can add many items on menu with actions for each item
        menubar = self.menuBar()
        fileMenu = menubar.addMenu('&File')
        # create open 
        fileMenu.addAction(openAct)
        # create exit 
        fileMenu.addAction(exitAct)
        
        # add submenu to menu 
        submenu = QMenu('Submenu',self)

        # some dummy actions
        submenu.addAction('Submenu 1')
        submenu.addAction('Submenu 2')
        
        # allow adding a sphere
        meshMenu = submenu.addMenu('Mesh')
        self.add_sphere_action = QAction('Add Sphere', self)
        self.add_sphere_action.triggered.connect(self.add_sphere)
        meshMenu.addAction(self.add_sphere_action)

        # add to the top menu
        menubar.addMenu(submenu)

        #layout = QtWidgets.QGridLayout(self)
        #layout.addWidget(self.editor_text)
        self.editor_text

        #resize the window application 
        self.resize(960, 720)
        #draw on center of the screen 
        self.center()
        #add title on windows application 
        self.setWindowTitle('Simple menu')
        #show the application
        self.show()
        #close the UI class

    def my_OpenDialog(self):
        path = QtWidgets.QFileDialog.getOpenFileName(
            self, 'Open file', '',
            'Text files (*.txt);;HTML files (*.html)')[0]
        if path:
            file = QtCore.QFile(path)
            if file.open(QtCore.QIODevice.ReadOnly):
                stream = QtCore.QTextStream(file)
                text = stream.readAll()
                info = QtCore.QFileInfo(path)
                if info.completeSuffix() == 'txt':
                    #self.editor_text.setHtml(text)
                    self.editor_text.insertPlainText(text)
                else:
                    self.editor_text.setPlainText(text)
                file.close()

    def add_sphere(self):
        """ add text with sphere """
        print("sphere")
        
if __name__ == '__main__':
    #create the application 
    app = QApplication(sys.argv)
    #use the UI with new  class
    ex = Example()
    #run the UI 
    sys.exit(app.exec_())

Python 3.8.2 : New release 2.3.2 for Pygame Menu.

Today, the development team come with this infos from the GitHub comes with a new release version 2.3.2.
Python library that can create a simple menu for the pygame application. Supports:

• Textual menus
  
• Buttons
• Lists of values (selectors) that can trigger functions when pressing return or changing the value
• Input text
• Color input

NOTE: pygame-menu v2 will not provide new widgets or functionalities, consider upgrading to the lastest version.
Let's start the tutorial with python install on Windows 10 using the installer from here.
Use these settings from images:


Download get-pip.py to a folder on your computer.
Open a command prompt and navigate to the folder containing get-pip.py.
Run the following command:

python get-pip.py

Then update the path:

C:\Projects\Python\pygame-menu>python -m pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431
  eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)
     |████████████████████████████████| 1.4MB 819kB/s
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.0.2

You need to install pygame python module:

C:\Projects\Python\pygame-menu>pip install pygame
Collecting pygame
  Downloading pygame-1.9.6-cp38-cp38-win_amd64.whl (4.8 MB)
     |████████████████████████████████| 4.8 MB 819 kB/s
Installing collected packages: pygame
Successfully installed pygame-1.9.6

The last step is to install the Pygame Menu with git tool and documentation with pip tool:

$ git clone https://github.com/ppizarror/pygame-menu
Cloning into 'pygame-menu'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 5649 (delta 3), reused 7 (delta 2), pack-reused 5640
Receiving objects: 100% (5649/5649), 12.99 MiB | 5.52 MiB/s, done.
Resolving deltas: 100% (4289/4289), done.
...
C:\Projects\Python\pygame-menu>pip install -e .[doc]

The result with of how these python module with a simple example:

C:\Projects\Python\pygame-menu\pygame_menu\examples>python game_selector.py
pygame 1.9.6
Hello from the pygame community. https://www.pygame.org/contribute.html
...

Python 3.7.5 : The this python package.

The this python package is simple to use:

[mythcat@desk ~]$ python3 
Python 3.7.5 (default, Dec 15 2019, 17:54:26) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Python 3.7.4 : About with the PyOpenCL python module.

PyOpenCL lets you access GPUs and other massively parallel compute devices from Python.
It is important to note that OpenCL is not restricted to GPUs.
In fact, no special hardware is required to use OpenCL for computation–your existing CPU is enough.
The documentation of this project can be found at this website.
Let's install the python module for python 3 version:

[mythcat@desk ~]$ pip3 install pyopencl --user
Collecting pyopencl
...
Successfully built pytools
Installing collected packages: pytools, pyopencl
Successfully installed pyopencl-2019.1.1 pytools-2019.1.1

The install of OpenCL driver can be done with these commands:

# get OpenCL driver automated installer (installs kernel 4.7)
curl https://software.intel.com/sites/default/files/managed/f6/77/install_OCL_driver.sh_.txt > install_OCL\
_driver.sh
chmod +x install_OCL_driver.sh
# install OpenCL driver
sudo ./install_OCL_driver.sh install
# check
ls /boot/vmlinuz-*intel*

This is a simple python script to test the opencl context:

import pyopencl as cl
import numpy as np
ctx = cl.create_some_context()
# cet platforms, both CPU and GPU
my_plat= cl.get_platforms()
CPU = my_plat[0].get_devices()
try:
    GPU = my_plat[1].get_devices()
except IndexError:
    GPU = "none"
# create context for GPU/CPU
if GPU != "none":
    ctx = cl.Context(GPU)
else:
    ctx = cl.Context(CPU)
# create queue for each kernel execution
queue = cl.CommandQueue(ctx)
mf = cl.mem_flags

This is another simple python script:

# -*- coding: utf-8 -*-
import pyopencl as cl 
import numpy
a = numpy.random.rand(50000).astype(numpy.float32)
ctx = cl.create_some_context()
queue = cl.CommandQueue(ctx)
a_buf = cl.Buffer(ctx ,cl.mem_flags.READ_WRITE,size=a.nbytes)
cl.enqueue_write_buffer(queue, a_buf , a)

prg= cl.Program(ctx,
"""
__kernel void twice(__global float ∗a)
{
 int gid=get_global_id(0);
 a[gid] ∗= 2;
}"""
). build()

prg.twice(queue, a.shape, None,a_buf ).wait()

Python 3.7.4 : Test the DHCP handshakes.

First, the DHCP is based on the earlier BOOTP protocol which uses well-known port numbers for both server and client instead of an ephemeral port. The server and the client communicate via broadcast and the server broadcasts the offered IP address to the client on UDP port 68.
This python example has a learning purpose and does not harm anyone.

import subprocess as sub
import re

def find_word(w):
    return re.compile(r'\b({0})\b'.format(w), flags=re.IGNORECASE).search

p = sub.Popen(('sudo', 'tcpdump', '-l', '-s 0', '-vvv', '-n', '((udp port 67) and (udp[8:1] = 0x1))'),
 stdout=sub.PIPE)
for row in iter(p.stdout.readline, b''):
    if find_word(row):
        print (row.split(' ')[-1])
    elif find_word(row):
        print (row.split(' ')[-1])

The result of my script ( I don't have inputs on this port).

[mythcat@desk scripts]$ python3 dhcpreq.py 
tcpdump: listening on ___, link-type EN10MB (Ethernet), capture size 262144 bytes
^CTraceback (most recent call last):
  File "dhcpreq.py", line 10, in 
    for row in iter(p.stdout.readline, b''):
KeyboardInterrupt
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[mythcat@desk scripts]$ vim dhcpreq.py 

Python 3.7.4 : Testing the PyUSB python module.

This python module named PyUSB can be found at pypi website.

[mythcat@desk scripts]$ pip3 install pyusb --user
Collecting pyusb
...
Successfully installed pyusb-1.0.2

Let' see some usb device with lsusb command:

[mythcat@desk scripts]$ lsusb
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 004: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 001 Device 003: ID 093a:2510 Pixart Imaging, Inc. Optical Mouse
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

First you need to set this to avoid the error: Access denied (insufficient permissions.

[mythcat@desk scripts]$ ll  /dev/bus/usb/001/004
crw-rw-r--. 1 root root 189, 3 Oct 10 20:34 /dev/bus/usb/001/004
[mythcat@desk scripts]$ chmod a+rw /dev/bus/usb/001/004
chmod: changing permissions of '/dev/bus/usb/001/004': Operation not permitted
[mythcat@desk scripts]$ sudo chmod a+rw /dev/bus/usb/001/004
[sudo] password for mythcat: 
[mythcat@desk scripts]$ ll  /dev/bus/usb/001/004
crw-rw-rw-. 1 root root 189, 3 Oct 10 20:34 /dev/bus/usb/001/004

The script is simple:

import sys
  
import usb.core
import usb.util
print(usb.__version__)
busses = usb.busses()
for bus in busses:
    devices = bus.devices
    for dev in devices:
        if dev != None:
            try:
                usd_dev = usb.core.find(idVendor=dev.idVendor, idProduct=dev.idProduct)
                print(usb_dev)
            except:
                pass
# 1a40:0101
dev  = usb.core.find(idVendor=0x1a40, idProduct=0x0101)
print ("The 8087:0024 is : ", dev)
if dev is None:
    raise ValueError("Device not found!")
else:

    if dev.is_kernel_driver_active(0):
        try:
                dev.detach_kernel_driver(0)
                print ("kernel driver detached")
        except usb.core.USBError as e:
                sys.exit("Could not detach kernel driver: %s" % str(e))
    else:
        print ("no kernel driver attached")
    try:
        usb.util.claim_interface(dev, 0)
        print ("claimed device")
    except:
        sys.exit("Could not claim the device: %s" % str(e))
    try:
        dev.set_configuration()
        dev.reset()
    except usb.core.USBError as e:
        sys.exit("Could not set configuration: %s" % str(e))

usb.util.release_interface(dev,interface)
dev.attach_kernel(interface

The result of this python script is this:

[mythcat@desk scripts]$ python3 usb_test.py 
1.0.2
The 8087:0024 is :  DEVICE ID 1a40:0101 on Bus 001 Address 004 =================
 bLength                :   0x12 (18 bytes)
 bDescriptorType        :    0x1 Device
 bcdUSB                 :  0x200 USB 2.0
 bDeviceClass           :    0x9 Hub
 bDeviceSubClass        :    0x0
 bDeviceProtocol        :    0x1
 bMaxPacketSize0        :   0x40 (64 bytes)
 idVendor               : 0x1a40
 idProduct              : 0x0101
 bcdDevice              :  0x111 Device 1.11
 iManufacturer          :    0x0 
 iProduct               :    0x1 USB 2.0 Hub
 iSerialNumber          :    0x0 
 bNumConfigurations     :    0x1
  CONFIGURATION 1: 100 mA ==================================
   bLength              :    0x9 (9 bytes)
   bDescriptorType      :    0x2 Configuration
   wTotalLength         :   0x19 (25 bytes)
   bNumInterfaces       :    0x1
   bConfigurationValue  :    0x1
   iConfiguration       :    0x0 
   bmAttributes         :   0xe0 Self Powered, Remote Wakeup
   bMaxPower            :   0x32 (100 mA)
    INTERFACE 0: Hub =======================================
     bLength            :    0x9 (9 bytes)
     bDescriptorType    :    0x4 Interface
     bInterfaceNumber   :    0x0
     bAlternateSetting  :    0x0
     bNumEndpoints      :    0x1
     bInterfaceClass    :    0x9 Hub
     bInterfaceSubClass :    0x0
     bInterfaceProtocol :    0x0
     iInterface         :    0x0 
      ENDPOINT 0x81: Interrupt IN ==========================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :   0x81 IN
       bmAttributes     :    0x3 Interrupt
       wMaxPacketSize   :    0x1 (1 bytes)
       bInterval        :    0xc
kernel driver detached
claimed device
Could not set configuration: [Errno 16] Resource busy

Set permisions for next usb:

[mythcat@desk scripts]$ ll /dev/bus/usb/001/003
crw-rw-rw-. 1 root root 189, 2 Oct 10 20:34 /dev/bus/usb/001/003

Te next source code will read the mouse device:

#!/usr/bin/python
import sys
import usb.core
import usb.util
# decimal vendor and product values
#dev = usb.core.find(idVendor=1118, idProduct=1917)
# or, uncomment the next line to search instead by the hexidecimal equivalent
# 093a:2510
dev = usb.core.find(idVendor=0x093a, idProduct=0x2510)
# first endpoint
interface = 0
endpoint = dev[0][(0,0)][0]
# if the OS kernel already claimed the device, which is most likely true
# thanks to http://stackoverflow.com/questions/8218683/pyusb-cannot-set-configuration
if dev.is_kernel_driver_active(interface) is True:
  # tell the kernel to detach
  dev.detach_kernel_driver(interface)
  # claim the device
  usb.util.claim_interface(dev, interface)
collected = 0
attempts = 50
while collected < attempts :
    try:
        data = dev.read(endpoint.bEndpointAddress,endpoint.wMaxPacketSize)
        collected += 1
        print (data)
    except usb.core.USBError as e:
        data = None
        if e.args == ('Operation timed out',):
            continue
# release the device
usb.util.release_interface(dev, interface)
# reattach the device to the OS kernel
dev.attach_kernel_driver(interface)

The output of mouse moves is this:

[mythcat@desk scripts]$ python3 usb_mouse.py 
[mythcat@desk scripts]$ python3 usb_mouse.py 
array('B', [0, 254, 255, 0])
array('B', [0, 253, 2, 0])
array('B', [0, 252, 3, 0])
array('B', [0, 251, 3, 0])
array('B', [0, 252, 3, 0])
array('B', [0, 254, 1, 0])
array('B', [0, 253, 2, 0])
array('B', [0, 255, 1, 0])
array('B', [0, 255, 4, 0])
array('B', [0, 0, 3, 0])
array('B', [0, 0, 3, 0])
array('B', [0, 0, 2, 0])
array('B', [0, 0, 2, 0])
array('B', [0, 2, 1, 0])
array('B', [0, 4, 1, 0])
array('B', [0, 3, 0, 0])
array('B', [0, 3, 0, 0])
array('B', [0, 1, 0, 0])

Python 3.7.4 : Example with subprocess - part 001.

This is a simple example with the python 3 subprocess package.
The source code is simple to understand.
The execute_proceess_with_communicate let run the ls command with the sudo user permissions:

import os
import sys
import string
import subprocess
import codecs

inp = ''
cmd = 'ls'
password = ''

def execute_proceess_with_communicate(inp):
    """Return a list of hops from traceroute command."""
    p = subprocess.Popen(
            ['sudo', cmd, inp],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            shell=False)
    text, _ = p.communicate(password)
    #print(type(text))
    outp = codecs.decode(text,'utf8')
    out_split=outp.split('\n')
    return out_split

def normalize_out(list_outp):
    """Extract information from traceroute line per line."""

    normalized_out = []
    for op in list_outp:
        # filer out if an empty line
        if len(op) is 0:
            continue
        op_split = op.split()
        normalized_out.append(op_split)
    return normalized_out

if __name__ == '__main__':
    inp = sys.argv[1]

    out = execute_proceess_with_communicate(inp)
    n_out = normalize_out(out)
    print(n_out)

The result is this:

[mythcat@desk scripts]$ python3 subprocess_001.py '/' 
[['bin'], ['boot'], ['dev'], ['etc'], ['home'], ['lib'], ['lib64'], ['media'], ['mnt'], ['opt'], ['proc'],
 ['root'], ['run'], ['sbin'], ['srv'], ['sys'], ['tmp'], ['usr'], ['var']]

Python 3.7.4 : Install the protobuf from sources on Fedora distro.

Today I will show you how to build protobuf from sources using the Fedora distro.
The google team comes with this intro:
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler...
This google project comes with these tutorials.
The GitHub project can be found here.
To install the compiler, download the package.

[mythcat@desk ~]$ cd Downloads/
[mythcat@desk Downloads]$ cp protobuf-python-3.9.2.tar.gz ~/
[mythcat@desk Downloads]$ cd ..
[mythcat@desk ~]$ tar xvzf protobuf-python-3.9.2.tar.gz 
...
protobuf-3.9.2/aclocal.m4
protobuf-3.9.2/install-sh
protobuf-3.9.2/generate_descriptor_proto.sh
protobuf-3.9.2/CHANGES.txt
protobuf-3.9.2/configure.ac
protobuf-3.9.2/configure

Let's see the content:

[mythcat@desk ~]$ cd protobuf-3.9.2/
[mythcat@desk protobuf-3.9.2]$ ls
aclocal.m4                   config.sub                    ltmain.sh            README.md
ar-lib                       configure                     m4                   six.BUILD
autogen.sh                   configure.ac                  Makefile.am          src
benchmarks                   conformance                   Makefile.in          test-driver
BUILD                        CONTRIBUTORS.txt              missing              third_party
CHANGES.txt                  depcomp                       objectivec           update_file_lists.sh
cmake                        editors                       protobuf.bzl         util
compile                      examples                      protobuf_deps.bzl    WORKSPACE
compiler_config_setting.bzl  generate_descriptor_proto.sh  protobuf-lite.pc.in
config.guess                 install-sh                    protobuf.pc.in
config.h.in                  LICENSE                       python

[mythcat@desk protobuf-3.9.2]$ ./configure
...
checking how to run the C preprocessor... gcc -E
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/mythcat/protobuf-3.9.2':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details
...
[root@desk protobuf-3.9.2]# dnf install g++
...
Installed:
  gcc-c++-9.2.1-1.fc30.x86_64                                                                               

Complete![root@desk protobuf-3.9.2]# exit
exit

Let's build again:

[mythcat@desk protobuf-3.9.2]$ ./configure
checking whether to enable maintainer-specific portions of Makefiles... yes
checking build system type... x86_64-pc-linux-gnu
...
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/gmock-config
config.status: creating build-aux/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
[mythcat@desk protobuf-3.9.2]$ nproc --all
2
[mythcat@desk protobuf-3.9.2]$ make -j2
...
  CXXLD    protoc
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2'
[mythcat@desk protobuf-3.9.2]$ make check -j2
...PASS: protobuf-lazy-descriptor-test
PASS: protobuf-lite-test
PASS: google/protobuf/compiler/zip_output_unittest.sh
PASS: google/protobuf/io/gzip_stream_unittest.sh
PASS: protobuf-lite-arena-test
PASS: no-warning-test
PASS: protobuf-test
============================================================================
Testsuite summary for Protocol Buffers 3.9.2
============================================================================
# TOTAL: 7
# PASS:  7
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'

The last steps is for install:

[mythcat@desk protobuf-3.9.2]$ sudo make install
[sudo] password for mythcat: 
...
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
[mythcat@desk protobuf-3.9.2]$ sudo ldconfig 

Let's test it:

[mythcat@desk protobuf-3.9.2]$ protoc --version
libprotoc 3.9.2

Now the next step comes for python module:

[mythcat@desk protobuf-3.9.2]$ cd python/
[mythcat@desk python]$ ls
google  MANIFEST.in  mox.py  README.md  release  release.sh  setup.cfg  setup.py  stubout.py  tox.ini
[mythcat@desk python]$ python setup.py build
...
testSerialize (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeMessageSetWireFormatUnknownExtension 
(google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeProto3 (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testByteSizeFunctions (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testPackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testUnpackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagDecode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagEncode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok

----------------------------------------------------------------------
Ran 818 tests in 5.343s

OK (skipped=10)

Let's test the python module:

[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import google
>>> from google.protobuf import descriptor as _descriptor
>>> from google.protobuf import message as _message
>>> from google.protobuf import reflection as _reflection
>>> from google.protobuf import symbol_database as _symbol_database
>>> from google.protobuf import descriptor_pb2
>>> from google.protobuf import text_format
>>> dir(_descriptor)
['Descriptor', 'DescriptorBase', 'DescriptorMetaclass', 'EnumDescriptor', 'EnumValueDescriptor', 
'Error', 'FieldDescriptor', 'FileDescriptor', 'MakeDescriptor', 'MethodDescriptor', 'OneofDescriptor', 
'ServiceDescriptor', 'TypeTransformationError', '_Lock', '_NestedDescriptorBase', '_OptionsOrNone', 
'_ParseOptions', '_ToCamelCase', '_ToJsonName', '_USE_C_DESCRIPTORS', '__author__', '__builtins__', 
'__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_lock', 
'api_implementation', 'six', 'threading']

Now I can to define the structure for the data structured as messages
These message is a small logical record of information containing a series of name-value pairs called fields.
The protobuffers compiler (protoc) to generate the source code in the language you need (from the .proto file).
To generate a Python file, you need to execute:
protoc -I=$SRC_DIR --python_out=$DST_DIR $SRC_DIR/my_example.proto
Let's test with am example (you can use any proto file) in default folder (use . for default folder):

[mythcat@desk python]$ protoc -I=. --python_out=. my_example.proto
[mythcat@desk python]$ ls my_example*
my_example_pb2.py  my_example.proto
[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_example_pb2
>>> dir(my_example_pb2)
['AddressBook', 'DESCRIPTOR', 'Person', '_ADDRESSBOOK', '_PERSON', '_PERSON_PHONENUMBER', '_PERSON_PHONETYPE',
 '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__',
 '_b', '_descriptor', '_message', '_reflection', '_sym_db', '_symbol_database', 'sys']