Python 3.7.4 : About with the PyOpenCL python module.

PyOpenCL lets you access GPUs and other massively parallel compute devices from Python.
It is important to note that OpenCL is not restricted to GPUs.
In fact, no special hardware is required to use OpenCL for computation–your existing CPU is enough.
The documentation of this project can be found at this website.
Let's install the python module for python 3 version:

[mythcat@desk ~]$ pip3 install pyopencl --user
Collecting pyopencl
...
Successfully built pytools
Installing collected packages: pytools, pyopencl
Successfully installed pyopencl-2019.1.1 pytools-2019.1.1

The install of OpenCL driver can be done with these commands:

# get OpenCL driver automated installer (installs kernel 4.7)
curl https://software.intel.com/sites/default/files/managed/f6/77/install_OCL_driver.sh_.txt > install_OCL\
_driver.sh
chmod +x install_OCL_driver.sh
# install OpenCL driver
sudo ./install_OCL_driver.sh install
# check
ls /boot/vmlinuz-*intel*

This is a simple python script to test the opencl context:

import pyopencl as cl
import numpy as np
ctx = cl.create_some_context()
# cet platforms, both CPU and GPU
my_plat= cl.get_platforms()
CPU = my_plat[0].get_devices()
try:
    GPU = my_plat[1].get_devices()
except IndexError:
    GPU = "none"
# create context for GPU/CPU
if GPU != "none":
    ctx = cl.Context(GPU)
else:
    ctx = cl.Context(CPU)
# create queue for each kernel execution
queue = cl.CommandQueue(ctx)
mf = cl.mem_flags

This is another simple python script:

# -*- coding: utf-8 -*-
import pyopencl as cl 
import numpy
a = numpy.random.rand(50000).astype(numpy.float32)
ctx = cl.create_some_context()
queue = cl.CommandQueue(ctx)
a_buf = cl.Buffer(ctx ,cl.mem_flags.READ_WRITE,size=a.nbytes)
cl.enqueue_write_buffer(queue, a_buf , a)

prg= cl.Program(ctx,
"""
__kernel void twice(__global float ∗a)
{
 int gid=get_global_id(0);
 a[gid] ∗= 2;
}"""
). build()

prg.twice(queue, a.shape, None,a_buf ).wait()

Python 3.7.4 : Usinge pytesseract for text recognition.

About this python module named tesseract, you can read here.
I tested with the tesseract tool install on my Fedora 30 distro and python module pytesseract version 0.3.0.

[root@desk mythcat]# dnf install tesseract
Last metadata expiration check: 0:24:18 ago on Sun 20 Oct 2019 10:56:23 AM EEST.
Package tesseract-4.1.0-1.fc30.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[root@desk mythcat]# whereis tesseract
tesseract: /usr/bin/tesseract /usr/share/tesseract
[mythcat@desk ~]$ pip3 install pytesseract --user
Collecting pytesseract
...
Installing collected packages: pytesseract
Successfully installed pytesseract-0.3.0

I test with many images and texts and works very well.
Text images with a printed font are very well recognized.
This test with this image does not have very good accuracy.

The result of the handwriting image.

[mythcat@desk ~]$ python3 ocr_image.py 001.png 
rake Yous mnislakes,
take you chances,
look silby,

bul hep. mv going
dont freeze up

Python 3.7.4 : Test the DHCP handshakes.

First, the DHCP is based on the earlier BOOTP protocol which uses well-known port numbers for both server and client instead of an ephemeral port. The server and the client communicate via broadcast and the server broadcasts the offered IP address to the client on UDP port 68.
This python example has a learning purpose and does not harm anyone.

import subprocess as sub
import re

def find_word(w):
    return re.compile(r'\b({0})\b'.format(w), flags=re.IGNORECASE).search

p = sub.Popen(('sudo', 'tcpdump', '-l', '-s 0', '-vvv', '-n', '((udp port 67) and (udp[8:1] = 0x1))'),
 stdout=sub.PIPE)
for row in iter(p.stdout.readline, b''):
    if find_word(row):
        print (row.split(' ')[-1])
    elif find_word(row):
        print (row.split(' ')[-1])

The result of my script ( I don't have inputs on this port).

[mythcat@desk scripts]$ python3 dhcpreq.py 
tcpdump: listening on ___, link-type EN10MB (Ethernet), capture size 262144 bytes
^CTraceback (most recent call last):
  File "dhcpreq.py", line 10, in 
    for row in iter(p.stdout.readline, b''):
KeyboardInterrupt
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[mythcat@desk scripts]$ vim dhcpreq.py 

Python 3.8.0 : New release of python development.

Good news from the python development area with the new release of python development:
Python 3.7.5 Oct. 15, 2019 and Python 3.8.0 Oct. 14, 2019

Now you can use the new python version 3.8.0 from the official webpage.

Major new features of the 3.8 series, compared to 3.7 - release Date: Oct. 14, 2019:

• PEP 572, Assignment expressions
• PEP 570, Positional-only arguments
• PEP 587, Python Initialization Configuration (improved embedding)
• PEP 590, Vectorcall: a fast calling protocol for CPython
• PEP 578, Runtime audit hooks
• PEP 574, Pickle protocol 5 with out-of-band data
• Typing-related: PEP 591 (Final qualifier), PEP 586 (Literal types), and PEP 589 (TypedDict)
• Parallel filesystem cache for compiled bytecode
• Debug builds share ABI as release builds
• f-strings support a handy = specifier for debugging
• continue is now legal in finally: blocks
• on Windows, the default asyncio event loop is now ProactorEventLoop
• on macOS, the spawn start method is now used by default in multiprocessing
• multiprocessing can now use shared memory segments to avoid pickling costs between processes
• typed_ast is merged back to CPython
• LOAD_GLOBAL is now 40% faster
• pickle now uses Protocol 4 by default, improving performance

Let's install on Fedora 30 Linux distro:

[mythcat@desk ~]$ cd Python-3.8.0/
[mythcat@desk Python-3.8.0]$ ls
aclocal.m4          Doc         m4               Parser         README.rst
CODE_OF_CONDUCT.md  Grammar     Mac              PC             setup.py
config.guess        Include     Makefile.pre.in  PCbuild        Tools
config.sub          install-sh  Misc             Programs
configure           Lib         Modules          pyconfig.h.in
configure.ac        LICENSE     Objects          Python
[mythcat@desk Python-3.8.0]$ ./configure 
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for python3.8... no
...
creating Makefile

If you want a release build with all stable optimizations active (PGO, etc),
please run ./configure --enable-optimizations

If you want then you can run the tool to prepare the build with optimizations:

[mythcat@desk Python-3.8.0]$ ./configure --enable-optimizations --with-ensurepip=install
...
creating Modules/Setup.local
creating Makefile

Use the make with the -j option to use building into parallel steps to speed up the compilation.

[mythcat@desk Python-3.8.0]$ make -j 2
...
make[1]: Leaving directory '/home/mythcat/Python-3.8.0'

Since you’re installing Python into /usr/bin, you’ll need to run as root:

[mythcat@desk Python-3.8.0]$ sudo make altinstall
...
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-19.2.3 setuptools-41.2.0

Let's test it in this folder and with new python3.8 :

[mythcat@desk Python-3.8.0]$ ./python 
Python 3.8.0 (default, Oct 15 2019, 23:45:20) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
[mythcat@desk Python-3.8.0]$ python3.8
Python 3.8.0 (default, Oct 15 2019, 23:45:20) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Python 3.7.4 : Testing python source code with streamlit tool.

The official webpage for this python package can be found at streamlit.io.
Let's install it with pip3 tool:

[mythcat@desk proiecte_github]$ mkdir streamlit_examples
[mythcat@desk proiecte_github]$ cd streamlit_examples/
[mythcat@desk streamlit_examples]$ pip3 install streamlit --user

Let's try some examples.
Create a file named 001.py
This simple example will show a map with randoms spots:

import pandas as pd
import numpy as np
import streamlit as st    
df = pd.DataFrame(
    np.random.randn(100, 2) / [50, 50] + [47.45, 26.3],
    columns=['lat', 'lon'])
st.map(df)

Let's run it with this command:

[mythcat@desk streamlit_examples]$ streamlit run 001.py 

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
...

The next source code will show just the map because the df variable is empty:

import streamlit as st
df = []
st.deck_gl_chart(
    viewport={
        'latitude': 47.45,
        'longitude': 26.3,
        'zoom': 13,
        'pitch': 50,
    },
    layers=[{
        'type': 'HexagonLayer',
        'data': df,
        'radius': 200,
        'elevationScale': 4,
        'elevationRange': [0, 1000],
        'pickable': True,
        'extruded': True,
    }, {
        'type': 'ScatterplotLayer',
        'data': df,
    }])

The source code is added into another file named 002.py and can be run with this command:

[mythcat@desk streamlit_examples]$ streamlit run 002.py 

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
...

You can see more about this tool at the official youtube channel:

Python 3.7.4 : Testing the PyUSB python module.

This python module named PyUSB can be found at pypi website.

[mythcat@desk scripts]$ pip3 install pyusb --user
Collecting pyusb
...
Successfully installed pyusb-1.0.2

Let' see some usb device with lsusb command:

[mythcat@desk scripts]$ lsusb
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 004: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 001 Device 003: ID 093a:2510 Pixart Imaging, Inc. Optical Mouse
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

First you need to set this to avoid the error: Access denied (insufficient permissions.

[mythcat@desk scripts]$ ll  /dev/bus/usb/001/004
crw-rw-r--. 1 root root 189, 3 Oct 10 20:34 /dev/bus/usb/001/004
[mythcat@desk scripts]$ chmod a+rw /dev/bus/usb/001/004
chmod: changing permissions of '/dev/bus/usb/001/004': Operation not permitted
[mythcat@desk scripts]$ sudo chmod a+rw /dev/bus/usb/001/004
[sudo] password for mythcat: 
[mythcat@desk scripts]$ ll  /dev/bus/usb/001/004
crw-rw-rw-. 1 root root 189, 3 Oct 10 20:34 /dev/bus/usb/001/004

The script is simple:

import sys
  
import usb.core
import usb.util
print(usb.__version__)
busses = usb.busses()
for bus in busses:
    devices = bus.devices
    for dev in devices:
        if dev != None:
            try:
                usd_dev = usb.core.find(idVendor=dev.idVendor, idProduct=dev.idProduct)
                print(usb_dev)
            except:
                pass
# 1a40:0101
dev  = usb.core.find(idVendor=0x1a40, idProduct=0x0101)
print ("The 8087:0024 is : ", dev)
if dev is None:
    raise ValueError("Device not found!")
else:

    if dev.is_kernel_driver_active(0):
        try:
                dev.detach_kernel_driver(0)
                print ("kernel driver detached")
        except usb.core.USBError as e:
                sys.exit("Could not detach kernel driver: %s" % str(e))
    else:
        print ("no kernel driver attached")
    try:
        usb.util.claim_interface(dev, 0)
        print ("claimed device")
    except:
        sys.exit("Could not claim the device: %s" % str(e))
    try:
        dev.set_configuration()
        dev.reset()
    except usb.core.USBError as e:
        sys.exit("Could not set configuration: %s" % str(e))

usb.util.release_interface(dev,interface)
dev.attach_kernel(interface

The result of this python script is this:

[mythcat@desk scripts]$ python3 usb_test.py 
1.0.2
The 8087:0024 is :  DEVICE ID 1a40:0101 on Bus 001 Address 004 =================
 bLength                :   0x12 (18 bytes)
 bDescriptorType        :    0x1 Device
 bcdUSB                 :  0x200 USB 2.0
 bDeviceClass           :    0x9 Hub
 bDeviceSubClass        :    0x0
 bDeviceProtocol        :    0x1
 bMaxPacketSize0        :   0x40 (64 bytes)
 idVendor               : 0x1a40
 idProduct              : 0x0101
 bcdDevice              :  0x111 Device 1.11
 iManufacturer          :    0x0 
 iProduct               :    0x1 USB 2.0 Hub
 iSerialNumber          :    0x0 
 bNumConfigurations     :    0x1
  CONFIGURATION 1: 100 mA ==================================
   bLength              :    0x9 (9 bytes)
   bDescriptorType      :    0x2 Configuration
   wTotalLength         :   0x19 (25 bytes)
   bNumInterfaces       :    0x1
   bConfigurationValue  :    0x1
   iConfiguration       :    0x0 
   bmAttributes         :   0xe0 Self Powered, Remote Wakeup
   bMaxPower            :   0x32 (100 mA)
    INTERFACE 0: Hub =======================================
     bLength            :    0x9 (9 bytes)
     bDescriptorType    :    0x4 Interface
     bInterfaceNumber   :    0x0
     bAlternateSetting  :    0x0
     bNumEndpoints      :    0x1
     bInterfaceClass    :    0x9 Hub
     bInterfaceSubClass :    0x0
     bInterfaceProtocol :    0x0
     iInterface         :    0x0 
      ENDPOINT 0x81: Interrupt IN ==========================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :   0x81 IN
       bmAttributes     :    0x3 Interrupt
       wMaxPacketSize   :    0x1 (1 bytes)
       bInterval        :    0xc
kernel driver detached
claimed device
Could not set configuration: [Errno 16] Resource busy

Set permisions for next usb:

[mythcat@desk scripts]$ ll /dev/bus/usb/001/003
crw-rw-rw-. 1 root root 189, 2 Oct 10 20:34 /dev/bus/usb/001/003

Te next source code will read the mouse device:

#!/usr/bin/python
import sys
import usb.core
import usb.util
# decimal vendor and product values
#dev = usb.core.find(idVendor=1118, idProduct=1917)
# or, uncomment the next line to search instead by the hexidecimal equivalent
# 093a:2510
dev = usb.core.find(idVendor=0x093a, idProduct=0x2510)
# first endpoint
interface = 0
endpoint = dev[0][(0,0)][0]
# if the OS kernel already claimed the device, which is most likely true
# thanks to http://stackoverflow.com/questions/8218683/pyusb-cannot-set-configuration
if dev.is_kernel_driver_active(interface) is True:
  # tell the kernel to detach
  dev.detach_kernel_driver(interface)
  # claim the device
  usb.util.claim_interface(dev, interface)
collected = 0
attempts = 50
while collected < attempts :
    try:
        data = dev.read(endpoint.bEndpointAddress,endpoint.wMaxPacketSize)
        collected += 1
        print (data)
    except usb.core.USBError as e:
        data = None
        if e.args == ('Operation timed out',):
            continue
# release the device
usb.util.release_interface(dev, interface)
# reattach the device to the OS kernel
dev.attach_kernel_driver(interface)

The output of mouse moves is this:

[mythcat@desk scripts]$ python3 usb_mouse.py 
[mythcat@desk scripts]$ python3 usb_mouse.py 
array('B', [0, 254, 255, 0])
array('B', [0, 253, 2, 0])
array('B', [0, 252, 3, 0])
array('B', [0, 251, 3, 0])
array('B', [0, 252, 3, 0])
array('B', [0, 254, 1, 0])
array('B', [0, 253, 2, 0])
array('B', [0, 255, 1, 0])
array('B', [0, 255, 4, 0])
array('B', [0, 0, 3, 0])
array('B', [0, 0, 3, 0])
array('B', [0, 0, 2, 0])
array('B', [0, 0, 2, 0])
array('B', [0, 2, 1, 0])
array('B', [0, 4, 1, 0])
array('B', [0, 3, 0, 0])
array('B', [0, 3, 0, 0])
array('B', [0, 1, 0, 0])

Python 3.7.4 : Example with subprocess - part 001.

This is a simple example with the python 3 subprocess package.
The source code is simple to understand.
The execute_proceess_with_communicate let run the ls command with the sudo user permissions:

import os
import sys
import string
import subprocess
import codecs

inp = ''
cmd = 'ls'
password = ''

def execute_proceess_with_communicate(inp):
    """Return a list of hops from traceroute command."""
    p = subprocess.Popen(
            ['sudo', cmd, inp],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            shell=False)
    text, _ = p.communicate(password)
    #print(type(text))
    outp = codecs.decode(text,'utf8')
    out_split=outp.split('\n')
    return out_split

def normalize_out(list_outp):
    """Extract information from traceroute line per line."""

    normalized_out = []
    for op in list_outp:
        # filer out if an empty line
        if len(op) is 0:
            continue
        op_split = op.split()
        normalized_out.append(op_split)
    return normalized_out

if __name__ == '__main__':
    inp = sys.argv[1]

    out = execute_proceess_with_communicate(inp)
    n_out = normalize_out(out)
    print(n_out)

The result is this:

[mythcat@desk scripts]$ python3 subprocess_001.py '/' 
[['bin'], ['boot'], ['dev'], ['etc'], ['home'], ['lib'], ['lib64'], ['media'], ['mnt'], ['opt'], ['proc'],
 ['root'], ['run'], ['sbin'], ['srv'], ['sys'], ['tmp'], ['usr'], ['var']]

Python Qt5 : the drag and drop feature.

Today I tested drag and drop feature with PyQt5.

Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux

This is a simple example using setAcceptDrops and setDragEnabled:

import sys
from PyQt5.QtWidgets import QApplication, QWidget, QListWidget, QHBoxLayout,QListWidgetItem
from PyQt5.QtGui import QIcon

class Window(QWidget):
    def __init__(self):
        super().__init__()

        self.myListWidget1 = QListWidget()
        self.myListWidget2 = QListWidget()
        self.myListWidget2.setViewMode(QListWidget.IconMode)
        self.myListWidget1.setAcceptDrops(True)
        self.myListWidget1.setDragEnabled(True)
        self.myListWidget2.setAcceptDrops(True)
        self.myListWidget2.setDragEnabled(True)
        self.setGeometry(480, 400, 640, 480)
        self.myLayout = QHBoxLayout()
        self.myLayout.addWidget(self.myListWidget1)
        self.myLayout.addWidget(self.myListWidget2)

        l1 = QListWidgetItem(QIcon('house.png'), "House")
        l2 = QListWidgetItem(QIcon('cloud.png'), "Clouds ")
        l3 = QListWidgetItem(QIcon('user.png'), "User")
        l4 = QListWidgetItem(QIcon('save.png'), "Save")

        self.myListWidget1.insertItem(1, l1)
        self.myListWidget1.insertItem(2, l2)
        self.myListWidget1.insertItem(3, l3)
        self.myListWidget1.insertItem(4, l4)

        QListWidgetItem(QIcon('house.png'), "House", self.
                        myListWidget2)
        QListWidgetItem(QIcon('cloud.png'), "Clouds", self.
                        myListWidget2)
        QListWidgetItem(QIcon('save.png'), "Save", self.
                        myListWidget2)

        self.setWindowTitle('Example: Drag and Drop');
        self.setLayout(self.myLayout)

        self.show()

App = QApplication(sys.argv)
window = Window()
sys.exit(App.exec())

The result is a windows with two QListWidget with an drag and drop feature.

Python 3.7.4 : Using the paramiko pakage.

Today I tested the paramiko package.
First, I install and check the version of this package.

[mythcat@desk my_network_tools]$ pip3 install paramiko --user
Collecting paramiko
...
  Running setup.py install for pycparser ... done
Successfully installed asn1crypto-0.24.0 bcrypt-3.1.7 cffi-1.12.3 cryptography-2.7 paramiko-2.6.0

 pycparser-2.19 pynacl-1.3.0
[mythcat@desk my_network_tools]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
>>> dir(paramiko)
['AUTH_FAILED', 'AUTH_PARTIALLY_SUCCESSFUL', 'AUTH_SUCCESSFUL', 'Agent', 'AgentKey', 'AuthHandler',

 'AuthenticationException', 'AutoAddPolicy', 'BadAuthenticationType', 'BadHostKeyException', 'BaseSFTP',

 'BufferedFile', 'Channel', 'ChannelException', 'ChannelFile', 'ChannelStderrFile', 'ChannelStdinFile',

 'DSSKey', 'ECDSAKey', 'Ed25519Key', 'GSSAuth', 'GSS_AUTH_AVAILABLE', 'GSS_EXCEPTIONS', 'HostKeys',

 'InteractiveQuery', 'Message', 'MissingHostKeyPolicy', 'OPEN_FAILED_ADMINISTRATIVELY_PROHIBITED',

 'OPEN_FAILED_CONNECT_FAILED', 'OPEN_FAILED_RESOURCE_SHORTAGE', 'OPEN_FAILED_UNKNOWN_CHANNEL_TYPE',

 'OPEN_SUCCEEDED', 'PKey', 'Packetizer', 'PasswordRequiredException', 'ProxyCommand', 'ProxyCommandFailure',

 'PublicBlob', 'RSAKey', 'RejectPolicy', 'SFTP', 'SFTPAttributes', 'SFTPClient', 'SFTPError', 'SFTPFile',

 'SFTPHandle', 'SFTPServer', 'SFTPServerInterface', 'SFTP_BAD_MESSAGE', 'SFTP_CONNECTION_LOST', 'SFTP_EOF',

 'SFTP_FAILURE', 'SFTP_NO_CONNECTION', 'SFTP_NO_SUCH_FILE', 'SFTP_OK', 'SFTP_OP_UNSUPPORTED', 

'SFTP_PERMISSION_DENIED', 'SSHClient', 'SSHConfig', 'SSHException', 'SecurityOptions', 'ServerInterface',

 'SubsystemHandler', 'Transport', 'WarningPolicy', '__all__', '__author__', '__builtins__', '__cached__', 

'__doc__', '__file__', '__license__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 

'__version__', '__version_info__', '_version', 'agent', 'auth_handler', 'ber', 'buffered_pipe', 'channel',

 'client', 'common', 'compress', 'config', 'dsskey', 'ecdsakey', 'ed25519key', 'file', 'hostkeys', 

'io_sleep', 'kex_curve25519', 'kex_ecdh_nist', 'kex_gex', 'kex_group1', 'kex_group14', 'kex_group16',

 'kex_gss', 'message', 'packet', 'pipe', 'pkey', 'primes', 'proxy', 'py3compat', 'rsakey', 'server', 

'sftp', 'sftp_attr', 'sftp_client', 'sftp_file', 'sftp_handle', 'sftp_server', 'sftp_si', 'ssh_exception',

 'ssh_gss', 'sys', 'transport', 'util']
>>> paramiko.__version__
'2.6.0'

The documentation for this version can be found at this webpage.
A simple example with this python package and ssh connection to 192.168.0.143 on port 22 can be see on the next source code

#!/usr/bin/env python
"""Return the ssh output with password connection and Linux commands"""
import paramiko

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
conn= client.connect('192.168.0.143', port='22', username='mythcat', password='the_pass')
# Obtain session
session = client.get_transport().open_session()
print("| Retcode: "+str(session)+"|")

stdin, stdout, stderr=client.exec_command('sudo hostname;w')
save_stdout = stdout.readlines()
retcode = stdout.channel.recv_exit_status()

stdin, stdout, stderr=client.exec_command('ss -nap;')
save_stdout2 = stdout.readlines()
print(save_stdout,save_stdout2)
#for line in stdout:
#    print (line)

Python 3.7.4 : Install the protobuf from sources on Fedora distro.

Today I will show you how to build protobuf from sources using the Fedora distro.
The google team comes with this intro:
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler...
This google project comes with these tutorials.
The GitHub project can be found here.
To install the compiler, download the package.

[mythcat@desk ~]$ cd Downloads/
[mythcat@desk Downloads]$ cp protobuf-python-3.9.2.tar.gz ~/
[mythcat@desk Downloads]$ cd ..
[mythcat@desk ~]$ tar xvzf protobuf-python-3.9.2.tar.gz 
...
protobuf-3.9.2/aclocal.m4
protobuf-3.9.2/install-sh
protobuf-3.9.2/generate_descriptor_proto.sh
protobuf-3.9.2/CHANGES.txt
protobuf-3.9.2/configure.ac
protobuf-3.9.2/configure

Let's see the content:

[mythcat@desk ~]$ cd protobuf-3.9.2/
[mythcat@desk protobuf-3.9.2]$ ls
aclocal.m4                   config.sub                    ltmain.sh            README.md
ar-lib                       configure                     m4                   six.BUILD
autogen.sh                   configure.ac                  Makefile.am          src
benchmarks                   conformance                   Makefile.in          test-driver
BUILD                        CONTRIBUTORS.txt              missing              third_party
CHANGES.txt                  depcomp                       objectivec           update_file_lists.sh
cmake                        editors                       protobuf.bzl         util
compile                      examples                      protobuf_deps.bzl    WORKSPACE
compiler_config_setting.bzl  generate_descriptor_proto.sh  protobuf-lite.pc.in
config.guess                 install-sh                    protobuf.pc.in
config.h.in                  LICENSE                       python

[mythcat@desk protobuf-3.9.2]$ ./configure
...
checking how to run the C preprocessor... gcc -E
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/mythcat/protobuf-3.9.2':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details
...
[root@desk protobuf-3.9.2]# dnf install g++
...
Installed:
  gcc-c++-9.2.1-1.fc30.x86_64                                                                               

Complete![root@desk protobuf-3.9.2]# exit
exit

Let's build again:

[mythcat@desk protobuf-3.9.2]$ ./configure
checking whether to enable maintainer-specific portions of Makefiles... yes
checking build system type... x86_64-pc-linux-gnu
...
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/gmock-config
config.status: creating build-aux/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
[mythcat@desk protobuf-3.9.2]$ nproc --all
2
[mythcat@desk protobuf-3.9.2]$ make -j2
...
  CXXLD    protoc
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2'
[mythcat@desk protobuf-3.9.2]$ make check -j2
...PASS: protobuf-lazy-descriptor-test
PASS: protobuf-lite-test
PASS: google/protobuf/compiler/zip_output_unittest.sh
PASS: google/protobuf/io/gzip_stream_unittest.sh
PASS: protobuf-lite-arena-test
PASS: no-warning-test
PASS: protobuf-test
============================================================================
Testsuite summary for Protocol Buffers 3.9.2
============================================================================
# TOTAL: 7
# PASS:  7
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'

The last steps is for install:

[mythcat@desk protobuf-3.9.2]$ sudo make install
[sudo] password for mythcat: 
...
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
[mythcat@desk protobuf-3.9.2]$ sudo ldconfig 

Let's test it:

[mythcat@desk protobuf-3.9.2]$ protoc --version
libprotoc 3.9.2

Now the next step comes for python module:

[mythcat@desk protobuf-3.9.2]$ cd python/
[mythcat@desk python]$ ls
google  MANIFEST.in  mox.py  README.md  release  release.sh  setup.cfg  setup.py  stubout.py  tox.ini
[mythcat@desk python]$ python setup.py build
...
testSerialize (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeMessageSetWireFormatUnknownExtension 
(google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeProto3 (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testByteSizeFunctions (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testPackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testUnpackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagDecode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagEncode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok

----------------------------------------------------------------------
Ran 818 tests in 5.343s

OK (skipped=10)

Let's test the python module:

[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import google
>>> from google.protobuf import descriptor as _descriptor
>>> from google.protobuf import message as _message
>>> from google.protobuf import reflection as _reflection
>>> from google.protobuf import symbol_database as _symbol_database
>>> from google.protobuf import descriptor_pb2
>>> from google.protobuf import text_format
>>> dir(_descriptor)
['Descriptor', 'DescriptorBase', 'DescriptorMetaclass', 'EnumDescriptor', 'EnumValueDescriptor', 
'Error', 'FieldDescriptor', 'FileDescriptor', 'MakeDescriptor', 'MethodDescriptor', 'OneofDescriptor', 
'ServiceDescriptor', 'TypeTransformationError', '_Lock', '_NestedDescriptorBase', '_OptionsOrNone', 
'_ParseOptions', '_ToCamelCase', '_ToJsonName', '_USE_C_DESCRIPTORS', '__author__', '__builtins__', 
'__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_lock', 
'api_implementation', 'six', 'threading']

Now I can to define the structure for the data structured as messages
These message is a small logical record of information containing a series of name-value pairs called fields.
The protobuffers compiler (protoc) to generate the source code in the language you need (from the .proto file).
To generate a Python file, you need to execute:
protoc -I=$SRC_DIR --python_out=$DST_DIR $SRC_DIR/my_example.proto
Let's test with am example (you can use any proto file) in default folder (use . for default folder):

[mythcat@desk python]$ protoc -I=. --python_out=. my_example.proto
[mythcat@desk python]$ ls my_example*
my_example_pb2.py  my_example.proto
[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_example_pb2
>>> dir(my_example_pb2)
['AddressBook', 'DESCRIPTOR', 'Person', '_ADDRESSBOOK', '_PERSON', '_PERSON_PHONENUMBER', '_PERSON_PHONETYPE',
 '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__',
 '_b', '_descriptor', '_message', '_reflection', '_sym_db', '_symbol_database', 'sys']

The tensorflow python module - part 004.

If you using the tensorflow then you can get some warnings.
You can use warnings python package to manage all of this:

[mythcat@desk ~]$ $ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> import tensorflow as tf
/home/mythcat/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: 
Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be 
understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/mythcat/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: 
Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be 
understood as (type, (1,)) / '(1,)type'.
...

/home/mythcat/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: 
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of 
numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
>>>
>>> warnings.filterwarnings('ignore')
>>>
[7]+  Stopped                 python3

Use it in this way to filter these warnings:

[mythcat@desk ~]$ $ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> warnings.filterwarnings('ignore',category=FutureWarning)
>>> import tensorflow as tf
>>> 

Python 3.7.4 : Print with random colors.

This is a simple example for custom print output.
The script detect the platform for color settings and then use print.
The first print will print with blue color the name of the script.
I used random to select a random color from colors array and used to print the -=RANDOM COLOR=- text.
The print (W+'') is used to set default white color for terminal

import sys
import random
if sys.platform == "linux" or sys.platform == "linux2":
        BB = "\033[34;1m" # Blue light
        YY = "\033[33;1m" # Yellow light
        GG = "\033[32;1m" # Green light
        WW = "\033[0;1m"  # White light
        RR = "\033[31;1m" # Red light
        CC = "\033[36;1m" # Cyan light
        B = "\033[34m"    # Blue
        Y = "\033[33m"    # Yellow
        G = "\033[32m"    # Green
        W = "\033[0m"     # White
        R = "\033[31m"    # Red
        C = "\033[36m"    # Cyan
colors = [BB,YY,GG,WW,RR,CC,B,Y,G,W,R,C]
print (B+"\033[2;2m "+sys.argv[0]+"\n"+B)

color=random.choice(colors)
print (color+"-=RANDOM COLOR=-"+color)
print (W+'')

For winodws platform you need to add this:

elif sys.platform == "win32":

 BB = '' # Blue light
 YY = '' # Yellow light
 GG = '' # Green light
 WW = '' # White light
 RR = '' # Red light
 CC = '' # Cyan light
 B = ''  # Blue
 Y = ''  # Yellow
 G = ''  # Green
 W = ''  # White
 R = ''  # Red
 C = ''  # Cyan
 P = ''  # Random color

Python 3.7.4 : Using the theano pakage.

If you want to test theano then you need to see this webpage.

[root@desk mythcat]# dnf search theano
======================== Name & Summary Matched: theano ========================
python-theano-doc.noarch : Theano documentation
============================= Name Matched: theano =============================
python3-theano.noarch : Mathematical expressions involving multidimensional
                      : arrays
=========================== Summary Matched: theano ============================
python3-lasagne.noarch : Lightweight library to build and train neural networks
                       : in Theano
[root@desk mythcat]# pip3 install Theano --user
WARNING: Running pip install with root privileges is generally not a good idea. 

Try `pip3 install --user` instead.
Collecting Theano
...
  Running setup.py install for Theano ... done
Successfully installed Theano-1.0.4 scipy-1.3.1

Let's see first example:

[mythcat@desk ~]$ python3 
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
/home/mythcat/.local/lib/python3.7/site-packages/theano/configdefaults.py:560: 

UserWarning: DeprecationWarning: there is no c++ compiler.This is deprecated and with 

Theano 0.11 a c++ compiler will be mandatory
  warnings.warn("DeprecationWarning: there is no c++ compiler."
WARNING (theano.configdefaults): g++ not detected ! Theano will be unable to execute 

optimized C-implementations (for both CPU and GPU) and will default to Python implementations.

 Performance will be severely degraded. To remove this warning, set Theano flags cxx to an empty

 string.
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
>>> import theano
>>> import theano.tensor as T
>>> x = T.dmatrix('x')
>>> s = 1 / (1 + T.exp(-x))
>>> logistic = theano.function([x], s)
>>> logistic([[0, 1], [-1, -2]])
array([[0.5       , 0.73105858],
       [0.26894142, 0.11920292]])
>>> ...