Python 3.7.4 : Testing the PyUSB python module.

This python module named PyUSB can be found at pypi website.

[mythcat@desk scripts]$ pip3 install pyusb --user
Collecting pyusb
...
Successfully installed pyusb-1.0.2

Let' see some usb device with lsusb command:

[mythcat@desk scripts]$ lsusb
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 004: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 001 Device 003: ID 093a:2510 Pixart Imaging, Inc. Optical Mouse
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

First you need to set this to avoid the error: Access denied (insufficient permissions.

[mythcat@desk scripts]$ ll  /dev/bus/usb/001/004
crw-rw-r--. 1 root root 189, 3 Oct 10 20:34 /dev/bus/usb/001/004
[mythcat@desk scripts]$ chmod a+rw /dev/bus/usb/001/004
chmod: changing permissions of '/dev/bus/usb/001/004': Operation not permitted
[mythcat@desk scripts]$ sudo chmod a+rw /dev/bus/usb/001/004
[sudo] password for mythcat: 
[mythcat@desk scripts]$ ll  /dev/bus/usb/001/004
crw-rw-rw-. 1 root root 189, 3 Oct 10 20:34 /dev/bus/usb/001/004

The script is simple:

import sys
  
import usb.core
import usb.util
print(usb.__version__)
busses = usb.busses()
for bus in busses:
    devices = bus.devices
    for dev in devices:
        if dev != None:
            try:
                usd_dev = usb.core.find(idVendor=dev.idVendor, idProduct=dev.idProduct)
                print(usb_dev)
            except:
                pass
# 1a40:0101
dev  = usb.core.find(idVendor=0x1a40, idProduct=0x0101)
print ("The 8087:0024 is : ", dev)
if dev is None:
    raise ValueError("Device not found!")
else:

    if dev.is_kernel_driver_active(0):
        try:
                dev.detach_kernel_driver(0)
                print ("kernel driver detached")
        except usb.core.USBError as e:
                sys.exit("Could not detach kernel driver: %s" % str(e))
    else:
        print ("no kernel driver attached")
    try:
        usb.util.claim_interface(dev, 0)
        print ("claimed device")
    except:
        sys.exit("Could not claim the device: %s" % str(e))
    try:
        dev.set_configuration()
        dev.reset()
    except usb.core.USBError as e:
        sys.exit("Could not set configuration: %s" % str(e))

usb.util.release_interface(dev,interface)
dev.attach_kernel(interface

The result of this python script is this:

[mythcat@desk scripts]$ python3 usb_test.py 
1.0.2
The 8087:0024 is :  DEVICE ID 1a40:0101 on Bus 001 Address 004 =================
 bLength                :   0x12 (18 bytes)
 bDescriptorType        :    0x1 Device
 bcdUSB                 :  0x200 USB 2.0
 bDeviceClass           :    0x9 Hub
 bDeviceSubClass        :    0x0
 bDeviceProtocol        :    0x1
 bMaxPacketSize0        :   0x40 (64 bytes)
 idVendor               : 0x1a40
 idProduct              : 0x0101
 bcdDevice              :  0x111 Device 1.11
 iManufacturer          :    0x0 
 iProduct               :    0x1 USB 2.0 Hub
 iSerialNumber          :    0x0 
 bNumConfigurations     :    0x1
  CONFIGURATION 1: 100 mA ==================================
   bLength              :    0x9 (9 bytes)
   bDescriptorType      :    0x2 Configuration
   wTotalLength         :   0x19 (25 bytes)
   bNumInterfaces       :    0x1
   bConfigurationValue  :    0x1
   iConfiguration       :    0x0 
   bmAttributes         :   0xe0 Self Powered, Remote Wakeup
   bMaxPower            :   0x32 (100 mA)
    INTERFACE 0: Hub =======================================
     bLength            :    0x9 (9 bytes)
     bDescriptorType    :    0x4 Interface
     bInterfaceNumber   :    0x0
     bAlternateSetting  :    0x0
     bNumEndpoints      :    0x1
     bInterfaceClass    :    0x9 Hub
     bInterfaceSubClass :    0x0
     bInterfaceProtocol :    0x0
     iInterface         :    0x0 
      ENDPOINT 0x81: Interrupt IN ==========================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :   0x81 IN
       bmAttributes     :    0x3 Interrupt
       wMaxPacketSize   :    0x1 (1 bytes)
       bInterval        :    0xc
kernel driver detached
claimed device
Could not set configuration: [Errno 16] Resource busy

Set permisions for next usb:

[mythcat@desk scripts]$ ll /dev/bus/usb/001/003
crw-rw-rw-. 1 root root 189, 2 Oct 10 20:34 /dev/bus/usb/001/003

Te next source code will read the mouse device:

#!/usr/bin/python
import sys
import usb.core
import usb.util
# decimal vendor and product values
#dev = usb.core.find(idVendor=1118, idProduct=1917)
# or, uncomment the next line to search instead by the hexidecimal equivalent
# 093a:2510
dev = usb.core.find(idVendor=0x093a, idProduct=0x2510)
# first endpoint
interface = 0
endpoint = dev[0][(0,0)][0]
# if the OS kernel already claimed the device, which is most likely true
# thanks to http://stackoverflow.com/questions/8218683/pyusb-cannot-set-configuration
if dev.is_kernel_driver_active(interface) is True:
  # tell the kernel to detach
  dev.detach_kernel_driver(interface)
  # claim the device
  usb.util.claim_interface(dev, interface)
collected = 0
attempts = 50
while collected < attempts :
    try:
        data = dev.read(endpoint.bEndpointAddress,endpoint.wMaxPacketSize)
        collected += 1
        print (data)
    except usb.core.USBError as e:
        data = None
        if e.args == ('Operation timed out',):
            continue
# release the device
usb.util.release_interface(dev, interface)
# reattach the device to the OS kernel
dev.attach_kernel_driver(interface)

The output of mouse moves is this:

[mythcat@desk scripts]$ python3 usb_mouse.py 
[mythcat@desk scripts]$ python3 usb_mouse.py 
array('B', [0, 254, 255, 0])
array('B', [0, 253, 2, 0])
array('B', [0, 252, 3, 0])
array('B', [0, 251, 3, 0])
array('B', [0, 252, 3, 0])
array('B', [0, 254, 1, 0])
array('B', [0, 253, 2, 0])
array('B', [0, 255, 1, 0])
array('B', [0, 255, 4, 0])
array('B', [0, 0, 3, 0])
array('B', [0, 0, 3, 0])
array('B', [0, 0, 2, 0])
array('B', [0, 0, 2, 0])
array('B', [0, 2, 1, 0])
array('B', [0, 4, 1, 0])
array('B', [0, 3, 0, 0])
array('B', [0, 3, 0, 0])
array('B', [0, 1, 0, 0])

Python 3.7.4 : Example with subprocess - part 001.

This is a simple example with the python 3 subprocess package.
The source code is simple to understand.
The execute_proceess_with_communicate let run the ls command with the sudo user permissions:

import os
import sys
import string
import subprocess
import codecs

inp = ''
cmd = 'ls'
password = ''

def execute_proceess_with_communicate(inp):
    """Return a list of hops from traceroute command."""
    p = subprocess.Popen(
            ['sudo', cmd, inp],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            shell=False)
    text, _ = p.communicate(password)
    #print(type(text))
    outp = codecs.decode(text,'utf8')
    out_split=outp.split('\n')
    return out_split

def normalize_out(list_outp):
    """Extract information from traceroute line per line."""

    normalized_out = []
    for op in list_outp:
        # filer out if an empty line
        if len(op) is 0:
            continue
        op_split = op.split()
        normalized_out.append(op_split)
    return normalized_out

if __name__ == '__main__':
    inp = sys.argv[1]

    out = execute_proceess_with_communicate(inp)
    n_out = normalize_out(out)
    print(n_out)

The result is this:

[mythcat@desk scripts]$ python3 subprocess_001.py '/' 
[['bin'], ['boot'], ['dev'], ['etc'], ['home'], ['lib'], ['lib64'], ['media'], ['mnt'], ['opt'], ['proc'],
 ['root'], ['run'], ['sbin'], ['srv'], ['sys'], ['tmp'], ['usr'], ['var']]

Python Qt5 : the drag and drop feature.

Today I tested drag and drop feature with PyQt5.

Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux

This is a simple example using setAcceptDrops and setDragEnabled:

import sys
from PyQt5.QtWidgets import QApplication, QWidget, QListWidget, QHBoxLayout,QListWidgetItem
from PyQt5.QtGui import QIcon

class Window(QWidget):
    def __init__(self):
        super().__init__()

        self.myListWidget1 = QListWidget()
        self.myListWidget2 = QListWidget()
        self.myListWidget2.setViewMode(QListWidget.IconMode)
        self.myListWidget1.setAcceptDrops(True)
        self.myListWidget1.setDragEnabled(True)
        self.myListWidget2.setAcceptDrops(True)
        self.myListWidget2.setDragEnabled(True)
        self.setGeometry(480, 400, 640, 480)
        self.myLayout = QHBoxLayout()
        self.myLayout.addWidget(self.myListWidget1)
        self.myLayout.addWidget(self.myListWidget2)

        l1 = QListWidgetItem(QIcon('house.png'), "House")
        l2 = QListWidgetItem(QIcon('cloud.png'), "Clouds ")
        l3 = QListWidgetItem(QIcon('user.png'), "User")
        l4 = QListWidgetItem(QIcon('save.png'), "Save")

        self.myListWidget1.insertItem(1, l1)
        self.myListWidget1.insertItem(2, l2)
        self.myListWidget1.insertItem(3, l3)
        self.myListWidget1.insertItem(4, l4)

        QListWidgetItem(QIcon('house.png'), "House", self.
                        myListWidget2)
        QListWidgetItem(QIcon('cloud.png'), "Clouds", self.
                        myListWidget2)
        QListWidgetItem(QIcon('save.png'), "Save", self.
                        myListWidget2)

        self.setWindowTitle('Example: Drag and Drop');
        self.setLayout(self.myLayout)

        self.show()

App = QApplication(sys.argv)
window = Window()
sys.exit(App.exec())

The result is a windows with two QListWidget with an drag and drop feature.

Python 3.7.4 : Using the paramiko pakage.

Today I tested the paramiko package.
First, I install and check the version of this package.

[mythcat@desk my_network_tools]$ pip3 install paramiko --user
Collecting paramiko
...
  Running setup.py install for pycparser ... done
Successfully installed asn1crypto-0.24.0 bcrypt-3.1.7 cffi-1.12.3 cryptography-2.7 paramiko-2.6.0

 pycparser-2.19 pynacl-1.3.0
[mythcat@desk my_network_tools]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
>>> dir(paramiko)
['AUTH_FAILED', 'AUTH_PARTIALLY_SUCCESSFUL', 'AUTH_SUCCESSFUL', 'Agent', 'AgentKey', 'AuthHandler',

 'AuthenticationException', 'AutoAddPolicy', 'BadAuthenticationType', 'BadHostKeyException', 'BaseSFTP',

 'BufferedFile', 'Channel', 'ChannelException', 'ChannelFile', 'ChannelStderrFile', 'ChannelStdinFile',

 'DSSKey', 'ECDSAKey', 'Ed25519Key', 'GSSAuth', 'GSS_AUTH_AVAILABLE', 'GSS_EXCEPTIONS', 'HostKeys',

 'InteractiveQuery', 'Message', 'MissingHostKeyPolicy', 'OPEN_FAILED_ADMINISTRATIVELY_PROHIBITED',

 'OPEN_FAILED_CONNECT_FAILED', 'OPEN_FAILED_RESOURCE_SHORTAGE', 'OPEN_FAILED_UNKNOWN_CHANNEL_TYPE',

 'OPEN_SUCCEEDED', 'PKey', 'Packetizer', 'PasswordRequiredException', 'ProxyCommand', 'ProxyCommandFailure',

 'PublicBlob', 'RSAKey', 'RejectPolicy', 'SFTP', 'SFTPAttributes', 'SFTPClient', 'SFTPError', 'SFTPFile',

 'SFTPHandle', 'SFTPServer', 'SFTPServerInterface', 'SFTP_BAD_MESSAGE', 'SFTP_CONNECTION_LOST', 'SFTP_EOF',

 'SFTP_FAILURE', 'SFTP_NO_CONNECTION', 'SFTP_NO_SUCH_FILE', 'SFTP_OK', 'SFTP_OP_UNSUPPORTED', 

'SFTP_PERMISSION_DENIED', 'SSHClient', 'SSHConfig', 'SSHException', 'SecurityOptions', 'ServerInterface',

 'SubsystemHandler', 'Transport', 'WarningPolicy', '__all__', '__author__', '__builtins__', '__cached__', 

'__doc__', '__file__', '__license__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 

'__version__', '__version_info__', '_version', 'agent', 'auth_handler', 'ber', 'buffered_pipe', 'channel',

 'client', 'common', 'compress', 'config', 'dsskey', 'ecdsakey', 'ed25519key', 'file', 'hostkeys', 

'io_sleep', 'kex_curve25519', 'kex_ecdh_nist', 'kex_gex', 'kex_group1', 'kex_group14', 'kex_group16',

 'kex_gss', 'message', 'packet', 'pipe', 'pkey', 'primes', 'proxy', 'py3compat', 'rsakey', 'server', 

'sftp', 'sftp_attr', 'sftp_client', 'sftp_file', 'sftp_handle', 'sftp_server', 'sftp_si', 'ssh_exception',

 'ssh_gss', 'sys', 'transport', 'util']
>>> paramiko.__version__
'2.6.0'

The documentation for this version can be found at this webpage.
A simple example with this python package and ssh connection to 192.168.0.143 on port 22 can be see on the next source code

#!/usr/bin/env python
"""Return the ssh output with password connection and Linux commands"""
import paramiko

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
conn= client.connect('192.168.0.143', port='22', username='mythcat', password='the_pass')
# Obtain session
session = client.get_transport().open_session()
print("| Retcode: "+str(session)+"|")

stdin, stdout, stderr=client.exec_command('sudo hostname;w')
save_stdout = stdout.readlines()
retcode = stdout.channel.recv_exit_status()

stdin, stdout, stderr=client.exec_command('ss -nap;')
save_stdout2 = stdout.readlines()
print(save_stdout,save_stdout2)
#for line in stdout:
#    print (line)

Python 3.7.4 : Install the protobuf from sources on Fedora distro.

Today I will show you how to build protobuf from sources using the Fedora distro.
The google team comes with this intro:
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler...
This google project comes with these tutorials.
The GitHub project can be found here.
To install the compiler, download the package.

[mythcat@desk ~]$ cd Downloads/
[mythcat@desk Downloads]$ cp protobuf-python-3.9.2.tar.gz ~/
[mythcat@desk Downloads]$ cd ..
[mythcat@desk ~]$ tar xvzf protobuf-python-3.9.2.tar.gz 
...
protobuf-3.9.2/aclocal.m4
protobuf-3.9.2/install-sh
protobuf-3.9.2/generate_descriptor_proto.sh
protobuf-3.9.2/CHANGES.txt
protobuf-3.9.2/configure.ac
protobuf-3.9.2/configure

Let's see the content:

[mythcat@desk ~]$ cd protobuf-3.9.2/
[mythcat@desk protobuf-3.9.2]$ ls
aclocal.m4                   config.sub                    ltmain.sh            README.md
ar-lib                       configure                     m4                   six.BUILD
autogen.sh                   configure.ac                  Makefile.am          src
benchmarks                   conformance                   Makefile.in          test-driver
BUILD                        CONTRIBUTORS.txt              missing              third_party
CHANGES.txt                  depcomp                       objectivec           update_file_lists.sh
cmake                        editors                       protobuf.bzl         util
compile                      examples                      protobuf_deps.bzl    WORKSPACE
compiler_config_setting.bzl  generate_descriptor_proto.sh  protobuf-lite.pc.in
config.guess                 install-sh                    protobuf.pc.in
config.h.in                  LICENSE                       python

[mythcat@desk protobuf-3.9.2]$ ./configure
...
checking how to run the C preprocessor... gcc -E
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/mythcat/protobuf-3.9.2':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details
...
[root@desk protobuf-3.9.2]# dnf install g++
...
Installed:
  gcc-c++-9.2.1-1.fc30.x86_64                                                                               

Complete![root@desk protobuf-3.9.2]# exit
exit

Let's build again:

[mythcat@desk protobuf-3.9.2]$ ./configure
checking whether to enable maintainer-specific portions of Makefiles... yes
checking build system type... x86_64-pc-linux-gnu
...
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/gmock-config
config.status: creating build-aux/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
[mythcat@desk protobuf-3.9.2]$ nproc --all
2
[mythcat@desk protobuf-3.9.2]$ make -j2
...
  CXXLD    protoc
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2'
[mythcat@desk protobuf-3.9.2]$ make check -j2
...PASS: protobuf-lazy-descriptor-test
PASS: protobuf-lite-test
PASS: google/protobuf/compiler/zip_output_unittest.sh
PASS: google/protobuf/io/gzip_stream_unittest.sh
PASS: protobuf-lite-arena-test
PASS: no-warning-test
PASS: protobuf-test
============================================================================
Testsuite summary for Protocol Buffers 3.9.2
============================================================================
# TOTAL: 7
# PASS:  7
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'

The last steps is for install:

[mythcat@desk protobuf-3.9.2]$ sudo make install
[sudo] password for mythcat: 
...
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
[mythcat@desk protobuf-3.9.2]$ sudo ldconfig 

Let's test it:

[mythcat@desk protobuf-3.9.2]$ protoc --version
libprotoc 3.9.2

Now the next step comes for python module:

[mythcat@desk protobuf-3.9.2]$ cd python/
[mythcat@desk python]$ ls
google  MANIFEST.in  mox.py  README.md  release  release.sh  setup.cfg  setup.py  stubout.py  tox.ini
[mythcat@desk python]$ python setup.py build
...
testSerialize (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeMessageSetWireFormatUnknownExtension 
(google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeProto3 (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testByteSizeFunctions (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testPackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testUnpackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagDecode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagEncode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok

----------------------------------------------------------------------
Ran 818 tests in 5.343s

OK (skipped=10)

Let's test the python module:

[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import google
>>> from google.protobuf import descriptor as _descriptor
>>> from google.protobuf import message as _message
>>> from google.protobuf import reflection as _reflection
>>> from google.protobuf import symbol_database as _symbol_database
>>> from google.protobuf import descriptor_pb2
>>> from google.protobuf import text_format
>>> dir(_descriptor)
['Descriptor', 'DescriptorBase', 'DescriptorMetaclass', 'EnumDescriptor', 'EnumValueDescriptor', 
'Error', 'FieldDescriptor', 'FileDescriptor', 'MakeDescriptor', 'MethodDescriptor', 'OneofDescriptor', 
'ServiceDescriptor', 'TypeTransformationError', '_Lock', '_NestedDescriptorBase', '_OptionsOrNone', 
'_ParseOptions', '_ToCamelCase', '_ToJsonName', '_USE_C_DESCRIPTORS', '__author__', '__builtins__', 
'__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_lock', 
'api_implementation', 'six', 'threading']

Now I can to define the structure for the data structured as messages
These message is a small logical record of information containing a series of name-value pairs called fields.
The protobuffers compiler (protoc) to generate the source code in the language you need (from the .proto file).
To generate a Python file, you need to execute:
protoc -I=$SRC_DIR --python_out=$DST_DIR $SRC_DIR/my_example.proto
Let's test with am example (you can use any proto file) in default folder (use . for default folder):

[mythcat@desk python]$ protoc -I=. --python_out=. my_example.proto
[mythcat@desk python]$ ls my_example*
my_example_pb2.py  my_example.proto
[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_example_pb2
>>> dir(my_example_pb2)
['AddressBook', 'DESCRIPTOR', 'Person', '_ADDRESSBOOK', '_PERSON', '_PERSON_PHONENUMBER', '_PERSON_PHONETYPE',
 '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__',
 '_b', '_descriptor', '_message', '_reflection', '_sym_db', '_symbol_database', 'sys']

The tensorflow python module - part 004.

If you using the tensorflow then you can get some warnings.
You can use warnings python package to manage all of this:

[mythcat@desk ~]$ $ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> import tensorflow as tf
/home/mythcat/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: 
Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be 
understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/mythcat/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: 
Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be 
understood as (type, (1,)) / '(1,)type'.
...

/home/mythcat/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: 
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of 
numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
>>>
>>> warnings.filterwarnings('ignore')
>>>
[7]+  Stopped                 python3

Use it in this way to filter these warnings:

[mythcat@desk ~]$ $ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> warnings.filterwarnings('ignore',category=FutureWarning)
>>> import tensorflow as tf
>>> 

Python 3.7.4 : Print with random colors.

This is a simple example for custom print output.
The script detect the platform for color settings and then use print.
The first print will print with blue color the name of the script.
I used random to select a random color from colors array and used to print the -=RANDOM COLOR=- text.
The print (W+'') is used to set default white color for terminal

import sys
import random
if sys.platform == "linux" or sys.platform == "linux2":
        BB = "\033[34;1m" # Blue light
        YY = "\033[33;1m" # Yellow light
        GG = "\033[32;1m" # Green light
        WW = "\033[0;1m"  # White light
        RR = "\033[31;1m" # Red light
        CC = "\033[36;1m" # Cyan light
        B = "\033[34m"    # Blue
        Y = "\033[33m"    # Yellow
        G = "\033[32m"    # Green
        W = "\033[0m"     # White
        R = "\033[31m"    # Red
        C = "\033[36m"    # Cyan
colors = [BB,YY,GG,WW,RR,CC,B,Y,G,W,R,C]
print (B+"\033[2;2m "+sys.argv[0]+"\n"+B)

color=random.choice(colors)
print (color+"-=RANDOM COLOR=-"+color)
print (W+'')

For winodws platform you need to add this:

elif sys.platform == "win32":

 BB = '' # Blue light
 YY = '' # Yellow light
 GG = '' # Green light
 WW = '' # White light
 RR = '' # Red light
 CC = '' # Cyan light
 B = ''  # Blue
 Y = ''  # Yellow
 G = ''  # Green
 W = ''  # White
 R = ''  # Red
 C = ''  # Cyan
 P = ''  # Random color

Python 3.7.4 : Using the theano pakage.

If you want to test theano then you need to see this webpage.

[root@desk mythcat]# dnf search theano
======================== Name & Summary Matched: theano ========================
python-theano-doc.noarch : Theano documentation
============================= Name Matched: theano =============================
python3-theano.noarch : Mathematical expressions involving multidimensional
                      : arrays
=========================== Summary Matched: theano ============================
python3-lasagne.noarch : Lightweight library to build and train neural networks
                       : in Theano
[root@desk mythcat]# pip3 install Theano --user
WARNING: Running pip install with root privileges is generally not a good idea. 

Try `pip3 install --user` instead.
Collecting Theano
...
  Running setup.py install for Theano ... done
Successfully installed Theano-1.0.4 scipy-1.3.1

Let's see first example:

[mythcat@desk ~]$ python3 
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
/home/mythcat/.local/lib/python3.7/site-packages/theano/configdefaults.py:560: 

UserWarning: DeprecationWarning: there is no c++ compiler.This is deprecated and with 

Theano 0.11 a c++ compiler will be mandatory
  warnings.warn("DeprecationWarning: there is no c++ compiler."
WARNING (theano.configdefaults): g++ not detected ! Theano will be unable to execute 

optimized C-implementations (for both CPU and GPU) and will default to Python implementations.

 Performance will be severely degraded. To remove this warning, set Theano flags cxx to an empty

 string.
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
>>> import theano
>>> import theano.tensor as T
>>> x = T.dmatrix('x')
>>> s = 1 / (1 + T.exp(-x))
>>> logistic = theano.function([x], s)
>>> logistic([[0, 1], [-1, -2]])
array([[0.5       , 0.73105858],
       [0.26894142, 0.11920292]])
>>> ... 

Python 3.7.4 : Using the sunpy - part 001.

I wrote about sunpy in the past on this website.
Now this package comes with new features, see the official webpage.
Let's install it.

[mythcat@desk ~]$ pip3 install sunpy --user
Collecting sunpy
...
Successfully installed aioftp-0.13.0 aiohttp-3.6.0 astropy-3.2.1 async-timeout-3.0.1 
multidict-4.5.2 parfive-1.0.0 scipy-1.3.1 sunpy-1.0.3 tqdm-4.35.0 yarl-1.3.0

If you search on web you can find example with spectra , but in the new package is not supported.

ModuleNotFoundError: No module named 'sunpy.spectra'

Let's test it:

[mythcat@desk ~]$ python3 
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sunpy
>>> dir(sunpy)
['SunPyTestRunner', 'UnsupportedPythonError', '__all__', '__builtins__', '__cached__', '__doc__',
 '__file__', '__loader__', '__minimum_python_version__', '__name__', '__package__', '__path__',
 '__spec__', '__version__', '_init_log', 'builtins', 'config', 'extern', 'load_config', 'log',
 'logging', 'os', 'print_config', 'self_test', 'sys', 'system_info', 'tests', 'util', 'version']
>>> sunpy.system_info()
==============================
SunPy Installation Information
==============================

#######
General
#######
Time : Monday, 09. September 2019 07:14PM UT
System : Linux
Processor : x86_64
Arch : 64bit
SunPy : 1.0.3
OS: Fedora 30 Thirty (Linux 5.2.11-200.fc30.x86_64 x86_64)


##################
Required Libraries
##################
Python: 3.7.4
NumPy: 1.16.4
SciPy: 1.3.1
matplotlib: 3.0.3
Astropy: 3.2.1
Pandas: 0.25.1
parfive: 1.0.0


#####################
Recommended Libraries
#####################
beautifulsoup: NOT INSTALLED
PyQt4: NOT INSTALLED
PyQt5: 5.12.2
Zeep: NOT INSTALLED
Sqlalchemy: 1.3.7
drms: NOT INSTALLED
>>> help(sunpy)

Help on package sunpy:

NAME
    sunpy

DESCRIPTION
    SunPy
    =====
    
    An open-source Python library for Solar Physics data analysis.
    
    Web Links
    ---------
    Homepage: https://sunpy.org
    Documentation: https://docs.sunpy.org/en/stable/

PACKAGE CONTENTS
    cm (package)
    compiler_version
    conftest
    coordinates (package)
    data (package)
    database (package)
    extern (package)
    image (package)
    instr (package)
    io (package)
    map (package)
    net (package)
    physics (package)
    roi (package)
    sun (package)
    tests (package)
    time (package)
    timeseries (package)
    util (package)
    version
    visualization (package)
... 

Let's install zeep, beautifulsoup4 and drms python packages:

[mythcat@desk ~]$ pip3 install zeep --user
Collecting zeep
...
Successfully installed appdirs-1.4.3 cached-property-1.5.1 isodate-0.6.0 lxml-4.4.1
 requests-toolbelt-0.9.1 zeep-3.4.0
[mythcat@desk ~]$ pip3 install --upgrade beautifulsoup4 --user
Collecting beautifulsoup4
...
Successfully installed beautifulsoup4-4.8.0 soupsieve-1.9.3
[mythcat@desk ~]$ pip3 install --upgrade drms --user
Collecting drms
...
Successfully installed drms-0.5.7

Now, we can see examples with this python package.
First, I will use the AIA_171_IMAGE image:

>>> from sunpy.data.sample import AIA_171_IMAGE 
Files Downloaded: 100%|███████████████████████| 26/26 [00:04<00:00 5.50file="" s="">>> import sunpy.map                                                            
>>> aiamap = sunpy.map.Map(AIA_171_IMAGE)
>>> aiamap.peek()                                                               
Attribute Qt::AA_EnableHighDpiScaling must be set before QCoreApplication is created.                                                                           

** (python3:4152): WARNING **: 22:53:15.329: AT-SPI: Could not obtain desktop path or name


** (python3:4152): WARNING **: 22:53:15.377: atk-bridge: GetRegisteredEvents returned message with unknown signature

Python 3.7.3 : Using the flask - part 018.

In this tutorial, I will show you how to fix auto increment in Flask SQLAlchemy.
The old source code for the user model from the server.py was this:

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    password = db.Column(db.String(120), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)

The server.sqlite will be this:

[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE user (
 id INTEGER NOT NULL, 
 username VARCHAR(80), 
 password VARCHAR(120), 
 email VARCHAR(120), 
 gender VARCHAR(5), 
 work VARCHAR(33), 
 city VARCHAR(15), 
 PRIMARY KEY (id), 
 UNIQUE (username), 
 UNIQUE (password), 
 UNIQUE (email), 
 UNIQUE (gender), 
 UNIQUE (work), 
 UNIQUE (city)
);
CREATE TABLE alembic_version (
 version_num VARCHAR(32) NOT NULL, 
 CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num)
);
CREATE TABLE sqlite_stat1(tbl,idx,stat);
sqlite> ^Z 

If you want to change the id into auto increment then you need to follow this steps:

class User(db.Model):
    __tablename__ = 'user'
    __table_args__ = {'sqlite_autoincrement': True}
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    password = db.Column(db.String(120), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)

Delete the server.sqlite file or rename it.
Open python3 and create a new server.sqlite file:

[mythcat@desk my_flask]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
>>> db.create_all()
>>> db.engine.table_names()
['sqlite_sequence', 'user']
>>> 
[5]+  Stopat                  python3 

Open the new file to see the changes:

[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE user (
 id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, 
 username VARCHAR(80), 
 password VARCHAR(120), 
 email VARCHAR(120), 
 gender VARCHAR(5), 
 work VARCHAR(33), 
 city VARCHAR(15), 
 UNIQUE (username), 
 UNIQUE (password), 
 UNIQUE (email), 
 UNIQUE (gender), 
 UNIQUE (work), 
 UNIQUE (city)
);
CREATE TABLE sqlite_sequence(name,seq);
sqlite>  

Python 3.7.4 : Create an Stand Alone Executable on Fedora distro.

In this tutorial I will show you how to create an Stand Alone Executable with Python version 3.7.4 and Fedora 30 distro.
First you need to install using the dnf tool the python3 package.
You can test it easy with this command:

[mythcat@desk dist]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information. 

I created a simple python script named test.py for testing:

import os
import sys
print("Hello!")

You need to install the pyinstaller python module.
Let's see this step with the output:

[mythcat@desk ~]$ pip3 install pyinstaller --user

Now you can use this python module to create Stand Alone Executable.

[mythcat@desk ~]$ pip3 install pyinstaller --user
Requirement already satisfied: pyinstaller in ./.local/lib/python3.7/site-packages (3.5)
Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from pyinstaller) (40.8.0)
Requirement already satisfied: altgraph in ./.local/lib/python3.7/site-packages (from pyinstaller) (0.16.1)
[mythcat@desk ~]$ pyinstaller --onefile test.py
561 INFO: PyInstaller: 3.5
561 INFO: Python: 3.7.4
571 INFO: Platform: Linux-5.2.9-200.fc30.x86_64-x86_64-with-fedora-30-Thirty
573 INFO: wrote /home/mythcat/test.spec
596 INFO: UPX is not available.
598 INFO: Extending PYTHONPATH with paths
['/home/mythcat', '/home/mythcat']
598 INFO: checking Analysis
624 INFO: Building because /home/mythcat/test.py changed
624 INFO: Initializing module dependency graph...
640 INFO: Initializing module graph hooks...
678 INFO: Analyzing base_library.zip ...
5247 INFO: running Analysis Analysis-00.toc
5352 INFO: Caching module hooks...
5376 INFO: Analyzing /home/mythcat/test.py
5396 INFO: Loading module hooks...
5397 INFO: Loading module hook "hook-xml.py"...
5775 INFO: Loading module hook "hook-pydoc.py"...
5800 INFO: Loading module hook "hook-encodings.py"...
5892 INFO: Looking for ctypes DLLs
5893 INFO: Analyzing run-time hooks ...
5901 INFO: Looking for dynamic libraries
6507 INFO: Looking for eggs
6507 INFO: Using Python library /lib64/libpython3.7m.so.1.0
6514 INFO: Warnings written to /home/mythcat/build/test/warn-test.txt
6547 INFO: Graph cross-reference written to /home/mythcat/build/test/xref-test.html
6604 INFO: checking PYZ
6607 INFO: Building because toc changed
6607 INFO: Building PYZ (ZlibArchive) /home/mythcat/build/test/PYZ-00.pyz
7030 INFO: Building PYZ (ZlibArchive) /home/mythcat/build/test/PYZ-00.pyz completed successfully.
7034 INFO: checking PKG
7035 INFO: Building because toc changed
7035 INFO: Building PKG (CArchive) PKG-00.pkg
10033 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
10036 INFO: Bootloader 
/home/mythcat/.local/lib/python3.7/site-packages/PyInstaller/bootloader/Linux-64bit/run
10036 INFO: checking EXE
10123 INFO: Building because toc changed
10124 INFO: Building EXE from EXE-00.toc
10163 INFO: Appending archive to ELF section in EXE /home/mythcat/dist/test
10337 INFO: Building EXE from EXE-00.toc completed successfully.

This command will create some folders: dist, build, ...
The dist folder will have the Stand Alone Executable named test:

[mythcat@desk ~]$ cd dist/
[mythcat@desk dist]$ ls
test
[mythcat@desk dist]$ ./test
Hello!

Python 3.7.3 : Using the flask - part 017.

Today I make some changes with my server.py and database and solve some issues, see old version at my old tutorial.
Firt issue was start script.
I create a linux script named start_server.sh to run the flask run command:

[mythcat@desk my_flask]$ ./start_server.sh 

I update the User with new fiels:

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    password = db.Column(db.String(120), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)

Let's see how I deal with this versus database and migrate process.

[mythcat@desk my_flask]$ rm server.sqlite 
[mythcat@desk my_flask]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
>>> db.create_all()
>>> db.engine.table_names()
['user']
>>> exit()
[mythcat@desk my_flask]$ ls server.sqlite 
server.sqlite
[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .tables
user
sqlite> .schema user
CREATE TABLE user (
        id INTEGER NOT NULL, 
        username VARCHAR(80), 
        password VARCHAR(120), 
        email VARCHAR(120), 
        gender VARCHAR(5), 
        work VARCHAR(33), 
        city VARCHAR(15), 
        PRIMARY KEY (id), 
        UNIQUE (username), 
        UNIQUE (password), 
        UNIQUE (email), 
        UNIQUE (gender), 
        UNIQUE (work), 
        UNIQUE (city)
);

I got a strange error:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: city

I fix it with this , but I'm not sure if the right way:

[mythcat@desk my_flask]$ rm migrations/ -r -f 
[mythcat@desk my_flask]$ python3 server.py db init 
  Creating directory /home/mythcat/project_github/my_flask/migrations ... done
  Creating directory /home/mythcat/project_github/my_flask/migrations/versions ... done
  Generating /home/mythcat/project_github/my_flask/migrations/script.py.mako ... done
  Generating /home/mythcat/project_github/my_flask/migrations/env.py ... done
  Generating /home/mythcat/project_github/my_flask/migrations/alembic.ini ... done
  Generating /home/mythcat/project_github/my_flask/migrations/README ... done
  Please edit configuration/connection/logging settings in '/home/mythcat/project_github/my_flask/migrations/alembic.ini'
  before proceeding.
[mythcat@desk my_flask]$ python3 server.py db migrate
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.env] No changes in schema detected.
[mythcat@desk my_flask]$ python3 server.py db upgrade
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.

Python 3.7.3 : Using the flask - part 016.

Today I tested a new feature of Flask version 1.1.1.

[mythcat@desk my_flask]$ pip list | grep Flask
Flask                    1.1.1            
Flask-Login              0.4.1            
Flask-Mail               0.9.1            
Flask-Migrate            2.5.2            
Flask-Script             2.0.6            
Flask-SQLAlchemy         2.4.0            
Flask-WTF                0.14.2 

This feature will remove the jsonify python module.
Let's start the blueprint blue_test:

[mythcat@desk my_flask]$ export FLASK_APP=blue_test
[mythcat@desk my_flask]$ flask run 
 * Serving Flask app "blue_test"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.

The output is this:

 {"result":"This is main page!"}

Into my blue_test folder I will remove this jsonify python module:
[

mythcat@desk my_flask]$ cd blue_test/
[mythcat@desk blue_test]$ ll
total 40
-rw-rw-r--. 1 mythcat mythcat  192 Aug 23 11:02 extensions.py
-rw-rw-r--. 1 mythcat mythcat 1826 Aug 23 11:02 forms.py
-rw-rw-r--. 1 mythcat mythcat  701 Aug 23 12:03 __init__.py
-rw-rw-r--. 1 mythcat mythcat 1491 Aug 23 11:02 models.py
drwxrwxr-x. 2 mythcat mythcat  184 Aug 25 14:36 __pycache__
-rw-rw-r--. 1 mythcat mythcat 1050 Aug 23 11:02 routes.py
-rw-rw-r--. 1 mythcat mythcat  142 Aug 23 11:02 settings.py
-rw-rw-r--. 1 mythcat mythcat   34 Aug 23 11:02 start.bat
drwxrwxr-x. 2 mythcat mythcat   81 Aug 23 11:02 templates
-rw-rw-r--. 1 mythcat mythcat 8192 Aug 23 11:02 texts.sqlite
-rw-rw-r--. 1 mythcat mythcat  677 Aug 23 11:02 views.py 

The routes.py will have these changes:

#from flask import Blueprint, jsonify, request
# new flask
from flask import Blueprint, request
...
def home():
    #return jsonify({'result' : 'This is main page!'})
    #this is the new source code:
    return {'result' : 'This is main page!'} 

The next step is to find all jsonify word into my source code using the grep tool and make changes:

[mythcat@desk my_flask]$ grep -nr jsonify *
Binary file blue_test/__pycache__/views.cpython-37.pyc matches
blue_test/views.py:3:from flask import Blueprint, jsonify, request
blue_test/views.py:20:    return jsonify ({'texts':texts})
blue_test/routes.py:1:#from flask import Blueprint, jsonify, request
crud.py:1:from flask import Flask, request, jsonify
crud.py:57:    return jsonify(new_user)
crud.py:64:    return jsonify(result.data)
crud.py:70:    return user_schema.jsonify(user)
crud.py:83:    return user_schema.jsonify(user)
crud.py:92:    return user_schema.jsonify(user)
Binary file __pycache__/crud.cpython-37.pyc matches
Binary file __pycache__/server.cpython-37.pyc matches
server.py:9:from flask import jsonify
server.py:101:    #return jsonify(new_user)
server.py:109:    #return users_schema.jsonify(users)
server.py:111:    return jsonify(all_users.data)
server.py:117:    return jsonify(result.data)
server.py:122:        return jsonify([d['keyword'] for d in result.data])
server.py:132:    return user_schema.jsonify(new_user)
tserv.py:13:from flask import jsonify
tserv.py:56:    #return users_schema.jsonify(users)
tserv.py:58:    return jsonify(all_users.data)
tserv.py:69:    return user_schema.jsonify(user_post)
tserv.py:80:    return user_schema.jsonify(user_put)
tserv.py:88:    return user_schema.jsonify(user_delete)
[mythcat@desk my_flask]$  

After I make changes the result of grep is this:

[mythcat@desk my_flask]$ grep -nr jsonify *
Binary file blue_test/__pycache__/views.cpython-37.pyc matches
blue_test/routes.py:1:#from flask import Blueprint, jsonify, request
blue_test/views.py:3:#from flask import Blueprint, jsonify, request
crud.py:1:#from flask import Flask, request, jsonify
Binary file __pycache__/crud.cpython-37.pyc matches
Binary file __pycache__/server.cpython-37.pyc matches
server.py:9:#from flask import jsonify
server.py:109:    #return users_schema.jsonify(users)
tserv.py:13:#from flask import jsonify
tserv.py:56:    #return users_schema.jsonify(users) 

You can see the last version of my project here.