Python 3.7.4 : Example with subprocess - part 001.

This is a simple example with the python 3 subprocess package.
The source code is simple to understand.
The execute_proceess_with_communicate let run the ls command with the sudo user permissions:

import os
import sys
import string
import subprocess
import codecs

inp = ''
cmd = 'ls'
password = ''

def execute_proceess_with_communicate(inp):
    """Return a list of hops from traceroute command."""
    p = subprocess.Popen(
            ['sudo', cmd, inp],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            shell=False)
    text, _ = p.communicate(password)
    #print(type(text))
    outp = codecs.decode(text,'utf8')
    out_split=outp.split('\n')
    return out_split

def normalize_out(list_outp):
    """Extract information from traceroute line per line."""

    normalized_out = []
    for op in list_outp:
        # filer out if an empty line
        if len(op) is 0:
            continue
        op_split = op.split()
        normalized_out.append(op_split)
    return normalized_out

if __name__ == '__main__':
    inp = sys.argv[1]

    out = execute_proceess_with_communicate(inp)
    n_out = normalize_out(out)
    print(n_out)

The result is this:

[mythcat@desk scripts]$ python3 subprocess_001.py '/' 
[['bin'], ['boot'], ['dev'], ['etc'], ['home'], ['lib'], ['lib64'], ['media'], ['mnt'], ['opt'], ['proc'],
 ['root'], ['run'], ['sbin'], ['srv'], ['sys'], ['tmp'], ['usr'], ['var']]

Python Qt5 : the drag and drop feature.

Today I tested drag and drop feature with PyQt5.

Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux

This is a simple example using setAcceptDrops and setDragEnabled:

import sys
from PyQt5.QtWidgets import QApplication, QWidget, QListWidget, QHBoxLayout,QListWidgetItem
from PyQt5.QtGui import QIcon

class Window(QWidget):
    def __init__(self):
        super().__init__()

        self.myListWidget1 = QListWidget()
        self.myListWidget2 = QListWidget()
        self.myListWidget2.setViewMode(QListWidget.IconMode)
        self.myListWidget1.setAcceptDrops(True)
        self.myListWidget1.setDragEnabled(True)
        self.myListWidget2.setAcceptDrops(True)
        self.myListWidget2.setDragEnabled(True)
        self.setGeometry(480, 400, 640, 480)
        self.myLayout = QHBoxLayout()
        self.myLayout.addWidget(self.myListWidget1)
        self.myLayout.addWidget(self.myListWidget2)

        l1 = QListWidgetItem(QIcon('house.png'), "House")
        l2 = QListWidgetItem(QIcon('cloud.png'), "Clouds ")
        l3 = QListWidgetItem(QIcon('user.png'), "User")
        l4 = QListWidgetItem(QIcon('save.png'), "Save")

        self.myListWidget1.insertItem(1, l1)
        self.myListWidget1.insertItem(2, l2)
        self.myListWidget1.insertItem(3, l3)
        self.myListWidget1.insertItem(4, l4)

        QListWidgetItem(QIcon('house.png'), "House", self.
                        myListWidget2)
        QListWidgetItem(QIcon('cloud.png'), "Clouds", self.
                        myListWidget2)
        QListWidgetItem(QIcon('save.png'), "Save", self.
                        myListWidget2)

        self.setWindowTitle('Example: Drag and Drop');
        self.setLayout(self.myLayout)

        self.show()

App = QApplication(sys.argv)
window = Window()
sys.exit(App.exec())

The result is a windows with two QListWidget with an drag and drop feature.

Python 3.7.4 : Using the paramiko pakage.

Today I tested the paramiko package.
First, I install and check the version of this package.

[mythcat@desk my_network_tools]$ pip3 install paramiko --user
Collecting paramiko
...
  Running setup.py install for pycparser ... done
Successfully installed asn1crypto-0.24.0 bcrypt-3.1.7 cffi-1.12.3 cryptography-2.7 paramiko-2.6.0

 pycparser-2.19 pynacl-1.3.0
[mythcat@desk my_network_tools]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
>>> dir(paramiko)
['AUTH_FAILED', 'AUTH_PARTIALLY_SUCCESSFUL', 'AUTH_SUCCESSFUL', 'Agent', 'AgentKey', 'AuthHandler',

 'AuthenticationException', 'AutoAddPolicy', 'BadAuthenticationType', 'BadHostKeyException', 'BaseSFTP',

 'BufferedFile', 'Channel', 'ChannelException', 'ChannelFile', 'ChannelStderrFile', 'ChannelStdinFile',

 'DSSKey', 'ECDSAKey', 'Ed25519Key', 'GSSAuth', 'GSS_AUTH_AVAILABLE', 'GSS_EXCEPTIONS', 'HostKeys',

 'InteractiveQuery', 'Message', 'MissingHostKeyPolicy', 'OPEN_FAILED_ADMINISTRATIVELY_PROHIBITED',

 'OPEN_FAILED_CONNECT_FAILED', 'OPEN_FAILED_RESOURCE_SHORTAGE', 'OPEN_FAILED_UNKNOWN_CHANNEL_TYPE',

 'OPEN_SUCCEEDED', 'PKey', 'Packetizer', 'PasswordRequiredException', 'ProxyCommand', 'ProxyCommandFailure',

 'PublicBlob', 'RSAKey', 'RejectPolicy', 'SFTP', 'SFTPAttributes', 'SFTPClient', 'SFTPError', 'SFTPFile',

 'SFTPHandle', 'SFTPServer', 'SFTPServerInterface', 'SFTP_BAD_MESSAGE', 'SFTP_CONNECTION_LOST', 'SFTP_EOF',

 'SFTP_FAILURE', 'SFTP_NO_CONNECTION', 'SFTP_NO_SUCH_FILE', 'SFTP_OK', 'SFTP_OP_UNSUPPORTED', 

'SFTP_PERMISSION_DENIED', 'SSHClient', 'SSHConfig', 'SSHException', 'SecurityOptions', 'ServerInterface',

 'SubsystemHandler', 'Transport', 'WarningPolicy', '__all__', '__author__', '__builtins__', '__cached__', 

'__doc__', '__file__', '__license__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 

'__version__', '__version_info__', '_version', 'agent', 'auth_handler', 'ber', 'buffered_pipe', 'channel',

 'client', 'common', 'compress', 'config', 'dsskey', 'ecdsakey', 'ed25519key', 'file', 'hostkeys', 

'io_sleep', 'kex_curve25519', 'kex_ecdh_nist', 'kex_gex', 'kex_group1', 'kex_group14', 'kex_group16',

 'kex_gss', 'message', 'packet', 'pipe', 'pkey', 'primes', 'proxy', 'py3compat', 'rsakey', 'server', 

'sftp', 'sftp_attr', 'sftp_client', 'sftp_file', 'sftp_handle', 'sftp_server', 'sftp_si', 'ssh_exception',

 'ssh_gss', 'sys', 'transport', 'util']
>>> paramiko.__version__
'2.6.0'

The documentation for this version can be found at this webpage.
A simple example with this python package and ssh connection to 192.168.0.143 on port 22 can be see on the next source code

#!/usr/bin/env python
"""Return the ssh output with password connection and Linux commands"""
import paramiko

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
conn= client.connect('192.168.0.143', port='22', username='mythcat', password='the_pass')
# Obtain session
session = client.get_transport().open_session()
print("| Retcode: "+str(session)+"|")

stdin, stdout, stderr=client.exec_command('sudo hostname;w')
save_stdout = stdout.readlines()
retcode = stdout.channel.recv_exit_status()

stdin, stdout, stderr=client.exec_command('ss -nap;')
save_stdout2 = stdout.readlines()
print(save_stdout,save_stdout2)
#for line in stdout:
#    print (line)

Python 3.7.4 : Install the protobuf from sources on Fedora distro.

Today I will show you how to build protobuf from sources using the Fedora distro.
The google team comes with this intro:
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler...
This google project comes with these tutorials.
The GitHub project can be found here.
To install the compiler, download the package.

[mythcat@desk ~]$ cd Downloads/
[mythcat@desk Downloads]$ cp protobuf-python-3.9.2.tar.gz ~/
[mythcat@desk Downloads]$ cd ..
[mythcat@desk ~]$ tar xvzf protobuf-python-3.9.2.tar.gz 
...
protobuf-3.9.2/aclocal.m4
protobuf-3.9.2/install-sh
protobuf-3.9.2/generate_descriptor_proto.sh
protobuf-3.9.2/CHANGES.txt
protobuf-3.9.2/configure.ac
protobuf-3.9.2/configure

Let's see the content:

[mythcat@desk ~]$ cd protobuf-3.9.2/
[mythcat@desk protobuf-3.9.2]$ ls
aclocal.m4                   config.sub                    ltmain.sh            README.md
ar-lib                       configure                     m4                   six.BUILD
autogen.sh                   configure.ac                  Makefile.am          src
benchmarks                   conformance                   Makefile.in          test-driver
BUILD                        CONTRIBUTORS.txt              missing              third_party
CHANGES.txt                  depcomp                       objectivec           update_file_lists.sh
cmake                        editors                       protobuf.bzl         util
compile                      examples                      protobuf_deps.bzl    WORKSPACE
compiler_config_setting.bzl  generate_descriptor_proto.sh  protobuf-lite.pc.in
config.guess                 install-sh                    protobuf.pc.in
config.h.in                  LICENSE                       python

[mythcat@desk protobuf-3.9.2]$ ./configure
...
checking how to run the C preprocessor... gcc -E
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/mythcat/protobuf-3.9.2':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details
...
[root@desk protobuf-3.9.2]# dnf install g++
...
Installed:
  gcc-c++-9.2.1-1.fc30.x86_64                                                                               

Complete![root@desk protobuf-3.9.2]# exit
exit

Let's build again:

[mythcat@desk protobuf-3.9.2]$ ./configure
checking whether to enable maintainer-specific portions of Makefiles... yes
checking build system type... x86_64-pc-linux-gnu
...
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/gmock-config
config.status: creating build-aux/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
[mythcat@desk protobuf-3.9.2]$ nproc --all
2
[mythcat@desk protobuf-3.9.2]$ make -j2
...
  CXXLD    protoc
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2'
[mythcat@desk protobuf-3.9.2]$ make check -j2
...PASS: protobuf-lazy-descriptor-test
PASS: protobuf-lite-test
PASS: google/protobuf/compiler/zip_output_unittest.sh
PASS: google/protobuf/io/gzip_stream_unittest.sh
PASS: protobuf-lite-arena-test
PASS: no-warning-test
PASS: protobuf-test
============================================================================
Testsuite summary for Protocol Buffers 3.9.2
============================================================================
# TOTAL: 7
# PASS:  7
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'

The last steps is for install:

[mythcat@desk protobuf-3.9.2]$ sudo make install
[sudo] password for mythcat: 
...
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
[mythcat@desk protobuf-3.9.2]$ sudo ldconfig 

Let's test it:

[mythcat@desk protobuf-3.9.2]$ protoc --version
libprotoc 3.9.2

Now the next step comes for python module:

[mythcat@desk protobuf-3.9.2]$ cd python/
[mythcat@desk python]$ ls
google  MANIFEST.in  mox.py  README.md  release  release.sh  setup.cfg  setup.py  stubout.py  tox.ini
[mythcat@desk python]$ python setup.py build
...
testSerialize (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeMessageSetWireFormatUnknownExtension 
(google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeProto3 (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testByteSizeFunctions (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testPackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testUnpackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagDecode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagEncode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok

----------------------------------------------------------------------
Ran 818 tests in 5.343s

OK (skipped=10)

Let's test the python module:

[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import google
>>> from google.protobuf import descriptor as _descriptor
>>> from google.protobuf import message as _message
>>> from google.protobuf import reflection as _reflection
>>> from google.protobuf import symbol_database as _symbol_database
>>> from google.protobuf import descriptor_pb2
>>> from google.protobuf import text_format
>>> dir(_descriptor)
['Descriptor', 'DescriptorBase', 'DescriptorMetaclass', 'EnumDescriptor', 'EnumValueDescriptor', 
'Error', 'FieldDescriptor', 'FileDescriptor', 'MakeDescriptor', 'MethodDescriptor', 'OneofDescriptor', 
'ServiceDescriptor', 'TypeTransformationError', '_Lock', '_NestedDescriptorBase', '_OptionsOrNone', 
'_ParseOptions', '_ToCamelCase', '_ToJsonName', '_USE_C_DESCRIPTORS', '__author__', '__builtins__', 
'__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_lock', 
'api_implementation', 'six', 'threading']

Now I can to define the structure for the data structured as messages
These message is a small logical record of information containing a series of name-value pairs called fields.
The protobuffers compiler (protoc) to generate the source code in the language you need (from the .proto file).
To generate a Python file, you need to execute:
protoc -I=$SRC_DIR --python_out=$DST_DIR $SRC_DIR/my_example.proto
Let's test with am example (you can use any proto file) in default folder (use . for default folder):

[mythcat@desk python]$ protoc -I=. --python_out=. my_example.proto
[mythcat@desk python]$ ls my_example*
my_example_pb2.py  my_example.proto
[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_example_pb2
>>> dir(my_example_pb2)
['AddressBook', 'DESCRIPTOR', 'Person', '_ADDRESSBOOK', '_PERSON', '_PERSON_PHONENUMBER', '_PERSON_PHONETYPE',
 '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__',
 '_b', '_descriptor', '_message', '_reflection', '_sym_db', '_symbol_database', 'sys']

The tensorflow python module - part 004.

If you using the tensorflow then you can get some warnings.
You can use warnings python package to manage all of this:

[mythcat@desk ~]$ $ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> import tensorflow as tf
/home/mythcat/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: 
Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be 
understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/mythcat/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: 
Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be 
understood as (type, (1,)) / '(1,)type'.
...

/home/mythcat/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: 
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of 
numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
>>>
>>> warnings.filterwarnings('ignore')
>>>
[7]+  Stopped                 python3

Use it in this way to filter these warnings:

[mythcat@desk ~]$ $ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> warnings.filterwarnings('ignore',category=FutureWarning)
>>> import tensorflow as tf
>>> 

Python 3.7.4 : Print with random colors.

This is a simple example for custom print output.
The script detect the platform for color settings and then use print.
The first print will print with blue color the name of the script.
I used random to select a random color from colors array and used to print the -=RANDOM COLOR=- text.
The print (W+'') is used to set default white color for terminal

import sys
import random
if sys.platform == "linux" or sys.platform == "linux2":
        BB = "\033[34;1m" # Blue light
        YY = "\033[33;1m" # Yellow light
        GG = "\033[32;1m" # Green light
        WW = "\033[0;1m"  # White light
        RR = "\033[31;1m" # Red light
        CC = "\033[36;1m" # Cyan light
        B = "\033[34m"    # Blue
        Y = "\033[33m"    # Yellow
        G = "\033[32m"    # Green
        W = "\033[0m"     # White
        R = "\033[31m"    # Red
        C = "\033[36m"    # Cyan
colors = [BB,YY,GG,WW,RR,CC,B,Y,G,W,R,C]
print (B+"\033[2;2m "+sys.argv[0]+"\n"+B)

color=random.choice(colors)
print (color+"-=RANDOM COLOR=-"+color)
print (W+'')

For winodws platform you need to add this:

elif sys.platform == "win32":

 BB = '' # Blue light
 YY = '' # Yellow light
 GG = '' # Green light
 WW = '' # White light
 RR = '' # Red light
 CC = '' # Cyan light
 B = ''  # Blue
 Y = ''  # Yellow
 G = ''  # Green
 W = ''  # White
 R = ''  # Red
 C = ''  # Cyan
 P = ''  # Random color

Python 3.7.4 : Using the theano pakage.

If you want to test theano then you need to see this webpage.

[root@desk mythcat]# dnf search theano
======================== Name & Summary Matched: theano ========================
python-theano-doc.noarch : Theano documentation
============================= Name Matched: theano =============================
python3-theano.noarch : Mathematical expressions involving multidimensional
                      : arrays
=========================== Summary Matched: theano ============================
python3-lasagne.noarch : Lightweight library to build and train neural networks
                       : in Theano
[root@desk mythcat]# pip3 install Theano --user
WARNING: Running pip install with root privileges is generally not a good idea. 

Try `pip3 install --user` instead.
Collecting Theano
...
  Running setup.py install for Theano ... done
Successfully installed Theano-1.0.4 scipy-1.3.1

Let's see first example:

[mythcat@desk ~]$ python3 
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
/home/mythcat/.local/lib/python3.7/site-packages/theano/configdefaults.py:560: 

UserWarning: DeprecationWarning: there is no c++ compiler.This is deprecated and with 

Theano 0.11 a c++ compiler will be mandatory
  warnings.warn("DeprecationWarning: there is no c++ compiler."
WARNING (theano.configdefaults): g++ not detected ! Theano will be unable to execute 

optimized C-implementations (for both CPU and GPU) and will default to Python implementations.

 Performance will be severely degraded. To remove this warning, set Theano flags cxx to an empty

 string.
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
>>> import theano
>>> import theano.tensor as T
>>> x = T.dmatrix('x')
>>> s = 1 / (1 + T.exp(-x))
>>> logistic = theano.function([x], s)
>>> logistic([[0, 1], [-1, -2]])
array([[0.5       , 0.73105858],
       [0.26894142, 0.11920292]])
>>> ... 

Python 3.7.4 : Using the sunpy - part 001.

I wrote about sunpy in the past on this website.
Now this package comes with new features, see the official webpage.
Let's install it.

[mythcat@desk ~]$ pip3 install sunpy --user
Collecting sunpy
...
Successfully installed aioftp-0.13.0 aiohttp-3.6.0 astropy-3.2.1 async-timeout-3.0.1 
multidict-4.5.2 parfive-1.0.0 scipy-1.3.1 sunpy-1.0.3 tqdm-4.35.0 yarl-1.3.0

If you search on web you can find example with spectra , but in the new package is not supported.

ModuleNotFoundError: No module named 'sunpy.spectra'

Let's test it:

[mythcat@desk ~]$ python3 
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sunpy
>>> dir(sunpy)
['SunPyTestRunner', 'UnsupportedPythonError', '__all__', '__builtins__', '__cached__', '__doc__',
 '__file__', '__loader__', '__minimum_python_version__', '__name__', '__package__', '__path__',
 '__spec__', '__version__', '_init_log', 'builtins', 'config', 'extern', 'load_config', 'log',
 'logging', 'os', 'print_config', 'self_test', 'sys', 'system_info', 'tests', 'util', 'version']
>>> sunpy.system_info()
==============================
SunPy Installation Information
==============================

#######
General
#######
Time : Monday, 09. September 2019 07:14PM UT
System : Linux
Processor : x86_64
Arch : 64bit
SunPy : 1.0.3
OS: Fedora 30 Thirty (Linux 5.2.11-200.fc30.x86_64 x86_64)


##################
Required Libraries
##################
Python: 3.7.4
NumPy: 1.16.4
SciPy: 1.3.1
matplotlib: 3.0.3
Astropy: 3.2.1
Pandas: 0.25.1
parfive: 1.0.0


#####################
Recommended Libraries
#####################
beautifulsoup: NOT INSTALLED
PyQt4: NOT INSTALLED
PyQt5: 5.12.2
Zeep: NOT INSTALLED
Sqlalchemy: 1.3.7
drms: NOT INSTALLED
>>> help(sunpy)

Help on package sunpy:

NAME
    sunpy

DESCRIPTION
    SunPy
    =====
    
    An open-source Python library for Solar Physics data analysis.
    
    Web Links
    ---------
    Homepage: https://sunpy.org
    Documentation: https://docs.sunpy.org/en/stable/

PACKAGE CONTENTS
    cm (package)
    compiler_version
    conftest
    coordinates (package)
    data (package)
    database (package)
    extern (package)
    image (package)
    instr (package)
    io (package)
    map (package)
    net (package)
    physics (package)
    roi (package)
    sun (package)
    tests (package)
    time (package)
    timeseries (package)
    util (package)
    version
    visualization (package)
... 

Let's install zeep, beautifulsoup4 and drms python packages:

[mythcat@desk ~]$ pip3 install zeep --user
Collecting zeep
...
Successfully installed appdirs-1.4.3 cached-property-1.5.1 isodate-0.6.0 lxml-4.4.1
 requests-toolbelt-0.9.1 zeep-3.4.0
[mythcat@desk ~]$ pip3 install --upgrade beautifulsoup4 --user
Collecting beautifulsoup4
...
Successfully installed beautifulsoup4-4.8.0 soupsieve-1.9.3
[mythcat@desk ~]$ pip3 install --upgrade drms --user
Collecting drms
...
Successfully installed drms-0.5.7

Now, we can see examples with this python package.
First, I will use the AIA_171_IMAGE image:

>>> from sunpy.data.sample import AIA_171_IMAGE 
Files Downloaded: 100%|███████████████████████| 26/26 [00:04<00:00 5.50file="" s="">>> import sunpy.map                                                            
>>> aiamap = sunpy.map.Map(AIA_171_IMAGE)
>>> aiamap.peek()                                                               
Attribute Qt::AA_EnableHighDpiScaling must be set before QCoreApplication is created.                                                                           

** (python3:4152): WARNING **: 22:53:15.329: AT-SPI: Could not obtain desktop path or name


** (python3:4152): WARNING **: 22:53:15.377: atk-bridge: GetRegisteredEvents returned message with unknown signature

Python 3.7.3 : Using the flask - part 018.

In this tutorial, I will show you how to fix auto increment in Flask SQLAlchemy.
The old source code for the user model from the server.py was this:

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    password = db.Column(db.String(120), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)

The server.sqlite will be this:

[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE user (
 id INTEGER NOT NULL, 
 username VARCHAR(80), 
 password VARCHAR(120), 
 email VARCHAR(120), 
 gender VARCHAR(5), 
 work VARCHAR(33), 
 city VARCHAR(15), 
 PRIMARY KEY (id), 
 UNIQUE (username), 
 UNIQUE (password), 
 UNIQUE (email), 
 UNIQUE (gender), 
 UNIQUE (work), 
 UNIQUE (city)
);
CREATE TABLE alembic_version (
 version_num VARCHAR(32) NOT NULL, 
 CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num)
);
CREATE TABLE sqlite_stat1(tbl,idx,stat);
sqlite> ^Z 

If you want to change the id into auto increment then you need to follow this steps:

class User(db.Model):
    __tablename__ = 'user'
    __table_args__ = {'sqlite_autoincrement': True}
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    password = db.Column(db.String(120), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)

Delete the server.sqlite file or rename it.
Open python3 and create a new server.sqlite file:

[mythcat@desk my_flask]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
>>> db.create_all()
>>> db.engine.table_names()
['sqlite_sequence', 'user']
>>> 
[5]+  Stopat                  python3 

Open the new file to see the changes:

[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE user (
 id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, 
 username VARCHAR(80), 
 password VARCHAR(120), 
 email VARCHAR(120), 
 gender VARCHAR(5), 
 work VARCHAR(33), 
 city VARCHAR(15), 
 UNIQUE (username), 
 UNIQUE (password), 
 UNIQUE (email), 
 UNIQUE (gender), 
 UNIQUE (work), 
 UNIQUE (city)
);
CREATE TABLE sqlite_sequence(name,seq);
sqlite>  

Python 3.7.4 : Create an Stand Alone Executable on Fedora distro.

In this tutorial I will show you how to create an Stand Alone Executable with Python version 3.7.4 and Fedora 30 distro.
First you need to install using the dnf tool the python3 package.
You can test it easy with this command:

[mythcat@desk dist]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information. 

I created a simple python script named test.py for testing:

import os
import sys
print("Hello!")

You need to install the pyinstaller python module.
Let's see this step with the output:

[mythcat@desk ~]$ pip3 install pyinstaller --user

Now you can use this python module to create Stand Alone Executable.

[mythcat@desk ~]$ pip3 install pyinstaller --user
Requirement already satisfied: pyinstaller in ./.local/lib/python3.7/site-packages (3.5)
Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from pyinstaller) (40.8.0)
Requirement already satisfied: altgraph in ./.local/lib/python3.7/site-packages (from pyinstaller) (0.16.1)
[mythcat@desk ~]$ pyinstaller --onefile test.py
561 INFO: PyInstaller: 3.5
561 INFO: Python: 3.7.4
571 INFO: Platform: Linux-5.2.9-200.fc30.x86_64-x86_64-with-fedora-30-Thirty
573 INFO: wrote /home/mythcat/test.spec
596 INFO: UPX is not available.
598 INFO: Extending PYTHONPATH with paths
['/home/mythcat', '/home/mythcat']
598 INFO: checking Analysis
624 INFO: Building because /home/mythcat/test.py changed
624 INFO: Initializing module dependency graph...
640 INFO: Initializing module graph hooks...
678 INFO: Analyzing base_library.zip ...
5247 INFO: running Analysis Analysis-00.toc
5352 INFO: Caching module hooks...
5376 INFO: Analyzing /home/mythcat/test.py
5396 INFO: Loading module hooks...
5397 INFO: Loading module hook "hook-xml.py"...
5775 INFO: Loading module hook "hook-pydoc.py"...
5800 INFO: Loading module hook "hook-encodings.py"...
5892 INFO: Looking for ctypes DLLs
5893 INFO: Analyzing run-time hooks ...
5901 INFO: Looking for dynamic libraries
6507 INFO: Looking for eggs
6507 INFO: Using Python library /lib64/libpython3.7m.so.1.0
6514 INFO: Warnings written to /home/mythcat/build/test/warn-test.txt
6547 INFO: Graph cross-reference written to /home/mythcat/build/test/xref-test.html
6604 INFO: checking PYZ
6607 INFO: Building because toc changed
6607 INFO: Building PYZ (ZlibArchive) /home/mythcat/build/test/PYZ-00.pyz
7030 INFO: Building PYZ (ZlibArchive) /home/mythcat/build/test/PYZ-00.pyz completed successfully.
7034 INFO: checking PKG
7035 INFO: Building because toc changed
7035 INFO: Building PKG (CArchive) PKG-00.pkg
10033 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
10036 INFO: Bootloader 
/home/mythcat/.local/lib/python3.7/site-packages/PyInstaller/bootloader/Linux-64bit/run
10036 INFO: checking EXE
10123 INFO: Building because toc changed
10124 INFO: Building EXE from EXE-00.toc
10163 INFO: Appending archive to ELF section in EXE /home/mythcat/dist/test
10337 INFO: Building EXE from EXE-00.toc completed successfully.

This command will create some folders: dist, build, ...
The dist folder will have the Stand Alone Executable named test:

[mythcat@desk ~]$ cd dist/
[mythcat@desk dist]$ ls
test
[mythcat@desk dist]$ ./test
Hello!

Python 3.7.3 : Using the flask - part 017.

Today I make some changes with my server.py and database and solve some issues, see old version at my old tutorial.
Firt issue was start script.
I create a linux script named start_server.sh to run the flask run command:

[mythcat@desk my_flask]$ ./start_server.sh 

I update the User with new fiels:

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    password = db.Column(db.String(120), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)

Let's see how I deal with this versus database and migrate process.

[mythcat@desk my_flask]$ rm server.sqlite 
[mythcat@desk my_flask]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
>>> db.create_all()
>>> db.engine.table_names()
['user']
>>> exit()
[mythcat@desk my_flask]$ ls server.sqlite 
server.sqlite
[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .tables
user
sqlite> .schema user
CREATE TABLE user (
        id INTEGER NOT NULL, 
        username VARCHAR(80), 
        password VARCHAR(120), 
        email VARCHAR(120), 
        gender VARCHAR(5), 
        work VARCHAR(33), 
        city VARCHAR(15), 
        PRIMARY KEY (id), 
        UNIQUE (username), 
        UNIQUE (password), 
        UNIQUE (email), 
        UNIQUE (gender), 
        UNIQUE (work), 
        UNIQUE (city)
);

I got a strange error:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: city

I fix it with this , but I'm not sure if the right way:

[mythcat@desk my_flask]$ rm migrations/ -r -f 
[mythcat@desk my_flask]$ python3 server.py db init 
  Creating directory /home/mythcat/project_github/my_flask/migrations ... done
  Creating directory /home/mythcat/project_github/my_flask/migrations/versions ... done
  Generating /home/mythcat/project_github/my_flask/migrations/script.py.mako ... done
  Generating /home/mythcat/project_github/my_flask/migrations/env.py ... done
  Generating /home/mythcat/project_github/my_flask/migrations/alembic.ini ... done
  Generating /home/mythcat/project_github/my_flask/migrations/README ... done
  Please edit configuration/connection/logging settings in '/home/mythcat/project_github/my_flask/migrations/alembic.ini'
  before proceeding.
[mythcat@desk my_flask]$ python3 server.py db migrate
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.env] No changes in schema detected.
[mythcat@desk my_flask]$ python3 server.py db upgrade
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.

Python 3.7.3 : Using the flask - part 016.

Today I tested a new feature of Flask version 1.1.1.

[mythcat@desk my_flask]$ pip list | grep Flask
Flask                    1.1.1            
Flask-Login              0.4.1            
Flask-Mail               0.9.1            
Flask-Migrate            2.5.2            
Flask-Script             2.0.6            
Flask-SQLAlchemy         2.4.0            
Flask-WTF                0.14.2 

This feature will remove the jsonify python module.
Let's start the blueprint blue_test:

[mythcat@desk my_flask]$ export FLASK_APP=blue_test
[mythcat@desk my_flask]$ flask run 
 * Serving Flask app "blue_test"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.

The output is this:

 {"result":"This is main page!"}

Into my blue_test folder I will remove this jsonify python module:
[

mythcat@desk my_flask]$ cd blue_test/
[mythcat@desk blue_test]$ ll
total 40
-rw-rw-r--. 1 mythcat mythcat  192 Aug 23 11:02 extensions.py
-rw-rw-r--. 1 mythcat mythcat 1826 Aug 23 11:02 forms.py
-rw-rw-r--. 1 mythcat mythcat  701 Aug 23 12:03 __init__.py
-rw-rw-r--. 1 mythcat mythcat 1491 Aug 23 11:02 models.py
drwxrwxr-x. 2 mythcat mythcat  184 Aug 25 14:36 __pycache__
-rw-rw-r--. 1 mythcat mythcat 1050 Aug 23 11:02 routes.py
-rw-rw-r--. 1 mythcat mythcat  142 Aug 23 11:02 settings.py
-rw-rw-r--. 1 mythcat mythcat   34 Aug 23 11:02 start.bat
drwxrwxr-x. 2 mythcat mythcat   81 Aug 23 11:02 templates
-rw-rw-r--. 1 mythcat mythcat 8192 Aug 23 11:02 texts.sqlite
-rw-rw-r--. 1 mythcat mythcat  677 Aug 23 11:02 views.py 

The routes.py will have these changes:

#from flask import Blueprint, jsonify, request
# new flask
from flask import Blueprint, request
...
def home():
    #return jsonify({'result' : 'This is main page!'})
    #this is the new source code:
    return {'result' : 'This is main page!'} 

The next step is to find all jsonify word into my source code using the grep tool and make changes:

[mythcat@desk my_flask]$ grep -nr jsonify *
Binary file blue_test/__pycache__/views.cpython-37.pyc matches
blue_test/views.py:3:from flask import Blueprint, jsonify, request
blue_test/views.py:20:    return jsonify ({'texts':texts})
blue_test/routes.py:1:#from flask import Blueprint, jsonify, request
crud.py:1:from flask import Flask, request, jsonify
crud.py:57:    return jsonify(new_user)
crud.py:64:    return jsonify(result.data)
crud.py:70:    return user_schema.jsonify(user)
crud.py:83:    return user_schema.jsonify(user)
crud.py:92:    return user_schema.jsonify(user)
Binary file __pycache__/crud.cpython-37.pyc matches
Binary file __pycache__/server.cpython-37.pyc matches
server.py:9:from flask import jsonify
server.py:101:    #return jsonify(new_user)
server.py:109:    #return users_schema.jsonify(users)
server.py:111:    return jsonify(all_users.data)
server.py:117:    return jsonify(result.data)
server.py:122:        return jsonify([d['keyword'] for d in result.data])
server.py:132:    return user_schema.jsonify(new_user)
tserv.py:13:from flask import jsonify
tserv.py:56:    #return users_schema.jsonify(users)
tserv.py:58:    return jsonify(all_users.data)
tserv.py:69:    return user_schema.jsonify(user_post)
tserv.py:80:    return user_schema.jsonify(user_put)
tserv.py:88:    return user_schema.jsonify(user_delete)
[mythcat@desk my_flask]$  

After I make changes the result of grep is this:

[mythcat@desk my_flask]$ grep -nr jsonify *
Binary file blue_test/__pycache__/views.cpython-37.pyc matches
blue_test/routes.py:1:#from flask import Blueprint, jsonify, request
blue_test/views.py:3:#from flask import Blueprint, jsonify, request
crud.py:1:#from flask import Flask, request, jsonify
Binary file __pycache__/crud.cpython-37.pyc matches
Binary file __pycache__/server.cpython-37.pyc matches
server.py:9:#from flask import jsonify
server.py:109:    #return users_schema.jsonify(users)
tserv.py:13:#from flask import jsonify
tserv.py:56:    #return users_schema.jsonify(users) 

You can see the last version of my project here.

Python 3.7.3 : Using the flask - part 015.

In this tutorial, I will show you how to migrate using the Database Migrations in flask project.
Because my laptop is gone I use my old Linux.
First you need to install these python modules with --user argument for Linux:

[mythcat@desk my_flask]$ pip3 install flask-migrate --user
...
[mythcat@desk my_flask]$ pip3 install flask-script --user

Let's test this new issue with server.py file by adding these python modules:

#migrate 
from flask_script import Manager
from flask_migrate import Migrate, MigrateCommand
...
# create migrate object with db 
migrate = Migrate(app, db)
# create manager 
manager = Manager(app)
# create db command for manager 
manager.add_command('db', MigrateCommand)
...
# add new columns into database 
class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    email = db.Column(db.String(120), unique=True)
    gender = db.Column(db.String(5), unique=True)
    work = db.Column(db.String(33), unique=True)
    city = db.Column(db.String(15), unique=True)
...
# the default name main
if __name__ == '__main__':
    manager.run()
    app.run(debug=True)

Let's fix this migrate issue with the new command:

[mythcat@desk my_flask]$ python3 server.py db init 
  Creating directory /home/mythcat/project_github/my_flask/migrations ... done
  Creating directory /home/mythcat/project_github/my_flask/migrations/versions ... done
  Generating /home/mythcat/project_github/my_flask/migrations/script.py.mako ... done
  Generating /home/mythcat/project_github/my_flask/migrations/env.py ... done
  Generating /home/mythcat/project_github/my_flask/migrations/alembic.ini ... done
  Generating /home/mythcat/project_github/my_flask/migrations/README ... done
  Please edit configuration/connection/logging settings in '/home/mythcat/project_github/my_flask/migrations
/alembic.ini'
  before proceeding.

[mythcat@desk my_flask]$ python3 server.py db migrate
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.autogenerate.compare] Detected added column 'user.city'
INFO  [alembic.autogenerate.compare] Detected added column 'user.gender'
INFO  [alembic.autogenerate.compare] Detected added column 'user.work'
INFO  [alembic.autogenerate.compare] Detected added unique constraint 'None' on '['city']'
INFO  [alembic.autogenerate.compare] Detected added unique constraint 'None' on '['gender']'
INFO  [alembic.autogenerate.compare] Detected added unique constraint 'None' on '['work']'
  Generating /home/mythcat/project_github/my_flask/migrations/versions/ca70c42b5b7a_.py ... done

[mythcat@desk my_flask]$ python3 server.py db upgrade 
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> ca70c42b5b7a, empty message
ERROR [root] Error: No support for ALTER of constraints in SQLite dialect

The database fiels is changed by this command.
Let's see with sqlite3 tool:

[mythcat@desk my_flask]$ sqlite3 server.sqlite 
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .tables
alembic_version  user           
sqlite> .schema user
CREATE TABLE user (
        id INTEGER NOT NULL, 
        username VARCHAR(80), 
        email VARCHAR(120), city VARCHAR(15), gender VARCHAR(5), work VARCHAR(33), 
        PRIMARY KEY (id), 
        UNIQUE (username), 
        UNIQUE (email)
);

You can see my source code here.