analitics

Pages

Sunday, December 27, 2020

Python 3.9.1 : Testing the drawSvg python package.

The tutorial for today is about drawSvg python package.
This python package let you to create SVG images (vector drawings) and rendering them or displaying them in a Jupyter notebook.
Let's install it with pip3 tool:
[mythcat@desk ~]$ cd PythonProjects/
[mythcat@desk PythonProjects]$ pip3 install drawSvg
...
Successfully installed drawSvg-1.7.0 imageio-2.9.0
A simple source code test added to svg_001.py python script:
import drawSvg as draw

# Draw a frame of the animation
def draw_frame(t):
    #Create draw surface and add a geometric shapes ...
    out = draw.Drawing(1, 1, origin=(0,0))
    out.setRenderSize(h=460)
    out.append(draw.Rectangle(0, 0, 5, 5, fill='white'))
    y = t + 0.3
    out.append(draw.Circle(0.5, y, 0.5, fill='blue'))
    return out

with draw.animate_video('example.gif', draw_frame, duration=0.01) as anim:
    # Add each frame to the animation
    for i in range(20):
        anim.draw_frame(i/10)
You can change the source code and open the GIF animation file:
[mythcat@desk PythonProjects]$ vi svg_001.py
[mythcat@desk PythonProjects]$ python svg_001.py 
[mythcat@desk PythonProjects]$ google-chrome example.gif
The result of the source code is this:

Monday, December 21, 2020

Python 3.6.9 : Ursina python game engine - part 001 .

I wrote the tutorial a few days ago and today I completed it with a video clip ...
The official webpage comes with this intro:
Ursina makes it easier to develop games, visualizations and other kinds of software.
The concise API combined with the power of the Python programming language, makes life easier for the developer so they can focus on what they are making.

Ursina can be used on Windows and Linux
Today I tested on Fedora Linux and works great.
First, the install process is easy with pip3 tool.
[mythcat@desk ~]$ pip3 install ursina
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: ursina in ./.local/lib/python3.9/site-packages (3.2.2)
Requirement already satisfied: screeninfo in ./.local/lib/python3.9/site-packages (from ursina) (0.6.6)
Requirement already satisfied: numpy in /usr/lib64/python3.9/site-packages (from ursina) (1.19.4)
Requirement already satisfied: panda3d in ./.local/lib/python3.9/site-packages (from ursina) (1.10.7)
Requirement already satisfied: pyperclip in ./.local/lib/python3.9/site-packages (from ursina) (1.8.1)
Requirement already satisfied: pillow in /usr/lib64/python3.9/site-packages (from ursina) (7.2.0)
Let's create some examples and see how this python package works.
Create a python script file:
[mythcat@desk ~]$ cd PythonProjects/
[mythcat@desk PythonProjects]$ vi ursina_001.py
Add this simple example:
from ursina import *
# create the application
app = Ursina()
# set some data , like a cube
ground = Entity(
    model = 'cube',
    color = color.blue,
    z = -.1,
    y = -3,
    origin = (0, .5),
    scale = (50, 1, 10),
    collider = 'box'
    )
# run the application
app.run()
And the last pas, run it with python:
[mythcat@desk PythonProjects]$ python ursina_001.py 
ursina version: 3.2.2
package_folder: /home/mythcat/.local/lib/python3.9/site-packages/ursina
asset_folder: .
psd-tools not installed
which: no blender in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:
/home/mythcat/.composer/vendor/bin:/home/mythcat/.dotnet/tools:/var/lib/snapd/snap/bin)
blender_paths:
{}
OS: posix
screen resolution: (1280, 1024)
Known pipe types:
  glxGraphicsPipe
(all display modules loaded.)
render mode: default
no settings.py file
no settings.py file
development mode: True
application successfully started
You can read on web!:

Sunday, December 6, 2020

Python 3.6.9 : My colab tutorials - part 010.

In this tutorial created with Colab online tool I used HTML and JavaScript source code.
The tutorial is easy to understand and use it.
You can see all my examples of my GitHub repo.

Thursday, December 3, 2020

Python 3.6.9 : My colab tutorials - part 009.

I update may colab work and I add new notebooks.
You can see all of these on my GitHub account.
These are examples:
  • catafest_009.ipynb - show you how to use %% colab features;
  • catafest_010.ipynb - example with Detectron2 is Facebook AI Research's with state-of-the-art object detection algorithms;
  • catafest_011.ipynb - test a sound classification with YAMNet from a web example - not very happy with the result;

Sunday, November 22, 2020

Python 3.9.0 : Physics simulation with PyBullet .

I took a look at the documentation of this python packet to give you a brief introduction:
PyBullet is a fast and easy to use Python module for robotics simulation and machine learning, with a focus on sim-to-real transfer. With PyBullet you can load articulated bodies from URDF, SDF, MJCF and other file formats. PyBullet provides forward dynamics simulation, inverse dynamics computation, forward and inverse kinematics, collision detection and ray intersection queries. The Bullet Physics SDK includes PyBullet robotic examples such as a simulated Minitaur quadruped, humanoids running using TensorFlow inference and KUKA arms grasping objects.
...
PyBullet can be easily used with TensorFlow and OpenAI Gym.
...
The GUI connection will create a new graphical user interface (GUI) with 3D OpenGL rendering, within the same process space as PyBullet. On Linux and Windows this GUI runs in a separate thread, while on OSX it runs in the same thread due to operating system limitations.
...
For UDP networking, there is a App_PhysicsServerUDP that listens to a certain UDP port.
...
By default, there is no gravitational force enabled. setGravity lets you set the default gravity force for all objects. The loadURDF will send a command to the physics server to load a physics model from a Universal Robot Description File (URDF). The URDF file is used by the ROS project (Robot Operating System) to describe robots and other objects, it was created by the WillowGarage and the Open Source Robotics Foundation (OSRF).

I tested with Fedora 33 and Python version 3.9.0.
The first step was installation.
[mythcat@desk ~]$ pip3 install pybullet --upgrade --user
Collecting pybullet
  Downloading pybullet-3.0.6.tar.gz (89.8 MB)
     |████████████████████████████████| 89.8 MB 51 kB/s 
Using legacy 'setup.py install' for pybullet, since package 'wheel' is not installed.
Installing collected packages: pybullet
    Running setup.py install for pybullet ... done
Successfully installed pybullet-3.0.6
The next step was to test with examples already built using urdf file type.
import pybullet as p
# Can alternatively pass in p.DIRECT 
client = p.connect(p.GUI)
p.setGravity(0, 0, -10, physicsClientId=client) 

import pybullet_data
p.setAdditionalSearchPath(pybullet_data.getDataPath())
planeId = p.loadURDF("plane.urdf")

my_racecar = p.loadURDF("racecar/racecar.urdf", basePosition=[0,0,0.2])

position, orientation = p.getBasePositionAndOrientation(my_racecar)

for _ in range(10000): 
    p.stepSimulation()
I run the script and all works well.
It can be seen that in the cycle for which I have the simulation for 10000 steps of time.
A smaller number will quickly interrupt the simulation.
You can create your own URDF files because they are formatted in XML format.
A good area for learning is this webpage.

Saturday, October 31, 2020

Python 3.9.0 : Testing twisted python module - part 001 .

Today I tested two python modules named: twisted and twisted[tls].
Twisted is an event-driven network programming framework written in Python and licensed under the MIT License. Twisted projects variously support TCP, UDP, SSL/TLS, IP multicast, Unix domain sockets, many protocols (including HTTP, XMPP, NNTP, IMAP, SSH, IRC, FTP, and others), and much more. Twisted is based on the event-driven programming paradigm, which means that users of Twisted write short callbacks which are called by the framework., see wikipedia webpage.
In this tutorial I will show you only some of these tests and how you can work with these python modules.
About twisted you can read more at the official webpage. In Fedora distro version 33 you can use the dnf tool to search for and install these python packages.
[root@desk mythcat]# dnf search twisted
...
python3-twisted.x86_64 : Twisted is a networking engine written in Python
python3-twisted+tls.x86_64 : Metapackage for python3-twisted: tls extras
You can also use the pip tool for installation:
[mythcat@desk ~]$ cd PythonProjects/
[mythcat@desk PythonProjects]$ pip3 install twisted
...
[mythcat@desk PythonProjects]$ pip3 install twisted[tls]
...
I used python 3.9.0 to test if this python package works:
[mythcat@desk PythonProjects]$ python3.9
Python 3.9.0 (default, Oct  6 2020, 00:00:00) 
[GCC 10.2.1 20200826 (Red Hat 10.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twisted.protocols import basic
Let's test with simple example using the reactor and protocol:
from twisted.internet import reactor, protocol

class ClientEcho(protocol.Protocol):
    def connectionMade(self):
        self.transport.write("Hello, world!".encode('utf-8'))

    def dataReceived(self, data):
        print ("Server: ", data)
        self.transport.loseConnection()

class FactoryEcho(protocol.ClientFactory):
    def buildProtocol(self, addr):
        return ClientEcho()

    def clientConnectionFailed(self, connector, reason):
        print ("Connection failed")
        reactor.stop()

    def clientConnectionLost(self, connector, reason):
        print ("Connection lost")
        reactor.stop()

reactor.connectTCP("localhost", 8080, FactoryEcho())
reactor.run()
Your protocol handling class will usually subclass twisted.internet.protocol.Protocol.
The default factory class twisted.internet.protocol.Factory just instantiates each Protocol and lets every Protocol access, and possibly modify, the persistent configuration.
This protocol responds to the initial connection with a well known quote, and then terminates the connection.
The protocol never waits for an event because handles data in an asynchronous manner.
The reactor interface lets many different loops handle the networking code.
The source code have two classes each is used to show a simple echo client on port 8080 - you can use any port.
This source code is the most simple example to understand the relation between factory , protocol and reactor.
The result is this:
[mythcat@desk PythonProjects]$ python3.9 echo_client_001.py 
Server:  b'Hello, world!'
Connection lost

Saturday, October 10, 2020

Python 3.9.0 : Union and in-place union operators

Python introduces two new operators for dictionaries named union used in code with pipe operator | and in-place union used in python code with this |=.

I this tutorial I will show you how can be used:
[mythcat@desk ~]$ python3.9 
Python 3.9.0 (default, Oct  6 2020, 00:00:00) 
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> step_one = {"worker_one":"task_one", "worker_two":"task_two"}
>>> step_two = {"worker_three":"task_one", "worker_fouw":"task_two"}
>>> merged_step_one_step_two = {**step_one, **step_two}
>>> empty = {}
>>> empty |= step_one
>>> empty
{'worker_one': 'task_one', 'worker_two': 'task_two'}
>>> step_one 
{'worker_one': 'task_one', 'worker_two': 'task_two'}
>>> step_two
{'worker_three': 'task_one', 'worker_fouw': 'task_two'}
>>> all = step_one | step_two 
>>> all 
{'worker_one': 'task_one', 'worker_two': 'task_two', 'worker_three': 'task_one', 'worker_fouw': 'task_two'} 
>>> copy_step_one = empty | step_one
>>> copy_step_one 
{'worker_one': 'task_one', 'worker_two': 'task_two'} 
>>> copy_step_one |= [("manager", "steps")]
>>> copy_step_one 
{'worker_one': 'task_one', 'worker_two': 'task_two', 'manager': 'steps'}
You can easily see how to use these operators with the dictionaries created and how to add lists to dictionaries. Operations with these operators always result in a dictionary. The order of these operations is important, see example:
>>> numbers = {0: "zero", 1: "one"}
>>> numbers_one = {0: "zero", 1: "one"}
>>> numbers_two = {0: "zero", 2: "two"}
>>> numbers_one | numbers_two
{0: 'zero', 1: 'one', 2: 'two'}
>>> numbers_two | numbers_one
{0: 'zero', 2: 'two', 1: 'one'}
>>> numbers_three = {0: 'zero', 11: 'error 1', 22: 'error 2', 33:'error 3'}
>>> numbers_three | numbers_one
{0: 'zero', 11: 'error 1', 22: 'error 2', 33: 'error 3', 1: 'one'}
>>> numbers_one | numbers_three
{0: 'zero', 1: 'one', 11: 'error 1', 22: 'error 2', 33: 'error 3'}
You can see how the dictionaries are overwritten by the union operation. These operations are implemented through dunder operations for most the dictionary objects except abstract classes. Dunder or magic methods in Python are the methods having two prefix and suffix underscores in the method name. You can find on this page.

Python 3.9.0 : Introduction to release 3.9.0.

This is a short introduction to release 3.9.0. 
Five days ago, a new release of version 3.9 appeared with a series of improvements and new python packages, see the official website
You can install easily on Fedora 32 with dnf tool.
[root@desk mythcat]# dnf install  python39.x86_64
...
Installing:
 python39                x86_64                3.9.0-1.fc32
...
Installed:
  python39-3.9.0-1.fc32.x86_64                                                                             

Complete!
A new article written about this release shows some of the advantages of this programming language, you can read it here
You can run easily with this command;
[root@desk mythcat]# python3.9
Python 3.9.0 (default, Oct  6 2020, 00:00:00) 
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information. 
You can configure python with this command, see an example:
[root@desk mythcat]# python3.9-x86_64-config --libs 
 -lcrypt -lpthread -ldl  -lutil -lm -lm 
This release of Python 3.9 uses a new parser, based on parsing expression grammar (PEG) instead of LL(1) know as LL parser (Left-to-right, Leftmost derivation). PEG parsers are more powerful than LL(1) parsers and avoid the need for special hacks. This the same abstract syntax tree (AST) as the old LL(1) parser. The PEG parser is the default, but you can run your program using the old parser, see the next example:
[mythcat@desk ~]$ python3.9 -X oldparser my_script_name.py
...
One of the most important improvements for me is PEP 614 -- Relaxing Grammar Restrictions On Decorators. In Python 3.9, these restrictions are lifted and you can now use any expression. 
For example including one accessing items in a dictionary, like this simple example:

buttons = {
  "hello": QPushButton("hello word!"),
  "bye": QPushButton("bye word!"),
  "buy": QPushButton("buy!!"),
}
...
@buttons["hello"].clicked.connect
def speak_hello():
... 
Yes, comes with another Python Enhancement Proposals - PEP 615 with support for the IANA Time Zone Database in the Standard Library. This acronym is similar to I.A.N.A known as the Internet Assigned Numbers Authority, but it is not the same. 
In this case, the zoneinfo module provides a concrete time zone implementation to support the IANA time zone database. 
This support contains code and data that represent the history of local time for many representative locations around the globe. Many features for math, strings, union operator for the dictionary, HTTP codes, and more. 
Completion of variable and module names is automatically enabled at interpreter startup so that the Tab key invokes the completion function. 
You can see a simple example with zoneinfo python module and completion feature.
[mythcat@desk ~]$ python3.9
Python 3.9.0 (default, Oct  6 2020, 00:00:00) 
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from zoneinfo import ZoneInfo
>>> import zoneinfo
>>> zoneinfo.available_timezones()
{'Hongkong', 'America/Iqaluit', 'America/Indianapolis', 'America/Louisville', 'America/New_York', 
'America/Mazatlan', 'Australia/Yancowinna', 'Africa/Ndjamena', 'Portugal', 'Africa/Bujumbura', 
'America/Rosario', 'America/Antigua','America/Indiana/Tell_City', 'America/Managua', 
'Europe/Paris', 'Europe/Oslo', 
...
>>> tz = ZoneInfo("Europe/Bucharest")
>>> tz.
tz.clear_cache(  tz.from_file(    tz.key           tz.tzname(       
tz.dst(          tz.fromutc(      tz.no_cache(     tz.utcoffset(    
>>> tz. 
On my desktop I got this result:
[mythcat@desk ~]$ python var_access_benchmark.py 
Variable and attribute read access:
   6.0 ns	read_local
   6.5 ns	read_nonlocal
  10.7 ns	read_global
  10.7 ns	read_builtin
  25.2 ns	read_classvar_from_class
  23.4 ns	read_classvar_from_instance
  34.3 ns	read_instancevar
  29.4 ns	read_instancevar_slots
  26.8 ns	read_namedtuple
  42.4 ns	read_boundmethod

Variable and attribute write access:
   6.6 ns	write_local
   7.0 ns	write_nonlocal
  23.3 ns	write_global
  54.1 ns	write_classvar
  46.4 ns	write_instancevar
  39.4 ns	write_instancevar_slots

Data structure read access:
  26.1 ns	read_list
  28.1 ns	read_deque
  27.5 ns	read_dict
  25.8 ns	read_strdict

Data structure write access:
  29.6 ns	write_list
  31.7 ns	write_deque
  34.4 ns	write_dict
  31.7 ns	write_strdict

Stack (or queue) operations:
  55.5 ns	list_append_pop
  49.7 ns	deque_append_pop
  51.0 ns	deque_append_popleft

Timing loop overhead:
   0.4 ns	loop_overhead

Tuesday, September 29, 2020

Python Qt5 : Use QStandardItem with Images.

This tutorial show you how to use QStandardItem with Images.

The source code is simple to understand.

import sys
from PyQt5.QtWidgets import QApplication, QMainWindow, QTreeView
from PyQt5.QtCore import  Qt
from PyQt5.QtGui import QFont, QColor, QImage, QStandardItemModel, QStandardItem

class ItemImage(QStandardItem):
    def __init__(self, txt='', image_path='', set_bold=False, color=QColor(0, 0, 0)):
        super().__init__()

        self.setEditable(False)
        self.setForeground(color)
        self.setText(txt)

        if image_path:
            image = QImage(image_path)
            self.setData(image, Qt.DecorationRole)

class MyApp(QMainWindow):
    def __init__(self):
        super().__init__()
        self.resize(1200, 1200)

        treeView = QTreeView()
        treeView.setHeaderHidden(True)
        treeView.header().setStretchLastSection(True)

        treeModel = QStandardItemModel()
        rootNode = treeModel.invisibleRootItem()

        robots = ItemImage('Robots', '', set_bold=True)

        aaa = ItemImage('aaa.jpg', 'aaa.jpg', 14)
        robots.appendRow(aaa)

        bbb = ItemImage('bbb.jpg', 'bbb.jpg', 16)
        robots.appendRow(bbb)

        robots2 = ItemImage('Robots 2', '', set_bold=False)

        aaa = ItemImage('ccc.png', 'ccc.png', 14)
        robots2.appendRow(aaa)

        bbb = ItemImage('ddd.jpg', 'ddd.jpg', 16)
        robots2.appendRow(bbb)

        rootNode.appendRow(robots)
        rootNode.appendRow(robots2)
        treeView.setModel(treeModel)
        treeView.expandAll()

        self.setCentralWidget(treeView)

app = QApplication(sys.argv)        
demo = MyApp()
demo.show()
sys.exit(app.exec_())

The result is this:

Monday, September 21, 2020

Python 3.8.5 : A sphere in Cartesian coordinates - part 001.

I like the equation of a sphere of radius R centered at the origin is given in Cartesian coordinates:

x*x + y*y + z*z = r*r

It is one of the first elements that helped me better understand mathematics and later the dynamics and theory of electromagnetic fields.

I did not find a graphical representation using python as accurately as possible without eliminating the discretion of the range from -1 and 1 and radius * radius = 1.

The main reason is the plot_surface from matplotlib python package.

This is output of my script:

[mythcat@desk ~]$ python sphere_xyz.py 
[-1.         -0.91666667 -0.83333333 -0.75       -0.66666667 -0.58333333
 -0.5        -0.41666667 -0.33333333 -0.25       -0.16666667 -0.08333333
  0.          0.08333333  0.16666667  0.25        0.33333333  0.41666667
  0.5         0.58333333  0.66666667  0.75        0.83333333  0.91666667
  1.        ]
sphere_xyz.py:7: RuntimeWarning: invalid value encountered in sqrt
  return np.sqrt(1-x**2 - y**2)
sphere_xyz.py:18: UserWarning: Z contains NaN values. This may result in rendering artifacts.
  surface1 = ax.plot_surface(X2, Y2, -Z2,rstride=1, cstride=1, linewidth=0,antialiased=True)
sphere_xyz.py:19: UserWarning: Z contains NaN values. This may result in rendering artifacts.
  surface2 = ax.plot_surface(X2, Y2, Z2,rstride=1, cstride=1, linewidth=0,antialiased=True)

The image result is this:

The source code is this:

import matplotlib.pyplot as plt
import numpy as np
from matplotlib.colors import LightSource

#@np.vectorize
def solve_Z2(x,y):
    return np.sqrt(1-x**2 - y**2)

fig = plt.figure()
ax = fig.gca(projection='3d')

xrange2 = np.linspace(-1.0, 1.0, 25)
yrange2 = np.linspace(-1.0, 1.0, 25)
print(xrange2)
X2, Y2 = np.meshgrid(xrange2, yrange2)
Z2 = solve_Z2(X2, Y2)

surface1 = ax.plot_surface(X2, Y2, -Z2,rstride=1, cstride=1, linewidth=0,antialiased=True)
surface2 = ax.plot_surface(X2, Y2, Z2,rstride=1, cstride=1, linewidth=0,antialiased=True)

plt.show()

Saturday, September 19, 2020

Python 3.8.5 : Linked List - part 001.

In computer science, a linked list is a linear collection of data elements whose order is not given by their physical placement in memory. see wikipedia.org.

In this tutorial I will show you how these linked list works and how can build with python programming language.
Let's start with some basic elements:
  • Linked List is a linear data structure;
  • Linked List are not array;
  • Linked List are not stored at a contiguous location because use pointes;
  • Each element use a linked pointe;
  • Linked List has a dynamic size;
  • Linked List use operations like insertion and deletion for each element;
  • The access to elements is sequentially;
  • Using reference to pointer foe each element needs extra memory space and is not cache friendly;

The Linked List consists of at least two parts: data and pointer to the next node.
The first element of the list is called the head.

The last node has a reference to null.
The each element named node of the list has a similar structure like Linked List: data node and pointer to the next node.
The data from Linked list is represent like a sequence.

A number of operations are required to use the Linked List.

The basic operations supported by a list can be:

  • insertion - adds an element at the beginning of the list;
  • deletion - deletes an element at the beginning of the list;
  • display - displays the complete list;
  • search - searches an element using the given key;
  • delete - deletes an element using the given key.

The Advantages of Linked List elements can be easily inserted or removed without reallocation or reorganization of the entire structure because the data items need not be stored contiguously in memory or on disk using operations.

The Linked List structure allows several ways to link nodes resulting in a number of types of Linked List:
  • Simple linked list;
  • Doubly linked list;
  • Multiply linked list;
  • Circular linked list;
  • Sentinel nodes;
  • Empty lists;
  • Hash linking;
  • List handles;
  • Combining alternatives;

Let's start with first source code for the first type of Linked List named Simple linked list.

Python 3.8.5 (default, Aug 12 2020, 00:00:00) 
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> # create a node class 
>>> class Node:
...     # first initialise of the node object
...     def __init__(self, data):
...             # assign the node data 
...             self.data = data
...             # initialize next element as null
...             self.next = None
... 
>>> # create a simple linked list
>>> class Simple_LinkedList:
...     # first initialize of element named head
...     def __init__(self):
...             self.head = None
>>> # use the list 
>>> if __name__=='__main__': 
...     simple_list = Simple_LinkedList()
...     simple_list.head = Node(1)
...     second = Node(2)
...     third = Node(3)
...     #link first node with the second
...     simple_list.head.next = second
...     #link second  node with the third
...     second.next = third
...     # now the list is linked as a simple Linked List

This source code don't include any basic operations.

If you put this source code into python script and run it you will see this:

[mythcat@desk ~]$ python simple_LinkedList.py 
<__main__.Node object at 0x7f35163e1ac0>
<__main__.Node object at 0x7f351638b0a0>
<__main__.Node object at 0x7f351638b130>

I create a basic operation for this simple Linked List to display the content of this list:

# create a simple linked list with a basic diplay operation
class Simple_LinkedList:
    # first initialize of element named head
    def __init__(self):
            self.head = None
    # use the basic operation for display
    def display_Simple_LinkedList(self): 
        points_to_node = self.head
	#will traverse the list to the last node. 
        while (points_to_node): 
            # print data linked to points_to_node
            print (points_to_node.data) 
            # print next node linked to points_to_node
            points_to_node = points_to_node.next

I run with this new python source code and result is this:

[mythcat@desk ~]$ python simple_LinkedList_basic_display.py 
1
2
3 

Thursday, September 10, 2020

Python 3.8.5 : Get Sentinel-3 satellite data from Eutelsat.

The tutorial for today is about Eutelsat satellites.
I used Sentinel-3 with these features:
  • Instrument: SLSTR;
  • Mode: EO;
  • Satellite: Sentinel-3

You need to install xarray and netcdf4 python packages:
[mythcat@desk TLauncher]$ pip install xarray
...
[mythcat@desk ~]$ pip install netcdf4

Defaulting to user installation because normal site-packages is not writeable
Collecting netcdf4
  Downloading netCDF4-1.5.4-cp38-cp38-manylinux1_x86_64.whl (4.3 MB)
     |████████████████████████████████| 4.3 MB 649 kB/s 
Collecting cftime
  Downloading cftime-1.2.1-cp38-cp38-manylinux1_x86_64.whl (271 kB)
     |████████████████████████████████| 271 kB 30.5 MB/s 
Requirement already satisfied: numpy>=1.9 in /usr/lib64/python3.8/site-packages (from netcdf4) (1.18.4)
Installing collected packages: cftime, netcdf4
Successfully installed cftime-1.2.1 netcdf4-1.5.4 
You need to get data from the official webpage.

After I download the nc file with all data I used this source code:
Python 3.8.5 (default, Aug 12 2020, 00:00:00) 
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xarray as xr
>>> dir = '/home/mythcat/Downloads/'
>>> file_xr = xr.open_dataset(dir+'FRP_in.nc')
>>> file_xr
<xarray .dataset="">
Dimensions:                 (columns: 1500, fires: 23, rows: 2000)
Dimensions without coordinates: columns, fires, rows
Data variables:
    i                       (fires) int16 ...
    j                       (fires) int32 ...
    time                    (fires) datetime64[ns] ...
    latitude                (fires) float64 ...
    longitude               (fires) float64 ...
    FRP_MWIR                (fires) float64 ...
    FRP_uncertainty_MWIR    (fires) float64 ...
    transmittance_MWIR      (fires) float64 ...
    FRP_SWIR                (fires) float64 ...
    FRP_uncertainty_SWIR    (fires) float64 ...
    FLAG_SWIR_SAA           (fires) int16 ...
    transmittance_SWIR      (fires) float64 ...
    confidence              (fires) float64 ...
    classification          (fires) uint8 ...
    S7_Fire_pixel_radiance  (fires) float32 ...
    F1_Fire_pixel_radiance  (fires) float32 ...
    used_channel            (fires) uint8 ...
    Radiance_window         (fires) float32 ...
    Glint_angle             (fires) float64 ...
    IFOV_area               (fires) float64 ...
    TCWV                    (fires) float64 ...
    n_window                (fires) int16 ...
    n_water                 (fires) int16 ...
    n_cloud                 (fires) int16 ...
    n_SWIR_fire             (fires) float32 ...
    flags                   (rows, columns) uint32 ...
Attributes:
    title:                  SLSTR Level 2 Product, Fire Radiative Power measu...
    comment:                 
    netCDF_version:         4.2 of Jul  5 2012 17:07:43 $
    product_name:           S3B_SL_2_FRP____20200910T082906_20200910T083406_2...
    institution:            MAR
    source:                 IPF-SL-2-FRP 02.00
    history:                 
    references:             S3MPC ACR FRP 003 - i1r2 - SLSTR L2 Product Data ...
    contact:                ops@eumetsat.int
    creation_time:          2020-09-10T10:38:54Z
    resolution:             [ 1000 1000 ]
    absolute_orbit_number:  12385
    start_time:             2020-09-10T08:29:06.288252Z
    stop_time:              2020-09-10T08:34:06.277355Z
    track_offset:           998
    start_offset:           14032
Let's parse size by latitude and longitude:
>>> lat = file_xr['latitude']
>>> long = file_xr['longitude']
>>> lat , long 
(<xarray.DataArray 'latitude' (fires: 23)>
array([52.992509, 52.98967 , 48.447335, 44.00415 , 48.443263, 48.439204,
       48.438141, 48.434069, 48.430012, 48.430012, 43.99905 , 43.993966,
       43.996166, 43.991064, 43.991064, 43.985972, 43.986568, 43.981467,
       43.976379, 43.978613, 43.973495, 43.973495, 43.968419])
Dimensions without coordinates: fires
Attributes:
    long_name:      Latitude
    standard_name:  latitude
    units:          degrees_north
    valid_min:      -90.0
    valid_max:      90.0, 
array([38.424813, 38.441404, 29.594317, 26.774829, 29.607258, 29.620235,
       29.592402, 29.605285, 29.618236, 29.618236, 26.787083, 26.799272,
       26.77044 , 26.782656, 26.782656, 26.7949  , 26.769496, 26.781764,
       26.794036, 26.765122, 26.777358, 26.777358, 26.789596])
Dimensions without coordinates: fires
Attributes:
    long_name:      Longitude
    standard_name:  longitude
    units:          degrees_east
    valid_min:      -180.0
    valid_max:      180.0) 
Get FRP_SWIR and FRP_MWIR data from satelite:
>>> FRP_SWIR = file_xr['FRP_SWIR']
>>> FRP_SWIR 

array([-1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1.,
       -1., -1., -1., -1., -1., -1., -1., -1., -1.])
Dimensions without coordinates: fires
Attributes:
    long_name:  Fire radiative power computed from SWIR channel (S6)
    units:      MW 
>>> FRP_MWIR = file_xr['FRP_MWIR']
>>> FRP_MWIR

array([ 10.290943,  11.447042, 179.555982,  84.84376 ,  48.277547,   9.320155,
        17.840467,  38.242334,  18.615514,  18.611452,  14.36118 ,   4.440371,
         3.06999 ,   5.008403,   5.005609,   4.452938,   8.228399,   8.442025,
         5.631591,   3.531036,   3.509205,   3.507225,   2.876292])
Dimensions without coordinates: fires
Attributes:
    long_name:  Fire radiative power computed from MWIR channels (S7 and F1)
    units:      MW 

Sunday, August 30, 2020

Python 3.8.5 : Testing with openpyxl - part 002 .

Today I will show you how can use Levenshtein ratio and distance between two strings, see wikipedia.
I used three files created with LibreOffice and save it like xlsx file type.
All of these files come with the column A fill with strings of characters, in this case, numbers.
The script will read all of these files from the folder named xlsx_files and will calculate Levenshtein ratio and distance between the strings of name of these files and column A.
Finally, the result is shown into a graph with matplotlib python package.
Let's see the python script:
import os
from glob import glob

from openpyxl import load_workbook
import numpy as np 
import matplotlib.pyplot as plt 

def levenshtein_ratio_and_distance(s, t, ratio_calc = False):
    """ levenshtein_ratio_and_distance - distance between two strings.
        If ratio_calc = True, the function computes the
        levenshtein distance ratio of similarity between two strings
        For all i and j, distance[i,j] will contain the Levenshtein
        distance between the first i characters of s and the
        first j characters of t
    """
    # Initialize matrix of zeros
    rows = len(s)+1
    cols = len(t)+1
    distance = np.zeros((rows,cols),dtype = int)

    # Populate matrix of zeros with the indeces of each character of both strings
    for i in range(1, rows):
        for k in range(1,cols):
            distance[i][0] = i
            distance[0][k] = k
    for col in range(1, cols):
        for row in range(1, rows):
            # check the characters are the same in the two strings in a given position [i,j] 
            # then the cost is 0
            if s[row-1] == t[col-1]:
                cost = 0 
            else:             
                # calculate distance, then the cost of a substitution is 1.
                if ratio_calc == True:
                    cost = 2
                else:
                    cost = 1
            distance[row][col] = min(distance[row-1][col] + 1,      # Cost of deletions
                                 distance[row][col-1] + 1,          # Cost of insertions
                                 distance[row-1][col-1] + cost)     # Cost of substitutions
    if ratio_calc == True:
        # Ration computation of the Levenshtein Distance Ratio
        Ratio = ((len(s)+len(t)) - distance[row][col]) / (len(s)+len(t))
        return Ratio
    else:
        return distance[row][col]


PATH = "/home/mythcat/xlsx_files/"
result = [y for x in os.walk(PATH) for y in glob(os.path.join(x[0], '*.xlsx'))]
result_files = [os.path.join(path, name) for path, subdirs, files in os.walk(PATH) for name in files]
#print(result)
row_0 = []

for r in result:
    n = 0
    wb = load_workbook(r)
    sheets = wb.sheetnames
    ws = wb[sheets[n]]
    for row in ws.rows:
            if (row[0].value) != None :
                rows = row[0].value
                row_0.append(rows)

print("All rows of column A ")
print(row_0)
files = []
for f in result_files:
    ff = str(f).split('/')[-1:][0]
    fff = str(ff).split('.xlsx')[0]
    files.append(fff)

print(files)
# define tree lists for levenshtein
list1 = []
list2 = []

for l in row_0:
    str(l).lower()
    for d in files:
        Distance = levenshtein_ratio_and_distance(str(l).lower(),str(d).lower())   
        Ratio = levenshtein_ratio_and_distance(str(l).lower(),str(d).lower(),ratio_calc = True)
        list1.append(Distance)
        list2.append(Ratio)
        
print(list1, list2)
# plotting the points  
plt.plot(list1,'g*', list2, 'ro' )
plt.show()
The result is this:
[mythcat@desk ~]$ python test_xlsx.py
All rows of column A 
[11, 2, 113, 4, 1111, 4, 4, 111, 2, 1111, 5, 4, 4, 3, 1111, 1, 2, 1113, 4, 115, 1, 2, 221, 1, 1,
 43536, 2, 34242, 3, 1]
['001', '002', '003']
[2, 3, 3, 3, 2, 3, 3, 3, 2, 3, 3, 3, 3, 4, 4, 3, 3, 3, 3, 3, 3, 2, 3, 3, 3, 2, 3, 3, 4, 4, 3, 3, 
3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 3, 4, 4, 2, 3, 3, 3, 2, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 3, 
2, 3, 2, 3, 3, 2, 3, 3, 2, 3, 3, 5, 5, 4, 3, 2, 3, 5, 4, 5, 3, 3, 2, 2, 3, 3] [0.4, 0.0, 0.0, 0.0, 
0.5, 0.0, 0.3333333333333333, 0.0, 0.3333333333333333, 0.0, 0.0, 0.0, 0.2857142857142857, 0.0, 0.0,
 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.3333333333333333, 0.0, 0.0, 0.0, 0.5, 0.0, 0.2857142857142857, 0.0,
 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5, 0.2857142857142857, 0.0, 0.0, 0.5,
 0.0, 0.0, 0.0, 0.5, 0.0, 0.2857142857142857, 0.0, 0.2857142857142857, 0.0, 0.0, 0.0, 0.3333333333333333,
 0.0, 0.0, 0.5, 0.0, 0.0, 0.0, 0.5, 0.0, 0.3333333333333333, 0.3333333333333333, 0.0, 0.5, 0.0, 0.0,
 0.5, 0.0, 0.0, 0.0, 0.0, 0.25, 0.0, 0.5, 0.0, 0.0, 0.25, 0.25, 0.0, 0.0, 0.5, 0.5, 0.0, 0.0]

Monday, August 24, 2020

Python Qt5 : Get item data from QTreeWidgets.

In this example, I create a tree view with QTreeView with all folders tree.
I add a context_menu with two options.
One option is to get the data from item and is the name of the folder.
The second option is to close the application.
Let's see the source code:
import sys
from PyQt5.QtWidgets import QApplication, QFileSystemModel, QDesktopWidget
from PyQt5.QtWidgets import QTreeView, QWidget, QVBoxLayout, QMenu
from PyQt5.QtGui import QIcon
from PyQt5 import QtCore
from PyQt5.QtCore import Qt, QObject

class my_app_tree(QWidget):

    def __init__(self):
        super().__init__()
        self.title = "show files and folders on tree view"
        #self.left = 0
        #self.top = 0
        #self.width = 640
        #self.height = 480
        self.center()
        self.resize(640,480)
        self.initUI()

    def center(self):
        frame_geometry = self.frameGeometry()
        center_position = QDesktopWidget().availableGeometry().center()
        frame_geometry.moveCenter(center_position)
        self.move(frame_geometry.topLeft())

    def context_menu(self, position):
        menu = QMenu()
        copy_action = menu.addAction("Get folder")
        quit_action = menu.addAction("Quit")
        action = menu.exec_(self.tree.mapToGlobal(position))
        # quit application
        if action == quit_action:
            my_application.quit()
        # copy folder name from item
        elif action == copy_action:
            item = self.tree.selectedIndexes()[0].data()
            print("name folder is: "+str(item))

    def initUI(self):
        self.setWindowTitle(self.title)
        #the next source code line is used with left, top, width, height from __init__
        #self.setGeometry(self.left, self.top, self.width, self.height)
        
        self.model = QFileSystemModel()
        self.model.setRootPath('')
        self.tree = QTreeView()
        self.tree.setModel(self.model)
        
        self.tree.setAnimated(False)
        self.tree.setIndentation(20)
        self.tree.setSortingEnabled(True)
        
        self.tree.setWindowTitle("Dir View")
        self.tree.resize(640, 480)
        
        windowLayout = QVBoxLayout()
        windowLayout.addWidget(self.tree)
        self.setLayout(windowLayout)

        self.tree.setContextMenuPolicy(Qt.CustomContextMenu)
        self.tree.customContextMenuRequested.connect(self.context_menu)
        
        self.show()

if __name__ == '__main__':
    my_application = QApplication(sys.argv)
    example = my_app_tree()
    sys.exit(my_application.exec_())
The result of this source code can be see in the next image:

Sunday, August 23, 2020

Python Qt5 : Add and remove items between two QTreeWidgets.

Today's tutorial will show you how to add and remove items between two QTreeWidgets.
The source code is very simple to understand: the user interface is created with two QTreeWidgets.
One is completed with elements and when the buttons are pressed, the elements are interchanged.
import sys
from PyQt5.QtWidgets import QApplication, QWidget, QDesktopWidget, QPushButton
from PyQt5.QtWidgets import QBoxLayout,QTreeWidget,QTreeWidgetItem

class my_app_class(QWidget):
    def __init__(self):
        super().__init__()

        self.add_button=QPushButton('Add item here')
        self.remove_button=QPushButton('Remove item')
        self.wishlist=QTreeWidget(self)
        self.tree_list=QTreeWidget(self)
        # init the UI
        self.initUI()

    def initUI(self):
        # set title of window
        self.setWindowTitle('add and remove items from QTreeWidget!')

        self.init_tree()

        self.resize(800, 480)
        self.center()
        self.show()

    def center(self):
        geometry_frame = self.frameGeometry()
        center_pos = QDesktopWidget().availableGeometry().center()
        geometry_frame.moveCenter(center_pos)
        self.move(geometry_frame.topLeft())


    def init_tree(self):
        headers = ['A','B','C','D']

        self.tree_list.setColumnCount(len(headers))
        self.tree_list.setHeaderLabels(headers)

        self.wishlist.setColumnCount(len(headers))
        self.wishlist.setHeaderLabels(headers)


        list_layout = QBoxLayout(QBoxLayout.LeftToRight)
        list_layout.addWidget(self.tree_list)
        list_layout.addWidget(self.wishlist)

        tree_root = QTreeWidget.invisibleRootItem(self.tree_list)
        # add data to QTreeWidget with QTreeWidgetItem
        my_data = ['1','2','3','4']
        item = QTreeWidgetItem()
        for idx, data in enumerate(my_data):
            item.setText(idx, data)

        tree_root.addChild(item)

        my_data = ['11','10','01','D']
        item = QTreeWidgetItem()
        for idx, data in enumerate(my_data):
            item.setText(idx, data)

        tree_root.addChild(item)

        my_data = ['s', 'c', 'c', 'c']
        item = QTreeWidgetItem()
        for idx, data in enumerate(my_data):
            item.setText(idx, data)

        tree_root.addChild(item)

        btn_layout = QBoxLayout(QBoxLayout.RightToLeft)
        btn_layout.addWidget(self.add_button)
        btn_layout.addWidget(self.remove_button)

        main_layout = QBoxLayout(QBoxLayout.TopToBottom)
        main_layout.addLayout(list_layout)
        main_layout.addLayout(btn_layout)

        self.add_button.clicked.connect(self.move_item)
        self.remove_button.clicked.connect(self.move_item)

        self.setLayout(main_layout)
        return main_layout


    def move_item(self):
        sender = self.sender()

        if self.add_button == sender:
            source = self.tree_list
            target = self.wishlist
        else:
            source = self.wishlist
            target = self.tree_list

        item = QTreeWidget.invisibleRootItem(source).takeChild(source.currentIndex().row())
        QTreeWidget.invisibleRootItem(target).addChild(item)

if __name__=='__main__':
    # start the QApplication
    my_application = QApplication(sys.argv)
    # create aplication with the class
    example = my_app_class()
    # use exit for QApplication
    sys.exit(my_application.exec_())

Python 3.8.5 : Testing with openpyxl - part 001 .

The Python executes the code line by line because is an interpreter language.
This allows users to solve issues in the programming area, fast and easy.
I use python versiono 3.8.5 build on Aug 12 2020 at 00:00:00, see the result of interactive mode:
[mythcat@desk ~]$ python
Python 3.8.5 (default, Aug 12 2020, 00:00:00) 
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Today I will show you how to start using openpyxl python package.
Another tutorial about python and documents can be found here.
The openpyxl was created by Eric Gazoni, Charlie Clark, and is a Python library to read/write Excel 2010 xlsx/xlsm/xltx/xltm files.
Let's install the openpyxl python package:
[mythcat@desk ~]$ pip3 install openpyxl --user
Collecting openpyxl
...
Installing collected packages: openpyxl
Successfully installed openpyxl-3.0.5
I tested with the default example source code and works well.
from openpyxl import Workbook
wb = Workbook()

# grab the active worksheet
ws = wb.active

# Data can be assigned directly to cells
ws['A1'] = 42

# Rows can also be appended
ws.append([1, 2, 3])

# Python types will automatically be converted
import datetime
ws['A2'] = datetime.datetime.now()

# Save the file
wb.save("sample.xlsx")
The next example gets all data about asteroids close to planet Earth and put into xlsx file type.
The rows with dangerous asteroids are fill with the red color:
# check asteroids close to planet Earth and add it to file
# import json python package
import json, urllib.request, time

# import openpyxl python package
from openpyxl import Workbook
from openpyxl.styles import PatternFill
# use active worksheet
wb = Workbook()
ws = wb.active

today = time.strftime('%Y-%m-%d', time.gmtime())
print("Time is: " + today)
now = today
# retrieve data about asteroids approaching planet Earth into json format
url = "https://api.nasa.gov/neo/rest/v1/feed?start_date=" + today + "&end_date=" + today + "&api_key=DEMO_KEY"
response = urllib.request.urlopen(url)
result = json.loads(response.read())

print("Now, " + str(result["element_count"]) + " asteroids is close to planet Earth.")
asteroids = result["near_earth_objects"]

no_data = ""
dangerous = ""

ws.append(['today', 'name', 'dangerous?', 'no_data'])
# parsing all the JSON data and add to file
for asteroid in asteroids:
    for field in asteroids[asteroid]:

      try:
        name = "Asteroid Name: " + field["name"]

        if field["is_potentially_hazardous_asteroid"]:   
          dangerous = "... dangerous to planet Earth!"

        else:
          dangerous = "... not threat to planet Earth!"

      except:
        no_data = "no data"
      ws.append([today, name, dangerous, no_data]) 

# create a red patern to fill
redFill = PatternFill(start_color='FFFF0000',
                   end_color='FFFF0000',
                   fill_type='solid')

# check the row with the dangerous asteroid and fill it
for row in ws.rows:
 if row[2].value == "... dangerous to planet Earth!":
  for cell in row:
      cell.fill = redFill

# write all data to file 
wb.save(str(now)+"_asteroids.xlsx")
I run it and result working well:
[mythcat@desk ~]$ python asteroid_data.py 
Time is: 2020-08-23
Now, 9 asteroids is close to planet Earth.
... see the next screenshot:


Saturday, August 22, 2020

Python 3.8.5 : Testing the pyre tool - part 001.

The Pyre is a static analysis tool to detect and prevent security issues in Python code that can be found on the official website.
The Pyre tool supports the Language Server Protocol and has an extension for VSCode.
The team development comes at August 7, 2020, with this intro:
Pyre is a performant type checker for Python. Statically typing what are essentially fully dynamic languages has a long tradition at Facebook. We've done this for PHP with Hack and for Javascript with Flow.
The install is easy to do with pip tool:
[mythcat@desk ~]$ pip install pyre-check
Defaulting to user installation because normal site-packages is not writeable
Collecting pyre-check
  Using cached pyre_check-0.0.52-py3-none-manylinux1_x86_64.whl (22.9 MB)
...
Installing collected packages: pyre-check
Successfully installed pyre-check-0.0.52
If you want to use a virtual environment:
[mythcat@desk ~]$ mkdir my_project && cd my_project
[mythcat@desk my_project]$ python3 -m venv ~/.venvs/venv
[mythcat@desk my_project]$ source ~/.venvs/venv/bin/activate
(venv) [mythcat@desk my_project]$ pip install pyre-check
Collecting pyre-check
...
(venv) [mythcat@desk my_project]$ pyre init
 ƛ Which directory should pyre be initialized in? (Default: `.`): 
(venv) [mythcat@desk my_project]$ cat .pyre_configuration
{
  "binary": "/home/mythcat/.venvs/venv/bin/pyre.bin",
  "source_directories": [
    "."
  ],
  "taint_models_path": "/home/mythcat/.venvs/venv/lib/pyre_check/taint/",
  "typeshed": "/home/mythcat/.venvs/venv/lib/pyre_check/typeshed/"
}
(venv) [mythcat@desk my_project]$ ls .pyre
my_project  pid_files  pyre.stderr
(venv) [mythcat@desk my_project]$ pyre
 ƛ No watchman binary found. 
To enable pyre incremental, you can install watchman: https://facebook.github.io/watchman/docs/install
 ƛ Defaulting to non-incremental check.
 ƛ No type errors found
Let's test with the default example from documentation:
(venv) [mythcat@desk my_project]$ echo "i: int = 'string'" > test.py
(venv) [mythcat@desk my_project]$ pyre
 ƛ No watchman binary found. 
To enable pyre incremental, you can install watchman: https://facebook.github.io/watchman/docs/install
 ƛ Defaulting to non-incremental check.
 ƛ Found 1 type error!
test.py:1:0 Incompatible variable type [9]: i is declared to have type `int` but is used as type `str`.
(venv) [mythcat@desk my_project]$ cat test.py 
i: int = 'string'
You can see is working well and detect the problem.
A short intro can found on the Facebook developers youtube channel:

Saturday, August 15, 2020

Python 3.8.5 : The hashlib python package - part 001.

The tutorial for today is about hashlib python module.
The official webpage comes for this python package has this intro:
This module implements a common interface to many different secure hash and message digest algorithms. Included are the FIPS secure hash algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 (defined in FIPS 180-2) as well as RSA’s MD5 algorithm (defined in Internet RFC 1321).
The example source code to test a simple hash is this:
import hashlib
import os

def file_sha1(filename):
    BUF_SIZE = 65536  # read stuff in 64kb chunks!
    get_sha1 = hashlib.sha1()
    with open(filename, 'rb') as f:
        while True:
            data = f.read(BUF_SIZE)
            if not data:
                break
            get_sha1.update(data)
    return get_sha1.hexdigest()

# I add this comment after first to see the hash difference.
files = [f for f in os.listdir('.') if os.path.isfile(f)]
for f in files:
    h = file_sha1(f)
    print(h) 
Let's test the source code with the default directory and two files.
I run it first with default source code and then I add a comment to test_hash_file.py file.
You can see the hash is changed from b222523567a8a806382b86578717ddbd00e0f4b4 to 2134660551cc67812413a3a75fd12efb05d591ef.
[mythcat@desk Projects_Python]$ ls
test_hash_file.py  test_numpy_001.py
[mythcat@desk Projects_Python]$ python test_hash_file.py 
98b2833527ad3d9fe263542c6aa06c04182d3dfb
b222523567a8a806382b86578717ddbd00e0f4b4
[mythcat@desk Projects_Python]$ python test_hash_file.py 
98b2833527ad3d9fe263542c6aa06c04182d3dfb
2134660551cc67812413a3a75fd12efb05d591ef

Sunday, August 9, 2020

Python 3.8.5 : Pearson Product Moment Correlation with corrcoef from numpy.

The python package named numpy come with corrcoef function to return Pearson product-moment correlation coefficients.
This method has a limitation in that it can compute the correlation matrix between two variables only.
The full name is the Pearson Product Moment Correlation (PPMC).
The PPMC is not able to tell the difference between dependent variables and independent variables.
The documentation about this function can be found here.
More examples of Pearson Correlation can be found on this website.
My example presented in this tutorial, use the random packet to randomly generate integers and then calculate the correlation coefficients.
All of these are calculated five times in a for a cycle and each time the seed parameters are changed randomly.
Each time the correlation matrices are printed and then the random number graphs are displayed.
Let's see the source code:
import random

import numpy as np

nr_integers = 100
size_integers = 100

import matplotlib
import matplotlib.pyplot as plt

# set from 0 to 4 seed for random and show result 
for e in range(5):
    # change random seed
    np.random.seed(e)
    # nr_integers random integers between 0 and size_integers
    x = np.random.randint(0, size_integers, nr_integers)
    # Positive Correlation with some noise created with
    # nr_integers random integers between 0 and size_integers
    positive_y = x + np.random.normal(0, size_integers, nr_integers)
    correlation_positive = np.corrcoef(x, positive_y)
    # show matrix for correlation_positive
    print(correlation_positive)
    # Negative Correlation with same noise created with 
    # nr_integers random integers between 0 and size_integers
    negative_y = 100 - x + np.random.normal(0, size_integers, nr_integers)
    correlation_negative = np.corrcoef(x, negative_y)
    # show matrix for output with plt
    print(correlation_negative)
    # set graphic for plt with two graphics for each output with subplot
    plt.subplot(1, 2, 1)
    plt.scatter(x,positive_y)
    plt.subplot(1, 2, 2)
    plt.scatter(x,negative_y)
    # show the graph 
    plt.show()


Tuesday, August 4, 2020

Python 3.6.9 : My colab tutorials - part 008.

Today I deal with these two python packages named selenium and chromium-chromedriver.
I used selenium to get pieces of information from webpages.
These examples can be found at my GitHub project colab on the notebook named catafest_008.

Friday, July 31, 2020

Python 3.8.5 : PyEphem astronomy library for Python - part 001.

About this python package, you can find it from the official website.
PyEphem provides an ephem Python package for performing high-precision astronomy computations. The underlying numeric routines are coded in C and are the same ones that drive the popular XEphem astronomy application, whose author, Elwood Charles Downey, generously gave permission for their use in PyEphem. The name ephem is short for the word ephemeris, which is the traditional term for a table giving the position of a planet, asteroid, or comet for a series of dates.
Because I like astronomy and lately a lot has happened in this field, I thought I would come up with some simple tutorials for those who are interested.
This tutorial is tested on a Linux distribution called Fedora 32 with Python version 3.8.5.
I installed this python packet with the pip3 tool.
[mythcat@desk ~]$ sudo pip3 install ephem --user
WARNING: Running pip install with root privileges is generally not a good idea. 
Try `pip3 install --user` instead.
Collecting ephem
...
Installing collected packages: ephem
Successfully installed ephem-3.7.7.1
You know that NASA has launched the Mars Perseverance rover, so let's play with this topic a bit.
Let's see current azimuth and elevation for planet Mars.
For this is need the position of the observer and then is compute the position of the planet Mars.
I used the position of the Greenwich for the observer, but you can create your oun observer and use it.
[mythcat@desk ~]$ python3
Python 3.8.5 (default, Jul 20 2020, 00:00:00) 
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ephem
>>> import math
>>> convert = math.pi / 180.
>>> mars = ephem.Mars()
>>> greenwich = ephem.Observer()
>>> greenwich.lat = "51.477928"
>>> greenwich.lon = "-0.001545"
>>> mars.compute(greenwich)
>>> az_deg, alt_deg = mars.az*convert, mars.alt*convert
>>> print(f"Mars' current azimuth and elevation: {az_deg:.2f} {alt_deg:.2f}")
Mars' current azimuth and elevation: 0.11 -0.01
Let's see another example with Mars to take ascension and declination for the epoch specified:
...
>>> import datetime
>>> now = datetime.datetime.now()
>>> print(now)
2020-07-31 19:11:42.312027
>>> mars.compute(now)
>>> print(mars.ra)
1:12:43.64
>>> print(mars.dec)
3:33:22.6
You can get the magnitude, size (diameter in arcseconds) and size (radius as an angle):
>>> print(mars.mag)
-1.11
>>> print(mars.size)
14.555475234985352
>>> print(mars.radius)
0:00:07.3
You can easily see which constellation Mars is on.
>>> ephem.constellation(mars)
('Psc', 'Pisces')

Sunday, July 26, 2020

Python 3.6.9 : My colab tutorials - parts 006 - 007.

This tutorial is called: My colab tutorials - parts 006 - 007.
The only reason for synchronization with the source code from my GitHub account on the Colab project.
I like collab more and more because I can quickly test the source code.
The example is taken from here and adapted to work on Colab and the new version of numba
Here is a simple example with the python numba package to creat that Mandelbrot fractal set.
import numba
from numba import jit

@jit
def mandel(x, y, max_iters):
  """
    Given the real and imaginary parts of a complex number,
    determine if it is a candidate for membership in the Mandelbrot
    set given a fixed number of iterations.
  """
  c = complex(x, y)
  z = 0.0j
  for i in range(max_iters):
    z = z*z + c
    if (z.real*z.real + z.imag*z.imag) >= 4:
      return i

  return max_iters

@jit
def create_fractal(min_x, max_x, min_y, max_y, image, iters):
  height = image.shape[0]
  width = image.shape[1]

  pixel_size_x = (max_x - min_x) / width
  pixel_size_y = (max_y - min_y) / height
    
  for x in range(width):
    real = min_x + x * pixel_size_x
    for y in range(height):
      imag = min_y + y * pixel_size_y
      color = mandel(real, imag, iters)
      image[y, x] = color

image = np.zeros((1024, 1536), dtype = np.uint8)
start = timer()
create_fractal(-2.0, 1.0, -1.0, 1.0, image, 20) 
dt = timer() - start

print ("Mandelbrot created in %f s" % dt)
imshow(image)
show() 

Saturday, July 25, 2020

Python 3.8.2 : The numba python package - part 001 .

The development of this python package comes with this short intro:
Numba is a just-in-time compiler for Python that works best on code that uses NumPy arrays and functions and loops. The most common way to use Numba is through its collection of decorators that can be applied to your functions to instruct Numba to compile them. When a call is made to a Numba decorated function it is compiled to machine code “just-in-time” for execution and all or part of your code can subsequently run at native machine code speed!
I installed this python package on my folder Python38:
D:\Python38>pip3 install numba
Collecting numba
...
Successfully installed numba-0.50.1
D:\Python38>python.exe
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 22:45:29) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
>>> numba.__version__
'0.50.1'
This package did not work with python install on the folder Python38_64:
D:\Python38_64>pip3 install numba
Collecting numba
...
Installing collected packages: numpy, llvmlite, numba
Successfully installed llvmlite-0.33.0 numba-0.50.1 numpy-1.19.1
WARNING: You are using pip version 20.1; however, version 20.1.1 is available.
...
D:\Python38_64>python.exe
Python 3.8.4 (tags/v3.8.4:dfa645a, Jul 13 2020, 16:46:45) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
Traceback (most recent call last):
...
ModuleNotFoundError: No module named 'numba'
You can write standard Python functions and run them on a CUDA-capable GPU.
First, I need to enable this feature:
D:\Python38>SET NUMBA_ENABLE_CUDASIM=1

D:\Python38>python
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 22:45:29) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from numba import cuda
>>> print(cuda.gpus)
...Managed Device 0...
Let's test with a simple example to create a data and use it:
D:\Python38>python
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 22:45:29) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> import numpy as np
>>> from numba import cuda
>>>
>>> @cuda.jit
... def create(data):
...     data[cuda.blockIdx.x, cuda.threadIdx.x] = cuda.blockIdx.x
...
>>> numBlocks = 4
>>> threadsPerBlock = 6
>>>
>>> data = np.ones((numBlocks, threadsPerBlock), dtype=np.uint8)
>>> create[numBlocks, threadsPerBlock](data)
>>> print(data)
[[0 0 0 0 0 0]
 [1 1 1 1 1 1]
 [2 2 2 2 2 2]
 [3 3 3 3 3 3]]

Saturday, July 18, 2020

Python Qt5 : Create a simple web browser.

This python package named PyQtWebEngine, see the official webpage for more infos:
The team development comes with this intro:
PyQtWebEngine is a set of Python bindings for The Qt Company’s Qt WebEngine framework. The framework provides the ability to embed web content in applications and is based on the Chrome browser. The bindings sit on top of PyQt5 and are implemented as three separate modules corresponding to the different libraries that make up the framework.
I used my python version: Python 3.8.3rc1 (tags/v3.8.3rc1:802eb67, Apr 29 2020, 21:39:14) [MSC v.1924 64 bit (AMD64)] on win32
... and PyQt5 version PyQt version: 5.15.0:
>>> from PyQt5.Qt import PYQT_VERSION_STR        
>>> print("PyQt version:", PYQT_VERSION_STR)  
PyQt version: 5.15.0
First, let's install this python package:
pip3 install PyQtWebEngine --user
Collecting PyQtWebEngine
  Using cached PyQtWebEngine-5.15.0-5.15.0-cp35.cp36.cp37.cp38-none-win_amd64.whl (57.9 MB)
Collecting PyQt5-sip<13>=12.8
  Using cached PyQt5_sip-12.8.0-cp38-cp38-win_amd64.whl (63 kB)
Collecting PyQt5>=5.15
  Using cached PyQt5-5.15.0-5.15.0-cp35.cp36.cp37.cp38-none-win_amd64.whl (64.5 MB)
Installing collected packages: PyQt5-sip, PyQt5, PyQtWebEngine
  WARNING: The scripts pylupdate5.exe, pyrcc5.exe and pyuic5.exe are installed in 
'C:\Users\catal\AppData\Roaming\Python\Python38\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, 
use --no-warn-script-location.
Successfully installed PyQt5-5.15.0 PyQt5-sip-12.8.0 PyQtWebEngine-5.15.0
The WARNING reflects the path for this tool, but you can find on user folder, see:
C:\Users\catal>pylupdate5.exe
Usage:
    pylupdate5 [options] project-file
    pylupdate5 [options] source-files -ts ts-files

Options:
    -help  Display this information and exit
    -version
           Display the version of pylupdate5 and exit
    -verbose
           Explain what is being done
    -noobsolete
           Drop all obsolete strings
    -tr-function name
           name() may be used instead of tr()
    -translate-function name
           name() may be used instead of translate()

C:\Users\catal>
This is a simple source code with an browser example:
import sys
from PyQt5.Qt import *
from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWidgets import QMainWindow, QWidget, QLabel, QLineEdit
from PyQt5.QtWidgets import QPushButton
from PyQt5.QtCore import QSize, QUrl
from PyQt5.QtWebEngineWidgets import *
from PyQt5.QtWebEngineWidgets import QWebEnginePage, QWebEngineView
class MainWindow(QMainWindow):
    def __init__(self):
        super(MainWindow, self).__init__()

        self.setWindowTitle('catafest browser')

        self.browser_toolbar = QToolBar()
        self.addToolBar(self.browser_toolbar)
        self.back_button = QPushButton()
        #self.back_button.setIcon(QIcon('left.png'))
        self.back_button.clicked.connect(self.back_page)
        self.browser_toolbar.addWidget(self.back_button)
        self.forward_button = QPushButton()
        #self.forward_button.setIcon(QIcon('right.png'))
        self.forward_button.clicked.connect(self.forward_page)
        self.browser_toolbar.addWidget(self.forward_button)

        self.web_address = QLineEdit()
        self.web_address.returnPressed.connect(self.load_page)
        self.browser_toolbar.addWidget(self.web_address)

        self.web_browser = QWebEngineView()
        self.setCentralWidget(self.web_browser)
        first_url = "https://pypi.org/project/PyQt5/"

        self.web_address.setText(first_url)
        self.web_browser.load(QUrl(first_url))
        self.web_browser.page().titleChanged.connect(self.setWindowTitle)
        self.web_browser.page().urlChanged.connect(self.changed_page)
        
    def load_page(self):
        url = QUrl.fromUserInput(self.web_address.text())
        if url.isValid():
            self.web_browser.load(url)

    def back_page(self):
        self.web_browser.page().triggerAction(QWebEnginePage.Back)

    def forward_page(self):
        self.web_browser.page().triggerAction(QWebEnginePage.Forward)

    def changed_page(self, url):
        self.web_address.setText(url.toString())
        
if __name__ == "__main__":
    app = QtWidgets.QApplication(sys.argv)
    mainWin = MainWindow()
    availableGeometry = app.desktop().availableGeometry(mainWin)
    mainWin.resize(availableGeometry.width(), availableGeometry.height())
    mainWin.show()
    sys.exit( app.exec_() )

Wednesday, July 15, 2020

Python 3.8.3 : Lists in Python 3 - part 001.

I am currently working on a project that involves the use of complex data structures and lists and my time is limited.
This led me to start a new series of python tutorials on python lists.
I realized that the lists had no substantial changes in the evolution of the python programming language, see the official documentation.
You will find on the internet a lot of questions related to lists, algorithms, and problems involving lists.
If you are not a beginner then it will seem boring at first but over time I will try to draw attention to really significant elements in python programming with lists.
Let's start taking significant steps in using lists.
Before using the methods of list objects, let's create some lists:

# create a empty list 
my_list_001 = []
# fill the list with 4 consecutive numbers from 0 to 3 
my_list_001 = list(range(0,4))
# show the list 
my_list_001
[0, 1, 2, 3]
# set the variable n with value 4
n = 4
# create a list with 4 zeroes 
list_of_zeros = [0] * n
# show the list
listofzeros
[0, 0, 0, 0]
# import string python package 
import string
# create a string with all ascii letters
my_letters = string.ascii_letters
# use list to create a list with all letters
list(my_letters)
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't',
 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O',
 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z']
# create a nested list is a list can also have another list as an item
my_list = ["catafest", [1,9,7,6], ["!"]]
# show the list
my_list
['catafest', [1, 9, 7, 6], ['!']]
# access elements from a list with index operator pozitive and negative [] 
my_list[0]
'catafest'
my_list[-1]
['!']
my_list[-2]
[1, 9, 7, 6]
my_list[-3]
'catafest'
# test conditions boolean
my_list[-3] == my_list[0]
True
my_list[-3] != my_list[0]
False
my_list_001[1] > listofzeros[1]
True
# sum an list element with another
sum_of_list_elemenet = my_list_001[1] + listofzeros[1]
# show result
sum_of_list_elemenet
1
# iterating through a list
for list_element in my_list:
...     print(list_element)
...
catafest
[1, 9, 7, 6]
['!']
# iterating each list  
for list_element in my_list:
...     for e in list_element:
...             print(e)
...
c
a
t
a
f
e
s
t
1
9
7
6
!
# create a list with all letters using chr and ord built-in functions
[chr(i) for i in range(ord('a'),ord('z')+1)]
# the result is
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't',
 'u', 'v', 'w', 'x', 'y', 'z']
# create a binary list from a integer
output = [int(x) for x in '{:08b}'.format(1976)]
# show the result of the list
output
[1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0]
# show the result just for on element of the list
output[3]
1
# convert a binary list to integer named result and set first with value 0
# using the bitwise left shift operator
result = 0
for digits in output:
    result = (result << 1) | digits
# show result of conversion from binary to integer
result
1976
# create a hexadecimal list from a list
hex_list = [hex(x) for x in my_list_001]
# show the hexadecimal list using the hex built-in function
hex_list
['0x0', '0x1', '0x2', '0x3']
# convert hexadecimal list to int list using the int 
int_list = [int(x,0) for x in hex_list]
# show the result
int_list
[0, 1, 2, 3]
# convert hexadecimal list to float list using the float.fromhex
float_list = [float.fromhex(x) for x in hex_list]
# show the result
float_list
[0.0, 1.0, 2.0, 3.0]
# create list using math python module 
import math
[2*x for x in my_list_001]
[0, 2, 4, 6]
[math.sin(x) for x in my_list_001]
[0.0, 0.8414709848078965, 0.9092974268256817, 0.1411200080598672]
# create sized list
sized_list = [1] * 4
# show the result
sized_list
[1, 1, 1, 1]
# list from 
>>> chars = ''.join(map(chr, range(32, 1048)))
>>> list(chars)
[' ', '!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5',
 '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K',
 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '[', '\\', ']', '^', '_', '`', 'a',
 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w',
 'x', 'y', 'z', '{', '|', '}', '~', '\x7f', '\x80', '\x81', '\x82', '\x83', '\x84', '\x85', '\x86', '\x87', 
'\x88', '\x89', '\x8a', '\x8b', '\x8c', '\x8d', '\x8e', '\x8f', '\x90', '\x91', '\x92', '\x93', '\x94', '\x95',
 '\x96', '\x97', '\x98', '\x99', '\x9a', '\x9b', '\x9c', '\x9d', '\x9e', '\x9f', '\xa0', '¡', '¢', '£', '¤', '¥',
 '¦', '§', '¨', '©', 'ª', '«', '¬', '\xad', '®', '¯', '°', '±', '²', '³', '´', 'µ', '¶', '·', '¸', '¹', 'º', '»',
 '¼', '½', '¾', '¿', 'À', 'Á', 'Â', 'Ã', 'Ä', 'Å', 'Æ', 'Ç', 'È', 'É', 'Ê', 'Ë', 'Ì', 'Í', 'Î', 'Ï', 'Ð', 'Ñ', 'Ò',
 'Ó', 'Ô', 'Õ', 'Ö', '×', 'Ø', 'Ù', 'Ú', 'Û', 'Ü', 'Ý', 'Þ', 'ß', 'à', 'á', 'â', 'ã', 'ä', 'å', 'æ', 'ç', 'è', 'é',
 'ê', 'ë', 'ì', 'í', 'î', 'ï', 'ð', 'ñ', 'ò', 'ó', 'ô', 'õ', 'ö', '÷', 'ø', 'ù', 'ú', 'û', 'ü', 'ý', 'þ', 'ÿ', 'Ā',
 'ā', 'Ă', 'ă', 'Ą', 'ą', 'Ć', 'ć', 'Ĉ', 'ĉ', 'Ċ', 'ċ', 'Č', 'č', 'Ď', 'ď', 'Đ', 'đ', 'Ē', 'ē', 'Ĕ', 'ĕ', 'Ė', 'ė',
 'Ę', 'ę', 'Ě', 'ě', 'Ĝ', 'ĝ', 'Ğ', 'ğ', 'Ġ', 'ġ', 'Ģ', 'ģ', 'Ĥ', 'ĥ', 'Ħ', 'ħ', 'Ĩ', 'ĩ', 'Ī', 'ī', 'Ĭ', 'ĭ', 'Į',
 'į', 'İ', 'ı', 'IJ', 'ij', 'Ĵ', 'ĵ', 'Ķ', 'ķ', 'ĸ', 'Ĺ', 'ĺ', 'Ļ', 'ļ', 'Ľ', 'ľ', 'Ŀ', 'ŀ', 'Ł', 'ł', 'Ń', 'ń', 'Ņ',
 'ņ', 'Ň', 'ň', 'ʼn', 'Ŋ', 'ŋ', 'Ō', 'ō', 'Ŏ', 'ŏ', 'Ő', 'ő', 'Œ', 'œ', 'Ŕ', 'ŕ', 'Ŗ', 'ŗ', 'Ř', 'ř', 'Ś', 'ś', 'Ŝ',
 'ŝ', 'Ş', 'ş', 'Š', 'š', 'Ţ', 'ţ', 'Ť', 'ť', 'Ŧ', 'ŧ', 'Ũ', 'ũ', 'Ū', 'ū', 'Ŭ', 'ŭ', 'Ů', 'ů', 'Ű', 'ű', 'Ų', 'ų', 
'Ŵ', 'ŵ', 'Ŷ', 'ŷ', 'Ÿ', 'Ź', 'ź', 'Ż', 'ż', 'Ž', 'ž', 'ſ', 'ƀ', 'Ɓ', 'Ƃ', 'ƃ', 'Ƅ', 'ƅ', 'Ɔ', 'Ƈ', 'ƈ', 'Ɖ', 'Ɗ', 'Ƌ', 'ƌ',
 'ƍ', 'Ǝ', 'Ə', 'Ɛ', 'Ƒ', 'ƒ', 'Ɠ', 'Ɣ', 'ƕ', 'Ɩ', 'Ɨ', 'Ƙ', 'ƙ', 'ƚ', 'ƛ', 'Ɯ', 'Ɲ', 'ƞ', 'Ɵ', 'Ơ', 'ơ', 'Ƣ', 'ƣ',
...
...
 'ϗ', 'Ϙ', 'ϙ', 'Ϛ', 'ϛ', 'Ϝ', 'ϝ', 'Ϟ', 'ϟ', 'Ϡ', 'ϡ', 'Ϣ', 'ϣ', 'Ϥ', 'ϥ', 'Ϧ', 'ϧ', 'Ϩ', 'ϩ', 'Ϫ', 'ϫ', 'Ϭ', 'ϭ',
 'Ϯ', 'ϯ', 'ϰ', 'ϱ', 'ϲ', 'ϳ', 'ϴ', 'ϵ', '϶', 'Ϸ', 'ϸ', 'Ϲ', 'Ϻ', 'ϻ', 'ϼ', 'Ͻ', 'Ͼ', 'Ͽ', 'Ѐ', 'Ё', 'Ђ', 'Ѓ', 'Є',
 'Ѕ', 'І', 'Ї', 'Ј', 'Љ', 'Њ', 'Ћ', 'Ќ', 'Ѝ', 'Ў', 'Џ', 'А', 'Б', 'В', 'Г', 'Д', 'Е', 'Ж', 'З']
This is not all about create and convert list, but it is quite close to the reality of the lists.
This tutorial remains open due to the complex issue and maybe I will complete it in the future.

Monday, July 13, 2020

Python 3.8.3 : Short intro to Appium-Python-Client python package.

This is a short intro of the Appium-Python-Client python package and Appium based on Client-Server Architecture.
The Appium Server can be installed using two ways: using NPM or using Appium Desktop.
I download and run the desktop version from here.
Appium-windows-1.18.0-beta.1>Appium.exe
The latest version of Java, needed for Android Studio ( you can use the installation of Android Studio with SDK) and mobile phone set on USB debugging.
The next step is to set all settings for android into Appium interface:

Using appium server, you can send commands to the Appium Server which translates it to platform-specific commands and executes on the devices.
The Appium-Python-Client python package is an extension library for use with the mobile testing framework Appium, see the official webpage.
Install Appium-Python-Client python package with pip3 tool.
pip3 install Appium-Python-Client
Collecting Appium-Python-Client
...
Successfully built Appium-Python-Client
Installing collected packages: Appium-Python-Client
Successfully installed Appium-Python-Client-1.0.1
Now you can test this python package with Appium and simple examples.


Thursday, July 9, 2020

Python 3.8.3 : About aiohttp python package.

This python package can help you to writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives, see the official documentation.
In this simple tutorial, I will show you in a few simple steps how to use it.
It is a complex module and there are multiple ways to use it.
First, on the Windows operating system users can install easily with:
pip3 install aiohttp
Collecting aiohttp
...
Installing collected packages: attrs, multidict, yarl, async-timeout, aiohttp
Successfully installed aiohttp-3.6.2 async-timeout-3.0.1 attrs-19.3.0 multidict-4.7.6 yarl-1.4.2
If you use a Linus operating system then you can use this command:
[mythcat@desk ~]$ pip3 install aiohttp --user 
...
Successfully installed aiohttp-3.6.2 async-timeout-3.0.1 multidict-4.7.5 yarl-1.4.2
This python package can be used as client or server.
The aiohttp.web implements a basic CLI for quickly serving an Application in development over TCP/IP.
You can find some example on this webpage.
These simple examples show how you can use handlers for web and servers and a request handler with a coroutine.
[mythcat@desk ~]$ python3 
Python 3.7.6 (default, Jan 30 2020, 09:44:41) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from aiohttp import web 
>>> def handler_req(request):
...     return web.Response(text='Handler')
... 
>>> handler_req
<function 0x7fc8a76aab00="" at="" handler_req="">
>>> async def handler_req_async(request):
...     return web.Response(text='async Handler')
... 
>>> handler_req_async
<function+0x7fc8a574edd0...
>>> import aiohttp
>>> from aiohttp import web
>>> from aiohttp.client import _RequestContextManager
>>> async def test_await(test_server, loop):
...  
...     async def handler(request):
...         return web.HTTPOk()
...
...     app = web.Application(loop=loop)
...     app.router.add_route('GET', '/', handler)
...     server = await test_server(app)
...     resp = await aiohttp.get(server.make_url('/'), loop=loop)
...     assert resp.status == 200
...     assert resp.connection is not None
...     await resp.release()
...     assert resp.connection is None
>>> test_await('htthttp://localhost:8080/',1000)   
<coroutine object test_await at 0x00000261FB96EA40>
Let's see the output of the dir:
dir(aiohttp)
['AsyncIterablePayload', 'AsyncResolver', 'BadContentDispositionHeader', 'BadContentDispositionParam', 
'BaseConnector', 'BasicAuth', 'BodyPartReader', 'BufferedReaderPayload', 'BytesIOPayload', 'BytesPayload',
 'ChainMapProxy', 'ClientConnectionError', 'ClientConnectorCertificateError', 'ClientConnectorError', 
'ClientConnectorSSLError', 'ClientError', 'ClientHttpProxyError', 'ClientOSError', 'ClientPayloadError', 
'ClientProxyConnectionError', 'ClientRequest', 'ClientResponse', 'ClientResponseError', 'ClientSSLError', 
'ClientSession', 'ClientTimeout', 'ClientWebSocketResponse', 'ContentTypeError', 'CookieJar', 'DataQueue', 
'DefaultResolver', 'DummyCookieJar', 'EMPTY_PAYLOAD', 'EofStream', 'Fingerprint', 'FlowControlDataQueue', 
'FormData', 'HttpVersion', 'HttpVersion10', 'HttpVersion11', 'IOBasePayload', 'InvalidURL', 'JsonPayload', 
'MultipartReader', 'MultipartWriter', 'NamedPipeConnector', 'PAYLOAD_REGISTRY', 'Payload', 'RequestInfo', 
'ServerConnectionError', 'ServerDisconnectedError', 'ServerFingerprintMismatch', 'ServerTimeoutError', 
'Signal', 'StreamReader', 'StringIOPayload', 'StringPayload', 'TCPConnector', 'TextIOPayload', 
'ThreadedResolver', 'TooManyRedirects', 'TraceConfig', 'TraceConnectionCreateEndParams', 
'TraceConnectionCreateStartParams', 'TraceConnectionQueuedEndParams', 'TraceConnectionQueuedStartParams', 
'TraceConnectionReuseconnParams', 'TraceDnsCacheHitParams', 'TraceDnsCacheMissParams', 
'TraceDnsResolveHostEndParams', 'TraceDnsResolveHostStartParams', 'TraceRequestChunkSentParams', 
'TraceRequestEndParams', 'TraceRequestExceptionParams', 'TraceRequestRedirectParams', 'TraceRequestStartParams',
 'TraceResponseChunkReceivedParams', 'Tuple', 'UnixConnector', 'WSCloseCode', 'WSMessage', 'WSMsgType', 
'WSServerHandshakeError', 'WebSocketError', '__all__', '__builtins__', '__cached__', '__doc__', '__file__',
 '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'abc', 'base_protocol', 
'client', 'client_exceptions', 'client_proto', 'client_reqrep', 'client_ws', 'connector', 
'content_disposition_filename', 'cookiejar', 'formdata', 'frozenlist', 'get_payload', 'hdrs', 'helpers', 
'http', 'http_exceptions', 'http_parser', 'http_websocket', 'http_writer', 'locks', 'log', 'multipart', 
'parse_content_disposition', 'payload', 'payload_streamer', 'payload_type', 'request', 'resolver', 'signals', 
'streamer', 'streams', 'tcp_helpers', 'tracing', 'typedefs']

Saturday, June 27, 2020

Python 3.8.3 : PyCryptodome python package - part 001.

In the last tutorial, I wrote on Sunday, June 16, 2019, you can see a simple example of this python package with KDF with PBKDF2 function.
I guess it should be interesting for visitors to this blog to read more about this package because it is very useful and interesting.
Today I come up with another tutorial covering how to use A.E.S. standard encryption and decrypting text in a binary file.
The A.E.S. is a standard?
The Federal Information Processing Standards Publications (FIPS PUBS) announcing the A.E.S. on November 26, 2001, on the Federal Information Processing Standards Publication 197.
A.E.S. known as Advanced Encryption Standard is a symmetric block cipher standardized by NIST.
The N.I.S.T is an abbreviation National Institute of Standards and Technology.
This python package named PyCryptodome is a self-contained Python package of low-level cryptographic primitives, see the readthedocs.io webpage.
First, you need to see if this python package is not on conflict with another one named PyCrypto.
Then use pip3 tool to install.
pip3 uninstall PyCrypto
WARNING: Skipping PyCrypto as it is not installed.
...
pip3 install pycryptodome
Collecting pycryptodome
...
Installing collected packages: pycryptodome
Successfully installed pycryptodome-3.9.8
Here is the source code commented for a better understanding of the encryption and decryption steps.
from Crypto.Cipher import AES
from Crypto.Random import get_random_bytes

# the data input needs to be encoded, else you will receive this error
# python TypeError: Object type  cannot be passed to C code

data = "Hello World! by catafest!".encode("utf8")
# first step - prepare for encrypt
key = get_random_bytes(16)
print("print key is: ", key)
cipher = AES.new(key, AES.MODE_EAX)
print("print cipher is: ", cipher)
ciphertext, tag = cipher.encrypt_and_digest(data)
print("print ciphertext is: ", ciphertext)

# open the binary file 
file_out = open("AES_encrypted.bin", "wb")
# write to binary file 
[ file_out.write(x) for x in (cipher.nonce, tag, ciphertext) ]

# the file is close 
file_out.close()

# next step - prepare for decrypt
new_cipher = AES.new(key, AES.MODE_EAX, cipher.nonce)

# open again the binary file 
file_in = open("AES_encrypted.bin", "rb")
# read all data from file
nonce, tag, ciphertext = [file_in.read(x) for x in (16, 16, -1) ]
# the file is close 
file_out.close()

# show data from file
print("print nonce is: ", nonce)
print("print tag is: ", tag)
print("print ciphertext is: ", ciphertext)

# create a new cipher
cipher = AES.new(key, AES.MODE_EAX, nonce)

# use this to decrypt
data = cipher.decrypt_and_verify(ciphertext, tag)
# show the result 
print(data)
The result of running the python script is this:
python.exe .\pycryptodome_AES_001.py
print key is:  b'\xbdEX\xf8\x1d!\xc5\xceI\x87\x81\xf1\xd5\xba\x8c\r'
print cipher is:  <Crypto.Cipher._mode_eax.EaxMode object at 0x000001D5F32DE100>
print cipher is:  <Crypto.Cipher._mode_eax.EaxMode object at 0x000001E062AFE100>
print ciphertext is:  b'ON\x1d\xb9\xb7\xa8\xf5\xd6\x0c\x91\xc5`B\xf4\x95u\xe1D\xb5\x88&I\x15\xc5\xc5'
print nonce is:  b'3\xa6R8\xbb\n \x9cimp$\xe4\xee\xf5-'
print tag is:  b'\xb8)\xe4\xe7\x08uE~\x84s]\xedX\xf5\xf9\xea'
print ciphertext is:  b'ON\x1d\xb9\xb7\xa8\xf5\xd6\x0c\x91\xc5`B\xf4\x95u\xe1D\xb5\x88&I\x15\xc5\xc5'
b'Hello World! by catafest!'
You can read about this process on this website.

Python 2.7.10 : IronPython and C# with Dynamic Language Runtime.

This is a simple tutorial about python and C# using the Dynamic Language Runtime with IronPython.
I use Visual Studio 2019 and .NET Framework 4.7.2 with my Console C# project named DynamicLanguageRuntime_001.
Let's install the package with Visual Studio by open the console using the main menu: Tools - NuGet Package Manager - Package Manager Console command.
PM> Install-Package DynamicLanguageRuntime
Package 'DynamicLanguageRuntime.1.2.3' already exists in project 'DynamicLanguageRuntime_001'
Time Elapsed: 00:00:01.2208674
Use Solution Explorer use right-click on References item from your project and use Add Reference ...
Into the new window dialog named Reference Manager on the Assemblies - Framework uses the edit box to search IronPython.
Then use the checkbox to select these two options: IronPython and IronPython.Modules.
See the screenshot from Visual Studio I.D.E.:

This is the source code I used:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using IronPython.Hosting;
using IronPython.Runtime;
using IronPython;
using Microsoft.Scripting.Hosting;
using Microsoft.Scripting;

namespace DynamicLanguageRuntime_001
{
    class Program
    {
        static void Main(string[] args)
        {
            // create python engine 
            ScriptEngine engine = Python.CreateEngine();
            // get and add paths to enfine
            var python_search_paths = engine.GetSearchPaths();
            python_search_paths.Add(@"C:\Program Files\IronPython 2.7\Lib");
            engine.SetSearchPaths(python_search_paths);
            // create a scope 
            ScriptScope scope = engine.CreateScope();
            // using CreateScriptSourceFromString
            engine.CreateScriptSourceFromString("print '... simple example with ironpython and C#'").Execute();
            // using Execute
            engine.Execute("print '                             by catafest!'", scope);
            // using ExecuteFile
            engine.ExecuteFile(@"D:\Projects\Python\testing\test_webpage_001.py", scope);
            dynamic testFunction = scope.GetVariable("GetFriends");
            var result = testFunction();
        }
    }
} 

Friday, June 26, 2020

Python 3.6.9 : My colab tutorials - part 005.

Today I tested google colab python features with google authentification and google.colab drive and files.
The first part of google colab code comes with authentification and you need to add the verification code for google account.
The google colab use the same version of python:
3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]
You can see all source code on my GitHub account.
The notebook can be found here.

Sunday, June 21, 2020

Python 3.8.3 : Using twitter application with python-twitter - part 002.

This is the second part of tutorials series with python-twitter.
Today I will show you how to get pieces of information about friends, users and save into a binary file with pickle known as cPickle.
I will use my old source code from the last tutorial.
import os
import twitter
# for save to file import by python version
try:
   import cPickle as pickle
except:
   import pickle

consumer_key=' '
consumer_secret=' '
token_key=' '
token_secret=' '

if __name__ == "__main__":
    api = twitter.Api(consumer_key=consumer_key,
                  consumer_secret=consumer_secret,
                  access_token_key=token_key,
                  access_token_secret=token_secret) 
    
    screen_name = 'catafest'
       
    # print all users of this account authentificated 
    # you can use GetFriends(screen_name=screen_name) 
    users = api.GetFriends()
    
    print([u.screen_name for u in users])
    # get followers 
    followers = api.GetFollowers(screen_name=screen_name)
    # print followers 
    print([f.screen_name for f in followers])
    
    # ... and save into a binary file 
    followers_file = "followers_file.bin"
    
    if not os.path.exists(followers_file):
        pickle.dump(followers, open(followers_file, "wb"), protocol=pickle.HIGHEST_PROTOCOL)
        
    # load binary file     
    if os.path.exists(followers_file):
        followers_read = pickle.load(open(followers_file, "rb"))
        print(followers_read)
The result is similar with this:
python.exe .\test_webpage_001.py
['SnapChick', 'NASA', 'andor_saga', 'blendermarket', 'Minehut', 'Aternos', 'axnro', 'Flexi23',
...
['PStackoverflow', 'SamLeac86078418', 'Sohanurr559', 'jasonalba', 'avkorablev', 'dotnetfiddle',
...
[User(ID=1260415029855256583, ScreenName=PStackoverflow), 
...

Saturday, June 20, 2020

Python 3.8.3 : Using twitter application with python-twitter - part 001.

You need to create a application for your twitter user developer on this webpage.
The next step is to get all keys and tokens from your application.
I used the python-twitter see the official webpage documentation.
Let's install this python module using the pip tool
pip install python-twitter
Collecting python-twitter
...
Installing collected packages: oauthlib, requests-oauthlib, python-twitter
Successfully installed oauthlib-3.1.0 python-twitter-3.5 requests-oauthlib-1.3.0
Let's see a simple source code:
import os
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import twitter
import datetime
from datetime import *

consumer_key=' '
consumer_secret=' '
token_key=' '
token_secret=' '

def get_tweets(api=None, screen_name=None):
    timeline = api.GetUserTimeline(screen_name=screen_name, count=200)
    earliest_tweet = min(timeline, key=lambda x: x.id).id
    print("getting tweets before:", earliest_tweet)

    while True:
        tweets = api.GetUserTimeline(
            screen_name=screen_name, max_id=earliest_tweet, count=200
        )
        new_earliest = min(tweets, key=lambda x: x.id).id

        if not tweets or new_earliest == earliest_tweet:
            break
        else:
            earliest_tweet = new_earliest
            print("getting tweets before:", earliest_tweet)
            timeline += tweets

    return timeline

if __name__ == "__main__":
    api = twitter.Api(consumer_key=consumer_key,
                  consumer_secret=consumer_secret,
                  access_token_key=token_key,
                  access_token_secret=token_secret) 
    # print api 
    #print(dir(api))
    
    # print all users of this account authentificated 
    #users = api.GetFriends()
    #print([u.screen_name for u in users])
    
    # print all tweets of my user catafest 
    screen_name = "catafest"
    timeline = get_tweets(api=api, screen_name=screen_name)
    dates = []
    for x in timeline:
        created = x.created_at
        dates.append(created)
        
    print(dates)
    dat = [datetime.strptime(d, "%a %b %d %H:%M:%S +0000 %Y") for d in dates]

    levels = np.tile([-8, 8, -4, 4, -1, 1],int(np.ceil(len(dat)/3)))[:len(dat)]
    print(levels)
    fig, ax = plt.subplots(figsize=(7.6, 5), constrained_layout=True)
    ax.set(title="Twitter dates")
    markerline, stemline, baseline = ax.stem(dat, levels,linefmt="C3-", basefmt="k-",use_line_collection=True)
    markerline.set_ydata(np.zeros(len(dat)))
    plt.setp(markerline, mec="k", mfc="w", zorder=1)
    plt.show()
The result of this script comes with this output:
python .\test_webpage_001.py
getting tweets before: 1123237192422367234
['Mon May 18 13:52:09 +0000 2020', 'Sat May 09 11:14:43 +0000 2020', 'Fri May 08 10:42:18 +0000 2020', 
'Fri May 08 10:41:37 +0000 2020', 'Sat May 02 17:41:07 +0000 2020', 'Sat May 02 17:39:15 +0000 2020', 
'Thu Apr 30 12:53:48 +0000 2020', 'Tue Apr 28 20:00:38 +0000 2020', 'Mon Apr 27 21:12:07 +0000 2020', 
'Fri Apr 24 16:39:58 +0000 2020', 'Fri Apr 24 16:09:26 +0000 2020', 'Sat Apr 11 16:56:40 +0000 2020', 
'Sun Mar 22 19:11:16 +0000 2020', 'Sat Mar 21 09:03:30 +0000 2020', 'Sat Mar 21 09:02:48 +0000 2020', 
'Sat Mar 21 08:59:18 +0000 2020', 'Mon Mar 16 06:29:34 +0000 2020', 'Fri Jan 24 19:59:38 +0000 2020', 
'Sat Jan 18 12:14:07 +0000 2020', 'Fri Jan 17 20:58:18 +0000 2020', 'Thu Jan 16 20:50:47 +0000 2020', 
'Thu Jan 16 20:49:16 +0000 2020', 'Fri Jan 03 17:57:33 +0000 2020', 'Sat Dec 28 10:14:11 +0000 2019', 
'Tue Apr 30 14:46:30 +0000 2019']
[-8  8 -4  4 -1  1 -8  8 -4  4 -1  1 -8  8 -4  4 -1  1 -8  8 -4  4 -1  1 -8]
The image show with matplotlib is this:

Python 3.7.5 : Django on Fedora distro.

[mythcat@desk django]$ source env/bin/activate
(env) [mythcat@desk django]$ python3 
Python 3.7.6 (default, Dec 19 2019, 22:52:49) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> django.VERSION
(3, 0, 1, 'final', 1) 

Static files are those files that can not be processed, generated or modified by the server.
Static files improve the performance of the website with the template inheritance method.
Static file management is an important factor in web development.
I will show you how static file works on Django project.
The new static files folder is set on settings.py file:
...
# Static files (CSS, JavaScript, Images)

STATIC_URL = '/static/'

STATIC_ROOT = os.path.join(BASE_DIR, '/home/mythcat/projects/django/mysite/test001/')
STATICFILES_DIRS = [
   os.path.join(BASE_DIR, 'static'),
 ]
...
Let's run the server:
(env) [mythcat@desk mysite]$ python3 manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
January 26, 2020 - 09:01:10
Django version 3.0.1, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C. 
If I try to use the admin area then we can see the bad result for static files. 
...
[26/Jan/2020 09:10:02] "GET /admin/test001/post/ HTTP/1.1" 200 5935
[26/Jan/2020 09:10:02] "GET /static/admin/css/changelists.css HTTP/1.1" 200 6190
[26/Jan/2020 09:10:02] "GET /admin/jsi18n/ HTTP/1.1" 200 3223
[26/Jan/2020 09:10:02] "GET /static/admin/js/jquery.init.js HTTP/1.1" 200 363
[26/Jan/2020 09:10:02] "GET /static/admin/js/urlify.js HTTP/1.1" 200 8941
[26/Jan/2020 09:10:02] "GET /static/admin/js/actions.js HTTP/1.1" 200 6766
[26/Jan/2020 09:10:02] "GET /static/admin/js/prepopulate.js HTTP/1.1" 200 1530
[26/Jan/2020 09:10:02] "GET /static/admin/js/core.js HTTP/1.1" 200 5723
[26/Jan/2020 09:10:02] "GET /static/admin/js/admin/RelatedObjectLookups.js HTTP/1.1" 200 6918
[26/Jan/2020 09:10:02] "GET /static/admin/css/fonts.css HTTP/1.1" 200 423
[26/Jan/2020 09:10:02] "GET /static/admin/js/vendor/jquery/jquery.js HTTP/1.1" 200 280364
[26/Jan/2020 09:10:02] "GET /static/admin/js/vendor/xregexp/xregexp.js HTTP/1.1" 200 128820
[26/Jan/2020 09:10:02] "GET /static/admin/img/tooltag-add.svg HTTP/1.1" 200 331
[26/Jan/2020 09:10:02] "GET /static/admin/img/sorting-icons.svg HTTP/1.1" 200 1097
Not Found: /favicon.ico
... 
Now I can try to run the command collectstatic.
When this command is executed, Django performs these operations:
  • it looks for static files in all the directories listed in STATICFILES_DIRS;
  • the static-files are then copied and saved in STATIC_ROOT directory;
  • when the server is requested for static content, it will fetch a file from STATIC_ROOT;
  • that file will have its URL modified with STATIC_URL.
These errors show us many informations about this process:
 (env) [mythcat@desk mysite]$ python3 manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings:

    /home/mythcat/projects/django/mysite

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
Found another file with the destination path 'admin/js/urlify.js'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
...
Found another file with the destination path 'django.png'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'favicon.ico'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'admin/js/urlify.js'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
...

Found another file with the destination path 'rest_framework/js/prettify-min.js'. 
It will be ignored since only the first encountered file is collected. If this is not what you want, make
 sure every static file has a unique path.
...
166 static files copied to '/home/mythcat/projects/django/mysite'. 
But, Django respects the order of your applications in settings.py file the INSTALLED_APPS area, when running collectstatic command.
If you have two installed apps that write the same static files then Django collectstatic command will write the static files for the app appearing first in the list.
In my case: django.contrib.admin , test001 and ... .
Also, is need to set this code source on urls.py to return the proper URL pattern for serving static files to your already defined pattern list.
from django.contrib.staticfiles.urls import staticfiles_urlpatterns
...
urlpatterns += staticfiles_urlpatterns()
Let's run the runserver:
 (env) [mythcat@desk mysite]$ python3 manage.py runserver
...
[26/Jan/2020 09:29:18] "GET /static/admin/css/fonts.css HTTP/1.1" 200 423
Not Found: /favicon.ico 
You can see the /favicon.ico is not found in the default path and need to move it at /home/mythcat/projects/django/mysite/test001.
After these changes let's fix all the problems I created with these learning steps on static issues.
Let's move the static folder into mysite folder.
Change the settings.py file for the static issue with this source of code:
# Static files (CSS, JavaScript, Images)

STATIC_URL = '/static/'

STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = [
   os.path.join(BASE_DIR, 'static'),
 ]
Run the (env) [mythcat@desk mysite]$ python3 manage.py collectstatic
(env) [mythcat@desk mysite]$ python3 manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings:

    /home/mythcat/projects/django/mysite/static

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes

163 static files copied to '/home/mythcat/projects/django/mysite/static', 2 unmodified.
(env) [mythcat@desk mysite]$ ls
db.sqlite3  favicon.ico  manage.py  mysite  rest_framework  static  test001
(env) [mythcat@desk mysite]$ cd static/
(env) [mythcat@desk static]$ ls
admin  django.png  favicon.ico  rest_framework
This is a simple base tutorial.
In this point you can select the way of your Django project.