analitics

Pages

Sunday, April 19, 2020

Python 3.8.2 : New release 2.3.2 for Pygame Menu.

Today, the development team come with this infos from the GitHub comes with a new release version 2.3.2.
Python library that can create a simple menu for the pygame application. Supports:
  • Textual menus
  • Buttons
  • Lists of values (selectors) that can trigger functions when pressing return or changing the value
  • Input text
  • Color input
NOTE: pygame-menu v2 will not provide new widgets or functionalities, consider upgrading to the lastest version.

Let's start the tutorial with python install on Windows 10 using the installer from here.
Use these settings from images:


Download get-pip.py to a folder on your computer.
Open a command prompt and navigate to the folder containing get-pip.py.
Run the following command:
python get-pip.py
Then update the path:
C:\Projects\Python\pygame-menu>python -m pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431
  eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)
     |████████████████████████████████| 1.4MB 819kB/s
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.0.2
You need to install pygame python module:
C:\Projects\Python\pygame-menu>pip install pygame
Collecting pygame
  Downloading pygame-1.9.6-cp38-cp38-win_amd64.whl (4.8 MB)
     |████████████████████████████████| 4.8 MB 819 kB/s
Installing collected packages: pygame
Successfully installed pygame-1.9.6
The last step is to install the Pygame Menu with git tool and documentation with pip tool:

$ git clone https://github.com/ppizarror/pygame-menu
Cloning into 'pygame-menu'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 5649 (delta 3), reused 7 (delta 2), pack-reused 5640
Receiving objects: 100% (5649/5649), 12.99 MiB | 5.52 MiB/s, done.
Resolving deltas: 100% (4289/4289), done.
...
C:\Projects\Python\pygame-menu>pip install -e .[doc]
The result with of how these python module with a simple example:
C:\Projects\Python\pygame-menu\pygame_menu\examples>python game_selector.py
pygame 1.9.6
Hello from the pygame community. https://www.pygame.org/contribute.html
...

Saturday, April 4, 2020

Python 2.7.8 : Using python scripts with Revit Dynamo.

Dynamo is a visual programming tool that extends the power of the Revit by providing access to Revit API (Application Programming Interface.
Dynamo works with node, each node have inputs and outputs and performs a specific task.
This is a short tutorial about how you can use your python skills with Revit and Dynamo software.
First, you need to start the Revit. I used Revit 2020 version.
Then from Main menu use Manage and click on Dynamo icon to open the Dynamo window and press on New project or Open an old project.
You can add node by typing in the left area named Library editbox the name o the node and click when is find it.
For example, type Watch and then double click to add to the work area.
Dynamo use Python version 2.7.8 and scripts works with python modules and Dynamo node.
For example, I add some node's to the working area and I link to see how these works using the click and drag mouse features.
If you want to test the python scripting issue, then use the editbox and type Python Script.
Use double click to add to working area.
Search again the Watch node and add it.
To see the editor , use right click on Python Script and select the Edit... .
Now, you can have a image like this with an editor and a node named Python Script and a node named Watch
Link the Python Script node with OUT with Watch, by click on OUT , drag with the mouse and then click on > input from Watch.

The script from the Python Script is this:
# Load the Python Standard and DesignScript Libraries
import sys
import clr
clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *

# The inputs to this node will be stored as a list in the IN variables.
dataEnteringNode = IN

# Place your code below this line

# Assign your output to the OUT variable.
OUT = 0

import time
from StringIO import StringIO
output = StringIO()
sys.stdout = output
t1 = time.time()
duration = time.time() - t1
print('Finished in {} seconds'.format(duration))
OUT = output.getvalue()

# let's the time module
print("Python version")
print (sys.version)
print("Version info.")
print (sys.version_info)

OUT = output.getvalue()
Press on Run button and you will see the result on Watch node: Finished in 0.0 seconds ...

Tuesday, March 17, 2020

Python 3.6.9 : My colab tutorials - part 003.

This tutorial refers to a python module named cirq.
The documentation of this python module can be found on this website.
The development team comes with this intro:
Cirq is a software library for writing, manipulating, and optimizing quantum circuits and then running them against quantum computers and simulators. Cirq attempts to expose the details of hardware, instead of abstracting them away, because, in the Noisy Intermediate-Scale Quantum (NISQ) regime, these details determine whether or not it is possible to execute a circuit at all.
I try to install on Ubuntu but not work:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
...
$ pip3 install cirq --user
Requirement already satisfied: cirq in ./.local/lib/python3.5/site-packages (0.5.556)
...
$ python3 -c 'import cirq; print(cirq.google.Foxtail)'
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named 'cirq'
I use it with colab notebook and works very well and I add some basic information about quantum computing with a few examples of this python module.
See my GitHub account with the catafest_004.ipynb notebook example and basic pieces of information.
A very short intro into quantum computing area can be found on this video:

Monday, March 16, 2020

Python 3.5.2 : Detect motion and save images with opencv.

This script is simple to use it with a webcam or to parse a video file.
The main goal of this script is to see the difference in various frames of a video or webcam output.
The first frame of our video file will contain no motion and just background and then is compute the absolute difference.
There is no need to process the large, raw images straight from the video stream and this is the reason I convert the image to grayscale.
Some text is put on the window to show us the status string to indicate it is detection.
With this script I detect cars and peoples from my window, see the screenshot with these files:

Let's see the python script:
import argparse
import datetime
import imutils

import cv2

import time
from time import sleep

def saveJpgImage(frame):
    #process image
    img_name = "opencv_frame_{}.jpg".format(time)
    cv2.imwrite(img_name, frame)

def savePngImage():
    #process image
    img_name = "opencv_frame_{}.png".format(time)
    cv2.imwrite(img_name, frame)

# get argument parse
ap = argparse.ArgumentParser()
ap.add_argument("-v", "--video", help="path to the video file")
ap.add_argument("-s", "--size", type=int, default=480, help="minimum area size , default 480")
args = vars(ap.parse_args())

# if no video use webcam
if args.get("video", None) is None:
    camera = cv2.VideoCapture(0)
    #time.sleep(1.5)

# use video file
else:
    camera = cv2.VideoCapture(args["video"])


# frame from video is none 
first_frame = None

# loop into frames of the video
while True:
    # grab the current frame 
    (grabbed, frame) = camera.read()
    text = "undetected"

    # is no frame grabbed the is end of video 
    if not grabbed:
        break

    # resize the frame 
    frame = imutils.resize(frame, width=640)
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    gray = cv2.GaussianBlur(gray, (21, 21), 0)

    # is first frame is none , make gray 
    if first_frame is None:
        first_frame = gray
        continue


    # compute difference from current frame and first frame 
    frameDelta = cv2.absdiff(first_frame, gray)
    first_frame = gray
    thresh = cv2.threshold(frameDelta, 1, 255, cv2.THRESH_BINARY)[1]

    # dilate the thresholded image to fill in holes
    # then find contours on thresholded image
    thresh = cv2.dilate(thresh, None, iterations=2)
    (cnts, _) = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL,
                                 cv2.CHAIN_APPROX_SIMPLE)

    # loop contours 
    for c in cnts:
        # if the contour is too small, ignore it
        if cv2.contourArea(c) < args["size"]:
            continue

        # compute the bounding box for the contour
        # draw it on the frame,
        # and update the text
        (x, y, w, h) = cv2.boundingRect(c)
        cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 0)
        roi = frame[y:y+h, x:x+w]
        ts = time.time()
        st = datetime.datetime.fromtimestamp(ts).strftime('%d-%m-%Y_%H-%M-%S')
        # if the detection is on sized then save the image 
        if (w > h ) and (y + h) > 50 and (y + h) < 550:
            cv2.imwrite(st+"opencv.jpg", roi)
        # set text to show on gui 
        text = "detected"
    
    # draw the text and timestamp on the frame
    cv2.putText(frame, "Detect: {}".format(text), (10, 20),
                cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
    cv2.putText(frame, datetime.datetime.now().strftime("%A %d %B %Y %I:%M:%S%p"),
                (10, frame.shape[0] - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.35, (0, 0, 255), 1)

    #show frame , thresh and frame_Delta
    cv2.imshow("Security Feed", frame)
    cv2.imshow("Thresh", thresh)
    cv2.imshow("Frame Delta", frameDelta)
    key = cv2.waitKey(1) &  0xFF

    # break from loop with q key 
    if key == ord("q"):
        break

# close camera and windows 
camera.release()
cv2.destroyAllWindows()

Wednesday, March 4, 2020

Python 3.6.9 : My colab tutorials - part 002.

This is another notebook with the Altair python package.
The development team comes with this intro:
Altair is a declarative statistical visualization library for Python, based on Vega and Vega-Lite.

Altair offers a powerful and concise visualization grammar that enables you to build a wide range of statistical visualizations quickly. Here is an example of using the Altair API to quickly visualize a dataset with an interactive scatter plot:

See the notebook at my GitHub account.

Sunday, March 1, 2020

Python 3.6.9 : My colab tutorials - part 001.

Today I start this tutorials series for the Colab tool.
To share my working with the Colab tool I created this GitHub project.
This project has two colab files :
catafest_001.ipynb  Created using Colaboratory  
catafest_002.ipynb  Created using Colaboratory 
First colab notebook come with a simple tutorial.
The next colab notebook is a little bit more complex and shares more information about how can deal with simples tasks on colab.
This is the table of contests for this colab notebook:
  • Table of contents
  • Select GPU for this notebook
  • Check with nvidia-smi
  • Check whether you have a visible GPU
  • Check with tensoflow test
  • Read information about hardware
  • Check cpuinfo
  • Check meminfo
  • Use Linux commands
  • Use python modules torch and fastai
  • Use python modules
  • Show and get information
  • Enter credentials with Username and Password:
  • Datatime fields
  • Raw fields
  • Number fields
  • Boolean fields
  • Pandas data fields
  • Upload files
  • Upload local files
  • Use the Jupyter Widgets

Python Qt5 : Create a spectrum equalizer.

I haven't written much for a while on these issues about python and PyQt5.
Today I will show a complex example of QtMultimedia and how to create a spectrum equalizer.
First, the PyQt5 bindings come with this python module named QtMultimedia.
The main reason was the lack of time and focus of my effort on more stringent elements of my life.
Let's start with the few lines of source code that show us how can use this python module.
[mythcat@desk ~]$ python3 
Python 3.7.6 (default, Jan 30 2020, 09:44:41) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import PyQt5
>>> from PyQt5.QtMultimedia import *
>>> dir(PyQt5.QtMultimedia)
['QAbstractVideoBuffer', 'QAbstractVideoFilter', 'QAbstractVideoSurface', 'QAudio', 'QAudioBuffer',
 'QAudioDecoder', 'QAudioDeviceInfo', 'QAudioEncoderSettings', 'QAudioFormat', 'QAudioInput', 'QAudioOutput',
 'QAudioProbe', 'QAudioRecorder', 'QCamera', 'QCameraExposure', 'QCameraFocus', 'QCameraFocusZone',
 'QCameraImageCapture', 'QCameraImageProcessing', 'QCameraInfo', 'QCameraViewfinderSettings', 
 'QImageEncoderSettings','QMediaBindableInterface', 'QMediaContent', 'QMediaControl', 'QMediaMetaData', 
 'QMediaObject', 'QMediaPlayer', 'QMediaPlaylist', 'QMediaRecorder', 'QMediaResource', 'QMediaService', 
 'QMediaTimeInterval', 'QMediaTimeRange', 'QMultimedia', 'QRadioData', 'QRadioTuner', 'QSound', 'QSoundEffect',
 'QVideoEncoderSettings', 'QVideoFilterRunnable', 'QVideoFrame', 'QVideoProbe', 'QVideoSurfaceFormat',
 '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']
The next issue is creating a random equalizer effect with bars.

This source doesn't use the QtMultimedia but can be implemented in the random area of equalizer set by update_values function:
import random
import sys
from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtCore import Qt
from PyQt5.QtCore import pyqtSignal as Signal

print("load all modules!")

class Equalizer_Bars(QtWidgets.QWidget):

    def __init__(self, bars, steps, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # this fit the size to equalizer bars area
        self.setSizePolicy(
            QtWidgets.QSizePolicy.MinimumExpanding,
            QtWidgets.QSizePolicy.MinimumExpanding
        )
        # set the size of the window
        self.setMinimumHeight(480)
        self.setMinimumWidth(640)
        
        if isinstance(steps, list):
            # list of colours.
            self.n_steps = len(steps)
            self.steps = steps

        elif isinstance(steps, int):
            # int number of bars, defaults to blue.
            self.n_steps = steps
            self.steps = ['blue'] * steps
        else:
            raise TypeError('steps variable set wrong, try with a list or int')
        
        # create bars
        self.set_bars = bars
        # define settings of bars
        self.set_space_between_bars = 1
        self.set_height_of_bars = 1
        # set color of background
        self.set_background_color = QtGui.QColor('white')
        # set padding by pixel .
        self.set_padding = 10  

        # create timer 
        self.set_timer = None
        # set decay for move bars 
        self.set_decay_frequency_ms(76)
        self.set_decay_value= 10

        # set values for minim and maxim 
        self.set_min_value = 0
        self.set_max_value = 100

        # store all current values in a list.
        self.set_all_values= [0.0] * bars


    def paintEvent(self, e):
        # create a painter
        painter = QtGui.QPainter(self)
        # create a brush for drawing 
        brush = QtGui.QBrush()
        brush.setColor(self.set_background_color)
        brush.setStyle(Qt.SolidPattern)
        rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height())
        painter.fillRect(rect, brush)
        
        # set the canvas.
        d_height = painter.device().height() - (self.set_padding * 2)
        d_width = painter.device().width() - (self.set_padding * 2)

        # set the bars.
        step_y = d_height / self.n_steps
        bar_height = step_y * self.set_height_of_bars
        bar_height_space = step_y * (1 - self.set_space_between_bars) / 2

        step_x = d_width / self.set_bars
        bar_width = step_x * self.set_space_between_bars
        bar_width_space = step_x * (1 - self.set_height_of_bars) / 2

        for i in range(self.set_bars):

            # calculating the y position for this bar for stop, from the value in range.
            c = (self.set_all_values[i] - self.set_min_value) / (self.set_max_value - self.set_min_value)
            n_steps_to_draw = int(c * self.n_steps)

            for n in range(n_steps_to_draw):
                brush.setColor(QtGui.QColor(self.steps[n]))
                rect = QtCore.QRect(
                    self.set_padding + (step_x * i) + bar_width_space,
                    self.set_padding + d_height - ((1 + n) * step_y) + bar_height_space,
                    bar_width,
                    bar_height
                )
                painter.fillRect(rect, brush)

        painter.end()

    def _trigger_refresh(self):
        self.update()

    def set_decay_trick(self, f):
        self.set_decay_value= float(f)

    def set_decay_frequency_ms(self, ms):
        if self.set_timer:
            self.set_timer.stop()

        if ms:
            self.set_timer = QtCore.QTimer()
            self.set_timer.setInterval(ms)
            self.set_timer.timeout.connect(self.set_decay_beat)
            self.set_timer.start()

    def set_decay_beat(self):
        self.set_all_values= [
            max(0, v - self.set_decay_value)
            for v in self.set_all_values
        ]
        self.update()  # Redraw new position.

    def size_values(self, v):
        self.set_all_values= v
        self.update()

    def values(self):
        return self.set_all_values

    def set_range(self, vmin, vmax):
        assert float(vmin) < float(vmax)
        self.set_min_value, self.set_max_value = float(vmin), float(vmax)
    # will be used with QBrush 
    def setColor(self, color):
        self.steps = [color] * self._bar.n_steps
        self.update()

    def set_color_bars(self, colors):
        self.n_steps = len(colors)
        self.steps = colors
        self.update()

    def set_bar_padding(self, i):
        self.set_padding = int(i)
        self.update()

    def set_bar_solid_percent(self, f):
        self._bar_solid_percent = float(f)
        self.update()

    def set_background_color(self, color):
        self.set_background_color = QtGui.QColor(color)
        self.update()


class Window(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.equalizer = Equalizer_Bars(8, \
        ['#00405e', '#3350a1', '#386787','#0088ba','#3396e6', '#00b9d0','#c1f9f9', '#cce2f7'])

        self.setCentralWidget(self.equalizer)

        self.set_timer = QtCore.QTimer()
        self.set_timer.setInterval(100)
        self.set_timer.timeout.connect(self.update_values)
        self.set_timer.start()

    def update_values(self):
        self.equalizer.size_values([
            min(100, i+random.randint(0, 500) if random.randint(0, 10) > 2 else i)
            for i in self.equalizer.values()
            ])
            
# start the program 
app = QtWidgets.QApplication([])
w = Window()
w.show()
app.exec_()

Thursday, February 27, 2020

Python 3.6.9 : Google give a new tool for python users.

Today I discovered a real surprise gift made by the team from Google for the evolution of programmers.
I say this because not everyone can afford hardware resources.
This gift is a new tool called Colab and uses these versions of python and sys:
Python version
3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0]
Version info.
sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0)
This utility allows you to run source code that requires online hardware resources using your google account.
Colab allows you to use and share Jupyter notebooks because is an open-source project on which Colab is based.
The types of GPUs that are available in Colab varies over time.
This is necessary for Colab to be able to provide access to these resources for free.
The GPUs available in Colab often include Nvidia K80s, T4s, P4s, and P100s.
This way you can test demanding modules like the python TensorFlow module.
The utility is free but you can pay extra for more hardware resources.
Colab notebooks are stored in Google Drive, or can be loaded from GitHub.
You can see a simple intro with a notebook on my GitHub account.

Tuesday, February 25, 2020

Python 3.7.6 : The new concepts of execution in python 3 - part 001.

The main goal of these tutorials series is learning to deal with python source code using the new concepts of execution in python 3.
When two or more events are concurrent it means that they are happening at the same time.
Concurrent programming is not equivalent to parallel execution.
In computing, concurrency is the execution of pieces of work or tasks by a computer at the same time.
Concurrency is a property which more than one operation can be run simultaneously.
When multiple computations or operations are carried out at the same time or in parallel with the goal of speeding up the computation process then this process is named parallelism.
Parallelism is a property which operations are actually being run simultaneously using the multiprocessing.
Multiprocessing, on the other hand, involves utilizing two or more processor units on a computer to achieve parallelism.
Multithreading is a property that refers to the ability of a CPU to execute multiple threads concurrently.
Python’s concurrency methods including threading, multiprocessing, and asyncio.
The difference between the threading and multiprocessing is this: the threading module uses threads, the multiprocessing module uses processes.
The threading is the package that provides API to create and manage threads.
With multiprocessing, Python creates new processes using an API similar to the threading module.
The asyncio is a library to write concurrent code using the async/await syntax.
The keyword async indicates that our function is a coroutine meaning they choose when to pause and let others execute and run coroutines multitask cooperatively.
The three fundamental advantages of async and await over threads are:
  • cooperative multitasking - you can reasonably have millions of concurrent tasks;
  • using await makes visible where the schedule points;
  • if a task doesn’t yield then it can accidentally block all other tasks from running;
  • tasks can support cancellation.
The next source of code show us how can deal with the execution code in Python 3 using the threading and multiprocessing python packages.
The timeit python package is used to benchmark the code write in the code_to_test variable:
Let's test the multi-threading execution with python:
[mythcat@desk ~]$ python3 
Python 3.7.6 (default, Jan 30 2020, 09:44:41) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> code_to_test = """
... import threading
... 
... text = "Hello World"
... 
... def print_text(text):
...     for char in text:
...             print (char)
... 
... # multi-threading execution
... def multi_threads():
...     thread_1 = threading.Thread(target=print_text(text))
...     thread_2 = threading.Thread(target=print_text(text))
...     thread_1.start()
...     thread_2.start()
...     thread_1.join()
...     thread_2.join()
... multi_threads()
... """
>>> 
>>> elapsed_time = timeit.timeit(code_to_test, number=1)
H
e
l
l
o
 
W
o
r
l
d
H
e
l
l
o
 
W
o
r
l
d
>>> print(elapsed_time)
0.010613240000566293
Let's test the serially execution with python:
[mythcat@desk ~]$ python3 
Python 3.7.6 (default, Jan 30 2020, 09:44:41) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> code_to_test = """
... import threading
... 
... text = "Hello World"
... 
... def print_text(text):
...     for char in text:
...             print (char)
... # serially execution
... def serially():
...     print_text(text)
...     print_text(text)
... serially()
... """
>>> elapsed_time = timeit.timeit(code_to_test, number=1)
H
e
l
l
o
 
W
o
r
l
d
H
e
l
l
o
 
W
o
r
l
d
>>> print(elapsed_time)
0.011771811000471644
Let's test the multiprocessing execution with python:
[mythcat@desk ~]$ python3 
Python 3.7.6 (default, Jan 30 2020, 09:44:41) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> code_to_test = """
... import multiprocessing
... 
... text = "Hello World"
... 
... def print_text(text):
...     for char in text:
...             print (char)
... 
... # multiprocessing execution
... def multiprocessing_test():
...      process_1 = multiprocessing.Process(target=print_text(text))
...      process_2 = multiprocessing.Process(target=print_text(text))
...      process_1.start()
...      process_2.start()
...      process_1.join()
...      process_2.join()
... multiprocessing_test()
... """
>>> elapsed_time = timeit.timeit(code_to_test, number=1)
H
e
l
l
o
 
W
o
r
l
d
H
e
l
l
o
 
W
o
r
l
d
>>> print(elapsed_time)
0.3649730779998208
Since asyncio is a little complex, I will write about this in the next tutorial.

Sunday, February 23, 2020

Python 3.7.6 : The SELinux python package.

The tutorial for today is about the SELinux python package.
The official webpage is this.
First, I update my pip tool and I used the python 3.7.6 version:
[mythcat@desk ~]$ pip install --upgrade pip --user
...
Successfully installed pip-20.0.2
Let's install the python package named selinux:
[mythcat@desk ~]$ pip3 install selinux --user
...
Requirement already satisfied: selinux in /usr/lib64/python3.7/site-packages (2.9)
Let's test it:
[mythcat@desk ~]$ python3 
Python 3.7.6 (default, Jan 30 2020, 09:44:41) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import selinux 
>>> from selinux import * 
>>> print(dir(selinux))
Now, I can test my SELinux with this python package:
>>> selinux.is_selinux_enabled()
1
>>> selinux.selinux_getenforcemode()
[0, 0]
>>> mode = selinux.security_getenforce()
>>> mode
0
>>> selinux.selinux_getpolicytype()
[0, 'mls']
>>> selinux.is_selinux_enabled()
1
>>> selinux.is_selinux_mls_enabled()
1
>>> selinux.get_default_context('mythcat','user_u:object_r:user_home_t:s0')
[-1, None]
>>> context = selinux.matchpathcon(os.path.normpath(/), 0)[1]
>>> context
'system_u:object_r:root_t:s0'
>>> selinux.restorecon('/')
>>> context_t = selinux.getfilecon('/')[1].split(":")[2]
>>> context_t 
'root_t'
>>> context_t = selinux.getfilecon('/home/mythcat')[1].split(":")[2]
>>> context_t 
'user_home_dir_t'
>>> selinux.matchpathcon('/', mode)
[0, 'system_u:object_r:root_t:s0']
>>> selinux.matchpathcon('/home/mythcat', mode)
[0, 'user_u:object_r:user_home_dir_t:s0']
>>> file_context=selinux.lgetfilecon('/home/mythcat/sel_001.py')
>>> file_context
[31, 'user_u:object_r:user_home_t:s0']
>>> selinux.security_policyvers()
32

Saturday, February 22, 2020

Python 3.7.5 : This python package can work with ArcGIS platform.

This python package is named like the ArcGIS platform and can be used for spatial analysis, mapping, and GIS.
The ArcGIS package uses the ArcGIS platform for organizations to create, manage, share, and analyze spatial data.
This platform has a server component, mobile and desktop applications, and developer tools.
[mythcat@desk projects]$ pip3 install arcgis --user
...
You can test it on your workstation or live using the notebooks esri.com, see the next screenshot:

Let's search all maps from world with the maps of disaster using the notebook from esri website.
from arcgis.gis import GIS
from arcgis.geocoding import geocode
from IPython.display import display
from arcgis.mapping import WebMap
from arcgis.geoprocessing import import_toolbox
gis = GIS()
map1 = gis.map()

webmap_search = gis.content.search("disaster", item_type="Web Map")
webmap_search
The result will be this list:
[<Item title:"Hurricane and Cyclone Web Map" type:Web Map owner:cfernerDRP>, 
<Item title:"Severe Weather Web Map" type:Web Map owner:cfernerDRP>, 
<Item title:"NYC OEM Hurricane Evacuation Zones" type:Web Map owner:nfurness>, 
<Item title:"County of San Diego Emergency Map (Live)" type:Web Map owner:oes_services>, 
<Item title:"Zombie US" type:Web Map owner:wjones_mt>, 
<Item title:"2009 Social Vulnerability in the United States (Mature Support)" type:Web Map owner:esri>, 
<Item title:"Waldo Canyon Fire - Consolidated Public Information" type:Web Map owner:jpfeffer>, 
<Item title:"US Wildfire Activity Web Map" type:Web Map owner:cfernerDRP>, 
<Item title:"Earthquake Web Map" type:Web Map owner:cfernerDRP>, 
<Item title:"Current Weather and Wind Station Information" type:Web Map owner:esri_livefeeds>]
Now you can show each map by usinh the index:
map_one = webmap_search[1]
map_one

Wednesday, February 19, 2020

Python 3.7.5 : The PyQtChart from python Qt5.

The PyQtChart is a set of Python bindings for The Qt Company’s Qt Charts library and is implemented as a single module.
Let's install this python package with the pip3 tool:
[mythcat@desk ~]$ pip3 install PyQtChart --user
...
Installing collected packages: PyQtChart
Successfully installed PyQtChart-5.14.0
Let's test with a simple example:
from PyQt5.QtWidgets import QApplication, QMainWindow
import sys
from PyQt5.QtChart import QChart, QChartView, QPieSeries, QPieSlice
from PyQt5.QtGui import QPainter, QPen
from PyQt5.QtCore import Qt

class Window(QMainWindow):
    def __init__(self):
        super().__init__()

        self.setWindowTitle("testing Pie Chart")
        self.setGeometry(100,100, 640,480)
        self.show()
        self.create_piechart()

    def create_piechart(self):

        series = QPieSeries()
        #append all values with a sum of 360 
        series.append("size 5", 5)
        series.append("size 10", 10)
        series.append("size 30", 30)
        series.append("size 45", 45)
        series.append("size 90", 90)
        series.append("size 180", 180)

        #adding slice 
        slice = QPieSlice()
        slice = series.slices()[2]
        slice.setExploded(True)
        slice.setLabelVisible(True)
        slice.setPen(QPen(Qt.darkGreen, 1))
        slice.setBrush(Qt.green)
        slice = series.slices()[4]
        slice.setExploded(False)
        slice.setLabelVisible(True)
        slice.setPen(QPen(Qt.red, 1))
        #slice.setBrush(Qt.blue)

        #create chart 
        chart = QChart()
        #chart.legend().hide()
        chart.addSeries(series)
        chart.createDefaultAxes()
        chart.setAnimationOptions(QChart.SeriesAnimations)
        chart.setTitle("The all 360 on  chart .")

        chart.legend().setVisible(True)
        chart.legend().setAlignment(Qt.AlignBottom)

        chartview = QChartView(chart)
        chartview.setRenderHint(QPainter.Antialiasing)

        self.setCentralWidget(chartview)

App = QApplication(sys.argv)
window = Window()
sys.exit(App.exec_())
The result can be seen in the next image:

Sunday, February 16, 2020

Python 3.7.5 : The httpx python package.

Today I will present a new python packet that can help you in developing web applications.
This is the next generation HTTP client for Python and is named httpx.
This python package comes with a nice logo: a butterfly.
The official webpage can be found at this webpage.
The development team come with this intro:
HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.
I install it on my Fedora 31 distro with the pip3 tool.
[mythcat@desk ~]$ pip3 install httpx --user
...
Successfully installed h11-0.9.0 h2-3.2.0 hpack-3.0.0 hstspreload-2020.2.15 httpx-0.11.1 
hyperframe-5.2.0 rfc3986-1.3.2 sniffio-1.1.0
With this python package, you can build a simple application with https API, migrate an application that uses web requests to make HTTP call.
use it as a test client for your web project, build a web spider and much more.
All HTTP methods get, post, patch, put, delete are implemented as coroutines in httpx python package and supports HTTP/2.
Let's start with few lines of source code:
[mythcat@desk ~]$ python3 
Python 3.7.6 (default, Jan 30 2020, 09:44:41) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import httpx
>>> out = httpx.get('https://www.google.com')
>>> out.status_code
200
>>> out.headers['content-type']
'text/html; charset=ISO-8859-1'
>>> out.text[:76]
'<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang='
>>> out.http_version
'HTTP/1.1'
>>> with httpx.Client() as client:
...     out_client = client.get('https://www.google.com')
... 
>>> out_client.http_version
'HTTP/1.1'
You can use asynchronous call to bost your python application using httpx python package.
I will make a tutorial about asyncio in the future.
For example, the website named example.com can return json data and using the next source code I can get it using the asyncio package.
import httpx
import asyncio
from typing import Dict
async def get_web() -> Dict:
    resp = await httpx.get("https://example.com/")
    if resp.status_code == httpx.codes.OK:
        return resp.json()
if __name__ == '__main__':   
    comments = asyncio.run(get_web())
Use async and await when you have to execute HTTP calls with httpx because is fully Requests compatible async HTTP Client.
The HTTPX comes with a lot of features, see the official GitHub webpage.

Saturday, February 15, 2020

Python 3.7.5 : Use Brython in web development to avoid javascript.

The tutorial for today is about how can avoid the javascript and use python script in webdevelopment using the Brython.
Brython's goal is to replace Javascript with Python, as the scripting language for web browsers. see the official webpage.
It is necessary to include brython.js and to run the brython() function upon page load using the onload attribute of the BODY tag.
You can use python language in the HTML file or you can write it in a separate file, and to load it using the src attribute of the script tag:
<html>
<head>
<script src="/brython.js"></script>
</head>
<body onload="brython()">
<script type="text/python" src="test.py"></script>
<input id="zone" autocomplete="off">
<button id="mybutton">click!</button>
</body>
</html>
Let's see one simple example with one edit and button bind
<html>
<head>
<script src="/brython.js"></script>
</head>
<body onload="brython()">
<script type="text/python">
from browser import document, alert
def echo(ev):
 alert(document["zone"].value)
document["mybutton"].bind("click", echo)
</script>
<input id="zone">
<button id="mybutton">click !</button>
</body>
</html>
When I click on the button, the onclick event calls and run the echo() function and gets the value of the INPUT element, through its id named zone and show the message.
Is fast and works great with javascripts libraries, see the example with THREE demo.
from browser import document, window

THREE = window.THREE

camera = THREE.PerspectiveCamera.new(75, 1, 1, 10000)
camera.position.z = 1000
scene = THREE.Scene.new()
geometry = THREE.CubeGeometry.new(200, 200, 200)
material = THREE.MeshBasicMaterial.new({"color": "#ff0000", "wireframe": True})
mesh = THREE.Mesh.new(geometry, material)
scene.add(mesh)

renderer = THREE.WebGLRenderer.new()
renderer.setSize(444, 444)

document <= renderer.domElement
renderer.render(scene, camera)

def animate(i):
    # note: three.js includes requestAnimationFrame shim
    window.requestAnimationFrame(animate)

    mesh.rotation.x += 0.01
    mesh.rotation.y += 0.02

    renderer.render(scene, camera)   

animate(0)

Tuesday, February 11, 2020

Python 3.7.5 : Using the hug framework - part 001.

Today I will come with another tutorial series about the hug framework.
The hug framework is on the top 3 performing web frameworks for Python and comes with the cleanest way to create HTTP REST APIs on Python 3.
The official webpage can be found hug web page with a good area for learn.
Let's install this python package.
[mythcat@desk projects]$ mkdir hug_001
[mythcat@desk projects]$ cd hug_001/
[mythcat@desk hug_001]$ pip3 install hug --user
...
Successfully installed hug-2.6.1
The Hug is a framework that allows you to expose a single code in several ways:
  • a local package;
  • as an A.P.I.;
  • as a C.L.I.;
Let's create an app.py python script to see all of these options:
import hug

# local pkg 
@hug.local()
def get_products(product:hug.types.text):
        """Get product name"""
        return {"product":product.upper()}

# API
@hug.get()

# CLI
@hug.cli()
To test it I used this source code:
import hug

# local pkg 
@hug.get()
@hug.local()
def get_products(product:hug.types.text):
        """Get product name"""
        return {"product":product.upper()}
The rerver can be run with this command:
[mythcat@desk hug_001]$ hug -f app.py

/#######################################################################\
          `.----``..-------..``.----.
         :/:::::--:---------:--::::://.
        .+::::----##/-/oo+:-##----:::://
        `//::-------/oosoo-------::://.       ##    ##  ##    ##    #####
          .-:------./++o/o-.------::-`   ```  ##    ##  ##    ##  ##
             `----.-./+o+:..----.     `.:///. ########  ##    ## ##
   ```        `----.-::::::------  `.-:::://. ##    ##  ##    ## ##   ####
  ://::--.``` -:``...-----...` `:--::::::-.`  ##    ##  ##   ##   ##    ##
  :/:::::::::-:-     `````      .:::::-.`     ##    ##    ####     ######
   ``.--:::::::.                .:::.`
         ``..::.                .::         EMBRACE THE APIs OF THE FUTURE
             ::-                .:-
             -::`               ::-                   VERSION 2.6.1
             `::-              -::`
              -::-`           -::-
\########################################################################/

 Copyright (C) 2016 Timothy Edmund Crosley
 Under the MIT License


Serving on :8000...
OPen the browser with the http://localhost:8000/ or http://localhost:8000/products:
If you want to use the CLI then you need to make these changes in the source code:
import hug
# cli 
@hug.cli()
# local pkg 
@hug.get('/products')
@hug.local()
def get_products(product:hug.types.text):
        """Get product name"""
        return {"product":product.upper()}
# the main CLI
if __name__ == '__main__':
        get_products.interface.cli() 
I can see the help area from python:
[mythcat@desk hug_001]$ hug -f app.py -c help
app

Available Commands:

 - get_products: Get product name 
The hug help show the all arguments for use:
[mythcat@desk hug_001]$ hug --help
usage: hug [-h] [-v] [-f FILE] [-m MODULE] [-ho HOST] [-p PORT] [-n] [-ma]
           [-i INTERVAL] [-c COMMAND] [-s]

Hug API Development Server

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -f FILE, --file FILE  file
  -m MODULE, --module MODULE
                        module
  -ho HOST, --host HOST
                        host
  -p PORT, --port PORT  A whole number
  -n, --no_404_documentation
                        Providing any value will set this to true
  -ma, --manual_reload  Providing any value will set this to true
  -i INTERVAL, --interval INTERVAL
                        A whole number
  -c COMMAND, --command COMMAND
                        command
  -s, --silent          Providing any value will set this to true 

Wednesday, January 22, 2020

Python 3.7.5 : Django security issues - part 003.

Let's update this subject today with another new tutorial.
In the last tutorial about Django security I wrote about python package named django-axes.
First, let's fix an old issue about a URL pattern that matches anything and expects an integer that generates errors like:
...
  File "/home/mythcat/.local/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 1772, 
in get_prep_value
    ) from e
ValueError: Field 'id' expected a number but got 'favicon.ico'.
[22/Jan/2020 21:50:06] "GET /favicon.ico/ HTTP/1.1" 500 130547
Now, let's start my project:
[mythcat@desk ~]$ cd projects/
[mythcat@desk projects]$ cd django/
[mythcat@desk django]$ source env/bin/activate
Create a new folder named static in the test001 folder and add a icon file named favicon.ico.
(env) [mythcat@desk django]$ cd mysite/test001/
(env) [mythcat@desk test001]$ mkdir static 
In the settings.py file you need to have this source code:

STATIC_URL = '/static/'

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, "static"),
]
Change in the urls.py this line of source code to fix the error:
path('<int:author_id>/',views.index_next, name = 'index_next'),
Let's run the Django project server with:
(env) [mythcat@desk django]$ cd mysite/
(env) [mythcat@desk mysite]$ python3 manage.py runserver
I login into my admin area with user catalin and password adminadmin.
If you try to login with a bad password then the account is locked by django-axes python package.
Use this command to reset all lockouts and access records.
(env) [mythcat@desk mysite]$ python3 manage.py axes_reset
No attempts found.
Into admin area you can see the AXES area with Access attempts and Access logs.
Axes listens to the following signals from django.contrib.auth.signals to log access attempts.
In this case Axes lockout responses on failed user authentication attempts from login views.
The Access logs shows access log, see examples:
Jan. 22, 2020, 8:46 p.m.-127.0.0.1catalinMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36/admin/login/
Jan. 21, 2020, 6:42 p.m.Jan. 22, 2020, 8:45 p.m.127.0.0.1catalinMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36/admin/login/
You can set the axes into settings.py file , see this link.


Tuesday, January 21, 2020

Python 3.7.5 : Use Django Formsets.

Django Formsets manage the complexity of multiple copies of a form in a view.
This simplifies the task of creating a formset for a form that handles multiple instances of a model.
Let's start with my old project:
[mythcat@desk ~]$ cd projects/
[mythcat@desk projects]$ cd django/
[mythcat@desk django]$ source env/bin/activate
Into models.py I add these classes:
#create Inline Form with book and author
class Author(models.Model):
    author_book = models.CharField(max_length = 100)
    def __str__(self):
        return self.author_book

class Book(models.Model):
    book_name = models.CharField(max_length = 100)
    author_book_name = models.ForeignKey(Author,on_delete=models.CASCADE)
    def __str__(self):
        return self.book_name

(env) [mythcat@desk mysite]$ python3 manage.py makemigrations
Migrations for 'test001':
  test001/migrations/0004_author_book.py
    - Create model Author
    - Create model Book
(env) [mythcat@desk mysite]$ python3 manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, axes, contenttypes, sessions, test001
Running migrations:
  Applying test001.0004_author_book... OK
Add the classes to admin.py:
from .models import Author, Book
...
admin.site.register(Author)
admin.site.register(Book)
Now you can log in into the admin area and add authors and then add books.

Add the source code to views.py:
...
# add to views.py Author and Book 
from .models import Author, Book
# Author , Book redirect 
from django.shortcuts import redirect
...
# author and book source code 
def index_next(request, author_id):
    author = Author.objects.get(pk=author_id)
    BookFormset = inlineformset_factory(Author,Book, fields=('book_name',))
    if request.method == 'POST':
        formset = BookFormset(request.POST,instance = author)
        if formset.is_valid():
            formset.save()
            return redirect('index_next',author_id = author_id)
    formset = BookFormset(instance = author)
    return render(request, 'index_next.html', {'formset': formset})
...
Let's fix the URL for the next step.
Add the source code to urls.py:
...
# add index_next to urls.py 
from test001.views import index_next
...
urlpatterns = [
    ...
    path('&lt author_id &gt',views.index_next, name = 'index_next'),
    ...
    ]
Add index_next.html file into the template folder and into this file write HTML5 with a form and one submit button.
In the form tag add this:
{{ formset.as_p }}
Run the runserver command:
(env) [mythcat@desk mysite]$ python3 manage.py runserver
Use this http://127.0.0.1:8000/1 to see the first on the database shown on the browser.
You can customize the output of inline form, see source code:
...
BookFormset = inlineformset_factory(Author,Book, fields=('book_name',), can_delete=False, extra=1)
...
See the full project on my GitHub account at django_chart project repo.

Monday, January 20, 2020

Python 3.7.5 : Django security issues - part 002.

The project can be found at this Github project.
Let's start with my default project and activate the env:
[mythcat@desk ~]$ cd projects/
[mythcat@desk projects]$ cd django/
[mythcat@desk django]$ source env/bin/activate
Let's install this python module:
(env) [mythcat@desk django]$ pip3 install django-axes --user
Make these changes into settings.py:
(env) [mythcat@desk django]$ cd mysite/
(env) [mythcat@desk mysite]$ ls
db.sqlite3  manage.py  mysite  test001
(env) [mythcat@desk mysite]$ cd mysite/
(env) [mythcat@desk mysite]$ vim settings.py 
Into your settings.py add axes:
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'test001',
    'rest_framework',
    'axes'
] 
Add this source code in settings.py:
AUTHENTICATION_BACKENDS = [
    # AxesBackend should be the first backend in the AUTHENTICATION_BACKENDS list.
    'axes.backends.AxesBackend',

    # Django ModelBackend is the default authentication backend.
    'django.contrib.auth.backends.ModelBackend',
] 
Add axes.middleware.AxesMiddleware into MIDDLEWARE area:
MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'axes.middleware.AxesMiddleware',
] 
Check the configuration with this command:
(env) [mythcat@desk mysite]$ cd ..
(env) [mythcat@desk mysite]$ python manage.py check
System check identified no issues (0 silenced).
Use this command to sync the database:
(env) [mythcat@desk mysite]$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, axes, contenttypes, sessions, test001
Running migrations:
  Applying axes.0001_initial... OK
  Applying axes.0002_auto_20151217_2044... OK
  Applying axes.0003_auto_20160322_0929... OK
  Applying axes.0004_auto_20181024_1538... OK
  Applying axes.0005_remove_accessattempt_trusted... OK
  Applying axes.0006_remove_accesslog_trusted... OK
Once Axes is is installed and configured, you can login and logout of your application via the django.contrib.auth views.
This python package can be integrated with some popular 3rd party packages such as Django Allauth, Django REST Framework, and other tools.
I will come with additional information about this python package in the future.

Saturday, January 18, 2020

Python 3.7.5 : Django security issues - part 001.

Django like any website development and framework implementation requires security settings and configurations.
Today I will present some aspects of this topic and then I will come back with other information.
1. First, check your security vulnerabilities by the following command:
[mythcat@desk django]$ source env/bin/activate
(env) [mythcat@desk django]$ cd mysite
(env) [mythcat@desk mysite]$ python3 manage.py check --deploy
...
  File "/home/mythcat/projects/django/mysite/mysite/settings.py", line 14
    <<<<<<< HEAD
     
This shows us the bad changes in source code, is added by GitHub features.
Let's run it again:
(env) [mythcat@desk mysite]$ python3 manage.py check --deploy
System check identified some issues:

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. 
If your entire site is served only over SSL, you may want to consider setting a value and 
enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling
 HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site 
should be available over both SSL and non-SSL connections, you may want to either set this 
setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes 
it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W016) You have 'django.middleware.csrf.CsrfViewMiddleware' in your MIDDLEWARE, but you have
 not set CSRF_COOKIE_SECURE to True. Using a secure-only CSRF cookie makes it more difficult for network
 traffic sniffers to steal the CSRF token.
?: (security.W018) You should not have DEBUG set to True in deployment.
?: (security.W020) ALLOWED_HOSTS must not be empty in deployment.
?: (security.W022) You have not set the SECURE_REFERRER_POLICY setting. Without this, your site 
will not send a Referrer-Policy header. You should consider enabling this header to protect user privacy.

System check identified 7 issues (0 silenced).
This output show us the security warnning problems.
2. Use the Observatory by Mozilla site to scan the security status of your Django website.
3. Django has built-in security against most forms of CSRF threats, but The CSRF protection cannot protect against man-in-the-middle attacks.
Use HTTPS with HTTP Strict Transport Security by add these lines in your settings.py file.
CSRF_COOKIE_SECURE = True #to avoid transmitting the CSRF cookie over HTTP accidentally.
SESSION_COOKIE_SECURE = True #to avoid transmitting the session cookie over HTTP accidentally.
4. A Cross-site Scripting (XSS) allows an attacker to inject a script into the content of a website or application.
In your settings.py use this:
django.middleware.security.SecurityMiddleware
...
SECURE_BROWSER_XSS_FILTER = True
SECURE_CONTENT_TYPE_NOSNIFF = True
5. You can create fake admin login page using django-admin-honeypot to attempt unauthorized access.
6. Use SSL Redirect on your settings.py file.
SECURE_SSL_REDIRECT = True
7. Add Content Security Policy (CSP) to your Django website with the installed django-csp, add following lines to your settings.py file:
# Content Security Policy
CSP_DEFAULT_SRC = ("'none'", )
CSP_STYLE_SRC = ("'self'", )
CSP_SCRIPT_SRC = ("'self'", )
CSP_IMG_SRC = ("'self'", )
CSP_FONT_SRC = ("'self'", )
# Google Tag Manager or Google Analytics should be allowed in your CSP policy. 
CSP_DEFAULT_SRC = ("'none'", )
CSP_STYLE_SRC = ("'self'", "fonts.googleapis.com", "'sha256-/3kWSXHts8LrwfemLzY9W0tOv5I4eLIhrf0pT8cU0WI='")
CSP_SCRIPT_SRC = ("'self'", "ajax.googleapis.com", "www.googletagmanager.com", "www.google-analytics.com")
CSP_IMG_SRC = ("'self'", "data:", "www.googletagmanager.com", "www.google-analytics.com")
CSP_FONT_SRC = ("'self'", "fonts.gstatic.com")
CSP_CONNECT_SRC = ("'self'", )
CSP_OBJECT_SRC = ("'none'", )
CSP_BASE_URI = ("'none'", )
CSP_FRAME_ANCESTORS = ("'none'", )
CSP_FORM_ACTION = ("'self'", )
CSP_INCLUDE_NONCE_IN = ('script-src',)
The HTTP Strict Transport Security can be set into your settings.py file:
SECURE_HSTS_SECONDS = 86400  # 1 day
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True
8. Use records login attempts to your Django powered site and prevents attackers from brute forcing using the django-axes.
This tutorial solves most of the security issues of any website built with Django and not just related to this framework.




Saturday, January 11, 2020

Python 3.7.5 : About asterisk operators in Python.

The asterisk known as the star operator is used in Python with more than one meaning attached to it.
Today I will show you some simple examples of how can be used.
Let's start with these issues.
You can merge two or more dictionaries by unpacking them in a new one:
>>> a = {'u': 1}
>>> b = {'v': 2}
>>> ab = {**a, **b, 'c':'d'}
>>> ab
{'u': 1, 'v': 2, 'c': 'd'}
Create multiple assignments:
>>> *x, y = 1, 2, 3, 4
>>> x
[1, 2, 3]
You can split into lists and generate sets:
>>> *a, = "Python"
>>> a
['P', 'y', 't', 'h', 'o', 'n']
>>> print(a)
['P', 'y', 't', 'h', 'o', 'n']
>>> *a,
('P', 'y', 't', 'h', 'o', 'n')
>>> zeros = (0,) * 8
>>> print(zeros)
(0, 0, 0, 0, 0, 0, 0, 0)
For numeric data types is used as multiplication operator:
>>> 2 * 3
6
You can use in mathematical function as an exponent:
>>> 3**2
9
You can create a sequences of strings using it like a repetition operator:
>>> t = 'Bye!'
>>> t * 4 
'Bye!Bye!Bye!Bye!'
>>> p = [0,1]
>>> p * 4 
[0, 1, 0, 1, 0, 1, 0, 1]
>>> r = (0,1)
>>> r * 4
(0, 1, 0, 1, 0, 1, 0, 1)
You can used into definition with arguments: *args and **kwargs:
def set_zero(*args):
    result = 0
The *args will give you all function parameters as a tuple.
The **kwargs will give you all keyword arguments except for those corresponding to a formal parameter as a dictionary.
See the PEP 3102 about Keyword-Only Arguments.

Thursday, January 9, 2020

Python 3.7.5 : The this python package.

The this python package is simple to use:
[mythcat@desk ~]$ python3 
Python 3.7.5 (default, Dec 15 2019, 17:54:26) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Monday, January 6, 2020

Python 3.7.5 : Post class and migration process.

Today I will solve some issues with the Django framework like:
  • create a new class for posts;
  • explain how the migration process works.
  • use the database with Django shell;
Let's activate the environment:
[mythcat@desk django]$ source env/bin/activate
I used my old project django-chart, see my old tutorials.
Let's add some source code to the models.py to create a class for the post into my website:
...
# Post data 
from django.utils import timezone
from django.contrib.auth.models import User
...
#create a post
class Post(models.Model):
  title = models.CharField(max_length = 100)
  content = models.TextField()
  date_posted = models.DateTimeField(default = timezone.now)
  #if the user is deleted their posts is deleted 
  author = models.ForeignKey(User, on_delete = models.CASCADE)
Let's use the migration process to update the database:
(env) [mythcat@desk django]$ cd mysite/
(env) [mythcat@desk mysite]$ python3 manage.py makemigrations
Migrations for 'test001':
  test001/migrations/0003_post.py
  - Create model Post
Before to run the migrate command I will show what happened with this migration output from SQL view.
This is great to solve issues and see the exact SQL code generated.
I have in the output of command makemigrations the application name test001 and the migration number 0003.
I will use all of this information output to show how you can see the SQL code:
(env) [mythcat@desk mysite]$ python3 manage.py sqlmigrate test001 0003
BEGIN;
--
-- Create model Post
--
CREATE TABLE "test001_post" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, 
"title" varchar(100) NOT NULL, "content" text NOT NULL, "date_posted" datetime NOT NULL, 
"author_id" integer NOT NULL REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED);
CREATE INDEX "test001_post_author_id_fed29ee6" ON "test001_post" ("author_id");
COMMIT;
This print SQL code that it's going to run when I will use the next command:
(env) [mythcat@desk mysite]$ python3 manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions, test001
Running migrations:
  Applying test001.0003_post... OK
The migration process solves issues linked to changes in the database.
Let's use the Django shell to see the User changes into the database:
(env) [mythcat@desk mysite]$ python3 manage.py shell
Python 3.7.5 (default, Dec 15 2019, 17:54:26) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from test001.models import Post
>>> from django.contrib.auth.models import User
>>> User.objects.all()
...
>>> User.objects.first()
...
>>> User.objects.filter(username='catalin')
...
>>> User.objects.filter(username='catalin').first()
...
>>> user = User.objects.filter(username='catalin').first()
>>> user
...
>>> user.id
1
>>> user.pk
1
>>> Post.objects.all()
...
Let's add a post to the database and show it:
>>> post_1=Post(title='First title', content='The first content!',author=user)
>>> post_1.save()
>>> Post.objects.all()
...
I can see the posts and information about the user with the Django shell.
>>> post = Post.objects.first()
>>> post.content
'The first content!'
>>> post.date_posted
...
>>> post.author
...
>>> post.author.email
...
>>> user.post_set.all()
...
You need to add to views.py the new model changes:
...
from .models import Post
...
def posts(request):
  context = {
  'posts':Post.objects.all()
  }
  return render(request, 'test001/posts.html', context)
...
Into the templates folder, I created the file named posts.html to load the data to HTML5.
{% extends 'base.html' %}
{% block content %}
  {% for post in posts %}
  <article>
  <div>{{ post.title }}</div>
  <div>{{ post.content }}</div>
  <div>{{ post.author }}</div>
  <div>{{ post.date_posted|date:"F d, Y" }}</div>
  </article>
  {% endfor %}
{% endblock content %}
To see these changes into the website I add the route to this HTML5 file posts.html into urls.py file:
app_name = 'test001'
urlpatterns = [
...
    path('posts/',posts, name = 'posts'),
...
]
Now I can run the server and see the output from the database into posts URL.
(env) [mythcat@desk mysite]$ python3 manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
January 06, 2020 - 17:52:29
Django version 3.0.1, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
...
This is the output of my running server...

Python 3.7.5 : Set json with settings in Django project.

[mythcat@desk django]$ source env/bin/activate
(env) [mythcat@desk django]$ cd mysite/
(env) [mythcat@desk mysite]$ ls
db.sqlite3  manage.py  mysite  test001
(env) [mythcat@desk mysite]$ pwd
/home/mythcat/projects/django/mysite
Create a file named config.json in the folder django:
(env) [mythcat@desk mysite]$ vim /home/mythcat/projects/django/config.json
Open your settings.py file from your Django project and copy your secret key from this file to config.json, see:
{ 
  "SECRET_KEY":"your_secret_key_from_settings.py"
}
Change your settings.py file with these changes:
...
import json 
...
with open('/home/mythcat/projects/django/config.json') as config_file:
    config = json.load(config_file)
...
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = config['SECRET_KEY']
This allows us to get the SECRET_KEY from the config.json file.
We can add many variables into file config.json, like EMAIL_USER and EMAIL_PASS.
You need to have a validate JSON file and the settings.py will change from this:
EMAIL_HOST_USER = os.environ.get('EMAIL_USER')
...
to this source code for EMAIL_USER variable:
EMAIL_HOST_USER = config.get('EMAIL_USER')
...

Sunday, January 5, 2020

Python 3.7.5 : Testing cryptography python package - part 001.

There are many python packets that present themselves as useful encryption and decryption solutions. I recommend before you test them, use them and spend time with them to focus on the correct study of cryptology because many disadvantages and problems can arise in the correct and safe writing of the source code.
Today I will show you a simple example with cryptography python package.
Let's install this with pip3 tool:
[mythcat@desk projects]$ pip3 install cryptography --user
You can read more about this python package on the documentation website.
Let's try a simple method of encryption and decryption.
I will show you how to solve this issue with the class cryptography.hazmat.primitives.ciphers.aead.AESGCM:
The AES-GCM construction is composed of the AES block cipher utilizing Galois Counter Mode (GCM).
The GCM (Galois Counter Mode) is a mode of operation for block ciphers.
An AEAD (authenticated encryption with additional data) mode is a type
of block cipher mode that simultaneously encrypts the message as well as authenticating it.

The Value of AESGCM key must be 128, 192, or 256 bits, see the size of the key:
[mythcat@desk projects]$ python3
Python 3.7.5 (default, Dec 15 2019, 17:54:26) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> from cryptography.hazmat.primitives.ciphers.aead import AESGCM
>>> aad = None
>>> key = 'catafestcatafestcatafestcatafest'
>>> aesgcm = AESGCM(key.encode('utf-8'))
>>> nonce = '12345678'
>>> data_binary = b'Hello world!'
>>> 
>>> cipher_text = aesgcm.encrypt(nonce.encode('utf-8'), data_binary, aad)
>>> print(cipher_text)
b'0\xab\xd2!mXe\xc3/\xdb\x15\xcaoT\x0f\x1d\xbb&\xc4\x92\xdf\\ZTMD\xa2\x9f'
>>> data_text = aesgcm.decrypt(nonce.encode('utf-8'), cipher_text, aad)
>>> print(data_text)
b'Hello world!'

Saturday, January 4, 2020

Python 3.7.5 : Testing the PyMongo package - part 001.

MongoDB and PyMongo are not my priorities for the new year 2020 but because they are quite evolved I thought to start presenting it within my free time.
The PyMongo python package is a Python distribution containing tools for working with MongoDB.
The full documentation can be found on this webpage.
You can see my tutorial about how to install the MongoDB into Fedora 31 on this webpage.
I used that webpage install to test this python package, see the result:
[mythcat@desk mongo_test]$ mongo
...
MongoDB server version: 4.2.2-rc1
> use admin 
switched to db admin
> show dbs
> db.auth('admin', 'admin')
1
> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
Let's install it with pip3 tool:
[mythcat@desk projects]$ pip3 install pymongo --user
Collecting pymongo
...
Successfully installed pymongo-3.10.0
Let's start with a simple example:
[mythcat@desk projects]$ mkdir mongo_test
[mythcat@desk projects]$ cd mongo_test/
[mythcat@desk mongo_test]$ vim mongo001.py
If you have already created the admin user, to run the next spython script you need to change the role like this:
> use admin;
switched to db admin
> db.grantRolesToUser('admin', [{ role: 'root', db: 'admin' }])
The script show you how to use a simple connection to the MongoDB:
import pymongo
from pymongo import MongoClient, errors

MONGO_URI = 'mongodb://admin:admin767779@127.0.0.1:27017/admin'
client = pymongo.MongoClient(MONGO_URI)
print("Server info : ")
print(client.server_info())
print("Databases : " + str(client.list_database_names()))
print("Connect to : admin database!")
db = client['admin']
db2 = client.config
print(db)
print(db2)
print("Collection admin : ")
collection = db['admin']
collection2 = db.config
print(collection)
print(collection2)
print("try call find_one method")
try:
    one_doc= collection2.find_one()
    print ("find_one():", one_doc)
except errors.ServerSelectionTimeoutError as err:
    print ("find_one() ERROR:", err)

print("Client close!")
client.close()
The output is this:
[mythcat@desk mongo_test]$ python3 mongo001.py 
Server info : 
{'version': '4.2.2-rc1', 'gitVersion': 'a0bbbff6ada159e19298d37946ac8dc4b497eadf', 'modules': [],
 'allocator': 'tcmalloc', 'javascriptEngine': 'mozjs', 'sysInfo': 'deprecated', 'versionArray': [4, 2, 2, -49],
 'openssl': {'running': 'OpenSSL 1.1.1d FIPS  10 Sep 2019', 'compiled': 'OpenSSL 1.1.1 FIPS  11 Sep 2018'},
 'buildEnvironment': {'distmod': 'rhel80', 'distarch': 'x86_64', 'cc': '/opt/mongodbtoolchain/v3/bin/gcc:
 gcc (GCC) 8.2.0', 'ccflags': '-fno-omit-frame-pointer -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare
 -Wno-unknown-pragmas -Winvalid-pch -Werror -O2 -Wno-unused-local-typedefs -Wno-unused-function
 -Wno-deprecated-declarations -Wno-unused-const-variable -Wno-unused-but-set-variable
 -Wno-missing-braces -fstack-protector-strong -fno-builtin-memcmp', 'cxx': '/opt/mongodbtoolchain/v3/bin/g++:
 g++ (GCC) 8.2.0', 'cxxflags': '-Woverloaded-virtual -Wno-maybe-uninitialized -fsized-deallocation -std=c++17',
 'linkflags': '-pthread -Wl,-z,now -rdynamic -Wl,--fatal-warnings -fstack-protector-strong -fuse-ld=gold -Wl,
--build-id -Wl,--hash-style=gnu -Wl,-z,noexecstack -Wl,--warn-execstack -Wl,-z,relro', 'target_arch':
 'x86_64', 'target_os': 'linux'}, 'bits': 64, 'debug': False, 'maxBsonObjectSize': 16777216, 'storageEngines':
 ['biggie', 'devnull', 'ephemeralForTest', 'wiredTiger'], 'ok': 1.0}
Databases : ['admin', 'config', 'local']
Connect to : admin database!
Database(MongoClient(host=['127.0.0.1:27017'], document_class=dict, tz_aware=False, connect=True), 'admin')
Database(MongoClient(host=['127.0.0.1:27017'], document_class=dict, tz_aware=False, connect=True), 'config')
Collection admin : 
Collection(Database(MongoClient(host=['127.0.0.1:27017'], document_class=dict, tz_aware=False, connect=True),
 'admin'), 'admin')
Collection(Database(MongoClient(host=['127.0.0.1:27017'], document_class=dict, tz_aware=False, connect=True),
 'admin'), 'config')
try call find_one method
find_one(): None
Client close!


Thursday, January 2, 2020

Python 3.7.5 : Testing the Falcon framework - part 001.

I start the new year with this python framework named Falcon.
The Falcon is a low-level, high-performance Python framework for building HTTP APIs, app backends, and higher-level frameworks.
The main reason was the speed of this python framework, see this article about falcon benchmark.
You can see is more faster like Flask and Django.
The instalation is easy with pip tool, you can read also the documenation webpage:
[mythcat@desk projects]$ mkdir falcon_test
[mythcat@desk projects]$ cd falcon_test/
[mythcat@desk falcon_test]$ pip3 install falcon --user
Collecting falcon
...
Successfully installed falcon-2.0.0
Falcon also fully supports CPython 2.7 and 3.5+.
If you want to install the latest beta or release candidate use:
[mythcat@desk falcon_test]$ pip3 install --pre falcon --user
The Falcon framework is easy to use.
First, I created a folder named test001 for my falcon application script named app.py:
[mythcat@desk falcon_test]$ mkdir test001
[mythcat@desk falcon_test]$ cd test001/
[mythcat@desk test001]$ vim app.py
In this python script I used python packages json and falcon with a class named req_class:
import json
import falcon

class req_class:
    def on_get(self,req,resp):
        print("on_get class")
my_falcon_api = falcon.API()
my_falcon_api.add_route('/test',req_class())
The url route is set to test.
To test the falcon framework we need the Gunicorn python package.
The Gunicorn is working on my Fedora Linux distro and I'm not sure if working Gunicorn on Windows.
You can try on Windows O.S. the waitress python package.
[mythcat@desk falcon_test]$ pip3 install gunicorn --user
Collecting gunicorn
...
Successfully installed gunicorn-20.0.4
To test this simple Falcon application use this command line where the app python script and my falcon variable A.P.I. named my_falcon_api is used.
[mythcat@desk test001]$ gunicorn app:my_falcon_api
[2020-01-02 18:57:48 +0200] [4401] [INFO] Starting gunicorn 20.0.4
[2020-01-02 18:57:48 +0200] [4401] [INFO] Listening at: http://127.0.0.1:8000 (4401)
[2020-01-02 18:57:48 +0200] [4401] [INFO] Using worker: sync
[2020-01-02 18:57:48 +0200] [4404] [INFO] Booting worker with pid: 4404
on_get class
on_get class
Open in the browser this URL with the route I set: http://127.0.0.1:8000/test.
You will don't see anything in the browser but will see the python output of the print function for my request.

Tuesday, December 31, 2019

News : The Python 2.7 no longer support from Python team.

The 1st of January 2020 will mark the sunset of Python 2.7.
It’s clear that Python 3 is more popular these days.
You can learn more about the popularity of both on Google Trends.
Python 3.0 was released in December 2008.
The main goal was to fix problems existing in Python 2.
Since the 1st January 2020, Python 2 will no longer receive any support whatsoever from the core Python team.
Migrating to Python 3 is recommended, including some of the top libraries, such as machine learning.

Saturday, December 28, 2019

Python 3.7.5 : Fix to python language the GitHub project.

I created a GitHub project with Django and I saw is detect like tcl programming language:

You need to create a file named .gitattributes in the root folder of my repository.
Use this source code to tell GitHub is a python project:
* linguist-vendored
*.py linguist-vendored=false
Now the project will be target with python language.