analitics

Pages

Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Saturday, September 2, 2023

Python 3.11.4 : Issues in Fedora with PyGobject and sway-tests

Today I wanted to test this repo named sway-tests.
I followed the steps there and received an error from gi.repository.
This error is related to another issue related to PyGobject.
In Fedora Linux distro, installing PyGobject is done with pip like this:
$ pip install PyGobject
In order to have no errors, the dnf or dnf5 tool should be used like this ...
I tested the functionality of this installation with a simple example:
import gi

gi.require_version("Gtk", "3.0")
from gi.repository import Gtk

win = Gtk.Window()
win.connect("destroy", Gtk.main_quit)
win.show_all()
Gtk.main()
It worked very well.
After solving this issue, I returned to the initial one and tested the sway-tests.
$ whereis sway
$ env/bin/pytest --sway=/usr/bin/sway
$ sudo env/bin/pytest --sway=/usr/bin/sway
I used the command both with and without sudo.
Both generated the same errors.
For the following command I had to install ... xorg-x11-server-Xephyr:
Xephyr is an X server which has been implemented as an ordinary X application. It runs in a window just like other X applications, but it is an X server ...
... the fixed centered black window specific to the xorg runtime appeared and somewhere on the side the terminal showed me a bunch of errors.
... obviously, I don't know how well sway-tests is implemented, now it's an archived repo, but I solved the use of PyGobject in python on the Fedora linux distribution.

Saturday, August 12, 2023

Python 3.10.12 : My colab tutorials and news from colab - part 036.

Today I recapitulated a bit the artificial intelligence part and a simple example for google drive.
I created two notebooks in collaboration and added them to my github repo.
The most interesting is the one with textgenrnn.
textgenrnn is an modern neural network architecture which utilizes new techniques as attention-weighting and skip-embedding to accelerate training
The last notebook colab is catafest_045.

Thursday, August 10, 2023

Python 3.10.12 : My colab tutorials and news from colab - part 035.

In this notebook I will show you how to use python to run a program written in the programming language for CUDA.
This allows you to use NVIDIA CUDA Compiler Driver NVCC, see this official webpage.
NVCC Plugin for Jupyter Notebook by https://github.com/andreinechaev/nvcc4jupyter.
The example I tested is simple:
# This is formatted as CUDA code
__global__ void cuda_hello(){
    printf("Hello World from GPU!\n");
}

int main() {
    cuda_hello<<<1,1>>>();
    return 0;
}

Tuesday, July 11, 2023

Python 3.8.10 : My colab tutorials and news from colab - part 034.

I add a new colab notebook with a simple source code to list all running VM processes from the colab notebook
You can see more examples on my GitHub colab google repo.
This is the source code:
%%sh
echo "List all running VM processes."
ps -ef
echo "Done"

Sunday, May 21, 2023

Python 3.11.3 : Using Jupyter Lab on Fedora linux distro.

JupyterLab is the latest web-based interactive development environment for notebooks, code, and data. Its flexible interface allows users to configure and arrange workflows in data science, scientific computing, computational journalism, and machine learning.
Follow these steps:
  1. Install the jupyterlab python package using pip. This will allow you to run Jupyter notebooks in the terminal.
    pip install jupyterlab
  2. Open a Jupyter Lab session in the terminal using the command:
    jupyter lab
  3. Create a new notebook file and save it with the .ipynb extension.
  4. In the notebook file, add the source code to work with the Python programming language and save the file notebook.
See the next screenshot how this works:

Wednesday, March 29, 2023

Python : Open3D cannot be used on Windows 10 and Fedora Linux Distro .

Open3D is an open-source library that supports rapid development of software that deals with 3D data. The Open3D frontend exposes a set of carefully selected data structures and algorithms in both C++ and Python. The backend is highly optimized and is set up for parallelization. Open3D was developed from a clean slate with a small and carefully considered set of dependencies. It can be set up on different platforms and compiled from source with minimal effort. The code is clean, consistently styled, and maintained via a clear code review mechanism. Open3D has been used in a number of published research projects and is actively deployed in the cloud. We welcome contributions from the open-source community.
Today I tested this python package with Windows 10 and Fedora Linux Distro with python versions 11 and 10 ...
This package does not work and you will see why ...
C:\PythonProjects\Open3D001>git clone https://github.com/isl-org/Open3D.git
Cloning into 'Open3D'...
remote: Enumerating objects: 67435, done.
remote: Counting objects: 100% (2280/2280), done.
remote: Compressing objects: 100% (1894/1894), done.
remote: Total 67435 (delta 886), reused 599 (delta 385), pack-reused 65155
Receiving objects: 100% (67435/67435), 237.23 MiB | 17.11 MiB/s, done.

Resolving deltas: 100% (50682/50682), done.
Updating files: 100% (2315/2315), done.

C:\PythonProjects\Open3D001>cd Open3D

C:\PythonProjects\Open3D001\Open3D>mkdir build

C:\PythonProjects\Open3D001\Open3D>cd build

C:\PythonProjects\Open3D001\Open3D\build>cmake -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=C:\open3d_install ..
-- Building for: Visual Studio 17 2022
-- Setting build type to Release as none was specified.
-- CMAKE_BUILD_TYPE is set to Release.
-- Downloading third-party dependencies to C:/PythonProjects/Open3D001/Open3D/3rdparty_downloads
CMake Deprecation Warning at CMakeLists.txt:189 (cmake_policy):
  The OLD behavior for policy CMP0072 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.

...
According to this issue number 4796 and my test on Windows 10 with a Python version greater than 10 and on Fedora Linux Distro you cannot use this python package.
You can try an older version of Python and try it, see this example:
C:\PythonProjects\Open3D001>C:\Python310\python.exe -m pip install --user open3d --no-warn-script-location
C:\PythonProjects\Open3D001>C:\Python310\python.exe -c "import open3d as o3d; print(o3d)"
Traceback (most recent call last):
 ...
    from open3d.cpu.pybind import (core, camera, data, geometry, io, pipelines,
ImportError: DLL load failed while importing pybind: A dynamic link library (DLL) initialization routine failed.
...
pip install pybind --user
Collecting pybind
  Using cached pybind-0.1.35.tar.gz (15.5 MB)
ERROR: Could not install packages due to an OSError: [WinError 206] The filename or extension is too 
long: 'C:\\Users\\catafest\\AppData\\Local\\Temp\\pip-install-7ccpzu3z\\pybind_
...
Basically, this python package cannot be used with an old python version in Windows 10.

Saturday, May 7, 2022

Python : Use django-allauth on heruko - part 001.

Today I am going to show you how to implement a google authentication using Django on the Heroku server.
The tutorial is almost complete. This introductory part is not finalized on the display side of the style and some elements ...
This python module is needed to run the web service.
pip install gunicorn
Collecting gunicorn
  Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
     ---------------------------------------- 79.5/79.5 KB 184.8 kB/s eta 0:00:00
Requirement already satisfied: setuptools>=3.0 in c:\python311alpha\lib\site-packages (from gunicorn) (58.1.0)
Installing collected packages: gunicorn
Successfully installed gunicorn-20.1.0
Create and activate a virtual environment:
python -m venv venv
venv\Scripts\activate
The next commands will be on the (venv)
Let's install the Django
(venv) python -m pip install django
Installing collected packages: tzdata, sqlparse, asgiref, django
Successfully installed asgiref-3.5.1 django-4.0.4 sqlparse-0.4.2 tzdata-2022.1
One way is to use this command to start the project
(venv) django-admin startproject herokuweb
I used it in this way, see the dot symbol:
(venv) django-admin startproject herokuweb . 
The next command will start an application
(venv) python manage.py startapp catafest
This python module needs to use google authentification:
(venv) pip install django-allauth
...
Successfully built django-allauth cffi
Installing collected packages: certifi, urllib3, tzdata, sqlparse, pyjwt, pycpar
ser, oauthlib, idna, defusedxml, charset-normalizer, asgiref, requests, python3-
openid, Django, cffi, requests-oauthlib, cryptography, django-allauth
Successfully installed Django-4.0.4 asgiref-3.5.1 certifi-2021.10.8 cffi-1.15.0
charset-normalizer-2.0.12 cryptography-37.0.2 defusedxml-0.7.1 django-allauth-0.
50.0 idna-3.3 oauthlib-3.2.0 pycparser-2.21 pyjwt-2.3.0 python3-openid-3.2.0 req
uests-2.27.1 requests-oauthlib-1.3.1 sqlparse-0.4.2 tzdata-2022.1 urllib3-1.26.9
Let's make changes in the settings.py file
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.sites',
    'catafest',
    'allauth',
    'allauth.account',
    'allauth.socialaccount',
    'allauth.socialaccount.providers.google',
]
...
AUTHENTIFICATION_BACKENDS = [
    'django.contrib.auth.backends.ModelBackend',
    'allauth.account.auth_backends.AuthenticationBackend',
]
...
STATIC_URL = '/static/'
SITE_ID = 1
LOGIN_REDIRECT_URL = '/'
SOCIALACCOUNT_PROVIDERS = {
'google': {
    'SCOPE': [
'profile',
'email',
    ],
'AUTH_PARAMS': {
    'access_type': 'online',
}
    }
}
Use Django features ...
python manage.py makemigrations
No changes detected
...
python manage.py migrate
...
  Applying sites.0002_alter_domain_unique... OK
  Applying socialaccount.0001_initial... OK
  Applying socialaccount.0002_token_max_lengths... OK
  Applying socialaccount.0003_extra_data_default_dict... OK
Create one superuser:
python manage.py createsuperuser
Username (leave blank to use 'catafest'):
Email address: catafest@yahoo.com
Password:
Password (again):
This password is too short. It must contain at least 8 characters.
This password is too common.
Bypass password validation and create user anyway? [y/N]: y
Superuser created successfully.
Create templates, catafest folders
Add the index.html file on the catafest folder
Make changes in url.py in the heroku web folder project:
from django.contrib import admin
    from django.urls import path, include
    from django.views.generic import TemplateView
    
    urlpatterns = [
        path('', TemplateView.as_view(template_name="catafest/index.html")),
        path('admin/', admin.site.urls),
        path('accounts/', include('allauth.urls')),
    ]
In the settings.py add templates feature make this change:
import os
...
'DIRS': [os.path.join(BASE_DIR, 'templates')],
Open a google console application and set Credentials to OAuth client ID for the web.
Follow the basic steps like for any basic project
I set for my project: Authorized JavaScript origins: https://catafest.herokuapp.com
... and: Authorized redirect URIs: https://catafest.herokuapp.com/accounts/google/login/callback
Finally, you need to have these: The 'OAuth client created' with 'Your Client ID' and 'Your Client Secret' .
You can test your Django project with this command:
python manage.py runserver
Open the admin area http://127.0.0.1:8000/admin/socialaccount/socialapp/ and add a social application
These commands will log in to the Heroku browser webpage and will create the application:
heroku login
 »   Warning: heroku update available from 7.53.0 to 7.60.2.
...
heroku create catafest
 »   Warning: heroku update available from 7.53.0 to 7.60.2.
Creating ⬢ catafest... done
https://catafest.herokuapp.com/ | https://git.heroku.com/catafest.git
Then create requirements.txt and fill it with this command:
pip freeze > requirements.txt
make this change in settings.py :

...
ALLOWED_HOSTS = ['.herokuapp.com','127.0.0.1']
...
STATIC_ROOT = os.path.join(BASE_DIR,"staticfiles")
...
Upload changes with these commands:
git add .
git commit -am "add requirements.txt and changes STATIC_ROOT"
git push heroku master
Enumerating objects: 39, done.
Counting objects: 100% (39/39), done.
...
To https://git.heroku.com/catafest.git
 * [new branch]      master -> master
Open the online Heroku application, and see all errors with the command: heroku logs --tail
This error is normal because is not set in one web service:
... at=error code=H14 desc="No web processes running" method=GET path="/" host
Create Procfile file in the web001 folder and fill it with:
Add this to the file:
web: gunicorn catafest.wsgi
Add changes and make changes all with these commands:
git add .
git commit -am "add Procfile"
Push to the Django application on the Heroku server:
git push heroku master
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 2 threads
...
You can test the result on my heroku application , see also this google gign in link.

Wednesday, May 4, 2022

Python 3.11.0a7 : Django-hypergen on Fedora 36 distro.

You can write server-rendered reactive HTML live views for Django in pure python, see the GitHub webpage.
I tested with python version 3.11.0a7 and Django version 4.0.4 on Fedora 36 Linux distro.
I started with the cloning process of the GitHub project and I set a virtual environment:
[mythcat@fedora ~]$ git clone http://github.com/runekaagaard/django-hypergen.git
Cloning into 'django-hypergen'
...
[mythcat@fedora ~]$ cd django-hypergen/
[mythcat@fedora django-hypergen]$ virtualenv -p python3.11 venv
created virtual environment
... 
[mythcat@fedora django-hypergen]$ source venv/bin/activate
I install the requirements from the project:
(venv) [mythcat@fedora django-hypergen]$
(venv) [mythcat@fedora django-hypergen]$ pip install -r requirements.txt
...Successfully installed Django-4.0.4 asgiref-3.5.1 attrs-21.4.0 beautifulsoup4-4.9.3 
iniconfig-1.1.1 packaging-21.3 pluggy-1.0.0 py-1.11.0 pyparsing-3.0.8 pyperclip-1.8.1 
pyrsistent-0.16.1 pytest-7.1.2 pyyaml-6.0 six-1.16.0 soupsieve-2.3.2.post1 sqlparse-0.4.2 
tomli-2.0.1 yapf-0.32.0WARNING: You are using pip version 21.3.1; 
however, version 22.0.4 is available.
You should consider upgrading via the '/home/mythcat/django-hypergen/venv/bin/python -m pip install --upgrade pip' command.
This gives some errors ...
(venv) [mythcat@fedora django-hypergen]$ pip install -r examples/requirements.txt
...Successfully built pyprof2calltreeFailed to build cymem numpy
ERROR: Could not build wheels for cymem, numpy, which is required to install pyproject.toml-based projects 
The last step is the migrate project and test the example
(venv) [mythcat@fedora django-hypergen]$ 
(venv) [mythcat@fedora django-hypergen]$ cd examples/
(venv) [mythcat@fedora examples]$ python manage.py migrate
... 
(venv) [mythcat@fedora examples]$ python manage.py runserver
Watching for file changes with ...
Starting development server at http://127.0.0.1:8000/
... and the result can be seen in this screenshot:

Friday, January 14, 2022

Python 3.10.1 : Django and channels on Fedora distro - sync and async features.

A consumer is a subclass of either channels.consumer.AsyncConsumer or channels.consumer.SyncConsumer.
Consumers do a couple of things in particular: 
  • Structures your code as a series of functions to be called whenever an event happens, rather than making you write an event loop. 
  • Allow you to write synchronous or async code and deals with handoffs and threading for you.
This is another tutorial about Django and channels, you can see the first one.
For testing area you need the postman tool and I install and used with snap tool.
[root@fedora mythcat]# dnf install snapd
Last metadata expiration check: 0:40:03 ago on Fri 14 Jan 2022 03:38:55 PM EET.
...
[root@fedora mythcat]# ln -s /var/lib/snapd/snap /snap
[root@fedora mythcat]# snap install postman
2022-01-14T16:22:15+02:00 INFO Waiting for automatic snapd restart...
postman (v9/stable) 9.8.3 from Postman, Inc. (postman-inc✓) installed
[mythcat@fedora ~]$ snap run postman
Let's go on the project folder:
[mythcat@fedora ~]$ cd djangotest001/
[mythcat@fedora djangotest001]$ cd website001/
In this folder I have two folders: appsite001 and website001.
In the appsite001 I add these scripts.
I create a new python script named consumers.py with this source code:
from channels.consumer import SyncConsumer, AsyncConsumer
from channels.exceptions import StopConsumer

class MySyncConsumer(SyncConsumer):
    def websocket_connect(self,event):
        print('Websocket Connected ...')
        self.send({
        'type':'websocket.accept',
        })
    def websocket_receive(self, event):
        print('Messaged Received ...')
        print(event['text'])
        self.send({
        'type':'websocket.send',
        'text':'Message sent to client'
        })
    def websocket_diconnect(self, event):
        print('Websocket Disconnected ...')
        raise StopConsumer
        
class MyAsyncConsumer(AsyncConsumer):
    async def websocket_connect(self,event):
        print('Websocket Connected ...')
    async def websocket_receive(self, event):
        print('Messaged Received ...')
    async def websocket_diconnect(self, event):
        print('Websocket Disconnected ...')
I created routing.py python script with this source code:
from django.urls import path
from . import consumers

websocket_urlpatterns = [
    path('ws/sc/',consumers.MySyncConsumer.as_asgi()),
    ]
In the website001 I change this script named asgi.py.
import os

from django.core.asgi import get_asgi_application

from channels.routing import ProtocolTypeRouter, URLRouter

import appsite001.routing

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'website001.settings')

application = ProtocolTypeRouter({
    'http':get_asgi_application(),
    'websocket':URLRouter(
        appsite001.routing.websocket_urlpatterns
    )
})
Run the Django project with :
[mythcat@fedora website001]$ python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
January 14, 2022 - 15:32:29
Django version 4.0.1, using settings 'website001.settings'
Starting ASGI/Channels version 3.0.4 development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
WebSocket HANDSHAKING /ws/sc/ [127.0.0.1:33944]
Websocket Connected ...
WebSocket CONNECT /ws/sc/ [127.0.0.1:33944]
Messaged Received ...
This is a message from mythcat
...
Use postman tool with websocket to send this message to Django project:
This is a message from mythcat
You can see how this works:

Saturday, January 8, 2022

Python 3.10.1 : Django and channels on Fedora distro.

Today I tested the Django version 4.0.1 with channels features on Fedora 35.
For the channels package, I used the pip tool and I install the version
The python package channels come with features like:
Channels augments Django to bring WebSocket, long-poll HTTP, task offloading, and other async support to your code, using familiar Django design patterns and a flexible underlying framework that lets you not only customize behaviors but also write support for your own protocols and needs. see the GitHub website.
Let's install the Django package
[mythcat@fedora ~]$ pip3 install django --user
Requirement already satisfied: django in /usr/local/lib/python3.10/site-packages (4.0.1)
Requirement already satisfied: sqlparse>=0.2.2 in /usr/local/lib/python3.10/site-packages (from django) (0.4.2)
Requirement already satisfied: asgiref<4>=3.4.1 in ./.local/lib/python3.10/site-packages (from django) (3.4.1)
The next step is to create the project named website001:
[mythcat@fedora ~]$ mkdir djangotest001
[mythcat@fedora ~]$ cd djangotest001/
[mythcat@fedora djangotest001]$ django-admin startproject website001
[mythcat@fedora djangotest001]$ cd website001/
[mythcat@fedora website001]$ python3 manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
January 08, 2022 - 13:26:21
Django version 4.0.1, using settings 'website001.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
...
Let's create the application named appsite001:
[mythcat@fedora website001]$ django-admin startapp  appsite001
[mythcat@fedora website001]$ ls
appsite001  db.sqlite3  manage.py  website00
The apps.py file is this:
[mythcat@fedora website001]$ cat  appsite001/apps.py 
from django.apps import AppConfig

class Appsite001Config(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'appsite001'
Let's add this on the settings.py file config:
[mythcat@fedora website001]$ vi website001/settings.py
    
# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'appsite001',
]
Use the migrate feature to fix all:
[mythcat@fedora website001]$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying sessions.0001_initial... OK
Create the superuser named admin with password admin and set the email address:
[mythcat@fedora website001]$ python manage.py createsuperuser
Username (leave blank to use 'mythcat'): admin
Email address: admin@server.com
Password: 
Password (again): 
The password is too similar to the username.
This password is too short. It must contain at least 8 characters.
This password is too common.
Bypass password validation and create user anyway? [y/N]: y
Superuser created successfully.

http://127.0.0.1:8000/admin/
...
Let's install the channels package for the Django project:
[mythcat@fedora website001]$ pip install channels
...
Successfully installed Automat-20.2.0 autobahn-21.11.1 channels-3.0.4 constantly-15.1.0 
daphne-3.0.2 hyperlink-21.0.0 incremental-21.3.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 
service-identity-21.1.0 twisted-21.7.0 txaio-21.2.1 zope.interface-5.4.0
Add this package into the settinngs.py config file:
[mythcat@fedora website001]$ vi website001/settings.py
INSTALLED_APPS = [
    'channels',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'appsite001',
]
...
#WSGI_APPLICATION = 'website001.wsgi.application'
ASGI_APPLICATION = 'website001.asgi.application'
...
Make these changes to switch from wsgi to asgi features for channels package:
[mythcat@fedora website001]$ cp website001/wsgi.py website001/asgi.py 
[mythcat@fedora website001]$ vi website001/asgi.py 
import os

from django.core.asgi import get_asgi_application
from channels.routing import ProtocolTypeRouter
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'website001.settings')

application = ProtocolTypeRouter({
    'http':get_asgi_application(),
})
I tested on the admin area how the settings for this packet will work:
[mythcat@fedora website001]$ python3 manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
January 08, 2022 - 14:20:53
Django version 4.0.1, using settings 'website001.settings'
Starting ASGI/Channels version 3.0.4 development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
...
The result shows that it works:
The next theoretical steps would be to determine how the channels package will work and routing for access to appsite001.

Saturday, October 31, 2020

Python 3.9.0 : Testing twisted python module - part 001 .

Today I tested two python modules named: twisted and twisted[tls].
Twisted is an event-driven network programming framework written in Python and licensed under the MIT License. Twisted projects variously support TCP, UDP, SSL/TLS, IP multicast, Unix domain sockets, many protocols (including HTTP, XMPP, NNTP, IMAP, SSH, IRC, FTP, and others), and much more. Twisted is based on the event-driven programming paradigm, which means that users of Twisted write short callbacks which are called by the framework., see wikipedia webpage.
In this tutorial I will show you only some of these tests and how you can work with these python modules.
About twisted you can read more at the official webpage. In Fedora distro version 33 you can use the dnf tool to search for and install these python packages.
[root@desk mythcat]# dnf search twisted
...
python3-twisted.x86_64 : Twisted is a networking engine written in Python
python3-twisted+tls.x86_64 : Metapackage for python3-twisted: tls extras
You can also use the pip tool for installation:
[mythcat@desk ~]$ cd PythonProjects/
[mythcat@desk PythonProjects]$ pip3 install twisted
...
[mythcat@desk PythonProjects]$ pip3 install twisted[tls]
...
I used python 3.9.0 to test if this python package works:
[mythcat@desk PythonProjects]$ python3.9
Python 3.9.0 (default, Oct  6 2020, 00:00:00) 
[GCC 10.2.1 20200826 (Red Hat 10.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twisted.protocols import basic
Let's test with simple example using the reactor and protocol:
from twisted.internet import reactor, protocol

class ClientEcho(protocol.Protocol):
    def connectionMade(self):
        self.transport.write("Hello, world!".encode('utf-8'))

    def dataReceived(self, data):
        print ("Server: ", data)
        self.transport.loseConnection()

class FactoryEcho(protocol.ClientFactory):
    def buildProtocol(self, addr):
        return ClientEcho()

    def clientConnectionFailed(self, connector, reason):
        print ("Connection failed")
        reactor.stop()

    def clientConnectionLost(self, connector, reason):
        print ("Connection lost")
        reactor.stop()

reactor.connectTCP("localhost", 8080, FactoryEcho())
reactor.run()
Your protocol handling class will usually subclass twisted.internet.protocol.Protocol.
The default factory class twisted.internet.protocol.Factory just instantiates each Protocol and lets every Protocol access, and possibly modify, the persistent configuration.
This protocol responds to the initial connection with a well known quote, and then terminates the connection.
The protocol never waits for an event because handles data in an asynchronous manner.
The reactor interface lets many different loops handle the networking code.
The source code have two classes each is used to show a simple echo client on port 8080 - you can use any port.
This source code is the most simple example to understand the relation between factory , protocol and reactor.
The result is this:
[mythcat@desk PythonProjects]$ python3.9 echo_client_001.py 
Server:  b'Hello, world!'
Connection lost

Sunday, August 30, 2020

Python 3.8.5 : Testing with openpyxl - part 002 .

Today I will show you how can use Levenshtein ratio and distance between two strings, see wikipedia.
I used three files created with LibreOffice and save it like xlsx file type.
All of these files come with the column A fill with strings of characters, in this case, numbers.
The script will read all of these files from the folder named xlsx_files and will calculate Levenshtein ratio and distance between the strings of name of these files and column A.
Finally, the result is shown into a graph with matplotlib python package.
Let's see the python script:
import os
from glob import glob

from openpyxl import load_workbook
import numpy as np 
import matplotlib.pyplot as plt 

def levenshtein_ratio_and_distance(s, t, ratio_calc = False):
    """ levenshtein_ratio_and_distance - distance between two strings.
        If ratio_calc = True, the function computes the
        levenshtein distance ratio of similarity between two strings
        For all i and j, distance[i,j] will contain the Levenshtein
        distance between the first i characters of s and the
        first j characters of t
    """
    # Initialize matrix of zeros
    rows = len(s)+1
    cols = len(t)+1
    distance = np.zeros((rows,cols),dtype = int)

    # Populate matrix of zeros with the indeces of each character of both strings
    for i in range(1, rows):
        for k in range(1,cols):
            distance[i][0] = i
            distance[0][k] = k
    for col in range(1, cols):
        for row in range(1, rows):
            # check the characters are the same in the two strings in a given position [i,j] 
            # then the cost is 0
            if s[row-1] == t[col-1]:
                cost = 0 
            else:             
                # calculate distance, then the cost of a substitution is 1.
                if ratio_calc == True:
                    cost = 2
                else:
                    cost = 1
            distance[row][col] = min(distance[row-1][col] + 1,      # Cost of deletions
                                 distance[row][col-1] + 1,          # Cost of insertions
                                 distance[row-1][col-1] + cost)     # Cost of substitutions
    if ratio_calc == True:
        # Ration computation of the Levenshtein Distance Ratio
        Ratio = ((len(s)+len(t)) - distance[row][col]) / (len(s)+len(t))
        return Ratio
    else:
        return distance[row][col]


PATH = "/home/mythcat/xlsx_files/"
result = [y for x in os.walk(PATH) for y in glob(os.path.join(x[0], '*.xlsx'))]
result_files = [os.path.join(path, name) for path, subdirs, files in os.walk(PATH) for name in files]
#print(result)
row_0 = []

for r in result:
    n = 0
    wb = load_workbook(r)
    sheets = wb.sheetnames
    ws = wb[sheets[n]]
    for row in ws.rows:
            if (row[0].value) != None :
                rows = row[0].value
                row_0.append(rows)

print("All rows of column A ")
print(row_0)
files = []
for f in result_files:
    ff = str(f).split('/')[-1:][0]
    fff = str(ff).split('.xlsx')[0]
    files.append(fff)

print(files)
# define tree lists for levenshtein
list1 = []
list2 = []

for l in row_0:
    str(l).lower()
    for d in files:
        Distance = levenshtein_ratio_and_distance(str(l).lower(),str(d).lower())   
        Ratio = levenshtein_ratio_and_distance(str(l).lower(),str(d).lower(),ratio_calc = True)
        list1.append(Distance)
        list2.append(Ratio)
        
print(list1, list2)
# plotting the points  
plt.plot(list1,'g*', list2, 'ro' )
plt.show()
The result is this:
[mythcat@desk ~]$ python test_xlsx.py
All rows of column A 
[11, 2, 113, 4, 1111, 4, 4, 111, 2, 1111, 5, 4, 4, 3, 1111, 1, 2, 1113, 4, 115, 1, 2, 221, 1, 1,
 43536, 2, 34242, 3, 1]
['001', '002', '003']
[2, 3, 3, 3, 2, 3, 3, 3, 2, 3, 3, 3, 3, 4, 4, 3, 3, 3, 3, 3, 3, 2, 3, 3, 3, 2, 3, 3, 4, 4, 3, 3, 
3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 3, 4, 4, 2, 3, 3, 3, 2, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 3, 
2, 3, 2, 3, 3, 2, 3, 3, 2, 3, 3, 5, 5, 4, 3, 2, 3, 5, 4, 5, 3, 3, 2, 2, 3, 3] [0.4, 0.0, 0.0, 0.0, 
0.5, 0.0, 0.3333333333333333, 0.0, 0.3333333333333333, 0.0, 0.0, 0.0, 0.2857142857142857, 0.0, 0.0,
 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.3333333333333333, 0.0, 0.0, 0.0, 0.5, 0.0, 0.2857142857142857, 0.0,
 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5, 0.2857142857142857, 0.0, 0.0, 0.5,
 0.0, 0.0, 0.0, 0.5, 0.0, 0.2857142857142857, 0.0, 0.2857142857142857, 0.0, 0.0, 0.0, 0.3333333333333333,
 0.0, 0.0, 0.5, 0.0, 0.0, 0.0, 0.5, 0.0, 0.3333333333333333, 0.3333333333333333, 0.0, 0.5, 0.0, 0.0,
 0.5, 0.0, 0.0, 0.0, 0.0, 0.25, 0.0, 0.5, 0.0, 0.0, 0.25, 0.25, 0.0, 0.0, 0.5, 0.5, 0.0, 0.0]

Sunday, August 23, 2020

Python 3.8.5 : Testing with openpyxl - part 001 .

The Python executes the code line by line because is an interpreter language.
This allows users to solve issues in the programming area, fast and easy.
I use python versiono 3.8.5 build on Aug 12 2020 at 00:00:00, see the result of interactive mode:
[mythcat@desk ~]$ python
Python 3.8.5 (default, Aug 12 2020, 00:00:00) 
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Today I will show you how to start using openpyxl python package.
Another tutorial about python and documents can be found here.
The openpyxl was created by Eric Gazoni, Charlie Clark, and is a Python library to read/write Excel 2010 xlsx/xlsm/xltx/xltm files.
Let's install the openpyxl python package:
[mythcat@desk ~]$ pip3 install openpyxl --user
Collecting openpyxl
...
Installing collected packages: openpyxl
Successfully installed openpyxl-3.0.5
I tested with the default example source code and works well.
from openpyxl import Workbook
wb = Workbook()

# grab the active worksheet
ws = wb.active

# Data can be assigned directly to cells
ws['A1'] = 42

# Rows can also be appended
ws.append([1, 2, 3])

# Python types will automatically be converted
import datetime
ws['A2'] = datetime.datetime.now()

# Save the file
wb.save("sample.xlsx")
The next example gets all data about asteroids close to planet Earth and put into xlsx file type.
The rows with dangerous asteroids are fill with the red color:
# check asteroids close to planet Earth and add it to file
# import json python package
import json, urllib.request, time

# import openpyxl python package
from openpyxl import Workbook
from openpyxl.styles import PatternFill
# use active worksheet
wb = Workbook()
ws = wb.active

today = time.strftime('%Y-%m-%d', time.gmtime())
print("Time is: " + today)
now = today
# retrieve data about asteroids approaching planet Earth into json format
url = "https://api.nasa.gov/neo/rest/v1/feed?start_date=" + today + "&end_date=" + today + "&api_key=DEMO_KEY"
response = urllib.request.urlopen(url)
result = json.loads(response.read())

print("Now, " + str(result["element_count"]) + " asteroids is close to planet Earth.")
asteroids = result["near_earth_objects"]

no_data = ""
dangerous = ""

ws.append(['today', 'name', 'dangerous?', 'no_data'])
# parsing all the JSON data and add to file
for asteroid in asteroids:
    for field in asteroids[asteroid]:

      try:
        name = "Asteroid Name: " + field["name"]

        if field["is_potentially_hazardous_asteroid"]:   
          dangerous = "... dangerous to planet Earth!"

        else:
          dangerous = "... not threat to planet Earth!"

      except:
        no_data = "no data"
      ws.append([today, name, dangerous, no_data]) 

# create a red patern to fill
redFill = PatternFill(start_color='FFFF0000',
                   end_color='FFFF0000',
                   fill_type='solid')

# check the row with the dangerous asteroid and fill it
for row in ws.rows:
 if row[2].value == "... dangerous to planet Earth!":
  for cell in row:
      cell.fill = redFill

# write all data to file 
wb.save(str(now)+"_asteroids.xlsx")
I run it and result working well:
[mythcat@desk ~]$ python asteroid_data.py 
Time is: 2020-08-23
Now, 9 asteroids is close to planet Earth.
... see the next screenshot:


Saturday, August 22, 2020

Python 3.8.5 : Testing the pyre tool - part 001.

The Pyre is a static analysis tool to detect and prevent security issues in Python code that can be found on the official website.
The Pyre tool supports the Language Server Protocol and has an extension for VSCode.
The team development comes at August 7, 2020, with this intro:
Pyre is a performant type checker for Python. Statically typing what are essentially fully dynamic languages has a long tradition at Facebook. We've done this for PHP with Hack and for Javascript with Flow.
The install is easy to do with pip tool:
[mythcat@desk ~]$ pip install pyre-check
Defaulting to user installation because normal site-packages is not writeable
Collecting pyre-check
  Using cached pyre_check-0.0.52-py3-none-manylinux1_x86_64.whl (22.9 MB)
...
Installing collected packages: pyre-check
Successfully installed pyre-check-0.0.52
If you want to use a virtual environment:
[mythcat@desk ~]$ mkdir my_project && cd my_project
[mythcat@desk my_project]$ python3 -m venv ~/.venvs/venv
[mythcat@desk my_project]$ source ~/.venvs/venv/bin/activate
(venv) [mythcat@desk my_project]$ pip install pyre-check
Collecting pyre-check
...
(venv) [mythcat@desk my_project]$ pyre init
 ƛ Which directory should pyre be initialized in? (Default: `.`): 
(venv) [mythcat@desk my_project]$ cat .pyre_configuration
{
  "binary": "/home/mythcat/.venvs/venv/bin/pyre.bin",
  "source_directories": [
    "."
  ],
  "taint_models_path": "/home/mythcat/.venvs/venv/lib/pyre_check/taint/",
  "typeshed": "/home/mythcat/.venvs/venv/lib/pyre_check/typeshed/"
}
(venv) [mythcat@desk my_project]$ ls .pyre
my_project  pid_files  pyre.stderr
(venv) [mythcat@desk my_project]$ pyre
 ƛ No watchman binary found. 
To enable pyre incremental, you can install watchman: https://facebook.github.io/watchman/docs/install
 ƛ Defaulting to non-incremental check.
 ƛ No type errors found
Let's test with the default example from documentation:
(venv) [mythcat@desk my_project]$ echo "i: int = 'string'" > test.py
(venv) [mythcat@desk my_project]$ pyre
 ƛ No watchman binary found. 
To enable pyre incremental, you can install watchman: https://facebook.github.io/watchman/docs/install
 ƛ Defaulting to non-incremental check.
 ƛ Found 1 type error!
test.py:1:0 Incompatible variable type [9]: i is declared to have type `int` but is used as type `str`.
(venv) [mythcat@desk my_project]$ cat test.py 
i: int = 'string'
You can see is working well and detect the problem.
A short intro can found on the Facebook developers youtube channel:

Saturday, June 27, 2020

Python 2.7.10 : IronPython and C# with Dynamic Language Runtime.

This is a simple tutorial about python and C# using the Dynamic Language Runtime with IronPython.
I use Visual Studio 2019 and .NET Framework 4.7.2 with my Console C# project named DynamicLanguageRuntime_001.
Let's install the package with Visual Studio by open the console using the main menu: Tools - NuGet Package Manager - Package Manager Console command.
PM> Install-Package DynamicLanguageRuntime
Package 'DynamicLanguageRuntime.1.2.3' already exists in project 'DynamicLanguageRuntime_001'
Time Elapsed: 00:00:01.2208674
Use Solution Explorer use right-click on References item from your project and use Add Reference ...
Into the new window dialog named Reference Manager on the Assemblies - Framework uses the edit box to search IronPython.
Then use the checkbox to select these two options: IronPython and IronPython.Modules.
See the screenshot from Visual Studio I.D.E.:

This is the source code I used:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using IronPython.Hosting;
using IronPython.Runtime;
using IronPython;
using Microsoft.Scripting.Hosting;
using Microsoft.Scripting;

namespace DynamicLanguageRuntime_001
{
    class Program
    {
        static void Main(string[] args)
        {
            // create python engine 
            ScriptEngine engine = Python.CreateEngine();
            // get and add paths to enfine
            var python_search_paths = engine.GetSearchPaths();
            python_search_paths.Add(@"C:\Program Files\IronPython 2.7\Lib");
            engine.SetSearchPaths(python_search_paths);
            // create a scope 
            ScriptScope scope = engine.CreateScope();
            // using CreateScriptSourceFromString
            engine.CreateScriptSourceFromString("print '... simple example with ironpython and C#'").Execute();
            // using Execute
            engine.Execute("print '                             by catafest!'", scope);
            // using ExecuteFile
            engine.ExecuteFile(@"D:\Projects\Python\testing\test_webpage_001.py", scope);
            dynamic testFunction = scope.GetVariable("GetFriends");
            var result = testFunction();
        }
    }
} 

Saturday, June 20, 2020

Python 3.7.5 : Django on Fedora distro.

[mythcat@desk django]$ source env/bin/activate
(env) [mythcat@desk django]$ python3 
Python 3.7.6 (default, Dec 19 2019, 22:52:49) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> django.VERSION
(3, 0, 1, 'final', 1) 

Static files are those files that can not be processed, generated or modified by the server.
Static files improve the performance of the website with the template inheritance method.
Static file management is an important factor in web development.
I will show you how static file works on Django project.
The new static files folder is set on settings.py file:
...
# Static files (CSS, JavaScript, Images)

STATIC_URL = '/static/'

STATIC_ROOT = os.path.join(BASE_DIR, '/home/mythcat/projects/django/mysite/test001/')
STATICFILES_DIRS = [
   os.path.join(BASE_DIR, 'static'),
 ]
...
Let's run the server:
(env) [mythcat@desk mysite]$ python3 manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
January 26, 2020 - 09:01:10
Django version 3.0.1, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C. 
If I try to use the admin area then we can see the bad result for static files. 
...
[26/Jan/2020 09:10:02] "GET /admin/test001/post/ HTTP/1.1" 200 5935
[26/Jan/2020 09:10:02] "GET /static/admin/css/changelists.css HTTP/1.1" 200 6190
[26/Jan/2020 09:10:02] "GET /admin/jsi18n/ HTTP/1.1" 200 3223
[26/Jan/2020 09:10:02] "GET /static/admin/js/jquery.init.js HTTP/1.1" 200 363
[26/Jan/2020 09:10:02] "GET /static/admin/js/urlify.js HTTP/1.1" 200 8941
[26/Jan/2020 09:10:02] "GET /static/admin/js/actions.js HTTP/1.1" 200 6766
[26/Jan/2020 09:10:02] "GET /static/admin/js/prepopulate.js HTTP/1.1" 200 1530
[26/Jan/2020 09:10:02] "GET /static/admin/js/core.js HTTP/1.1" 200 5723
[26/Jan/2020 09:10:02] "GET /static/admin/js/admin/RelatedObjectLookups.js HTTP/1.1" 200 6918
[26/Jan/2020 09:10:02] "GET /static/admin/css/fonts.css HTTP/1.1" 200 423
[26/Jan/2020 09:10:02] "GET /static/admin/js/vendor/jquery/jquery.js HTTP/1.1" 200 280364
[26/Jan/2020 09:10:02] "GET /static/admin/js/vendor/xregexp/xregexp.js HTTP/1.1" 200 128820
[26/Jan/2020 09:10:02] "GET /static/admin/img/tooltag-add.svg HTTP/1.1" 200 331
[26/Jan/2020 09:10:02] "GET /static/admin/img/sorting-icons.svg HTTP/1.1" 200 1097
Not Found: /favicon.ico
... 
Now I can try to run the command collectstatic.
When this command is executed, Django performs these operations:
  • it looks for static files in all the directories listed in STATICFILES_DIRS;
  • the static-files are then copied and saved in STATIC_ROOT directory;
  • when the server is requested for static content, it will fetch a file from STATIC_ROOT;
  • that file will have its URL modified with STATIC_URL.
These errors show us many informations about this process:
 (env) [mythcat@desk mysite]$ python3 manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings:

    /home/mythcat/projects/django/mysite

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
Found another file with the destination path 'admin/js/urlify.js'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
...
Found another file with the destination path 'django.png'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'favicon.ico'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'admin/js/urlify.js'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
...

Found another file with the destination path 'rest_framework/js/prettify-min.js'. 
It will be ignored since only the first encountered file is collected. If this is not what you want, make
 sure every static file has a unique path.
...
166 static files copied to '/home/mythcat/projects/django/mysite'. 
But, Django respects the order of your applications in settings.py file the INSTALLED_APPS area, when running collectstatic command.
If you have two installed apps that write the same static files then Django collectstatic command will write the static files for the app appearing first in the list.
In my case: django.contrib.admin , test001 and ... .
Also, is need to set this code source on urls.py to return the proper URL pattern for serving static files to your already defined pattern list.
from django.contrib.staticfiles.urls import staticfiles_urlpatterns
...
urlpatterns += staticfiles_urlpatterns()
Let's run the runserver:
 (env) [mythcat@desk mysite]$ python3 manage.py runserver
...
[26/Jan/2020 09:29:18] "GET /static/admin/css/fonts.css HTTP/1.1" 200 423
Not Found: /favicon.ico 
You can see the /favicon.ico is not found in the default path and need to move it at /home/mythcat/projects/django/mysite/test001.
After these changes let's fix all the problems I created with these learning steps on static issues.
Let's move the static folder into mysite folder.
Change the settings.py file for the static issue with this source of code:
# Static files (CSS, JavaScript, Images)

STATIC_URL = '/static/'

STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = [
   os.path.join(BASE_DIR, 'static'),
 ]
Run the (env) [mythcat@desk mysite]$ python3 manage.py collectstatic
(env) [mythcat@desk mysite]$ python3 manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings:

    /home/mythcat/projects/django/mysite/static

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes

163 static files copied to '/home/mythcat/projects/django/mysite/static', 2 unmodified.
(env) [mythcat@desk mysite]$ ls
db.sqlite3  favicon.ico  manage.py  mysite  rest_framework  static  test001
(env) [mythcat@desk mysite]$ cd static/
(env) [mythcat@desk static]$ ls
admin  django.png  favicon.ico  rest_framework
This is a simple base tutorial.
In this point you can select the way of your Django project.

Wednesday, January 22, 2020

Python 3.7.5 : Django security issues - part 003.

Let's update this subject today with another new tutorial.
In the last tutorial about Django security I wrote about python package named django-axes.
First, let's fix an old issue about a URL pattern that matches anything and expects an integer that generates errors like:
...
  File "/home/mythcat/.local/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 1772, 
in get_prep_value
    ) from e
ValueError: Field 'id' expected a number but got 'favicon.ico'.
[22/Jan/2020 21:50:06] "GET /favicon.ico/ HTTP/1.1" 500 130547
Now, let's start my project:
[mythcat@desk ~]$ cd projects/
[mythcat@desk projects]$ cd django/
[mythcat@desk django]$ source env/bin/activate
Create a new folder named static in the test001 folder and add a icon file named favicon.ico.
(env) [mythcat@desk django]$ cd mysite/test001/
(env) [mythcat@desk test001]$ mkdir static 
In the settings.py file you need to have this source code:

STATIC_URL = '/static/'

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, "static"),
]
Change in the urls.py this line of source code to fix the error:
path('<int:author_id>/',views.index_next, name = 'index_next'),
Let's run the Django project server with:
(env) [mythcat@desk django]$ cd mysite/
(env) [mythcat@desk mysite]$ python3 manage.py runserver
I login into my admin area with user catalin and password adminadmin.
If you try to login with a bad password then the account is locked by django-axes python package.
Use this command to reset all lockouts and access records.
(env) [mythcat@desk mysite]$ python3 manage.py axes_reset
No attempts found.
Into admin area you can see the AXES area with Access attempts and Access logs.
Axes listens to the following signals from django.contrib.auth.signals to log access attempts.
In this case Axes lockout responses on failed user authentication attempts from login views.
The Access logs shows access log, see examples:
Jan. 22, 2020, 8:46 p.m.-127.0.0.1catalinMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36/admin/login/
Jan. 21, 2020, 6:42 p.m.Jan. 22, 2020, 8:45 p.m.127.0.0.1catalinMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36/admin/login/
You can set the axes into settings.py file , see this link.


Tuesday, January 21, 2020

Python 3.7.5 : Use Django Formsets.

Django Formsets manage the complexity of multiple copies of a form in a view.
This simplifies the task of creating a formset for a form that handles multiple instances of a model.
Let's start with my old project:
[mythcat@desk ~]$ cd projects/
[mythcat@desk projects]$ cd django/
[mythcat@desk django]$ source env/bin/activate
Into models.py I add these classes:
#create Inline Form with book and author
class Author(models.Model):
    author_book = models.CharField(max_length = 100)
    def __str__(self):
        return self.author_book

class Book(models.Model):
    book_name = models.CharField(max_length = 100)
    author_book_name = models.ForeignKey(Author,on_delete=models.CASCADE)
    def __str__(self):
        return self.book_name

(env) [mythcat@desk mysite]$ python3 manage.py makemigrations
Migrations for 'test001':
  test001/migrations/0004_author_book.py
    - Create model Author
    - Create model Book
(env) [mythcat@desk mysite]$ python3 manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, axes, contenttypes, sessions, test001
Running migrations:
  Applying test001.0004_author_book... OK
Add the classes to admin.py:
from .models import Author, Book
...
admin.site.register(Author)
admin.site.register(Book)
Now you can log in into the admin area and add authors and then add books.

Add the source code to views.py:
...
# add to views.py Author and Book 
from .models import Author, Book
# Author , Book redirect 
from django.shortcuts import redirect
...
# author and book source code 
def index_next(request, author_id):
    author = Author.objects.get(pk=author_id)
    BookFormset = inlineformset_factory(Author,Book, fields=('book_name',))
    if request.method == 'POST':
        formset = BookFormset(request.POST,instance = author)
        if formset.is_valid():
            formset.save()
            return redirect('index_next',author_id = author_id)
    formset = BookFormset(instance = author)
    return render(request, 'index_next.html', {'formset': formset})
...
Let's fix the URL for the next step.
Add the source code to urls.py:
...
# add index_next to urls.py 
from test001.views import index_next
...
urlpatterns = [
    ...
    path('&lt author_id &gt',views.index_next, name = 'index_next'),
    ...
    ]
Add index_next.html file into the template folder and into this file write HTML5 with a form and one submit button.
In the form tag add this:
{{ formset.as_p }}
Run the runserver command:
(env) [mythcat@desk mysite]$ python3 manage.py runserver
Use this http://127.0.0.1:8000/1 to see the first on the database shown on the browser.
You can customize the output of inline form, see source code:
...
BookFormset = inlineformset_factory(Author,Book, fields=('book_name',), can_delete=False, extra=1)
...
See the full project on my GitHub account at django_chart project repo.

Monday, January 20, 2020

Python 3.7.5 : Django security issues - part 002.

The project can be found at this Github project.
Let's start with my default project and activate the env:
[mythcat@desk ~]$ cd projects/
[mythcat@desk projects]$ cd django/
[mythcat@desk django]$ source env/bin/activate
Let's install this python module:
(env) [mythcat@desk django]$ pip3 install django-axes --user
Make these changes into settings.py:
(env) [mythcat@desk django]$ cd mysite/
(env) [mythcat@desk mysite]$ ls
db.sqlite3  manage.py  mysite  test001
(env) [mythcat@desk mysite]$ cd mysite/
(env) [mythcat@desk mysite]$ vim settings.py 
Into your settings.py add axes:
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'test001',
    'rest_framework',
    'axes'
] 
Add this source code in settings.py:
AUTHENTICATION_BACKENDS = [
    # AxesBackend should be the first backend in the AUTHENTICATION_BACKENDS list.
    'axes.backends.AxesBackend',

    # Django ModelBackend is the default authentication backend.
    'django.contrib.auth.backends.ModelBackend',
] 
Add axes.middleware.AxesMiddleware into MIDDLEWARE area:
MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'axes.middleware.AxesMiddleware',
] 
Check the configuration with this command:
(env) [mythcat@desk mysite]$ cd ..
(env) [mythcat@desk mysite]$ python manage.py check
System check identified no issues (0 silenced).
Use this command to sync the database:
(env) [mythcat@desk mysite]$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, axes, contenttypes, sessions, test001
Running migrations:
  Applying axes.0001_initial... OK
  Applying axes.0002_auto_20151217_2044... OK
  Applying axes.0003_auto_20160322_0929... OK
  Applying axes.0004_auto_20181024_1538... OK
  Applying axes.0005_remove_accessattempt_trusted... OK
  Applying axes.0006_remove_accesslog_trusted... OK
Once Axes is is installed and configured, you can login and logout of your application via the django.contrib.auth views.
This python package can be integrated with some popular 3rd party packages such as Django Allauth, Django REST Framework, and other tools.
I will come with additional information about this python package in the future.

Saturday, January 18, 2020

Python 3.7.5 : Django security issues - part 001.

Django like any website development and framework implementation requires security settings and configurations.
Today I will present some aspects of this topic and then I will come back with other information.
1. First, check your security vulnerabilities by the following command:
[mythcat@desk django]$ source env/bin/activate
(env) [mythcat@desk django]$ cd mysite
(env) [mythcat@desk mysite]$ python3 manage.py check --deploy
...
  File "/home/mythcat/projects/django/mysite/mysite/settings.py", line 14
    <<<<<<< HEAD
     
This shows us the bad changes in source code, is added by GitHub features.
Let's run it again:
(env) [mythcat@desk mysite]$ python3 manage.py check --deploy
System check identified some issues:

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. 
If your entire site is served only over SSL, you may want to consider setting a value and 
enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling
 HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site 
should be available over both SSL and non-SSL connections, you may want to either set this 
setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes 
it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W016) You have 'django.middleware.csrf.CsrfViewMiddleware' in your MIDDLEWARE, but you have
 not set CSRF_COOKIE_SECURE to True. Using a secure-only CSRF cookie makes it more difficult for network
 traffic sniffers to steal the CSRF token.
?: (security.W018) You should not have DEBUG set to True in deployment.
?: (security.W020) ALLOWED_HOSTS must not be empty in deployment.
?: (security.W022) You have not set the SECURE_REFERRER_POLICY setting. Without this, your site 
will not send a Referrer-Policy header. You should consider enabling this header to protect user privacy.

System check identified 7 issues (0 silenced).
This output show us the security warnning problems.
2. Use the Observatory by Mozilla site to scan the security status of your Django website.
3. Django has built-in security against most forms of CSRF threats, but The CSRF protection cannot protect against man-in-the-middle attacks.
Use HTTPS with HTTP Strict Transport Security by add these lines in your settings.py file.
CSRF_COOKIE_SECURE = True #to avoid transmitting the CSRF cookie over HTTP accidentally.
SESSION_COOKIE_SECURE = True #to avoid transmitting the session cookie over HTTP accidentally.
4. A Cross-site Scripting (XSS) allows an attacker to inject a script into the content of a website or application.
In your settings.py use this:
django.middleware.security.SecurityMiddleware
...
SECURE_BROWSER_XSS_FILTER = True
SECURE_CONTENT_TYPE_NOSNIFF = True
5. You can create fake admin login page using django-admin-honeypot to attempt unauthorized access.
6. Use SSL Redirect on your settings.py file.
SECURE_SSL_REDIRECT = True
7. Add Content Security Policy (CSP) to your Django website with the installed django-csp, add following lines to your settings.py file:
# Content Security Policy
CSP_DEFAULT_SRC = ("'none'", )
CSP_STYLE_SRC = ("'self'", )
CSP_SCRIPT_SRC = ("'self'", )
CSP_IMG_SRC = ("'self'", )
CSP_FONT_SRC = ("'self'", )
# Google Tag Manager or Google Analytics should be allowed in your CSP policy. 
CSP_DEFAULT_SRC = ("'none'", )
CSP_STYLE_SRC = ("'self'", "fonts.googleapis.com", "'sha256-/3kWSXHts8LrwfemLzY9W0tOv5I4eLIhrf0pT8cU0WI='")
CSP_SCRIPT_SRC = ("'self'", "ajax.googleapis.com", "www.googletagmanager.com", "www.google-analytics.com")
CSP_IMG_SRC = ("'self'", "data:", "www.googletagmanager.com", "www.google-analytics.com")
CSP_FONT_SRC = ("'self'", "fonts.gstatic.com")
CSP_CONNECT_SRC = ("'self'", )
CSP_OBJECT_SRC = ("'none'", )
CSP_BASE_URI = ("'none'", )
CSP_FRAME_ANCESTORS = ("'none'", )
CSP_FORM_ACTION = ("'self'", )
CSP_INCLUDE_NONCE_IN = ('script-src',)
The HTTP Strict Transport Security can be set into your settings.py file:
SECURE_HSTS_SECONDS = 86400  # 1 day
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True
8. Use records login attempts to your Django powered site and prevents attackers from brute forcing using the django-axes.
This tutorial solves most of the security issues of any website built with Django and not just related to this framework.