Saturday, June 27, 2020

Python 3.8.3 : PyCryptodome python package - part 001.

In the last tutorial, I wrote on Sunday, June 16, 2019, you can see a simple example of this python package with KDF with PBKDF2 function.
I guess it should be interesting for visitors to this blog to read more about this package because it is very useful and interesting.
Today I come up with another tutorial covering how to use A.E.S. standard encryption and decrypting text in a binary file.
The A.E.S. is a standard?
The Federal Information Processing Standards Publications (FIPS PUBS) announcing the A.E.S. on November 26, 2001, on the Federal Information Processing Standards Publication 197.
A.E.S. known as Advanced Encryption Standard is a symmetric block cipher standardized by NIST.
The N.I.S.T is an abbreviation National Institute of Standards and Technology.
This python package named PyCryptodome is a self-contained Python package of low-level cryptographic primitives, see the readthedocs.io webpage.
First, you need to see if this python package is not on conflict with another one named PyCrypto.
Then use pip3 tool to install.
pip3 uninstall PyCrypto
WARNING: Skipping PyCrypto as it is not installed.
...
pip3 install pycryptodome
Collecting pycryptodome
...
Installing collected packages: pycryptodome
Successfully installed pycryptodome-3.9.8
Here is the source code commented for a better understanding of the encryption and decryption steps.
from Crypto.Cipher import AES
from Crypto.Random import get_random_bytes

# the data input needs to be encoded, else you will receive this error
# python TypeError: Object type  cannot be passed to C code

data = "Hello World! by catafest!".encode("utf8")
# first step - prepare for encrypt
key = get_random_bytes(16)
print("print key is: ", key)
cipher = AES.new(key, AES.MODE_EAX)
print("print cipher is: ", cipher)
ciphertext, tag = cipher.encrypt_and_digest(data)
print("print ciphertext is: ", ciphertext)

# open the binary file 
file_out = open("AES_encrypted.bin", "wb")
# write to binary file 
[ file_out.write(x) for x in (cipher.nonce, tag, ciphertext) ]

# the file is close 
file_out.close()

# next step - prepare for decrypt
new_cipher = AES.new(key, AES.MODE_EAX, cipher.nonce)

# open again the binary file 
file_in = open("AES_encrypted.bin", "rb")
# read all data from file
nonce, tag, ciphertext = [file_in.read(x) for x in (16, 16, -1) ]
# the file is close 
file_out.close()

# show data from file
print("print nonce is: ", nonce)
print("print tag is: ", tag)
print("print ciphertext is: ", ciphertext)

# create a new cipher
cipher = AES.new(key, AES.MODE_EAX, nonce)

# use this to decrypt
data = cipher.decrypt_and_verify(ciphertext, tag)
# show the result 
print(data)
The result of running the python script is this:
python.exe .\pycryptodome_AES_001.py
print key is:  b'\xbdEX\xf8\x1d!\xc5\xceI\x87\x81\xf1\xd5\xba\x8c\r'
print cipher is:  <Crypto.Cipher._mode_eax.EaxMode object at 0x000001D5F32DE100>
print cipher is:  <Crypto.Cipher._mode_eax.EaxMode object at 0x000001E062AFE100>
print ciphertext is:  b'ON\x1d\xb9\xb7\xa8\xf5\xd6\x0c\x91\xc5`B\xf4\x95u\xe1D\xb5\x88&I\x15\xc5\xc5'
print nonce is:  b'3\xa6R8\xbb\n \x9cimp$\xe4\xee\xf5-'
print tag is:  b'\xb8)\xe4\xe7\x08uE~\x84s]\xedX\xf5\xf9\xea'
print ciphertext is:  b'ON\x1d\xb9\xb7\xa8\xf5\xd6\x0c\x91\xc5`B\xf4\x95u\xe1D\xb5\x88&I\x15\xc5\xc5'
b'Hello World! by catafest!'
You can read about this process on this website.

Python 2.7.10 : IronPython and C# with Dynamic Language Runtime.

This is a simple tutorial about python and C# using the Dynamic Language Runtime with IronPython.
I use Visual Studio 2019 and .NET Framework 4.7.2 with my Console C# project named DynamicLanguageRuntime_001.
Let's install the package with Visual Studio by open the console using the main menu: Tools - NuGet Package Manager - Package Manager Console command.
PM> Install-Package DynamicLanguageRuntime
Package 'DynamicLanguageRuntime.1.2.3' already exists in project 'DynamicLanguageRuntime_001'
Time Elapsed: 00:00:01.2208674
Use Solution Explorer use right-click on References item from your project and use Add Reference ...
Into the new window dialog named Reference Manager on the Assemblies - Framework uses the edit box to search IronPython.
Then use the checkbox to select these two options: IronPython and IronPython.Modules.
See the screenshot from Visual Studio I.D.E.:

This is the source code I used:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using IronPython.Hosting;
using IronPython.Runtime;
using IronPython;
using Microsoft.Scripting.Hosting;
using Microsoft.Scripting;

namespace DynamicLanguageRuntime_001
{
    class Program
    {
        static void Main(string[] args)
        {
            // create python engine 
            ScriptEngine engine = Python.CreateEngine();
            // get and add paths to enfine
            var python_search_paths = engine.GetSearchPaths();
            python_search_paths.Add(@"C:\Program Files\IronPython 2.7\Lib");
            engine.SetSearchPaths(python_search_paths);
            // create a scope 
            ScriptScope scope = engine.CreateScope();
            // using CreateScriptSourceFromString
            engine.CreateScriptSourceFromString("print '... simple example with ironpython and C#'").Execute();
            // using Execute
            engine.Execute("print '                             by catafest!'", scope);
            // using ExecuteFile
            engine.ExecuteFile(@"D:\Projects\Python\testing\test_webpage_001.py", scope);
            dynamic testFunction = scope.GetVariable("GetFriends");
            var result = testFunction();
        }
    }
} 

Friday, June 26, 2020

Python 3.6.9 : My colab tutorials - part 005.

Today I tested google colab python features with google authentification and google.colab drive and files.
The first part of google colab code comes with authentification and you need to add the verification code for google account.
The google colab use the same version of python:
3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]
You can see all source code on my GitHub account.
The notebook can be found here.

Sunday, June 21, 2020

Python 3.8.3 : Using twitter application with python-twitter - part 002.

This is the second part of tutorials series with python-twitter.
Today I will show you how to get pieces of information about friends, users and save into a binary file with pickle known as cPickle.
I will use my old source code from the last tutorial.
import os
import twitter
# for save to file import by python version
try:
   import cPickle as pickle
except:
   import pickle

consumer_key=' '
consumer_secret=' '
token_key=' '
token_secret=' '

if __name__ == "__main__":
    api = twitter.Api(consumer_key=consumer_key,
                  consumer_secret=consumer_secret,
                  access_token_key=token_key,
                  access_token_secret=token_secret) 
    
    screen_name = 'catafest'
       
    # print all users of this account authentificated 
    # you can use GetFriends(screen_name=screen_name) 
    users = api.GetFriends()
    
    print([u.screen_name for u in users])
    # get followers 
    followers = api.GetFollowers(screen_name=screen_name)
    # print followers 
    print([f.screen_name for f in followers])
    
    # ... and save into a binary file 
    followers_file = "followers_file.bin"
    
    if not os.path.exists(followers_file):
        pickle.dump(followers, open(followers_file, "wb"), protocol=pickle.HIGHEST_PROTOCOL)
        
    # load binary file     
    if os.path.exists(followers_file):
        followers_read = pickle.load(open(followers_file, "rb"))
        print(followers_read)
The result is similar with this:
python.exe .\test_webpage_001.py
['SnapChick', 'NASA', 'andor_saga', 'blendermarket', 'Minehut', 'Aternos', 'axnro', 'Flexi23',
...
['PStackoverflow', 'SamLeac86078418', 'Sohanurr559', 'jasonalba', 'avkorablev', 'dotnetfiddle',
...
[User(ID=1260415029855256583, ScreenName=PStackoverflow), 
...

Saturday, June 20, 2020

Python 3.8.3 : Using twitter application with python-twitter - part 001.

You need to create a application for your twitter user developer on this webpage.
The next step is to get all keys and tokens from your application.
I used the python-twitter see the official webpage documentation.
Let's install this python module using the pip tool
pip install python-twitter
Collecting python-twitter
...
Installing collected packages: oauthlib, requests-oauthlib, python-twitter
Successfully installed oauthlib-3.1.0 python-twitter-3.5 requests-oauthlib-1.3.0
Let's see a simple source code:
import os
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import twitter
import datetime
from datetime import *

consumer_key=' '
consumer_secret=' '
token_key=' '
token_secret=' '

def get_tweets(api=None, screen_name=None):
    timeline = api.GetUserTimeline(screen_name=screen_name, count=200)
    earliest_tweet = min(timeline, key=lambda x: x.id).id
    print("getting tweets before:", earliest_tweet)

    while True:
        tweets = api.GetUserTimeline(
            screen_name=screen_name, max_id=earliest_tweet, count=200
        )
        new_earliest = min(tweets, key=lambda x: x.id).id

        if not tweets or new_earliest == earliest_tweet:
            break
        else:
            earliest_tweet = new_earliest
            print("getting tweets before:", earliest_tweet)
            timeline += tweets

    return timeline

if __name__ == "__main__":
    api = twitter.Api(consumer_key=consumer_key,
                  consumer_secret=consumer_secret,
                  access_token_key=token_key,
                  access_token_secret=token_secret) 
    # print api 
    #print(dir(api))
    
    # print all users of this account authentificated 
    #users = api.GetFriends()
    #print([u.screen_name for u in users])
    
    # print all tweets of my user catafest 
    screen_name = "catafest"
    timeline = get_tweets(api=api, screen_name=screen_name)
    dates = []
    for x in timeline:
        created = x.created_at
        dates.append(created)
        
    print(dates)
    dat = [datetime.strptime(d, "%a %b %d %H:%M:%S +0000 %Y") for d in dates]

    levels = np.tile([-8, 8, -4, 4, -1, 1],int(np.ceil(len(dat)/3)))[:len(dat)]
    print(levels)
    fig, ax = plt.subplots(figsize=(7.6, 5), constrained_layout=True)
    ax.set(title="Twitter dates")
    markerline, stemline, baseline = ax.stem(dat, levels,linefmt="C3-", basefmt="k-",use_line_collection=True)
    markerline.set_ydata(np.zeros(len(dat)))
    plt.setp(markerline, mec="k", mfc="w", zorder=1)
    plt.show()
The result of this script comes with this output:
python .\test_webpage_001.py
getting tweets before: 1123237192422367234
['Mon May 18 13:52:09 +0000 2020', 'Sat May 09 11:14:43 +0000 2020', 'Fri May 08 10:42:18 +0000 2020', 
'Fri May 08 10:41:37 +0000 2020', 'Sat May 02 17:41:07 +0000 2020', 'Sat May 02 17:39:15 +0000 2020', 
'Thu Apr 30 12:53:48 +0000 2020', 'Tue Apr 28 20:00:38 +0000 2020', 'Mon Apr 27 21:12:07 +0000 2020', 
'Fri Apr 24 16:39:58 +0000 2020', 'Fri Apr 24 16:09:26 +0000 2020', 'Sat Apr 11 16:56:40 +0000 2020', 
'Sun Mar 22 19:11:16 +0000 2020', 'Sat Mar 21 09:03:30 +0000 2020', 'Sat Mar 21 09:02:48 +0000 2020', 
'Sat Mar 21 08:59:18 +0000 2020', 'Mon Mar 16 06:29:34 +0000 2020', 'Fri Jan 24 19:59:38 +0000 2020', 
'Sat Jan 18 12:14:07 +0000 2020', 'Fri Jan 17 20:58:18 +0000 2020', 'Thu Jan 16 20:50:47 +0000 2020', 
'Thu Jan 16 20:49:16 +0000 2020', 'Fri Jan 03 17:57:33 +0000 2020', 'Sat Dec 28 10:14:11 +0000 2019', 
'Tue Apr 30 14:46:30 +0000 2019']
[-8  8 -4  4 -1  1 -8  8 -4  4 -1  1 -8  8 -4  4 -1  1 -8  8 -4  4 -1  1 -8]
The image show with matplotlib is this:

Python 3.7.5 : Django on Fedora distro.

[mythcat@desk django]$ source env/bin/activate
(env) [mythcat@desk django]$ python3 
Python 3.7.6 (default, Dec 19 2019, 22:52:49) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> django.VERSION
(3, 0, 1, 'final', 1) 

Static files are those files that can not be processed, generated or modified by the server.
Static files improve the performance of the website with the template inheritance method.
Static file management is an important factor in web development.
I will show you how static file works on Django project.
The new static files folder is set on settings.py file:
...
# Static files (CSS, JavaScript, Images)

STATIC_URL = '/static/'

STATIC_ROOT = os.path.join(BASE_DIR, '/home/mythcat/projects/django/mysite/test001/')
STATICFILES_DIRS = [
   os.path.join(BASE_DIR, 'static'),
 ]
...
Let's run the server:
(env) [mythcat@desk mysite]$ python3 manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
January 26, 2020 - 09:01:10
Django version 3.0.1, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C. 
If I try to use the admin area then we can see the bad result for static files. 
...
[26/Jan/2020 09:10:02] "GET /admin/test001/post/ HTTP/1.1" 200 5935
[26/Jan/2020 09:10:02] "GET /static/admin/css/changelists.css HTTP/1.1" 200 6190
[26/Jan/2020 09:10:02] "GET /admin/jsi18n/ HTTP/1.1" 200 3223
[26/Jan/2020 09:10:02] "GET /static/admin/js/jquery.init.js HTTP/1.1" 200 363
[26/Jan/2020 09:10:02] "GET /static/admin/js/urlify.js HTTP/1.1" 200 8941
[26/Jan/2020 09:10:02] "GET /static/admin/js/actions.js HTTP/1.1" 200 6766
[26/Jan/2020 09:10:02] "GET /static/admin/js/prepopulate.js HTTP/1.1" 200 1530
[26/Jan/2020 09:10:02] "GET /static/admin/js/core.js HTTP/1.1" 200 5723
[26/Jan/2020 09:10:02] "GET /static/admin/js/admin/RelatedObjectLookups.js HTTP/1.1" 200 6918
[26/Jan/2020 09:10:02] "GET /static/admin/css/fonts.css HTTP/1.1" 200 423
[26/Jan/2020 09:10:02] "GET /static/admin/js/vendor/jquery/jquery.js HTTP/1.1" 200 280364
[26/Jan/2020 09:10:02] "GET /static/admin/js/vendor/xregexp/xregexp.js HTTP/1.1" 200 128820
[26/Jan/2020 09:10:02] "GET /static/admin/img/tooltag-add.svg HTTP/1.1" 200 331
[26/Jan/2020 09:10:02] "GET /static/admin/img/sorting-icons.svg HTTP/1.1" 200 1097
Not Found: /favicon.ico
... 
Now I can try to run the command collectstatic.
When this command is executed, Django performs these operations:
  • it looks for static files in all the directories listed in STATICFILES_DIRS;
  • the static-files are then copied and saved in STATIC_ROOT directory;
  • when the server is requested for static content, it will fetch a file from STATIC_ROOT;
  • that file will have its URL modified with STATIC_URL.
These errors show us many informations about this process:
 (env) [mythcat@desk mysite]$ python3 manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings:

    /home/mythcat/projects/django/mysite

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
Found another file with the destination path 'admin/js/urlify.js'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
...
Found another file with the destination path 'django.png'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'favicon.ico'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'admin/js/urlify.js'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
...

Found another file with the destination path 'rest_framework/js/prettify-min.js'. 
It will be ignored since only the first encountered file is collected. If this is not what you want, make
 sure every static file has a unique path.
...
166 static files copied to '/home/mythcat/projects/django/mysite'. 
But, Django respects the order of your applications in settings.py file the INSTALLED_APPS area, when running collectstatic command.
If you have two installed apps that write the same static files then Django collectstatic command will write the static files for the app appearing first in the list.
In my case: django.contrib.admin , test001 and ... .
Also, is need to set this code source on urls.py to return the proper URL pattern for serving static files to your already defined pattern list.
from django.contrib.staticfiles.urls import staticfiles_urlpatterns
...
urlpatterns += staticfiles_urlpatterns()
Let's run the runserver:
 (env) [mythcat@desk mysite]$ python3 manage.py runserver
...
[26/Jan/2020 09:29:18] "GET /static/admin/css/fonts.css HTTP/1.1" 200 423
Not Found: /favicon.ico 
You can see the /favicon.ico is not found in the default path and need to move it at /home/mythcat/projects/django/mysite/test001.
After these changes let's fix all the problems I created with these learning steps on static issues.
Let's move the static folder into mysite folder.
Change the settings.py file for the static issue with this source of code:
# Static files (CSS, JavaScript, Images)

STATIC_URL = '/static/'

STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = [
   os.path.join(BASE_DIR, 'static'),
 ]
Run the (env) [mythcat@desk mysite]$ python3 manage.py collectstatic
(env) [mythcat@desk mysite]$ python3 manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings:

    /home/mythcat/projects/django/mysite/static

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes

163 static files copied to '/home/mythcat/projects/django/mysite/static', 2 unmodified.
(env) [mythcat@desk mysite]$ ls
db.sqlite3  favicon.ico  manage.py  mysite  rest_framework  static  test001
(env) [mythcat@desk mysite]$ cd static/
(env) [mythcat@desk static]$ ls
admin  django.png  favicon.ico  rest_framework
This is a simple base tutorial.
In this point you can select the way of your Django project.