analitics

Pages

Thursday, August 1, 2019

Python 3.7.3 : Using the flask - part 005.

In the last tutorial, I used the flask-sqlalchemy python module.
Today I will show you how to use the flask_marshmallow python module.
First, let's take a look at this python module, see the official webpage:
Flask-Marshmallow is a thin integration layer for Flask (a Python web framework) and marshmallow (an object serialization/deserialization library) that adds additional features to marshmallow, including URL and Hyperlinks fields for HATEOAS-ready APIs. It also (optionally) integrates with Flask-SQLAlchemy.
The base class User will need to be integrated with this python module:
from flask import Flask
from flask import render_template
from forms import SignUpForm
from flask import request

from flask import jsonify
from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow
import os
from forms import AddUser

app = Flask (__name__)
app.config['SECRET_KEY'] = 'abcdefg'
# set SQLAlchemy
basedir = os.path.abspath(os.path.dirname(__file__))
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'server.sqlite')
db = SQLAlchemy(app)
ma = Marshmallow(app)
class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    email = db.Column(db.String(120), unique=True)

    def __init__(self, username, email):
        self.username = username
        self.email = email
    '''
    def __rep_(self):
        return '' % self.username
    '''
class UserSchema(ma.ModelSchema):
    class Meta:
        model = User
user_schema = UserSchema()
users_schema = UserSchema(many=True)

@app.route("/users/", methods=['GET'])
def users():
    users = User.query.all()
    #return users_schema.jsonify(users)
    all_users = users_schema.dump(users)
    return jsonify(all_users.data)

# the default name main
if __name__ == '__main__':
    app.run(debug=True)
Let's take a look at http://127.0.0.1:5000/users/ and see the result:
[
  {
    "email": "catafest@yahoo.com", 
    "id": 1, 
    "username": "catafest"
  }, 
  {
    "email": "test@test.com", 
    "id": 2, 
    "username": "user_test"
  }
]

Tuesday, July 30, 2019

Python 3.7.3 : Using the flask - part 004.

The goal of this tutorial is to interact with the database in order to use it with flask_sqlalchemy python module.
The db.Model is used to interact with the database.
A database doesn't need a primary key but if you using the flask-sqlalchemy you need to have it for each one table in order to connect it.
Let's see the database:
C:\Python373\my_flask>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD6
4)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
>>> db.create_all()
>>> db.engine.table_names()
['user']
Let's add some data into database on user table:
C:\Python373\my_flask>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD6
4)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
>>> from server import User
>>> first_user = User(username='catafest',email='catafest@yahoo.com')
>>> db.session.add(first_user)
>>> db.session.commit()
>>> test_user = User(username='test',email='test@test.com')
>>> db.session.add(test_user)
>>> db.session.commit()
Update is a simple issue.
Let's update the username test_user from test to user_test:
>>> test_user.username = 'user_test'
>>> db.session.commit()
The delete is simple like the add:
>>> db.session.delete(test_user)
>>> db.session.commit()
Let's use query:
>>> results=User.query.all()
>>> results[0].username
'catafest'
>>> results[0].email
'catafest@yahoo.com'
>>> results
The next step is an important issue because let you to see how result by content and query and filter by first result:
>>> q1 = User.query.filter_by(username='catafest')
>>> q1
...flask_sqlalchemy .basequery= ...
>>> print(q1)
SELECT user.id AS user_id, user.username AS user_username, user.email AS user_email
FROM user
WHERE user.username = ?
>>> q2 = User.query.filter_by(username='catafest').first()
>>> q2
< User 1 >
>>> print(q2)
< User 1 >
>>> print(q1.username)
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'BaseQuery' object has no attribute 'username'
>>> print(q2.username)
catafest
>>> print(q2.username,q2.email)
catafest catafest@yahoo.com 
In this case because the first is limited to one result the print of q2 is the correct way.

Monday, July 29, 2019

Python 3.7.3 : Using the twitter python module - part 003.

Today I will speak about twitter python module with the new changes of the A.P.I.
This two tutorial will not work now because the twitter A.P.I is changed.
The reason I don't delete it is the similar flow programming and access the A.P.I.:
Let's start with the install of this python module with Python version 3.7.3:
C:\Python373>cd Scripts

C:\Python373\Scripts>pip install python-twitter
...
Installing collected packages: python-twitter
Successfully installed python-twitter-3.5
Let's test the GetSearch.
You need to create a twitter application to have access to the tokens and secret keys:
import os
import json
import twitter
from twitter import *
CONSUMER_KEY=""
CONSUMER_SECRET=""

ACCESS_TOKEN=""
ACCESS_TOKEN_SECRET=""
LANGUAGES="En"
at=input("ACCESS_TOKEN: ", )
ats=input("ACCESS_TOKEN_SECRET: ", )
ck=input("CONSUMER_KEY: ", )
cs=input("CONSUMER_SECRET: ", )
api = Api(ck, cs, at, ats)
def main():
    print("Search by query using the GetSearch ")
    r = api.GetSearch(raw_query="q=twitter%20&result_type=recent&since=1999-03-07&count=100")
    print(r)
if __name__ == '__main__':
    main()
The result will be something like this:
...
@GeorgePapa19 @realDonaldTrump Sure Did https://t.co/GblrSOsaJg'), Status(ID=115
5787444525993984, ScreenName=MD__PCY, Created=Mon Jul 29 10:29:34 +0000 2019, Te
xt='mal ako ni twitter https://t.co/lRA1BOyf6a'), Status(ID=1155787444517838849,
 ScreenName=Elyse95, Created=Mon Jul 29 10:29:34 +0000 2019, Text='????? ????? ?
??? ..??? ???? ????????? ????? ????? ????? ?????? ??? ???????? .. !!    ... http
s://t.co/weChoAPDnC. https://t.co/34D2nmtbOz'), Status(ID=1155787444517781504, S
creenName=chrichacham123, Created=Mon Jul 29 10:29:34 +0000 2019, Text='RT @John
JCrace: Project Blind Faith! https://t.co/2mpB0MC540'), Status(ID=11557874445134
72514, ScreenName=318520_mu, Created=Mon Jul 29 10:29:34 +0000 2019, Text='31-85
20 ????????\n???????? → ???twitter??????????????\n??...?\n\nhttps://t.co/oVET5Z9
wXq')]

Sunday, July 28, 2019

Python 3.7.3 : Using the flask - part 003.

Another tutorial with python 3.7.3 and flask python module.
In the last tutorial, I speak about some tricks and tips.
Today, I will show some steps for fixing and run with flask-sqlalchemy.
The source code from my GitHub account can be updated with this source code.
app = Flask (__name__)
app.config['SECRET_KEY'] = 'abcdefg'
# set SQLAlchemy
basedir = os.path.abspath(os.path.dirname(__file__))
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'server.sqlite')
db = SQLAlchemy(app)
ma = Marshmallow(app)
class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    email = db.Column(db.String(120), unique=True)

    def __init__(self, username, email):
        self.username = username
        self.email = email
If you run it then you get this:
C:\Python373\my_flask>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD6
4)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
C:\Python373\lib\site-packages\flask_sqlalchemy\__init__.py:835: FSADeprecationW
arning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be dis
abled by default in the future.  Set it to True or False to suppress this warnin
g.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
You will have a database file named server.sqlite.
#app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
Let's create and show the database:
C:\Python373\my_flask>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD6
4)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
>>> db.create_all()
>>> db.engine.table_names()
['user']
A good approach is to create a config.py file and used into the application area.
The config.py file will have this source code:
SQLALCHEMY_TRACK_MODIFICATIONS = False
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'server.sqlite')
DEBUG=True
The server.py has this source code:
...
app = Flask (__name__)
app.config.from_pyfile('config.py')
db = SQLAlchemy(app)
...

Saturday, July 27, 2019

Python 3.7.3 : About pytweening python module.

This is a simple module of tweening and easing functions implemented in Python, see the GitHub webpage.
C:\Python373\Scripts>pip install pytweening
Collecting pytweening
Downloading https://files.pythonhosted.org/packages/b9/f8/c32a58d6e4dff8aa5c27
e907194d69f3b57e525c2e4af96f39c6e9c854d2/PyTweening-1.0.3.zip
Building wheels for collected packages: pytweening
Building wheel for pytweening (setup.py) ... done
Created wheel for pytweening: filename=PyTweening-1.0.3-cp37-none-any.whl size
=3821 sha256=6655c055d779982ff1259a5266bf1300c1cd02046e45e92cf18b20053a326531
Stored in directory: C:\Users\catafest\AppData\Local\pip\Cache\wheels\7b\92\30
\06e21159eed2709436bfb6d7c690959e578cf74f029643866e
Successfully built pytweening
Installing collected packages: pytweening
Successfully installed pytweening-1.0.3
Let's test it:
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD6
4)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytweening
>>> dir(pytweening)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__',
'__package__', '__path__', '__spec__', '__version__', '_checkRange', 'division',
 'easeInBack', 'easeInBounce', 'easeInCirc', 'easeInCubic', 'easeInElastic', 'ea
seInExpo', 'easeInOutBack', 'easeInOutBounce', 'easeInOutCirc', 'easeInOutCubic'
, 'easeInOutElastic', 'easeInOutExpo', 'easeInOutQuad', 'easeInOutQuart', 'easeI
nOutQuint', 'easeInOutSine', 'easeInQuad', 'easeInQuart', 'easeInQuint', 'easeIn
Sine', 'easeOutBack', 'easeOutBounce', 'easeOutCirc', 'easeOutCubic', 'easeOutEl
astic', 'easeOutExpo', 'easeOutQuad', 'easeOutQuart', 'easeOutQuint', 'easeOutSi
ne', 'getLine', 'getPointOnLine', 'linear', 'math']
>>> pytweening.linear(0.1)
0.1
>>> pytweening.easeInOutSine(0.1)
0.024471741852423234
>>> pytweening.easeInQuad(0.1)
0.010000000000000002
>>> pytweening.getLine(0, 0, 0.1, 0.1)
[(0, 0)]
>>> pytweening.getLine(0, 0, 1.1, 1.1)
[(0, 0), (1, 1)]
For example, this module can help with some math function:
  • simple linear tweening - no easing, no acceleration;
  • quadratic easing in/out - acceleration until halfway, then deceleration;
  • cubic easing in/out - acceleration until halfway, then deceleration;
  • quartic easing in/out - acceleration until halfway, then deceleration;
  • sinusoidal easing in/out - accelerating until halfway, then decelerating;
  • ...

Friday, July 26, 2019

Python 3.7.3 : Tonny I.D.E. for python programmers.

Today I tested the Thonny I.D.E. from thonny.org official webpage.
Yesterday I tried several editors for python programming language and did not work.
One of these is the spyder editor that does not work with python 3.7.3 - we have not discovered why.
The Mu is a simple Python editor for beginner programmers and has a strange working I.D.E. for good and fast development.
The PyCharm, it is developed by the Czech company JetBrains is not fully free ...

The Thonny I.D.E. is fast and comes with two good features:
Code completion and Highlights syntax errors.
The install steps are easy with a wizard application.
After that, you can test with any python source code.
The source code from the image is a simple example from this webpage.
Let's see a screenshot:

This is a video tutorial with this I.D.E. from Aivar Annamaa user:



Thursday, July 25, 2019

Python 3.7.3 : Using the flask - part 002.

Let's see some tips for starting any project with flask python module.
Use these python modules to work with databases: flask-sqlalchemy and flask_marshmallow.
The Flask-SQLAlchemy is an extension for Flask that adds support for SQLAlchemy to your application.
The marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, to and from native Python datatypes.
Let's install it:
C:\Python373>cd Scripts

C:\Python373\Scripts>pip install flask-sqlalchemy
Collecting flask_sqlalchemy
...
Successfully installed flask-sqlalchemy-2.4.0
C:\Python373\Scripts>pip install flask_marshmallow
Collecting flask_marshmallow
...
Successfully installed flask-marshmallow-0.10.1 marshmallow-2.19.5
C:\Python373\Scripts>pip install marshmallow-sqlalchemy
Collecting marshmallow-sqlalchemy
...
Successfully installed marshmallow-sqlalchemy-0.17.0
Use these python modules for login and authorization issues: flask-login
The Flask-Login provides user session management for Flask. It handles the common tasks of logging in, logging out, and remembering your users’ sessions over extended periods of time.
Flask-login doesn't actually have a user backend, it just handles the session machinery to help you to log in and logout users.
Flask-Login can work with user models that are based on any database system with four required items: is_authenticated, is_active, is_anonymous and get_id.
Let's install it:
C:\Python373\Scripts>pip install flask-login
Collecting flask-login
...
Successfully installed flask-login-0.4.1
This python module named python-dotenv let us to reads the key-value pair from .env file and adds them to an environment variable.
C:\Python373\Scripts>pip install python-dotenv
Collecting python-dotenv
...
Successfully installed python-dotenv-0.10.3
Flask-Login provides a UserMixin which makes it easy to create a user class, see example class User:
from flask_login import UserMixin
...
class User(UserMixin, db.Model):
Another tip is to use the get_or_404 and first_or_404 to avoid show errors on the webpage.
Reduce the passing of multiple variables to render_template, let's see the example:
from flask import render_template
...
return render_template('index.html', fn=name.first, ln=name.last, day=student.day, 
.... ')
This can de easy fix with this code to send value I want to pass:
context = {
'fn': name.first,
'ln': name.last,
'day': student.day,
...
}
return render_template('index.html', **context)
Use query to solve complex tasks.
Let's see a simple example of delete from a Student database class:
...
class Student(db.Model):
id=db.Column(db.Integer, primary_key=True)
...
country=db.Column(db.String(2))
...
Let's delete this with a query and return the number of deleted by delete_count:
del_ids= db.session.query(Student.id).filter(Student.country=='Ro')
delete_count=db.session.query(Student).filter(Student.id.in_(del_ids.subquery())).delete(synchronize_session=False)
I used two files in the main project named .env and .flaskenv:
The file .env comes with:
SQLALCHEMY_DATABASE_URI=sqlite:///db.sqlite3
SECRET_KEY = abcdef
The file .flaskenv has these lines of source code:
FLASK_ENV = development
FLASK_APP = my_flask_app
This can be used into a new file named settings.py:
import os 
SQLALCHEMY_DATABASE_URI = os.environ.get('SQLALCHEMY_DATABASE_URI')
SECRET_KEY = os.environ.get('SECRET_KEY')
SQLALCHEMY_TRACK_MODIFICATIONS = False
Using this the main flask application will start like this:
from flask import Flask
def create_my_app(config_file='settings.py'):
    app = Flask(__name__)
    app.config.from_pyfile(config_file)
    return app
There are plenty of tips and tricks that can be found based on the project in the bottle.
These are the most used.


Wednesday, July 24, 2019

Python 3.7.3 : Testing the timeit and Bokeh python module.

The tutorial today has several goals:
  • testing the timeit function to measure the execution time of a line of print code;
  • using python lists to record data;
  • generate lists using the range function;
  • multiplying a number with an entire list of numbers;
  • how to use the bokeh module with CustomJSHover and HoverTool
Let's test with the timeit python module, see the official webpage.
This module provides a simple way to time small bits of Python code. It has both a Command-Line Interface as well as a callable one. It avoids a number of common traps for measuring execution times.
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> import random
>>> timeit.timeit('print("a")',number=1)
a
0.00013114200010022614
>>> timeit.timeit('print("a")',number=100)
...
a
0.007177434000027461
>>> timeit.timeit('print("a")',number=1000)
...
a
0.07585798000002342
Let make more visible the results with the bokeh python module:
import timeit
import random

# if you want to use numpy 
#import numpy as np

# import bokeh python module 
from bokeh.plotting import figure, show, output_file
# for show values
from bokeh.models.tools import CustomJSHover
from bokeh.models import HoverTool
print ('''
You can tests a number of times named l
with a number of print named i
and see the result value with bokeh python module
timeit.timeit('print("a")',number = i)
''')
i = int(input('The increment number i:'))
l = int(input('The number of test l:'))

# create the list with the output values
timit_list = []
# create the test function for l and i and return the list
def get_timeit(i,l):
 while i < l:
  i += 1
  out=timeit.timeit('print("a")',number = i)
  timit_list.append(out)
 return timit_list

# run the test function with l and i
# this will be coordinate y
yt=get_timeit(i,l)

# show result of the test 
print(yt)

# create the coordinate x
xt = [i for i in range(0, len(yt))]

#xt = np.linspace(0, l, l)
# print the coordinate x
#print(xt)

# create the output HTML file to see the result
output_file("test.html")

# create a figure with a timeit type y-axis
fig = figure(title='timit_list values for print("a")',
             plot_height=400, plot_width=700,
             x_axis_label='x has each incrementation of i', y_axis_label='value of timeit of print the char a',
             x_minor_ticks=3, 
             toolbar_location=None)

# create a circle for each value
# see new multiplication with a list 
# y=[i * 100 for i in yt]
fig.circle(x=xt, y=[i * 100 for i in yt], 
         color='blue', size=5,
         legend='Values')

x_custom = CustomJSHover(code="""
    return '' + special_vars.data_x""")

y_custom = CustomJSHover(code="""
    return '' + special_vars.data_y""")

fig.add_tools(
    HoverTool(
        show_arrow=True, 
        tooltips=[
 ('xt', '$data_x'),
 ('yt', '$data_y')
        ],
        formatters=dict(
            xt=x_custom,
            yt=y_custom
        )
    )
)

# Put the legend in the upper left corner
fig.legend.location = 'top_left'

# Let's check it out
show(fig)
The result of this python source code can be found on my YouTube channel:


Tuesday, July 23, 2019

Python 3.7.3 : Using the flask - part 001.

A short intro into this python module can be found at the PyPI website:
Flask is a lightweight WSGI web application framework. It is designed to make getting started quick and easy, with the ability to scale up to complex applications. It began as a simple wrapper around Werkzeug and Jinja and has become one of the most popular Python web application frameworks.

Flask offers suggestions but doesn’t enforce any dependencies or project layout. It is up to the developer to choose the tools and libraries they want to use. There are many extensions provided by the community that makes adding new functionality easy.

The reason I used a series of tutorials with this python module is the complexity of the features of this python module.
Let's briefly outline some of the essential aspects of flask programming.
  • Flask is a simple, lightweight, and minimalist web framework;
  • Flask is developed based on the Jinja2 template engine;
  • Flask depends on the Jinja template engine and the Werkzeug WSGI toolkit;
  • Flask does not provide a built-in ORM system (Object Relation Mapping);
  • in Flask web applications to perform CRUD operations on a database can be tedious (see: ORM techniques of Flask-SQLAlchemy);
  • The Flask has a simple and customizable architecture;
  • the Flask to accelerate the development of simple websites that use static content;
  • has the option to extend and customize Flask according to precise project requirements;
A list of companies using the Flask framework - who is using Flask?
The install process is very simple using the pip tool:
C:\Python373>cd Scripts

C:\Python373\Scripts>pip install flask
Collecting flask
...
Installing collected packages: itsdangerous, Werkzeug, flask
Successfully installed Werkzeug-0.15.4 flask-1.1.1 itsdangerous-1.1.0
Let's start initializing the first flask application into the server.py file:
from flask import Flask 
app = Flask (__name__)
# create a wrarp of localhost
@app.route('/')
def home():
 return 'Hello world'
# the default name main 
if __name__ == '__main__':
 app.run()
About Routing and Variable Rules.
This can change it into the server.py script like this:
@app.route('/about')
def about():
 return 'The about page'
@app.route('/blog')
def blog():
 return 'This is the blog'
The next step is the variable rule issue:
# creaza a regula variabila ( variable rules) 
# the type of the variable can be set, see int 
# @app.route('/blog/')
@app.route('/blog/')
def blogpost(blog_id):
 return 'This is the post '+str(blog_id)
To run it, just use:
C:\Python373\my_flask>python server.py
 * Serving Flask app "server" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
The full source code can be found at my GitHub account.

Monday, July 22, 2019

Python 3.7.3 : The sip python module.

The official webpage pypi.org comes with this intro:
One of the features of Python that makes it so powerful is the ability to take existing libraries, written in C or C++, and make them available as Python extension modules. Such extension modules are often called bindings for the library.

SIP is a tool that makes it very easy to create Python bindings for C and C++ libraries. It was originally developed to create PyQt, the Python bindings for the Qt toolkit, but can be used to create bindings for any C or C++ library.

SIP comprises a code generator and a Python module. The code generator processes a set of specification files and generates C or C++ code which is then compiled to create the bindings extension module. The sip Python module provides support functions to the automatically generated code.

The SIP is copyright (c) Riverbank Computing Limited and its homepage is this webpage.
Support may be obtained from the PyQt mailing list at here.
The SIP is a tool for quickly writing Python modules that interface with C++ and C libraries.
The SIP comprises a code generator and a Python module.
About the code generator:
First, the install:
C:\Python373\Scripts>pip3 install sip
Collecting sip
...
Installing collected packages: sip
Successfully installed sip-4.19.8
If you using the PyQt5 then this version includes a private copy of the module.
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyQt5 import sip
If you want to see backward compatibility the module then needs to imported and will only work if another PyQt5 module is imported first.
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyQt5 import QtCore
>>> import sip
A good example of how can you build with SIP can be seen here.
About the SIP Python module:
This provides support functions to the automatically generated code.
The import python module sip let you use all functions build by the code generator:
>>> dir(sip)
['SIP_VERSION', 'SIP_VERSION_STR', '_C_API', '__doc__', '__file__', '__loader__'
, '__name__', '__package__', '__spec__', '_unpickle_enum', '_unpickle_type', 'as
sign', 'cast', 'delete', 'dump', 'enableautoconversion', 'enableoverflowchecking
', 'getapi', 'isdeleted', 'ispycreated', 'ispyowned', 'setapi', 'setdeleted', 's
etdestroyonexit', 'settracemask', 'simplewrapper', 'transferback', 'transferto',
 'unwrapinstance', 'voidptr', 'wrapinstance', 'wrapper', 'wrappertype']
If you want to improve or create python modules with C or C ++, then this tool can help.

Sunday, July 21, 2019

Python 3.7.3 : The IMDbPY python module version 6.8.

The GitHub official webpage comes with this intro:
IMDbPY is a Python package for retrieving and managing the data of the IMDb movie database about movies, people and companies.
The last release version 6.8 was at 2019 Jul 20.
The official webpage tells us:
In the release 6.8 (codename "Apollo 11") of IMDbPY, multiple parsers were added and fixes; the new search_movie_advanced method allows advanced movie searches...
The changes of the version 6.8 can be found at GitHub webpage and come with these new features:
#224: introduce the search_movie_advanced(title, adult=None, results=None, sort=None, sort_dir=None) method
#145: names are stored in normal format (Name Surname)
#225: remove the obsolete cookie
#182: box office information
#168: parse series and episode number searching for movies
#217: grab poster from search
#218: extract MPAA rating
#220: extract actor headshot from full credits
The install on Python 3.7.3 is easy with pip3 tool:
C:\Python373\Scripts>pip3 install imdbpy
Collecting imdbpy
...
Installing collected packages: imdbpy
Successfully installed imdbpy-6.8
Let's test the new features:
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import imdb
>>> ia = imdb.IMDb()
>>> movies = ia.search_movie_advanced('debby does dallas', adult=True)
>>> print(movies)
...
>>> people = ia.search_person('Clark Gregg')
>>> print(people)
Let's test it:
import imdb
from imdb import IMDb

ia = imdb.IMDb()

# create a file to put the output 
file1 = open("_imdb_data.txt","w", encoding='utf-8') 

# get movies by movie 
# example: Alien 

def get_by_movie():
 my_movie = str(input('Type the movie name: '))
 movies = ia.search_movie_advanced(my_movie, adult=True)
 print(type(movies))
 return movies

# get filmography by id 
filmography_list = []
def get_filmography_by_id(id):
 actor_results = ia.get_person_filmography(id)
 for item in actor_results['data']['filmography']:
  filmography_list.append(str(item))
 return filmography_list

# the main function 
def main():
 a = get_by_movie()
 for i in a:
  print("________________________")
  print("i: ",i)
  # you can uncomment this to test Movie class functions
  #print("Type:",type(i))
  #print("Summary:",i.summary())
  #print("ID: ",i.getID())
  #print("Smart cannonical title: ",i.smartCanonicalTitle())
  #print("caracters ref: ",i.get_charactersRefs())
  #print("current info: ",i.get_current_info())
  #print("cinematographic process: ",i.get('cinematographic process'))
  #print(i["title"])
  #print informations items from Movie class
  print("~~~~~~~~~~~~~~~~~~~~~~~~")
  for k, v in i.items():
   print(k, v)
   # write to the file the value of a
   txt = str(k)+":"+str(v)+"\n" 
   file1.write(txt)
  print("------------------------")
  # add a new line on each movie
  file1.write('-----^-----\n')
 #get filmography by id 
 id_filmography=get_filmography_by_id('0078748')
 # print the filmography
 for item in id_filmography:
  print(item)

 #after write, close the file 
 file1.close()

if __name__ == '__main__':
    main()
This is the first part of the output file named _imdb_data:
title:Alien
certificates:['R']
runtimes:['117']
genres:['Horror', 'Sci-Fi']
rating:8.5
votes:719508
metascore:89
gross:78900000
plot:After a space merchant vessel perceives an unknown transmission as a distress call, its landing on the source moon finds one of the crew attacked by a mysterious lifeform, and they soon realize that its life cycle has merely begun.
directors:[]
cast:[, , , ]
cover url:https://m.media-amazon.com/images/M/MV5BMmQ2MmU3NzktZjAxOC00ZDZhLTk4YzEtMDMyMzcxY2IwMDAyXkEyXkFqcGdeQXVyNzkwMjQ5NzM@._V1_UX67_CR0,0,67,98_AL_.jpg
year:1979
kind:movie
canonical title:Alien
long imdb title:Alien (1979)
long imdb canonical title:Alien (1979)
smart canonical title:Alien
smart long imdb canonical title:Alien (1979)
full-size cover url:https://m.media-amazon.com/images/M/MV5BMmQ2MmU3NzktZjAxOC00ZDZhLTk4YzEtMDMyMzcxY2IwMDAyXkEyXkFqcGdeQXVyNzkwMjQ5NzM@.jpg
-----^-----
title:Aliens
certificates:['R']
runtimes:['137']
...

Saturday, July 20, 2019

Python 3.7.3 : Use BeautifulSoup to parse Instagram account.

This example is a bit more complex because it parses the source code in a more particular way depending on it.
The basic idea of this script is to take the content of an Instagram account in the same way as a web browser.
For my account I found a parsing error, I guess the reason is using the points, see festila.george.catalin.
    scripts_content = json.loads(scripts[0].text.strip())
IndexError: list index out of range
In this case comment this line of code and will work:
For the other accounts I've tried, it works very well with the default script.
This is the script I used:
import requests
from bs4 import BeautifulSoup
import json
import re

from pprint import pprint

instagram_url = 'https://instagram.com'
#example user instagram profile_url = sherwoodseries
profile_url=str(input("name of the instagram user: "))


#UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f48b' in posit ion 5022: character maps to 
#fix write text file with  encoding='utf-8'
file1 = open("_shared_data.txt","w", encoding='utf-8') 

#profile_url = 'festila.george.catalin'
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(f"{instagram_url}/{profile_url}", headers = headers)

if response.ok:
    html = response.text
    bs_html = BeautifulSoup(html, "html.parser")
    print(bs_html)
    # get info from ... type="application/ld+json">{"@context":"http:\/\/schema.org","@type":"Person","name":
    scripts = bs_html.select('script[type="application/ld+json"]')
    #scripts_content = json.loads(scripts[0].text.strip())
    #pprint(scripts_content)

    #print scripts_content like json 
    #print(json.dumps(scripts_content,indent = 4,sort_keys = True))

    #print just part of source code get by 'script' (0 .. n), see n = 6 
    #print(bs_html.find_all('script')[6])
    script_tag = bs_html.find('script', text=re.compile('window\._sharedData'))
    shared_data = script_tag.string.partition('=')[-1].strip(' ;')

    #get item from shared data, see "language_code":"en"
    rex_item  = re.compile('(?<=\"language_code\":\")[a-zA-Z_\- ]+(?=\")')
    rex_get_item = rex_item.findall(shared_data)  
    print(rex_get_item)
    #get url image from shared data
    rex_url  = re.compile('(?<=\"display_url\":\")[^\s\"]+(?=\")')
    rex_get_url = rex_url.findall(shared_data)  
    print(rex_get_url)
 
    # load like a json 
    result_json = json.loads(shared_data)
    pprint(result_json)
    
    data = bs_html.find_all('meta', attrs={'property': 'og:description'})
    bb = data[0].get('content').split()
    user = '%s %s %s' % (bb[-3], bb[-2], bb[-1])
    # get from bb parts 
    posts = bb[4]
    print('all string: ',bb)
    print('number of posts: ',posts)
    print('name and the user: ',user)

    # write any output show by print into _a.txt file, see example
    #file1.write(str(bs_html.find_all('script')[4]))
    #example: write to _shared_data.txt file the shared_data
    #file1.write(str(shared_data))
#after write, close the file 
#file1.close() 
This is a part of the output for sherwoodseries account:
...
all string:  ['95', 'Followers,', '24', 'Following,', '56', 'Posts', '-', 'See',
 'Instagram', 'photos', 'and', 'videos', 'from', 'Sherwood', 'Series', '(@sherwo
odseries)']
number of posts:  56
name and the user:  Sherwood Series (@sherwoodseries)

Thursday, July 18, 2019

Python 3.7.3 : The pandas python module.

Since I started learning python programming language I have not found a more complex and complete module for viewing complex data.
The official documentation of this python module tells us:
pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real-world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open-source data analysis/manipulation tool available in any language. It is already well on its way toward this goal.
The official webpage can be found here.
This python module is one of the most popular Python libraries for Data Science and Analytics.
You can install this python module with pip tool:
C:\Python373\Scripts>pip install pandas
Requirement already satisfied: pandas in c:\python373\lib\site-packages (0.24.2)
You can find many tutorials on web with this python module.
Today I will show you a short tutorial about this python module.
Most users use this both python modules:
import numpy as np
import pandas as pd
Most area of the pandas python module has a target into this list:
Window Functions, Aggregations, Missing Data, GroupBy, Merging/Joining, Concatenation, Date Functionality, Timedelta, Categorical Data,
Visualization, IO Tools, Sparse Data, Caveats & Gotchas, Comparison with SQL
There are two types of data structures in pandas: Series and DataFrames.
The pandas Series is a one-dimensional data structure.
The pandas DataFrame is a two (or more) dimensional data structure, like a table
Pandas provide few variants rolling, expanding and exponentially moving weights for window statistics.
Also have the sum, mean, median, variance, covariance, correlation, etc.
The several methods are available to perform aggregations on data.
Pandas provide functions for missing data like the isnull() and notnull().
Let's test the DataFrames with pandas and the Wikipedia example from my tutorial
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> # get table from wikipedia
... import requests
>>> from bs4 import BeautifulSoup
>>> website_url = requests.get('https://en.wikipedia.org/w/index.php?title=Table
_of_food_nutrients').text
>>> soup = BeautifulSoup(website_url,'lxml')
>>>
>>> my_table = soup.find('table',{'class':'wikitable collapsible collapsed'})
>>> links = my_table.findAll('a')
>>> Food = []
>>> for link in links:
...     Food.append(link.get('title'))
...
>>> print(Food)
["Cows' milk (page does not exist)", 'Buttermilk', 'Fortified milk (page does no
t exist)', 'Powdered milk', "Goats' milk", 'Malted milk', 'Hot chocolate', 'Yogu
rt', 'Milk pudding (page does not exist)', 'Custard', 'Ice cream', 'Ice milk', '
Cream', 'Cheese', 'Cheddar cheese', 'American cheese', 'Processed cheese', 'Egg
(food)', 'Scrambled', 'Omelet', 'Yolk']
>>> import pandas
>>> import pandas as pd
>>> df = pd.DataFrame()
>>> df['Foods'] = Food
>>> print(df)
                                   Foods
0       Cows' milk (page does not exist)
1                             Buttermilk
2   Fortified milk (page does not exist)
3                          Powdered milk
4                            Goats' milk
5                            Malted milk
6                          Hot chocolate
7                                 Yogurt
8     Milk pudding (page does not exist)
9                                Custard
10                             Ice cream
11                              Ice milk
12                                 Cream
13                                Cheese
14                        Cheddar cheese
15                       American cheese
16                      Processed cheese
17                            Egg (food)
18                             Scrambled
19                                Omelet
20                                  Yolk
>>> df.describe()
              Foods
count            21
unique           21
top     Malted milk
freq              1
>>> df.apply(pd.Series.value_counts)
                                      Foods
Malted milk                               1
Ice milk                                  1
Omelet                                    1
Goats' milk                               1
Custard                                   1
Cheddar cheese                            1
American cheese                           1
Ice cream                                 1
Yolk                                      1
Cream                                     1
Cows' milk (page does not exist)          1
Yogurt                                    1
Fortified milk (page does not exist)      1
Egg (food)                                1
Powdered milk                             1
Milk pudding (page does not exist)        1
Cheese                                    1
Hot chocolate                             1
Buttermilk                                1
Processed cheese                          1
Scrambled                                 1
The last example is to show data with
>>> import numpy as np
>>> import pandas as pd
>>> import matplotlib.pyplot as plt
>>> ts = pd.Series(np.random.randn(76), index=pd.date_range('1/1/76', periods=76
))
>>> ts.plot()

>>> plt.show()

Wednesday, July 17, 2019

Python 3.7.3 : Using the pipenv tool.

The goal of this tutorial is how to use pipenv and manage dependencies and development environments on a collaborative Python project.
The documentation area can be found here.
C:\Python373>cd Scripts

C:\Python373\Scripts>pip install --user pipenv
Collecting pipenv
...
Successfully installed pipenv-2018.11.26 virtualenv-clone-0.5.3
If you see errors like this:
WARNING: The scripts pipenv-resolver.exe and pipenv.exe are installed in 'C:\Users\....'
which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, 
use --no-warn-script-location.
You need to add this path show into the WARNING to the windows path, see command:
setx path "%path%;C:\Users\..\Python37\Scripts"
Let use this tool to install a requests package:
C:\Python373\Scripts>cd ..

C:\Python373>cd my_flask

C:\Python373\my_flask>pipenv install requests
Creating a virtualenv for this project.
Pipfile: C:\Python373\my_flask\Pipfile
Using c:\python373\python.exe (3.7.3) to create virtualenv.
[  ==] Creating virtual environment...Already using interpreter c:\python373\pyt
hon.exe
Using base prefix 'c:\\python373'
New python executable in C:\Users\catafest\.virtualenvs\my_flask-j9hGDZgP\Script
s\python.exe
Installing setuptools, pip, wheel...
done.
Successfully created virtual environment!
Virtualenv location: C:\Users\catafest\.virtualenvs\my_flask-j9hGDZgP
Creating a Pipfile for this project.
Installing requests.
Adding requests to Pipfile's [packages].
Installation Succeeded
Pipfile.lock not found, creating.
Locking [dev-packages] dependencies.
Locking [packages] dependencies.
Success!
Updated Pipfile.lock (444a6d)!
Installing dependencies from Pipfile.lock (444a6d).
  ================================ 5/5 - 00:00:04
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
The syntax for the Pipfile is TOML, and the file is separated into sections. [dev-packages] for development-only packages, [packages] for minimally required packages, and [requires] for other requirements like a specific version of Python
If you got this error you need restarts the command prompt:
'pipenv' is not recognized as an internal or external command,
operable program or batch file.
The next command let us see output diagnostic information for use in GitHub issues:
C:\Python373\my_flask>pipenv --support | clip
C:\Python373\my_flask>pipenv --support
Check for security vulnerabilities (and PEP 508 requirements) in your environment with:
C:\Python373\my_flask>pipenv check
Checking PEP 508 requirements.
Passed!
Checking installed package safety.
All good!
Let's use the example from the documentation webpage:
import requests

response = requests.get('https://httpbin.org/ip')

print('Your IP is {0}'.format(response.json()['origin']))
The result works great.
C:\Python373\my_flask>pipenv run python my_ip.py
Your IP is ...
You can print out a tree-like structure showing your dependencies.
C:\Python373\my_flask>pipenv graph
requests==2.22.0
  - certifi [required: >=2017.4.17, installed: 2019.6.16]
...
Using this command will see a reversed tree may be more useful when you are trying to figure out conflicting sub-dependencies.
C:\Python373\my_flask>pipenv graph --reverse
certifi==2019.6.16
  - requests==2.22.0 [requires: certifi>=2017.4.17]
chardet==3.0.4
...
This command installs all the dependencies needed for development:
C:\Python373\my_flask>pipenv install --dev
Installing dependencies from Pipfile.lock (444a6d).
  ================================ 5/5 - 00:00:10
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
You can uninstall a python module:
pipenv uninstall numpy
You can replace --all with --all-dev to just remove dev packages.
Let’s start a shell in a virtual environment to isolate the development of this app:
C:\Python373\my_flask>pipenv shell
Launching subshell in virtual environment.
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

(my_flask-j9hGDZgP) C:\Python373\my_flask>
The package name, together with its version and a list of its own dependencies, can be frozen by updating the Pipfile.lock with the lock keyword.
(my_flask-j9hGDZgP) C:\Python373\my_flask>pipenv lock
Locking [dev-packages] dependencies.
Locking [packages] dependencies.
Success!
Updated Pipfile.lock (444a6d)!
Clears caches (pipenv, pip, and pip-tools) with:
pipenv --clear
Remove the virtualenv created:
(my_flask-j9hGDZgP) C:\Python373\my_flask>pipenv --rm
Close the pipenv shell:
(my_flask-j9hGDZgP) C:\Python373>exit

C:\Python373\my_flask>
This tool requires special attention in developing and programming with Python, so we have only traveled some of its real possibilities.


Tuesday, July 16, 2019

Python 3.7.3 : Using the werkzeug.

From the official webpage, you can see all the features of this python module:
Werkzeug is a comprehensive WSGI web application library. It began as a simple collection of various utilities for WSGI applications and has become one of the most advanced WSGI utility libraries.
It includes:
An interactive debugger that allows inspecting stack traces and source code in the browser with an interactive interpreter for any frame in the stack.
A full-featured request object with objects to interact with headers, query args, form data, files, and cookies.
A response object that can wrap other WSGI applications and handle streaming data.
A routing system for matching URLs to endpoints and generating URLs for endpoints, with an extensible system for capturing variables from URLs.
HTTP utilities to handle entity tags, cache control, dates, user agents, cookies, files, and more.
A threaded WSGI server for use while developing applications locally.
A test client for simulating HTTP requests during testing without requiring running a server.

The documentation of this python module can be found here.
You can install this python module with the pip tool:
pip install -U Werkzeug
This module can install it if you install the flask python module with python 3.7.3.
Let's test it:
C:\Python373>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from werkzeug.wrappers import Request, Response
>>>
>>> @Request.application
... def application(request):
...     return Response('Hello, World!')
...
>>> if __name__ == '__main__':
...     from werkzeug.serving import run_simple
...     run_simple('localhost', 4000, application)
...
 * Running on http://localhost:4000/ (Press CTRL+C to quit)
127.0.0.1 - - [16/Jul/2019 13:29:44] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [16/Jul/2019 13:29:46] "GET /favicon.ico HTTP/1.1" 200 -
If you run this source code and open your browser with this http://localhost:4000/ address then you will see a text with Hello, World!.
The common uses of this python package are:
>>> from werkzeug.wrappers import Request, Response
>>> from werkzeug.routing import Map, Rule
>>> from werkzeug.exceptions import HTTPException, NotFound
>>> from werkzeug.wsgi import SharedDataMiddleware
>>> from werkzeug.formparser import parse_form_data
>>> from werkzeug.utils import redirect
>>> from werkzeug.utils import escape
>>> from werkzeug.serving import run_simple
Let's test another simple application with werkzeug python module:
from werkzeug.serving import run_simple
from werkzeug.wrappers import Request
from werkzeug.wrappers import Response

class Application(object):
    def __init__(self):
        print('init Application class')

    def dispatch_request(self, request):
        return Response("Logged in as %s" % request.authorization)

    def __call__(self, environ, start_response):
        request = Request(environ)
        auth = request.authorization
        response = self.dispatch_request(request)
        return response(environ, start_response)

if __name__ == "__main__":
    application = Application()
    run_simple("localhost", 5000, application)
The result will be this:
C:\Python373>python.exe werkzeug_001.py
init Application class
 * Running on http://localhost:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [16/Jul/2019 14:05:31] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [16/Jul/2019 14:05:32] "GET /favicon.ico HTTP/1.1" 200 -
You can add more features to this source code and check with werkzeug features of Web Server Gateway Interface (WSGI).
A full list with examples can be found here.

Monday, July 15, 2019

Python 3.7.3 : Programming Krita.

Today I wrote a python tutorial about Krita software and programming python.
The Krita software use python version 3.6.2.
==== Warning: Script not saved! ====
3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:57:36) [MSC v.1900 64 bit (AMD64)]
The full tutorial can be found at my Blogspot (a Blogspot about the graphics area).

Sunday, July 14, 2019

Python 3.7.3 : Simple tests with zipfile python module.

You can read about this python module here.
The ZIP file format is a common archive and compression standard. This module provides tools to create, read, write, append, and list a ZIP file. Any advanced use of this module will require an understanding of the format, as defined in PKZIP Application Note.

This module does not currently handle multi-disk ZIP files. It can handle ZIP files that use the ZIP64 extensions (that is ZIP files that are more than 4 GiB in size). It supports decryption of encrypted files in ZIP archives, but it currently cannot create an encrypted file. Decryption is extremely slow as it is implemented in native Python rather than C.

C:\Python373> python -m zipfile -c test.zip test.html  textalongpath.pdf
C:\Python373> python -m zipfile -c test_folder.zip test.html  temp
C:\Python373>python -m zipfile -l  test.zip
File Name                                             Modified             Size
test.html                                      2019-07-11 10:46:58         6115
textalongpath.pdf                              2019-06-08 22:55:50           84

C:\Python373>python -m zipfile -l  test_folder.zip
File Name                                             Modified             Size
test.html                                      2019-07-11 10:46:58         6115
temp/                                          2019-07-07 21:36:42            0
temp/wlop/                                     2019-07-07 21:36:42            0
This lines of code will create two archives named test.zip and test_folder.zip with the files shown on each command.
For extraction, is need to use the -e argument:
C:\Python373>python -m zipfile -e test.zip zipfiles/

C:\Python373>python -m zipfile -e test_folder.zip zipfiles/

C:\Python373>cd zipfiles

C:\Python373\zipfiles>dir
...
Let's using this python module inside python:
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import zipfile
>>> import datetime
>>> my_zip = zipfile.ZipFile('test.zip','r')
>>> print (my_zip.namelist())
['test.html', 'textalongpath.pdf']
>>> def print_info(archive_name):
...     my_zip = zipfile.ZipFile(archive_name)
...     for info in my_zip.infolist():
...             print (info.filename)
...             print ('Comment: ', info.comment)
...             print ('Modified: ', datetime.datetime(*info.date_time))
...             print ('System: ', info.create_system, '(0 = Windows, 3 = Unix)'
)
...             print ('ZIP version: ', info.create_version)
...             print ('Compressed: ', info.compress_size, 'bytes')
...             print ('Uncompressed: ', info.file_size, 'bytes')
...
>>> print_info('test_folder.zip')
test.html
Comment:  b''
Modified:  2019-07-11 10:46:58
System:  0 (0 = Windows, 3 = Unix)
ZIP version:  20
Compressed:  1679 bytes
Uncompressed:  6115 bytes
temp/
Comment:  b''
Modified:  2019-07-07 21:36:42
...
Extract all files from an archive:
>>> from zipfile import ZipFile
>>> with ZipFile('test.zip','r') as zipObj:
...     zipObj.extractall()
Extract files by extension:
>>> with ZipFile('test.zip', 'r') as zipObj:
...    listOfFileNames = zipObj.namelist()
...    for fileName in listOfFileNames:
...        if fileName.endswith('.html'):
...            zipObj.extract(fileName, 'new.html')
...
'new.html\\test.html'
Create a new arhive named The_new.zip and add the new.html file on it.
>>> zipObj = ZipFile('The_new.zip','w')
>>> zipObj.write('new.html')
>>> zipObj.close()
>>> print_info('The_new.zip')
new.html/
Comment:  b''
Modified:  2019-07-14 22:15:58
System:  0 (0 = Windows, 3 = Unix)
ZIP version:  20
Compressed:  0 bytes
Uncompressed:  0 bytes

Thursday, July 11, 2019

Python 3.7.3 : Three examples with BeautifulSoup.

Beautiful Soup is a library that makes it easy to scrape information from web pages. It sits atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying the parse tree., see the pypi webpage.
This python module was created by Leonard Richardson.
A large definition can be this:
Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting, etc.) is a technique employed to extract large amounts of data from websites whereby the data is extracted and saved to a local file in your computer or to a database in table (spreadsheet) format.
This python module can do that but the input format and output format is different.
The input can be a webpage like an URL or webpage with all pieces of information and the output depends by the this and the user choices.
Les's see some examples:
First example show you how to take content of the first row table from a wikipedia webpage.
# get table from wikipedia 
import requests
from bs4 import BeautifulSoup
website_url = requests.get('https://en.wikipedia.org/w/index.php?title=Table_of_food_nutrients').text
soup = BeautifulSoup(website_url,'lxml')

my_table = soup.find('table',{'class':'wikitable collapsible collapsed'})
links = my_table.findAll('a')
Food = []
for link in links:
    Food.append(link.get('title'))

print(Food)
The next example takes all files from a page

# get links using the url
import urllib
from bs4 import BeautifulSoup
page = urllib.request.urlopen('http://____share.net/filmes/').read()
soup = BeautifulSoup(page)
soup.prettify()
for anchor in soup.findAll('a', href=True):
    print (anchor['href'])
The last example takes all images from the search query of imgur website:
# get images from imgur search query
import urllib
from bs4 import BeautifulSoup
url = 'https://imgur.com/search/score?q=cyborg'
with urllib.request.urlopen(url) as f:
    soup = BeautifulSoup(f.read(),'lxml')

a_tags = soup.findAll("a",{"class":"image-list-link"})
img_tags = [a.find("img") for a in a_tags]
print(img_tags)
srcs = []
for s in img_tags:
    src_tags=('http:'+s['src'])
    srcs.append(src_tags)

print(srcs)
As a conclusion, this module will pose problems for those who do not understand how to scroll through the source code, the content of web pages, how to read 'lxml', 'page', etc.
It will greatly help your Chrome F12 key to access parts of web content.

Python 3.7.3 : Testing the Bokeh python module.

This python module has a beautiful website:
Bokeh is an interactive visualization library that targets modern web browsers for presentation. Its goal is to provide elegant, concise construction of versatile graphics, and to extend this capability with high-performance interactivity over very large or streaming datasets. Bokeh can help anyone who would like to quickly and easily create interactive plots, dashboards, and data applications.
Let's install this python module with the pip tool:
C:\Python373>cd Scripts

C:\Python373\Scripts>pip install bokeh
Collecting bokeh
...
Successfully built bokeh
Installing collected packages: PyYAML, tornado, bokeh
Successfully installed PyYAML-5.1.1 bokeh-1.2.0 tornado-6.0.3
Let's test it with a simple example:
from bokeh.plotting import figure, output_file, show

output_file("test.html")
plot = figure()
plot.line([1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 7, 6, 1, 5, 6, 7, 9, 1], line_width=2)
show(plot)
This will create a file into my python folder C:/Python373/test.html.
The HTML webpage comes with the graph and additional tool like Pan, Box Zoom, Wheel Zoom, Save, Reset and help.
If you need sample data that is not included in the Bokeh GitHub repository or released packages then you need to download it.
>>> import bokeh.sampledata
>>> bokeh.sampledata.download()
Creating C:\Users\catafest\.bokeh directory
Creating C:\Users\catafest\.bokeh\data directory
Using data directory: C:\Users\catafest\.bokeh\data
Downloading: CGM.csv (1589982 bytes)
   1589982 [100.00%]
Downloading: US_Counties.zip (3171836 bytes)
    229376 [  7.23%]
Bokeh comes with support for working with Geographical data: Mercator, Google Maps, GeoJSON Data.
You can see all example into the webpage gallery.

Wednesday, July 10, 2019

Python 3.7.3 : About python version 3.7.3.

All versions of python come with many features and changes with every released version.
A full list of these changes can be found at PEP official webpage and this documentation webpage.
The goal of these tutorials is to fix the learning area by each python version and have a good picture of these features.
Let's start with the first step - python modules.
Several of the standard library Python packages have been reorganized or moved with a few notable changes:
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> import profile
>>> import urllib
>>> import urllib3
The division math operation has new features to explicitly convert integers to floats when working with integer variables:
>>> one = 1
>>> two = 2
>>> three = 3
>>> float(three)
3.0
>>> one/two
0.5
>>> one//three
0
>>> one//two
0
>>> float(one)/three
0.3333333333333333
>>>
You can use the pathlib library which provides the Path() object to fulfill all your path manipulation needs.
>>> from pathlib import Path
>>> folder1 = Path('/folder1')
>>> config_path = folder1 / 'subfolder1'
>>> config_path
WindowsPath('/folder1/subfolder1')
>>> str(config_path)
'\\folder1\\subfolder1'
>>> config_path.name
'subfolder1'
You can use operators with matrix:
>>> import numpy as np
>>> x = np.array([[11, 33], [22, 55]])
>>> y = np.array([[1, 3], [2, 5]])
>>> x @ y
array([[ 77, 198],
       [132, 341]])
>>> x * y
array([[ 11,  99],
       [ 44, 275]])
The list and dictionaries can easily be emptied using the .clear method:
>>> my_list = ['a','b','c']
>>> my_list.clear()
The print function is changed.
>>> print('hello')
hello
>>> a = ''
>>> f = open('my_file.txt', 'w')
>>> print(a, file=f)
>>> f.close()
The function annotations can provide information on inputs/outputs:
>>> def a_to_b(x: str) -> str:
...     return x.replace('a','b')
...
>>> a_to_b("abcdcba!?!")
'bbcdcbb!?!'
Fix the sensible comparison:
>>> 'True' > True
Traceback (most recent call last):
  File "", line 1, in 
TypeError: '>' not supported between instances of 'str' and 'bool'
With Python 3.6 we have a new type of strings: f-strings and string interpolation:
>>> var = 76/3
>>> f'The value is {var}.'
'The value is 25.333333333333332.'
You can use underscores in numbers:
>>> int_a = 1_000_000_000
>>> hex_b = 0b_0011_1111_0100_1110
>>> print(int_a,hex_b)
1000000000 16206
The new Unicode strings and variable (including emoji) names to be used.
An LRU cache decorator for your functions: functools.lru_cache.
An enumerated type in the standard library: Enum.
Use the standard ipaddress:
>>> import ipaddress
>>> ipaddress.ip_address('192.168.0.1')
IPv4Address('192.168.0.1')
>>> ipaddress.ip_address('2001:db8::')
IPv6Address('2001:db8::')
In Python 3, decimals are rounded to the nearest even number (.5).
The input() function was fixed in Python 3 so that it always stores the user inputs as str objects.
In Python 3, the range() was implemented like the xrange() in older version.
That range got a new __contains__ method in Python 3.x.
You can simply convert the iterable object into a list via the list() function.
>>> print(range(3))
range(0, 3)
>>> print(type(range(3)))

>>> print(list(range(3)))
[0, 1, 2]
With advanced unpacking and range you can do this:
>>> a, b = range(2)
>>> print(a,b)
0 1
>>> a, b, *rest = range(6)
>>> print(a,b)
0 1
>>> print(rest)
[2, 3, 4, 5]
>>> a, *rest, b = range(6)
>>> print( a,b, rest)
0 5 [1, 2, 3, 4]
Get the first and the last of the open file with:
first, *_, last = f.readlines()
Keyword only arguments can be done with:
def f(a, b, *args, option=True):
The only way to access it is to explicitly call f(a, b, option=True).
If you don't want to collect *args the use this:
def f(a, b, *, option=True):
You can just use os.stat(file, follow_symlinks=False) instead of os.lstat.
The next function can be call only into this way:
>>> my_generator = (letter for letter in 'abcd')
>>> next(my_generator)
'a'
>>> next(my_generator)
'b'
The for-loop variables don’t leak into the global namespace anymore:
>>> i = 1
>>> print('comprehension:', [i for i in range(6)])
comprehension: [0, 1, 2, 3, 4, 5]
>>> print('i is ', i)
i is  1
The async and await are now reserved keywords.
More useful exceptions and also change the comma with the keyword as:
>>> try:
...     f = open('my_file.txt')
... except OSError as e:
...     if e.errno == errno.ENOENT:
...             #

...     else:

...     raise 
In Python 3, the .keys() method instead returns an iterator object instead of a list.
>>> my_dict = {'a': 11, 'b': 12, 'c': 13, 'd': 14}
>>> my_dict.keys()
dict_keys(['a', 'b', 'c', 'd'])
>>> my_dict_keys=list(my_dict.keys())
>>> my_dict_keys[3]
'd'
>>> my_dict.keys()[3]
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'dict_keys' object is not subscriptable
Keyword-only arguments and positional parameters are valid in Python 3.7.3.
The chained Exceptions provide by python 3.7.3 has more information and the original exception is printed out, along with the original traceback.
>>> raise exception from e
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'exception' is not defined
>>> raise NotImplementedError from OSError
OSError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "", line 1, in 
NotImplementedError
These were briefly some information about python 3.7.3 that might be of use to you.



Tuesday, July 9, 2019

Python 3.7.3 : Testing Wagtail.

The Wagtail is a beautiful project and can be integrated with Django.
In this tutorial, I will show you the steps for the first install of a basic website.
C:\Python373>python -m virtualenv venv_wagtail
Using base prefix 'C:\\Python373'
New python executable in C:\Python373\venv_wagtail\Scripts\python.exe
Installing setuptools, pip, wheel...
done.

C:\Python373>venv_wagtail\Scripts\activate.bat

(venv_wagtail) C:\Python373>pip install wagtail
Collecting wagtail
...
Successfully built django-treebeard draftjs-exporter
Installing collected packages: sqlparse, pytz, Django, django-taggit, chardet, i
dna, urllib3, certifi, requests, django-treebeard, draftjs-exporter, beautifulso
up4, django-modelcluster, djangorestframework, Willow, six, Pillow, webencodings
, html5lib, Unidecode, wagtail
Successfully installed Django-2.2.3 Pillow-5.4.1 Unidecode-1.1.1 Willow-1.1 beau
tifulsoup4-4.6.0 certifi-2019.6.16 chardet-3.0.4 django-modelcluster-4.4 django-
taggit-0.24.0 django-treebeard-4.3 djangorestframework-3.9.4 draftjs-exporter-2.
1.6 html5lib-1.0.1 idna-2.8 pytz-2019.1 requests-2.22.0 six-1.12.0 sqlparse-0.3.
0 urllib3-1.25.3 wagtail-2.5.1 webencodings-0.5.1

(venv_wagtail) C:\Python373>wagtail start mysite
Creating a Wagtail project called mysite

(venv_wagtail) C:\Python373>cd mysite

(venv_wagtail) C:\Python373\mysite>python manage.py migrate

(venv_wagtail) C:\Python373\mysite>python manage.py createsuperuser
Username (leave blank to use 'catafest'):
Email address: catafest@yahoo.com
Password:
Password (again):
This password is too short. It must contain at least 8 characters.
Bypass password validation and create user anyway? [y/N]: n
Password:
Password (again):
Superuser created successfully.

(venv_wagtail) C:\Python373\mysite>manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
July 09, 2019 - 21:20:46
Django version 2.2.3, using settings 'mysite.settings.dev'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.

Success! mysite has been created
Let's see the result of the first Wagtail test on my YouTube channel:


Python 3.7.3 : The python-slugify python module.

This python module named python-slugify can handle Unicode.
You can see this python module source code and examples at GITHUB webpage.
The install step with pip python tool is easy:
C:\Python373>cd Scripts
C:\Python373\Scripts>pip install python-slugify
Collecting python-slugify
...
Successfully built python-slugify
Installing collected packages: text-unidecode, python-slugify
Successfully installed python-slugify-3.0.2 text-unidecode-1.2
Let's see some simple example.
>>> from slugify import slugify
>>> txt = "___This is a test___"
>>> regex_pattern = r'[^-a-z0-9_]+'
>>> r = slugify(txt, regex_pattern=regex_pattern)
>>> print(r)
___this-is-a-test___
Remove an email address from a string:
>>> txt_email = "___My mail is catafest@yahoo.com *&@!@#$$76"
>>> regex_pattern =r'[\w\.]+\@[\w]+(?:\.[\w]{3}|\.[\w]{2}\.[\w]{2})\b'
>>> r = slugify(txt_email, regex_pattern=regex_pattern)
>>> print(r)
___my mail is - *&@!@#$$76
You can write your data into one text file:
>>> ftxt = open('C:\\Python373\\soup.txt','w')
>>> ftxt.write(soup)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: write() argument must be str, not BeautifulSoup
>>> ftxt.write(str(soup))
If you got errors the use the slugify to fix it:
ftxt.write(slugify(str(soup)))

Monday, July 8, 2019

Python 3.7.3 : Using attrgetter from operator python module.

Return a callable object that fetches attr from its operand. If more than one attribute is requested, returns a tuple of attributes. The attribute names can also contain dots. see documentation.
The attrgetter operator works more or less similar to itemgetter, except that it looks up an attribute instead of an index.
For example to test this operator we need to create a class to access the attribute on the class.
I create a class named Person with name and surname initialization.
I will get the attributes from this class and show it.
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import operator
>>> class Person:
...     def __init__(self, name, surname):
...             self.n = name
...             self.s = surname
...
>>> my_per = Person('Festila', 'Catalin George')
>>> getter = operator.attrgetter("n")
>>> print(getter)
operator.attrgetter('n')
>>> print(getter(my_per))
Festila
>>> another_getter = operator.attrgetter("s")
>>> print(another_getter)
operator.attrgetter('s')
>>> print(another_getter(my_per))
Catalin George
>>> all = operator.attrgetter("n","s")
>>> print(all)
operator.attrgetter('n', 's')
>>> print(all(my_per))
('Festila', 'Catalin George')
To restrict setting an attribute outside of constructor you need to use a private attribute starting with an underscore, and a read-only property for public access.
See the new class with the attribute s restricted:
class Person:
    def __init__(self, name, surname):
            self.n = name
            self.s = surname
s = property(operator.attrgetter("_s"))
Let's test it in another way:
>>> from operator import attrgetter
>>> test=attrgetter('something')
>>> print(test)
operator.attrgetter('something')
>>> print(type(test))

>>> print(test(something))
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'something' is not defined
>>> print(type(test(something)))
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'something' is not defined
Because something is not initialized correctly the error is expected for both: test(something) and type(test(something)
>>> something = 'all'
>>> print(type(test))

>>> print(type(test(something)))
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'str' object has no attribute 'something'
>>> something = 1
>>> print(type(test(something)))
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'int' object has no attribute 'something'

Friday, July 5, 2019

Python 3.7.3 : Use python with MySQL and MariaDB.

If you want to use MariaDB databases with python then you need to install the MySQL.
Use the pip tool to install the mysql-connector-python python module:
C:\Python373\Scripts>pip install mysql-connector-python
Collecting mysql-connector-python
...
Installing collected packages: mysql-connector-python
Successfully installed mysql-connector-python-8.0.16
The information about python module named mysql-connector-python version 8.0.16 can be found at this web page.
For the test, I install MariaDB from the official website with user root and my password.
Let's test the python module for MariaDB:
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mysql
>>> import mysql.connector as mariadb
After install, start the command prompt of MariaDB and test it:
C:\Windows\System32>mysql -u root -p
Enter password: ****
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.4.6-MariaDB mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.001 sec)
MariaDB [information_schema]> USE information_schema;
Database changed
MariaDB [information_schema]> SHOW TABLES;
...
| INNODB_TABLESPACES_SCRUBBING          |
| INNODB_SYS_SEMAPHORE_WAITS            |
+---------------------------------------+
77 rows in set (0.001 sec)

C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mysql.connector as mariadb
>>> mariadb_connection = mariadb.connect(user='root', password='test', database=
'information_schema')
>>> cursor = mariadb_connection.cursor()
>>> cursor.execute("USE information_schema")
>>> cursor.execute("SHOW TABLES")
>>> result = cursor.fetchall()
>>> print(result)
[('ALL_PLUGINS',), ('APPLICABLE_ROLES',), ('CHARACTER_SETS',), ('CHECK_CONSTRAIN
TS',), ('COLLATIONS',), ('COLLATION_CHARACTER_SET_APPLICABILITY',), ('COLUMNS',)
, ('COLUMN_PRIVILEGES',), ('ENABLED_ROLES',), ('ENGINES',), ('EVENTS',), ('FILES
',), ('GLOBAL_STATUS',), ('GLOBAL_VARIABLES',), ('KEY_CACHES',), ('KEY_COLUMN_US
AGE',), ('OPTIMIZER_TRACE',), ('PARAMETERS',), ('PARTITIONS',), ('PLUGINS',), ('
PROCESSLIST',), ('PROFILING',), ('REFERENTIAL_CONSTRAINTS',), ('ROUTINES',), ('S
CHEMATA',), ('SCHEMA_PRIVILEGES',), ('SESSION_STATUS',), ('SESSION_VARIABLES',),
 ('STATISTICS',), ('SYSTEM_VARIABLES',), ('TABLES',), ('TABLESPACES',), ('TABLE_
CONSTRAINTS',), ('TABLE_PRIVILEGES',), ('TRIGGERS',), ('USER_PRIVILEGES',), ('VI
EWS',), ('GEOMETRY_COLUMNS',), ('SPATIAL_REF_SYS',), ('CLIENT_STATISTICS',), ('I
NDEX_STATISTICS',), ('INNODB_SYS_DATAFILES',), ('USER_STATISTICS',), ('INNODB_SY
S_TABLESTATS',), ('INNODB_LOCKS',), ('INNODB_MUTEXES',), ('INNODB_CMPMEM',), ('I
NNODB_CMP_PER_INDEX',), ('INNODB_CMP',), ('INNODB_FT_DELETED',), ('INNODB_CMP_RE
SET',), ('INNODB_LOCK_WAITS',), ('TABLE_STATISTICS',), ('INNODB_TABLESPACES_ENCR
YPTION',), ('INNODB_BUFFER_PAGE_LRU',), ('INNODB_SYS_FIELDS',), ('INNODB_CMPMEM_
RESET',), ('INNODB_SYS_COLUMNS',), ('INNODB_FT_INDEX_TABLE',), ('INNODB_CMP_PER_
INDEX_RESET',), ('user_variables',), ('INNODB_FT_INDEX_CACHE',), ('INNODB_SYS_FO
REIGN_COLS',), ('INNODB_FT_BEING_DELETED',), ('INNODB_BUFFER_POOL_STATS',), ('IN
NODB_TRX',), ('INNODB_SYS_FOREIGN',), ('INNODB_SYS_TABLES',), ('INNODB_FT_DEFAUL
T_STOPWORD',), ('INNODB_FT_CONFIG',), ('INNODB_BUFFER_PAGE',), ('INNODB_SYS_TABL
ESPACES',), ('INNODB_METRICS',), ('INNODB_SYS_INDEXES',), ('INNODB_SYS_VIRTUAL',
), ('INNODB_TABLESPACES_SCRUBBING',), ('INNODB_SYS_SEMAPHORE_WAITS',)]

Thursday, July 4, 2019

Python 3.7.3 : Using itemgetter from operator python module.

The operator module exports a set of efficient functions corresponding to the intrinsic operators of Python. see documentation.
Today I will show you how to use itemgetter from this python module with python 3.7.3.
Let's see how to sort my two dictionaries named my_dict and my_dict2, using the classical lambda function;
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> my_dict = {'a': 11, 'b': 12, 'c': 13, 'd': 14}
>>> sorted(my_dict.items(), key = lambda x:x[1])
[('a', 11), ('b', 12), ('c', 13), ('d', 14)]
>>> my_dict2 = {'a': 31, 'b': 12, 'c': 33, 'd': 14}
>>> sorted(my_dict2.items(), key = lambda x:x[1])
[('b', 12), ('d', 14), ('a', 31), ('c', 33)]
Using the operator python module with my examples for both dictionaries:
>>> import operator
>>> sorted(my_dict.items(), key = operator.itemgetter(1))
[('a', 11), ('b', 12), ('c', 13), ('d', 14)]
>>> sorted(my_dict2.items(), key = operator.itemgetter(1))
[('b', 12), ('d', 14), ('a', 31), ('c', 33)]
>>> sorted(my_dict2.items(), key = operator.itemgetter(1), reverse=True)
[('c', 33), ('a', 31), ('d', 14), ('b', 12)]
The operator.itemgetter returns a callable object that fetches the item from its operand using the operand’s __getitem__() method.
If multiple items are specified, returns a tuple of lookup values.
Let's see another example:
>>> my_arr = []
>>> my_arr.append(["A","Z",77])
>>> my_arr.append(["bA","Zc",11])
>>> my_arr.append(["d","c",111])
>>> print(my_arr)
[['A', 'Z', 77], ['bA', 'Zc', 11], ['d', 'c', 111]]
>>> my_arr.sort(key=operator.itemgetter(1))
>>> my_arr
[['A', 'Z', 77], ['bA', 'Zc', 11], ['d', 'c', 111]]
>>> my_arr.sort(key=operator.itemgetter(1,2))
>>> my_arr
[['A', 'Z', 77], ['bA', 'Zc', 11], ['d', 'c', 111]]
>>> my_arr.sort(key=operator.itemgetter(2))
>>> my_arr
[['bA', 'Zc', 11], ['A', 'Z', 77], ['d', 'c', 111]]
>>> my_arr.sort(key=operator.itemgetter(2,1))
>>> my_arr
[['bA', 'Zc', 11], ['A', 'Z', 77], ['d', 'c', 111]]
After using the sort with itemgetter the array is changed it:

>>> i0 = operator.itemgetter(0)
>>> i0(my_arr)
['bA', 'Zc', 11]>>> i1 = operator.itemgetter(1)
>>> i1(my_arr)
['A', 'Z', 77]
>>> i2 = operator.itemgetter(2)
>>> i2(my_arr)
['d', 'c', 111]
>>> print(my_arr)
[['bA', 'Zc', 11], ['A', 'Z', 77], ['d', 'c', 111]]

Monday, July 1, 2019

Python 3.7.3 : Using the Pony python module.

The development team of the Pony project comes with this intro:
Using Pony object-relational mapper you can concentrate on writing the business logic of your application and use Python syntax for interacting with the database. Pony translates such queries into SQL and executes them in the database in the most efficient way.
The Pony python module can be installed on Python 2.7 or Python 3.
Today I tested with the Python 3.7.3 version:
C:\Python373>cd Scripts
C:\Python373\Scripts>pip install pony
Collecting pony
...
Successfully built pony
Installing collected packages: pony
Successfully installed pony-0.7.10
Let's start with one simple example:
C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pony
>>> from pony import orm
>>> from pony.orm import *
>>> mydb = Database()
>>> class Person(mydb.Entity):
...     name = Required(str)
...     surname = Required(str)
...     age = Required(int)
...     house = Set('Home')
...
>>> class Home(mydb.Entity):
...     owner = Required(Person)
...     address = Required(str)
...     floors = Required(int)
...     perimeter = Required(int)
...
>>> show(Person)
class Person(Entity):
    id = PrimaryKey(int, auto=True)
    name = Required(str)
    surname = Required(str)
    age = Required(int)
    house = Set(Home)
>>> show(Home)
class Home(Entity):
    id = PrimaryKey(int, auto=True)
    owner = Required(Person)
    address = Required(str)
    floors = Required(int)
    perimeter = Required(int)
Let's see the relationships of this example.
The house variable is set to the type named Home.
The owner variable has type Person.
The Pony supports 4 database types: SQLite, MySQL, PostgreSQL, and Oracle.
Use absolute path, see the error generated by bind function when saving it to the database file:
>>> mydb.bind(provider='sqlite', filename='err_my.sqlite', create_db=True)
...
ValueError: When in interactive mode, please provide absolute file path. Got: 'err_my.sqlite'
The correct way to bind the database is:
mydb.bind(provider='sqlite', filename='C:\\Python373\\mydb.sqlite', create_db=True)
Using the Pony python module with databases is simple to use, see the documentation webpage:
# SQLite
db.bind(provider='sqlite', filename=':memory:')
# or
db.bind(provider='sqlite', filename='database.sqlite', create_db=True)
# PostgreSQL
db.bind(provider='postgres', user='', password='', host='', database='')
# MySQL
db.bind(provider='mysql', host='', user='', passwd='', db='')
# Oracle
db.bind(provider='oracle', user='', password='', dsn='')
The next step is the mapping process for declared entities to the corresponding tables in the database, like: creates tables, foreign key references, and indexes if necessary:
>>> mydb.generate_mapping(create_tables=True)
Let's fill the database using these objects:
data001 = Person(name='John',surname = 'Unknown', age=18)
data002 = Person(name='Unknown',surname = 'Unknown', age=81)
add001 = Home(owner = data001,address = '18', floors = 1, perimeter=0)
add002 = Home(owner = data002,address = '11', floors = 0, perimeter=1)
These objects will be saved only after the commit function is called.
>>> commit()
Let's see the next steps:
- Writing queries:
>>> select (var_p for var_p in Person)

>>> select (var_p for var_p in Person)[:]
[Person[1], Person[2]]
>>> select (var_p for var_p in Person)[:].show()
id|name   |surname|age
--+-------+-------+---
1 |John   |Unknown|18
2 |Unknown|Unknown|81
- Getting objects:
>>> p1 = Person[1]
>>> print(p1)
Person[1]
>>> print(p1.name)
John
- Updating an object:
>>> m = Person.get(name='John')
>>> print(m.age)
18
Even I used an SQLite database, the databases can be used writing queries according to the databases
>>> x = 25
>>> Person.select_by_sql('SELECT * FROM Person p WHERE p.age < $x')
[Person[1]]
Instead of creating models manually, you can use the examples from the Pony distribution package:
>>> from pony.orm.examples.estore import *
...
SELECT "Product"."id", "Product"."name", "Product"."description", "Product"."pic
ture", "Product"."price", "Product"."quantity"
FROM "Product" "Product"
WHERE 0 = 1

COMMIT
COMMIT
PRAGMA foreign_keys = true
CLOSE CONNECTION
RELEASE CONNECTION
You can see the database diagram at the eStore webpage.
This tutorial can be continued with additional pieces of information and examples.
You can read more at Pony ORM webpage.

Thursday, June 20, 2019

Python 3.7.3 : Read and save RSS data from goodreads website.

Today I will show you how to parse data from www.goodreads.com using the feedparser and save all into CSV file.
The Goodreads website comes with hundreds of great book recommendations from fellow readers, beloved authors, and let you add your favorite books.
The main goal was to have a structured link from the RSS file and the CSV file.
This issue was solve with arrays for each type of data.
First, let's install the feedparser python module with the pip tool:
C:\Python373\Scripts>pip install feedparser
Collecting feedparser
...
Successfully built feedparser
Installing collected packages: feedparser
Successfully installed feedparser-5.2.1
You need to get the RSS link with your books from your account.
The example is simple and has commented lines to understand easily how can I solve this issue.
This is the source code for reading all RSS data and put on the CSV file:
import feedparser
import csv
bookread_rss = "your RSS with data account"
feeds = feedparser.parse(bookread_rss)
print ("aditional RSS data")
print (feeds['feed']['title'])
print (feeds['feed']['link'])
print (feeds.feed.subtitle)
print (len(feeds['entries']))
print (feeds.version)
print (feeds.headers)
print (feeds.headers.get('content-type'))
print ("read RSS items")
# empty arrays for values by type
dates = []
titles = []
authors = []
links = []
pages =[]

# create the name of the CSV file
file_csv = 'my_goodreads_books.csv'

# prepare the CSV file with fix for error
# UnicodeEncodeError: 'charmap' codec can't encode character '\u0435' in position
# 30: character maps to 
cvs_out = csv.writer(open(file_csv, 'w',newline='',encoding="utf-8"))

#print(feeds)
for post in feeds.entries:
    date = "%d/%02d/%02d" % (post.published_parsed.tm_year,\
        post.published_parsed.tm_mon, \
        post.published_parsed.tm_mday,)
    # uncomment and will print on console
    #print("___")
    #print("post date: " + date)
    #print("post title: " + post.title)
    #print("post author: " + post.author_name)
    #print("post link: " + post.link)
    #print("post pages: " + post.num_pages)

    dates.append(date)
    titles.append(post.title)
    authors.append(post.author_name)
    links.append(post.link)
    pages.append(post.num_pages)

for d,t,a,l,p in zip(dates,titles,authors,links,pages):
    cvs_out.writerow((d,t,a,l,p))
The result will print you some info, see my example:
C:\Python373>python bookreader_rss_001.py
aditional RSS data
Catalin's bookshelf: all
https://www.goodreads.com/review/list_rss/52019632?key=pyfTLqvJXpg-_ghi4a6ZTZfJV
gLVXC8TcWyaBSyoiScgfXq3&shelf=%23ALL%23
Catalin's bookshelf: all
100
rss20
{'Server': 'Server', 'Date': 'Thu, 20 Jun 2019 12:22:55 GMT', 'Content-Type': 'a
pplication/xml; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'c
lose', 'Status': '200 OK', 'X-Frame-Options': 'ALLOWALL', 'X-XSS-Protection': '1
...
All date will be put into the my_goodreads_books.csv file.

Tuesday, June 18, 2019

Python 3.7.3 : Stemming with nltk.

Today I will start another tutorial about nltk python module and stemming.
The stemming is the process of producing morphological variants of a root/base word.
Stemming programs are commonly referred to as stemming algorithms or stemmers to reduces the words.
Errors in Stemming can be overstemming and understemming.
These two words are stemmed to the same root that are of different stems then the term is overstemming.
When two words are stemmed to same root that are not of different stems then the term used is understemming.
Applications of stemming are used in information retrieval systems like search engines or is used to determine domain vocabularies in domain analysis.
Let install this python module named nltk with pip tool:
C:\Python373\Scripts>pip install nltk
Collecting nltk
...
Successfully installed nltk-3.4.1 six-1.12.0
The nltk python module work with human language data for applying in statistical natural language processing (NLP).
It contains text processing libraries for tokenization, parsing, classification, stemming, tagging, graphical demonstrations, sample data sets, and semantic reasoning.
The next step is to download the models and data, see more at this official webpage.
First run this lines of code to update the nltk python module.
import nltk
nltk.download()
Let's test a simple implementation of stemming words using nltk python module:
from nltk.stem import PorterStemmer 
from nltk.tokenize import word_tokenize 
   
my_porter = PorterStemmer() 
   
quote = "Deep in the human unconscious is a pervasive need for a logical universe that makes sense."

words = word_tokenize(quote) 
   
for w in words: 
    print(w, " : ", my_porter.stem(w))
The result is something like this:
C:\Users\catafest>python stemming_001.py
Deep  :  deep
in  :  in
the  :  the
human  :  human
unconscious  :  unconsci
is  :  is
a  :  a
pervasive  :  pervas
need  :  need
for  :  for
a  :  a
logical  :  logic
universe  :  univers
that  :  that
makes  :  make
sense  :  sens
.  :  .

C:\Users\catafest>
You can read more about the stemming at Wikipedia.

Python 3.7.3 : Using getters and setters in object-oriented.

The main purpose of using getters and setters in object-oriented programs is to ensure data encapsulation.
Let's start with a simple example.
I created a class named my_class init with one variable named my_variable:
self._my_variable = my_variable
A new, initialized instance can be obtained by this line of code:
test_it = my_class()
The example use getter and setter methods to use this variable.
C:\Users\catafest>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> class my_class:
...     def __init__(self, my_variable = 0):
...          self._my_variable = my_variable
...
...     # getter method
...     def get_my_variable(self):
...         return self._my_variable
...
...     # setter method
...     def set_my_variable(self, x):
...         self._my_variable = x
...
>>> test_it = my_class()
>>> test_it.set_my_variable(1976)
>>> print(test_it.get_my_variable())
1976
>>> print(test_it._my_variable)
1976
In Python property() is a built-in function that creates and returns a property object.
Python has four arguments property: fget, fset, fdel, doc.
  • fget is a function for retrieving an attribute value;
  • fset is a fuction for setting an attribute value;
  • fdel is a function for deleting an attribute value;
  • doc creates a docstring for attribute.
This line of code will setting the my_variable using setter:
test_it.set_my_variable(1976)
This line of code will retrieving my_variable using getter:
print(test_it.get_my_variable())
A property object has three methods, getter(), setter(), and delete() to specify fget, fset and fdel individually.
How my example changes it:
>>> class my_class:
...      def __init__(self):
...           self._my_variable = 0
...
...      # function to get value of _my_variable
...      def get_my_variable(self):
...          print("getter method called")
...          return self._my_variable
...
...      # function to set value of _my_variable
...      def set_my_variable(self, a):
...          print("setter method called")
...          self._my_variable = a
...
...      # function to delete _my_variable attribute
...      def del_my_variable(self):
...          del self._my_variable
...
...      my_variable = property(get_my_variable, set_my_variable, del_my_variabl
e)
...
>>> test_it = my_class()
>>> test_it.my_variable = 1976
setter method called
>>> print(test_it.my_variable)
getter method called
1976
Using decorator with python @property is one of the built-in decorators.
The main purpose of any decorator is to change your class methods or attributes.
The user of your class no need to make any change in their code.
This is the final result:
>>> class my_class:
...      def __init__(self):
...           self._my_variable = 0
...
...      # using property decorator
...      # a getter function
...      @property
...      def my_variable(self):
...          print("getter method called")
...          return self._my_variable
...
...      # a setter function
...      @my_variable.setter
...      def my_variable(self, my_out):
...          if(my_out < 1976):
...             raise ValueError("... this my_variable has a criteria!!")
...          print("setter method called")
...          self._my_variable = my_out
...
>>> test_it = my_class()
>>> test_it.my_variable = 1979
setter method called
>>> print(test_it.my_variable)
getter method called
1979
>>> test_it.my_variable = 1975
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 16, in my_variable
ValueError: ... this my_variable has a criteria!!
This last example show you how to use @property decorator to create getters and setters in pythonic way.