Python 3.7.3 : Using the twitter python module - part 003.

Today I will speak about twitter python module with the new changes of the A.P.I.
This two tutorial will not work now because the twitter A.P.I is changed.
The reason I don't delete it is the similar flow programming and access the A.P.I.:

• Use the twitter python module - part 001.
• Use the twitter python module - part 002.

Let's start with the install of this python module with Python version 3.7.3:

C:\Python373>cd Scripts

C:\Python373\Scripts>pip install python-twitter
...
Installing collected packages: python-twitter
Successfully installed python-twitter-3.5

Let's test the GetSearch.
You need to create a twitter application to have access to the tokens and secret keys:

import os
import json
import twitter
from twitter import *
CONSUMER_KEY=""
CONSUMER_SECRET=""

ACCESS_TOKEN=""
ACCESS_TOKEN_SECRET=""
LANGUAGES="En"
at=input("ACCESS_TOKEN: ", )
ats=input("ACCESS_TOKEN_SECRET: ", )
ck=input("CONSUMER_KEY: ", )
cs=input("CONSUMER_SECRET: ", )
api = Api(ck, cs, at, ats)
def main():
    print("Search by query using the GetSearch ")
    r = api.GetSearch(raw_query="q=twitter%20&result_type=recent&since=1999-03-07&count=100")
    print(r)
if __name__ == '__main__':
    main()

The result will be something like this:

...
@GeorgePapa19 @realDonaldTrump Sure Did https://t.co/GblrSOsaJg'), Status(ID=115
5787444525993984, ScreenName=MD__PCY, Created=Mon Jul 29 10:29:34 +0000 2019, Te
xt='mal ako ni twitter https://t.co/lRA1BOyf6a'), Status(ID=1155787444517838849,
 ScreenName=Elyse95, Created=Mon Jul 29 10:29:34 +0000 2019, Text='????? ????? ?
??? ..??? ???? ????????? ????? ????? ????? ?????? ??? ???????? .. !!    ... http
s://t.co/weChoAPDnC. https://t.co/34D2nmtbOz'), Status(ID=1155787444517781504, S
creenName=chrichacham123, Created=Mon Jul 29 10:29:34 +0000 2019, Text='RT @John
JCrace: Project Blind Faith! https://t.co/2mpB0MC540'), Status(ID=11557874445134
72514, ScreenName=318520_mu, Created=Mon Jul 29 10:29:34 +0000 2019, Text='31-85
20 ????????\n???????? → ???twitter??????????????\n??...?\n\nhttps://t.co/oVET5Z9
wXq')]

Python 3.7.3 : Using the flask - part 003.

Another tutorial with python 3.7.3 and flask python module.
In the last tutorial, I speak about some tricks and tips.
Today, I will show some steps for fixing and run with flask-sqlalchemy.
The source code from my GitHub account can be updated with this source code.

app = Flask (__name__)
app.config['SECRET_KEY'] = 'abcdefg'
# set SQLAlchemy
basedir = os.path.abspath(os.path.dirname(__file__))
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'server.sqlite')
db = SQLAlchemy(app)
ma = Marshmallow(app)
class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    email = db.Column(db.String(120), unique=True)

    def __init__(self, username, email):
        self.username = username
        self.email = email

If you run it then you get this:

C:\Python373\my_flask>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD6
4)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
C:\Python373\lib\site-packages\flask_sqlalchemy\__init__.py:835: FSADeprecationW
arning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be dis
abled by default in the future.  Set it to True or False to suppress this warnin
g.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '

You will have a database file named server.sqlite.

#app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False

Let's create and show the database:

C:\Python373\my_flask>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD6
4)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import db
>>> db.create_all()
>>> db.engine.table_names()
['user']

A good approach is to create a config.py file and used into the application area.
The config.py file will have this source code:

SQLALCHEMY_TRACK_MODIFICATIONS = False
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'server.sqlite')
DEBUG=True

The server.py has this source code:

...
app = Flask (__name__)
app.config.from_pyfile('config.py')
db = SQLAlchemy(app)
...

Python 3.7.3 : About pytweening python module.

This is a simple module of tweening and easing functions implemented in Python, see the GitHub webpage.

C:\Python373\Scripts>pip install pytweening
Collecting pytweening
Downloading https://files.pythonhosted.org/packages/b9/f8/c32a58d6e4dff8aa5c27
e907194d69f3b57e525c2e4af96f39c6e9c854d2/PyTweening-1.0.3.zip
Building wheels for collected packages: pytweening
Building wheel for pytweening (setup.py) ... done
Created wheel for pytweening: filename=PyTweening-1.0.3-cp37-none-any.whl size
=3821 sha256=6655c055d779982ff1259a5266bf1300c1cd02046e45e92cf18b20053a326531
Stored in directory: C:\Users\catafest\AppData\Local\pip\Cache\wheels\7b\92\30
\06e21159eed2709436bfb6d7c690959e578cf74f029643866e
Successfully built pytweening
Installing collected packages: pytweening
Successfully installed pytweening-1.0.3

Let's test it:

C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD6
4)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytweening
>>> dir(pytweening)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__',
'__package__', '__path__', '__spec__', '__version__', '_checkRange', 'division',
 'easeInBack', 'easeInBounce', 'easeInCirc', 'easeInCubic', 'easeInElastic', 'ea
seInExpo', 'easeInOutBack', 'easeInOutBounce', 'easeInOutCirc', 'easeInOutCubic'
, 'easeInOutElastic', 'easeInOutExpo', 'easeInOutQuad', 'easeInOutQuart', 'easeI
nOutQuint', 'easeInOutSine', 'easeInQuad', 'easeInQuart', 'easeInQuint', 'easeIn
Sine', 'easeOutBack', 'easeOutBounce', 'easeOutCirc', 'easeOutCubic', 'easeOutEl
astic', 'easeOutExpo', 'easeOutQuad', 'easeOutQuart', 'easeOutQuint', 'easeOutSi
ne', 'getLine', 'getPointOnLine', 'linear', 'math']
>>> pytweening.linear(0.1)
0.1
>>> pytweening.easeInOutSine(0.1)
0.024471741852423234
>>> pytweening.easeInQuad(0.1)
0.010000000000000002
>>> pytweening.getLine(0, 0, 0.1, 0.1)
[(0, 0)]
>>> pytweening.getLine(0, 0, 1.1, 1.1)
[(0, 0), (1, 1)]

For example, this module can help with some math function:

• simple linear tweening - no easing, no acceleration;
• quadratic easing in/out - acceleration until halfway, then deceleration;
• cubic easing in/out - acceleration until halfway, then deceleration;
• quartic easing in/out - acceleration until halfway, then deceleration;
• sinusoidal easing in/out - accelerating until halfway, then decelerating;
• ...

Python 3.7.3 : Tonny I.D.E. for python programmers.

Today I tested the Thonny I.D.E. from thonny.org official webpage.
Yesterday I tried several editors for python programming language and did not work.
One of these is the spyder editor that does not work with python 3.7.3 - we have not discovered why.
The Mu is a simple Python editor for beginner programmers and has a strange working I.D.E. for good and fast development.
The PyCharm, it is developed by the Czech company JetBrains is not fully free ...

---

The Thonny I.D.E. is fast and comes with two good features:
Code completion and Highlights syntax errors.
The install steps are easy with a wizard application.
After that, you can test with any python source code.
The source code from the image is a simple example from this webpage.
Let's see a screenshot:

This is a video tutorial with this I.D.E. from Aivar Annamaa user:

Python 3.7.3 : Using the flask - part 002.

Let's see some tips for starting any project with flask python module.
Use these python modules to work with databases: flask-sqlalchemy and flask_marshmallow.
The Flask-SQLAlchemy is an extension for Flask that adds support for SQLAlchemy to your application.
The marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, to and from native Python datatypes.
Let's install it:

C:\Python373>cd Scripts

C:\Python373\Scripts>pip install flask-sqlalchemy
Collecting flask_sqlalchemy
...
Successfully installed flask-sqlalchemy-2.4.0
C:\Python373\Scripts>pip install flask_marshmallow
Collecting flask_marshmallow
...
Successfully installed flask-marshmallow-0.10.1 marshmallow-2.19.5
C:\Python373\Scripts>pip install marshmallow-sqlalchemy
Collecting marshmallow-sqlalchemy
...
Successfully installed marshmallow-sqlalchemy-0.17.0

Use these python modules for login and authorization issues: flask-login
The Flask-Login provides user session management for Flask. It handles the common tasks of logging in, logging out, and remembering your users’ sessions over extended periods of time.
Flask-login doesn't actually have a user backend, it just handles the session machinery to help you to log in and logout users.
Flask-Login can work with user models that are based on any database system with four required items: is_authenticated, is_active, is_anonymous and get_id.
Let's install it:

C:\Python373\Scripts>pip install flask-login
Collecting flask-login
...
Successfully installed flask-login-0.4.1

This python module named python-dotenv let us to reads the key-value pair from .env file and adds them to an environment variable.

C:\Python373\Scripts>pip install python-dotenv
Collecting python-dotenv
...
Successfully installed python-dotenv-0.10.3

Flask-Login provides a UserMixin which makes it easy to create a user class, see example class User:

from flask_login import UserMixin
...
class User(UserMixin, db.Model):

Another tip is to use the get_or_404 and first_or_404 to avoid show errors on the webpage.
Reduce the passing of multiple variables to render_template, let's see the example:

from flask import render_template
...
return render_template('index.html', fn=name.first, ln=name.last, day=student.day, 
.... ')

This can de easy fix with this code to send value I want to pass:

context = {
'fn': name.first,
'ln': name.last,
'day': student.day,
...
}
return render_template('index.html', **context)

Use query to solve complex tasks.
Let's see a simple example of delete from a Student database class:

...
class Student(db.Model):
id=db.Column(db.Integer, primary_key=True)
...
country=db.Column(db.String(2))
...

Let's delete this with a query and return the number of deleted by delete_count:

del_ids= db.session.query(Student.id).filter(Student.country=='Ro')
delete_count=db.session.query(Student).filter(Student.id.in_(del_ids.subquery())).delete(synchronize_session=False)

I used two files in the main project named .env and .flaskenv:
The file .env comes with:

SQLALCHEMY_DATABASE_URI=sqlite:///db.sqlite3
SECRET_KEY = abcdef

The file .flaskenv has these lines of source code:

FLASK_ENV = development
FLASK_APP = my_flask_app

This can be used into a new file named settings.py:

import os 
SQLALCHEMY_DATABASE_URI = os.environ.get('SQLALCHEMY_DATABASE_URI')
SECRET_KEY = os.environ.get('SECRET_KEY')
SQLALCHEMY_TRACK_MODIFICATIONS = False

Using this the main flask application will start like this:

from flask import Flask
def create_my_app(config_file='settings.py'):
    app = Flask(__name__)
    app.config.from_pyfile(config_file)
    return app

There are plenty of tips and tricks that can be found based on the project in the bottle.
These are the most used.

Python 3.7.3 : Testing the timeit and Bokeh python module.

The tutorial today has several goals:

• testing the timeit function to measure the execution time of a line of print code;
• using python lists to record data;
• generate lists using the range function;
• multiplying a number with an entire list of numbers;
• how to use the bokeh module with CustomJSHover and HoverTool

Let's test with the timeit python module, see the official webpage.
This module provides a simple way to time small bits of Python code. It has both a Command-Line Interface as well as a callable one. It avoids a number of common traps for measuring execution times.

C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> import random
>>> timeit.timeit('print("a")',number=1)
a
0.00013114200010022614
>>> timeit.timeit('print("a")',number=100)
...
a
0.007177434000027461
>>> timeit.timeit('print("a")',number=1000)
...
a
0.07585798000002342

Let make more visible the results with the bokeh python module:

import timeit
import random

# if you want to use numpy 
#import numpy as np

# import bokeh python module 
from bokeh.plotting import figure, show, output_file
# for show values
from bokeh.models.tools import CustomJSHover
from bokeh.models import HoverTool
print ('''
You can tests a number of times named l
with a number of print named i
and see the result value with bokeh python module
timeit.timeit('print("a")',number = i)
''')
i = int(input('The increment number i:'))
l = int(input('The number of test l:'))

# create the list with the output values
timit_list = []
# create the test function for l and i and return the list
def get_timeit(i,l):
 while i < l:
  i += 1
  out=timeit.timeit('print("a")',number = i)
  timit_list.append(out)
 return timit_list

# run the test function with l and i
# this will be coordinate y
yt=get_timeit(i,l)

# show result of the test 
print(yt)

# create the coordinate x
xt = [i for i in range(0, len(yt))]

#xt = np.linspace(0, l, l)
# print the coordinate x
#print(xt)

# create the output HTML file to see the result
output_file("test.html")

# create a figure with a timeit type y-axis
fig = figure(title='timit_list values for print("a")',
             plot_height=400, plot_width=700,
             x_axis_label='x has each incrementation of i', y_axis_label='value of timeit of print the char a',
             x_minor_ticks=3, 
             toolbar_location=None)

# create a circle for each value

# see new multiplication with a list 

# y=[i * 100 for i in yt]
fig.circle(x=xt, y=[i * 100 for i in yt], 
         color='blue', size=5,
         legend='Values')

x_custom = CustomJSHover(code="""
    return '' + special_vars.data_x""")

y_custom = CustomJSHover(code="""
    return '' + special_vars.data_y""")

fig.add_tools(
    HoverTool(
        show_arrow=True, 
        tooltips=[
 ('xt', '$data_x'),
 ('yt', '$data_y')
        ],
        formatters=dict(
            xt=x_custom,
            yt=y_custom
        )
    )
)

# Put the legend in the upper left corner
fig.legend.location = 'top_left'

# Let's check it out
show(fig)

The result of this python source code can be found on my YouTube channel:

Python 3.7.3 : Using the flask - part 001.

A short intro into this python module can be found at the PyPI website:
Flask is a lightweight WSGI web application framework. It is designed to make getting started quick and easy, with the ability to scale up to complex applications. It began as a simple wrapper around Werkzeug and Jinja and has become one of the most popular Python web application frameworks.

Flask offers suggestions but doesn’t enforce any dependencies or project layout. It is up to the developer to choose the tools and libraries they want to use. There are many extensions provided by the community that makes adding new functionality easy.
The reason I used a series of tutorials with this python module is the complexity of the features of this python module.
Let's briefly outline some of the essential aspects of flask programming.

• Flask is a simple, lightweight, and minimalist web framework;
• Flask is developed based on the Jinja2 template engine;
• Flask depends on the Jinja template engine and the Werkzeug WSGI toolkit;
• Flask does not provide a built-in ORM system (Object Relation Mapping);
• in Flask web applications to perform CRUD operations on a database can be tedious (see: ORM techniques of Flask-SQLAlchemy);
• The Flask has a simple and customizable architecture;
• the Flask to accelerate the development of simple websites that use static content;
• has the option to extend and customize Flask according to precise project requirements;

A list of companies using the Flask framework - who is using Flask?
The install process is very simple using the pip tool:

C:\Python373>cd Scripts

C:\Python373\Scripts>pip install flask
Collecting flask
...
Installing collected packages: itsdangerous, Werkzeug, flask
Successfully installed Werkzeug-0.15.4 flask-1.1.1 itsdangerous-1.1.0

Let's start initializing the first flask application into the server.py file:

from flask import Flask 
app = Flask (__name__)
# create a wrarp of localhost
@app.route('/')
def home():
 return 'Hello world'
# the default name main 
if __name__ == '__main__':
 app.run()

About Routing and Variable Rules.
This can change it into the server.py script like this:

@app.route('/about')
def about():
 return 'The about page'
@app.route('/blog')
def blog():
 return 'This is the blog'

The next step is the variable rule issue:

# creaza a regula variabila ( variable rules) 
# the type of the variable can be set, see int 
# @app.route('/blog/')
@app.route('/blog/')
def blogpost(blog_id):
 return 'This is the post '+str(blog_id)

To run it, just use:

C:\Python373\my_flask>python server.py
 * Serving Flask app "server" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

The full source code can be found at my GitHub account.

Python 3.7.3 : The sip python module.

The official webpage pypi.org comes with this intro:
One of the features of Python that makes it so powerful is the ability to take existing libraries, written in C or C++, and make them available as Python extension modules. Such extension modules are often called bindings for the library.

SIP is a tool that makes it very easy to create Python bindings for C and C++ libraries. It was originally developed to create PyQt, the Python bindings for the Qt toolkit, but can be used to create bindings for any C or C++ library.

SIP comprises a code generator and a Python module. The code generator processes a set of specification files and generates C or C++ code which is then compiled to create the bindings extension module. The sip Python module provides support functions to the automatically generated code.
The SIP is copyright (c) Riverbank Computing Limited and its homepage is this webpage.
Support may be obtained from the PyQt mailing list at here.
The SIP is a tool for quickly writing Python modules that interface with C++ and C libraries.
The SIP comprises a code generator and a Python module.
About the code generator:
First, the install:

C:\Python373\Scripts>pip3 install sip
Collecting sip
...
Installing collected packages: sip
Successfully installed sip-4.19.8

If you using the PyQt5 then this version includes a private copy of the module.

C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyQt5 import sip

If you want to see backward compatibility the module then needs to imported and will only work if another PyQt5 module is imported first.

C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyQt5 import QtCore
>>> import sip

A good example of how can you build with SIP can be seen here.
About the SIP Python module:
This provides support functions to the automatically generated code.
The import python module sip let you use all functions build by the code generator:

>>> dir(sip)
['SIP_VERSION', 'SIP_VERSION_STR', '_C_API', '__doc__', '__file__', '__loader__'
, '__name__', '__package__', '__spec__', '_unpickle_enum', '_unpickle_type', 'as
sign', 'cast', 'delete', 'dump', 'enableautoconversion', 'enableoverflowchecking
', 'getapi', 'isdeleted', 'ispycreated', 'ispyowned', 'setapi', 'setdeleted', 's
etdestroyonexit', 'settracemask', 'simplewrapper', 'transferback', 'transferto',
 'unwrapinstance', 'voidptr', 'wrapinstance', 'wrapper', 'wrappertype']

If you want to improve or create python modules with C or C ++, then this tool can help.

Python 3.7.3 : The IMDbPY python module version 6.8.

The GitHub official webpage comes with this intro:
IMDbPY is a Python package for retrieving and managing the data of the IMDb movie database about movies, people and companies.
The last release version 6.8 was at 2019 Jul 20.
The official webpage tells us:
In the release 6.8 (codename "Apollo 11") of IMDbPY, multiple parsers were added and fixes; the new search_movie_advanced method allows advanced movie searches...
The changes of the version 6.8 can be found at GitHub webpage and come with these new features:
#224: introduce the search_movie_advanced(title, adult=None, results=None, sort=None, sort_dir=None) method
#145: names are stored in normal format (Name Surname)
#225: remove the obsolete cookie
#182: box office information
#168: parse series and episode number searching for movies
#217: grab poster from search
#218: extract MPAA rating
#220: extract actor headshot from full credits
The install on Python 3.7.3 is easy with pip3 tool:

C:\Python373\Scripts>pip3 install imdbpy
Collecting imdbpy
...
Installing collected packages: imdbpy
Successfully installed imdbpy-6.8

Let's test the new features:

C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import imdb
>>> ia = imdb.IMDb()
>>> movies = ia.search_movie_advanced('debby does dallas', adult=True)
>>> print(movies)
...
>>> people = ia.search_person('Clark Gregg')
>>> print(people)

Let's test it:

import imdb
from imdb import IMDb

ia = imdb.IMDb()

# create a file to put the output 
file1 = open("_imdb_data.txt","w", encoding='utf-8') 

# get movies by movie 
# example: Alien 

def get_by_movie():
 my_movie = str(input('Type the movie name: '))
 movies = ia.search_movie_advanced(my_movie, adult=True)
 print(type(movies))
 return movies

# get filmography by id 
filmography_list = []
def get_filmography_by_id(id):
 actor_results = ia.get_person_filmography(id)
 for item in actor_results['data']['filmography']:
  filmography_list.append(str(item))
 return filmography_list

# the main function 
def main():
 a = get_by_movie()
 for i in a:
  print("________________________")
  print("i: ",i)
  # you can uncomment this to test Movie class functions
  #print("Type:",type(i))
  #print("Summary:",i.summary())
  #print("ID: ",i.getID())
  #print("Smart cannonical title: ",i.smartCanonicalTitle())
  #print("caracters ref: ",i.get_charactersRefs())
  #print("current info: ",i.get_current_info())
  #print("cinematographic process: ",i.get('cinematographic process'))
  #print(i["title"])
  #print informations items from Movie class
  print("~~~~~~~~~~~~~~~~~~~~~~~~")
  for k, v in i.items():
   print(k, v)
   # write to the file the value of a
   txt = str(k)+":"+str(v)+"\n" 
   file1.write(txt)
  print("------------------------")
  # add a new line on each movie
  file1.write('-----^-----\n')
 #get filmography by id 
 id_filmography=get_filmography_by_id('0078748')
 # print the filmography
 for item in id_filmography:
  print(item)

 #after write, close the file 
 file1.close()

if __name__ == '__main__':
    main()

This is the first part of the output file named _imdb_data:

title:Alien
certificates:['R']
runtimes:['117']
genres:['Horror', 'Sci-Fi']
rating:8.5
votes:719508
metascore:89
gross:78900000
plot:After a space merchant vessel perceives an unknown transmission as a distress call, its landing on the source moon finds one of the crew attacked by a mysterious lifeform, and they soon realize that its life cycle has merely begun.
directors:[]
cast:[, , , ]
cover url:https://m.media-amazon.com/images/M/MV5BMmQ2MmU3NzktZjAxOC00ZDZhLTk4YzEtMDMyMzcxY2IwMDAyXkEyXkFqcGdeQXVyNzkwMjQ5NzM@._V1_UX67_CR0,0,67,98_AL_.jpg
year:1979
kind:movie
canonical title:Alien
long imdb title:Alien (1979)
long imdb canonical title:Alien (1979)
smart canonical title:Alien
smart long imdb canonical title:Alien (1979)
full-size cover url:https://m.media-amazon.com/images/M/MV5BMmQ2MmU3NzktZjAxOC00ZDZhLTk4YzEtMDMyMzcxY2IwMDAyXkEyXkFqcGdeQXVyNzkwMjQ5NzM@.jpg
-----^-----
title:Aliens
certificates:['R']
runtimes:['137']
...

Python 3.7.3 : Use BeautifulSoup to parse Instagram account.

This example is a bit more complex because it parses the source code in a more particular way depending on it.
The basic idea of this script is to take the content of an Instagram account in the same way as a web browser.
For my account I found a parsing error, I guess the reason is using the points, see festila.george.catalin.

    scripts_content = json.loads(scripts[0].text.strip())
IndexError: list index out of range

In this case comment this line of code and will work:
For the other accounts I've tried, it works very well with the default script.
This is the script I used:

import requests
from bs4 import BeautifulSoup
import json
import re

from pprint import pprint

instagram_url = 'https://instagram.com'
#example user instagram profile_url = sherwoodseries
profile_url=str(input("name of the instagram user: "))


#UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f48b' in posit ion 5022: character maps to 
#fix write text file with  encoding='utf-8'
file1 = open("_shared_data.txt","w", encoding='utf-8') 

#profile_url = 'festila.george.catalin'
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(f"{instagram_url}/{profile_url}", headers = headers)

if response.ok:
    html = response.text
    bs_html = BeautifulSoup(html, "html.parser")
    print(bs_html)
    # get info from ... type="application/ld+json">{"@context":"http:\/\/schema.org","@type":"Person","name":
    scripts = bs_html.select('script[type="application/ld+json"]')
    #scripts_content = json.loads(scripts[0].text.strip())
    #pprint(scripts_content)

    #print scripts_content like json 
    #print(json.dumps(scripts_content,indent = 4,sort_keys = True))

    #print just part of source code get by 'script' (0 .. n), see n = 6 
    #print(bs_html.find_all('script')[6])
    script_tag = bs_html.find('script', text=re.compile('window\._sharedData'))
    shared_data = script_tag.string.partition('=')[-1].strip(' ;')

    #get item from shared data, see "language_code":"en"
    rex_item  = re.compile('(?<=\"language_code\":\")[a-zA-Z_\- ]+(?=\")')
    rex_get_item = rex_item.findall(shared_data)  
    print(rex_get_item)
    #get url image from shared data
    rex_url  = re.compile('(?<=\"display_url\":\")[^\s\"]+(?=\")')
    rex_get_url = rex_url.findall(shared_data)  
    print(rex_get_url)
 
    # load like a json 
    result_json = json.loads(shared_data)
    pprint(result_json)
    
    data = bs_html.find_all('meta', attrs={'property': 'og:description'})
    bb = data[0].get('content').split()
    user = '%s %s %s' % (bb[-3], bb[-2], bb[-1])
    # get from bb parts 
    posts = bb[4]
    print('all string: ',bb)
    print('number of posts: ',posts)
    print('name and the user: ',user)

    # write any output show by print into _a.txt file, see example
    #file1.write(str(bs_html.find_all('script')[4]))
    #example: write to _shared_data.txt file the shared_data
    #file1.write(str(shared_data))
#after write, close the file 
#file1.close() 

This is a part of the output for sherwoodseries account:

...
all string:  ['95', 'Followers,', '24', 'Following,', '56', 'Posts', '-', 'See',
 'Instagram', 'photos', 'and', 'videos', 'from', 'Sherwood', 'Series', '(@sherwo
odseries)']
number of posts:  56
name and the user:  Sherwood Series (@sherwoodseries)

Python 3.7.3 : The pandas python module.

Since I started learning python programming language I have not found a more complex and complete module for viewing complex data.
The official documentation of this python module tells us:
pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real-world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open-source data analysis/manipulation tool available in any language. It is already well on its way toward this goal.
The official webpage can be found here.
This python module is one of the most popular Python libraries for Data Science and Analytics.
You can install this python module with pip tool:

C:\Python373\Scripts>pip install pandas
Requirement already satisfied: pandas in c:\python373\lib\site-packages (0.24.2)

You can find many tutorials on web with this python module.
Today I will show you a short tutorial about this python module.
Most users use this both python modules:

import numpy as np
import pandas as pd

Most area of the pandas python module has a target into this list:
Window Functions, Aggregations, Missing Data, GroupBy, Merging/Joining, Concatenation, Date Functionality, Timedelta, Categorical Data,
Visualization, IO Tools, Sparse Data, Caveats & Gotchas, Comparison with SQL
There are two types of data structures in pandas: Series and DataFrames.
The pandas Series is a one-dimensional data structure.
The pandas DataFrame is a two (or more) dimensional data structure, like a table
Pandas provide few variants rolling, expanding and exponentially moving weights for window statistics.
Also have the sum, mean, median, variance, covariance, correlation, etc.
The several methods are available to perform aggregations on data.
Pandas provide functions for missing data like the isnull() and notnull().
Let's test the DataFrames with pandas and the Wikipedia example from my tutorial

C:\Python373>python.exe
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Inte
l)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> # get table from wikipedia
... import requests
>>> from bs4 import BeautifulSoup
>>> website_url = requests.get('https://en.wikipedia.org/w/index.php?title=Table
_of_food_nutrients').text
>>> soup = BeautifulSoup(website_url,'lxml')
>>>
>>> my_table = soup.find('table',{'class':'wikitable collapsible collapsed'})
>>> links = my_table.findAll('a')
>>> Food = []
>>> for link in links:
...     Food.append(link.get('title'))
...
>>> print(Food)
["Cows' milk (page does not exist)", 'Buttermilk', 'Fortified milk (page does no
t exist)', 'Powdered milk', "Goats' milk", 'Malted milk', 'Hot chocolate', 'Yogu
rt', 'Milk pudding (page does not exist)', 'Custard', 'Ice cream', 'Ice milk', '
Cream', 'Cheese', 'Cheddar cheese', 'American cheese', 'Processed cheese', 'Egg
(food)', 'Scrambled', 'Omelet', 'Yolk']
>>> import pandas
>>> import pandas as pd
>>> df = pd.DataFrame()
>>> df['Foods'] = Food
>>> print(df)
                                   Foods
0       Cows' milk (page does not exist)
1                             Buttermilk
2   Fortified milk (page does not exist)
3                          Powdered milk
4                            Goats' milk
5                            Malted milk
6                          Hot chocolate
7                                 Yogurt
8     Milk pudding (page does not exist)
9                                Custard
10                             Ice cream
11                              Ice milk
12                                 Cream
13                                Cheese
14                        Cheddar cheese
15                       American cheese
16                      Processed cheese
17                            Egg (food)
18                             Scrambled
19                                Omelet
20                                  Yolk
>>> df.describe()
              Foods
count            21
unique           21
top     Malted milk
freq              1
>>> df.apply(pd.Series.value_counts)
                                      Foods
Malted milk                               1
Ice milk                                  1
Omelet                                    1
Goats' milk                               1
Custard                                   1
Cheddar cheese                            1
American cheese                           1
Ice cream                                 1
Yolk                                      1
Cream                                     1
Cows' milk (page does not exist)          1
Yogurt                                    1
Fortified milk (page does not exist)      1
Egg (food)                                1
Powdered milk                             1
Milk pudding (page does not exist)        1
Cheese                                    1
Hot chocolate                             1
Buttermilk                                1
Processed cheese                          1
Scrambled                                 1

The last example is to show data with

>>> import numpy as np
>>> import pandas as pd
>>> import matplotlib.pyplot as plt
>>> ts = pd.Series(np.random.randn(76), index=pd.date_range('1/1/76', periods=76
))
>>> ts.plot()

>>> plt.show()

Python 3.7.3 : Using the pipenv tool.

The goal of this tutorial is how to use pipenv and manage dependencies and development environments on a collaborative Python project.
The documentation area can be found here.

C:\Python373>cd Scripts

C:\Python373\Scripts>pip install --user pipenv
Collecting pipenv
...
Successfully installed pipenv-2018.11.26 virtualenv-clone-0.5.3

If you see errors like this:
WARNING: The scripts pipenv-resolver.exe and pipenv.exe are installed in 'C:\Users\....'

which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, 
use --no-warn-script-location.

You need to add this path show into the WARNING to the windows path, see command:

setx path "%path%;C:\Users\..\Python37\Scripts"

Let use this tool to install a requests package:

C:\Python373\Scripts>cd ..

C:\Python373>cd my_flask

C:\Python373\my_flask>pipenv install requests
Creating a virtualenv for this project.
Pipfile: C:\Python373\my_flask\Pipfile
Using c:\python373\python.exe (3.7.3) to create virtualenv.
[  ==] Creating virtual environment...Already using interpreter c:\python373\pyt
hon.exe
Using base prefix 'c:\\python373'
New python executable in C:\Users\catafest\.virtualenvs\my_flask-j9hGDZgP\Script
s\python.exe
Installing setuptools, pip, wheel...
done.
Successfully created virtual environment!
Virtualenv location: C:\Users\catafest\.virtualenvs\my_flask-j9hGDZgP
Creating a Pipfile for this project.
Installing requests.
Adding requests to Pipfile's [packages].
Installation Succeeded
Pipfile.lock not found, creating.
Locking [dev-packages] dependencies.
Locking [packages] dependencies.
Success!
Updated Pipfile.lock (444a6d)!
Installing dependencies from Pipfile.lock (444a6d).
  ================================ 5/5 - 00:00:04
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

The syntax for the Pipfile is TOML, and the file is separated into sections. [dev-packages] for development-only packages, [packages] for minimally required packages, and [requires] for other requirements like a specific version of Python
If you got this error you need restarts the command prompt:

'pipenv' is not recognized as an internal or external command,
operable program or batch file.

The next command let us see output diagnostic information for use in GitHub issues:

C:\Python373\my_flask>pipenv --support | clip
C:\Python373\my_flask>pipenv --support

Check for security vulnerabilities (and PEP 508 requirements) in your environment with:

C:\Python373\my_flask>pipenv check
Checking PEP 508 requirements.
Passed!
Checking installed package safety.
All good!

Let's use the example from the documentation webpage:

import requests

response = requests.get('https://httpbin.org/ip')

print('Your IP is {0}'.format(response.json()['origin']))

The result works great.

C:\Python373\my_flask>pipenv run python my_ip.py
Your IP is ...

You can print out a tree-like structure showing your dependencies.

C:\Python373\my_flask>pipenv graph
requests==2.22.0
  - certifi [required: >=2017.4.17, installed: 2019.6.16]
...

Using this command will see a reversed tree may be more useful when you are trying to figure out conflicting sub-dependencies.

C:\Python373\my_flask>pipenv graph --reverse
certifi==2019.6.16
  - requests==2.22.0 [requires: certifi>=2017.4.17]
chardet==3.0.4
...

This command installs all the dependencies needed for development:

C:\Python373\my_flask>pipenv install --dev
Installing dependencies from Pipfile.lock (444a6d).
  ================================ 5/5 - 00:00:10
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

You can uninstall a python module:

pipenv uninstall numpy

You can replace --all with --all-dev to just remove dev packages.
Let’s start a shell in a virtual environment to isolate the development of this app:

C:\Python373\my_flask>pipenv shell
Launching subshell in virtual environment.
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

(my_flask-j9hGDZgP) C:\Python373\my_flask>

The package name, together with its version and a list of its own dependencies, can be frozen by updating the Pipfile.lock with the lock keyword.

(my_flask-j9hGDZgP) C:\Python373\my_flask>pipenv lock
Locking [dev-packages] dependencies.
Locking [packages] dependencies.
Success!
Updated Pipfile.lock (444a6d)!

Clears caches (pipenv, pip, and pip-tools) with:

pipenv --clear

Remove the virtualenv created:

(my_flask-j9hGDZgP) C:\Python373\my_flask>pipenv --rm

Close the pipenv shell:

(my_flask-j9hGDZgP) C:\Python373>exit

C:\Python373\my_flask>

This tool requires special attention in developing and programming with Python, so we have only traveled some of its real possibilities.

Python 3.7.3 : Using the werkzeug.

From the official webpage, you can see all the features of this python module:
Werkzeug is a comprehensive WSGI web application library. It began as a simple collection of various utilities for WSGI applications and has become one of the most advanced WSGI utility libraries.
It includes:
An interactive debugger that allows inspecting stack traces and source code in the browser with an interactive interpreter for any frame in the stack.
A full-featured request object with objects to interact with headers, query args, form data, files, and cookies.
A response object that can wrap other WSGI applications and handle streaming data.
A routing system for matching URLs to endpoints and generating URLs for endpoints, with an extensible system for capturing variables from URLs.
HTTP utilities to handle entity tags, cache control, dates, user agents, cookies, files, and more.
A threaded WSGI server for use while developing applications locally.
A test client for simulating HTTP requests during testing without requiring running a server.
The documentation of this python module can be found here.
You can install this python module with the pip tool:

pip install -U Werkzeug

This module can install it if you install the flask python module with python 3.7.3.
Let's test it:

C:\Python373>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from werkzeug.wrappers import Request, Response
>>>
>>> @Request.application
... def application(request):
...     return Response('Hello, World!')
...
>>> if __name__ == '__main__':
...     from werkzeug.serving import run_simple
...     run_simple('localhost', 4000, application)
...
 * Running on http://localhost:4000/ (Press CTRL+C to quit)
127.0.0.1 - - [16/Jul/2019 13:29:44] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [16/Jul/2019 13:29:46] "GET /favicon.ico HTTP/1.1" 200 -

If you run this source code and open your browser with this http://localhost:4000/ address then you will see a text with Hello, World!.
The common uses of this python package are:

>>> from werkzeug.wrappers import Request, Response
>>> from werkzeug.routing import Map, Rule
>>> from werkzeug.exceptions import HTTPException, NotFound
>>> from werkzeug.wsgi import SharedDataMiddleware
>>> from werkzeug.formparser import parse_form_data
>>> from werkzeug.utils import redirect
>>> from werkzeug.utils import escape
>>> from werkzeug.serving import run_simple

Let's test another simple application with werkzeug python module:

from werkzeug.serving import run_simple
from werkzeug.wrappers import Request
from werkzeug.wrappers import Response

class Application(object):
    def __init__(self):
        print('init Application class')

    def dispatch_request(self, request):
        return Response("Logged in as %s" % request.authorization)

    def __call__(self, environ, start_response):
        request = Request(environ)
        auth = request.authorization
        response = self.dispatch_request(request)
        return response(environ, start_response)

if __name__ == "__main__":
    application = Application()
    run_simple("localhost", 5000, application)

The result will be this:

C:\Python373>python.exe werkzeug_001.py
init Application class
 * Running on http://localhost:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [16/Jul/2019 14:05:31] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [16/Jul/2019 14:05:32] "GET /favicon.ico HTTP/1.1" 200 -

You can add more features to this source code and check with werkzeug features of Web Server Gateway Interface (WSGI).
A full list with examples can be found here.