Python 3.6.9 : My colab tutorials - part 005.

Today I tested google colab python features with google authentification and google.colab drive and files.
The first part of google colab code comes with authentification and you need to add the verification code for google account.
The google colab use the same version of python:

3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]

You can see all source code on my GitHub account.
The notebook can be found here.

Python 3.8.3 : Using twitter application with python-twitter - part 002.

This is the second part of tutorials series with python-twitter.

Today I will show you how to get pieces of information about friends, users and save into a binary file with pickle known as cPickle.

I will use my old source code from the last tutorial.

import os
import twitter
# for save to file import by python version
try:
   import cPickle as pickle
except:
   import pickle

consumer_key=' '
consumer_secret=' '
token_key=' '
token_secret=' '

if __name__ == "__main__":
    api = twitter.Api(consumer_key=consumer_key,
                  consumer_secret=consumer_secret,
                  access_token_key=token_key,
                  access_token_secret=token_secret) 
    
    screen_name = 'catafest'
       
    # print all users of this account authentificated 
    # you can use GetFriends(screen_name=screen_name) 
    users = api.GetFriends()
    
    print([u.screen_name for u in users])
    # get followers 
    followers = api.GetFollowers(screen_name=screen_name)
    # print followers 
    print([f.screen_name for f in followers])
    
    # ... and save into a binary file 
    followers_file = "followers_file.bin"
    
    if not os.path.exists(followers_file):
        pickle.dump(followers, open(followers_file, "wb"), protocol=pickle.HIGHEST_PROTOCOL)
        
    # load binary file     
    if os.path.exists(followers_file):
        followers_read = pickle.load(open(followers_file, "rb"))
        print(followers_read)

The result is similar with this:

python.exe .\test_webpage_001.py
['SnapChick', 'NASA', 'andor_saga', 'blendermarket', 'Minehut', 'Aternos', 'axnro', 'Flexi23',
...
['PStackoverflow', 'SamLeac86078418', 'Sohanurr559', 'jasonalba', 'avkorablev', 'dotnetfiddle',
...
[User(ID=1260415029855256583, ScreenName=PStackoverflow), 
...

Python 3.8.3 : Using twitter application with python-twitter - part 001.

You need to create a application for your twitter user developer on this webpage.

The next step is to get all keys and tokens from your application.

I used the python-twitter see the official webpage documentation.

Let's install this python module using the pip tool

pip install python-twitter
Collecting python-twitter
...
Installing collected packages: oauthlib, requests-oauthlib, python-twitter
Successfully installed oauthlib-3.1.0 python-twitter-3.5 requests-oauthlib-1.3.0

Let's see a simple source code:

import os
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import twitter
import datetime
from datetime import *

consumer_key=' '
consumer_secret=' '
token_key=' '
token_secret=' '

def get_tweets(api=None, screen_name=None):
    timeline = api.GetUserTimeline(screen_name=screen_name, count=200)
    earliest_tweet = min(timeline, key=lambda x: x.id).id
    print("getting tweets before:", earliest_tweet)

    while True:
        tweets = api.GetUserTimeline(
            screen_name=screen_name, max_id=earliest_tweet, count=200
        )
        new_earliest = min(tweets, key=lambda x: x.id).id

        if not tweets or new_earliest == earliest_tweet:
            break
        else:
            earliest_tweet = new_earliest
            print("getting tweets before:", earliest_tweet)
            timeline += tweets

    return timeline

if __name__ == "__main__":
    api = twitter.Api(consumer_key=consumer_key,
                  consumer_secret=consumer_secret,
                  access_token_key=token_key,
                  access_token_secret=token_secret) 
    # print api 
    #print(dir(api))
    
    # print all users of this account authentificated 
    #users = api.GetFriends()
    #print([u.screen_name for u in users])
    
    # print all tweets of my user catafest 
    screen_name = "catafest"
    timeline = get_tweets(api=api, screen_name=screen_name)
    dates = []
    for x in timeline:
        created = x.created_at
        dates.append(created)
        
    print(dates)
    dat = [datetime.strptime(d, "%a %b %d %H:%M:%S +0000 %Y") for d in dates]

    levels = np.tile([-8, 8, -4, 4, -1, 1],int(np.ceil(len(dat)/3)))[:len(dat)]
    print(levels)
    fig, ax = plt.subplots(figsize=(7.6, 5), constrained_layout=True)
    ax.set(title="Twitter dates")
    markerline, stemline, baseline = ax.stem(dat, levels,linefmt="C3-", basefmt="k-",use_line_collection=True)
    markerline.set_ydata(np.zeros(len(dat)))
    plt.setp(markerline, mec="k", mfc="w", zorder=1)
    plt.show()

The result of this script comes with this output:

python .\test_webpage_001.py
getting tweets before: 1123237192422367234
['Mon May 18 13:52:09 +0000 2020', 'Sat May 09 11:14:43 +0000 2020', 'Fri May 08 10:42:18 +0000 2020', 
'Fri May 08 10:41:37 +0000 2020', 'Sat May 02 17:41:07 +0000 2020', 'Sat May 02 17:39:15 +0000 2020', 
'Thu Apr 30 12:53:48 +0000 2020', 'Tue Apr 28 20:00:38 +0000 2020', 'Mon Apr 27 21:12:07 +0000 2020', 
'Fri Apr 24 16:39:58 +0000 2020', 'Fri Apr 24 16:09:26 +0000 2020', 'Sat Apr 11 16:56:40 +0000 2020', 
'Sun Mar 22 19:11:16 +0000 2020', 'Sat Mar 21 09:03:30 +0000 2020', 'Sat Mar 21 09:02:48 +0000 2020', 
'Sat Mar 21 08:59:18 +0000 2020', 'Mon Mar 16 06:29:34 +0000 2020', 'Fri Jan 24 19:59:38 +0000 2020', 
'Sat Jan 18 12:14:07 +0000 2020', 'Fri Jan 17 20:58:18 +0000 2020', 'Thu Jan 16 20:50:47 +0000 2020', 
'Thu Jan 16 20:49:16 +0000 2020', 'Fri Jan 03 17:57:33 +0000 2020', 'Sat Dec 28 10:14:11 +0000 2019', 
'Tue Apr 30 14:46:30 +0000 2019']
[-8  8 -4  4 -1  1 -8  8 -4  4 -1  1 -8  8 -4  4 -1  1 -8  8 -4  4 -1  1 -8]

The image show with matplotlib is this:

Python 3.7.5 : Django on Fedora distro.

[mythcat@desk django]$ source env/bin/activate
(env) [mythcat@desk django]$ python3 
Python 3.7.6 (default, Dec 19 2019, 22:52:49) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> django.VERSION
(3, 0, 1, 'final', 1) 

Static files are those files that can not be processed, generated or modified by the server.
Static files improve the performance of the website with the template inheritance method.
Static file management is an important factor in web development.
I will show you how static file works on Django project.
The new static files folder is set on settings.py file:

...
# Static files (CSS, JavaScript, Images)

STATIC_URL = '/static/'

STATIC_ROOT = os.path.join(BASE_DIR, '/home/mythcat/projects/django/mysite/test001/')
STATICFILES_DIRS = [
   os.path.join(BASE_DIR, 'static'),
 ]
...

Let's run the server:

(env) [mythcat@desk mysite]$ python3 manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
January 26, 2020 - 09:01:10
Django version 3.0.1, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C. 

If I try to use the admin area then we can see the bad result for static files. 
...

[26/Jan/2020 09:10:02] "GET /admin/test001/post/ HTTP/1.1" 200 5935
[26/Jan/2020 09:10:02] "GET /static/admin/css/changelists.css HTTP/1.1" 200 6190
[26/Jan/2020 09:10:02] "GET /admin/jsi18n/ HTTP/1.1" 200 3223
[26/Jan/2020 09:10:02] "GET /static/admin/js/jquery.init.js HTTP/1.1" 200 363
[26/Jan/2020 09:10:02] "GET /static/admin/js/urlify.js HTTP/1.1" 200 8941
[26/Jan/2020 09:10:02] "GET /static/admin/js/actions.js HTTP/1.1" 200 6766
[26/Jan/2020 09:10:02] "GET /static/admin/js/prepopulate.js HTTP/1.1" 200 1530
[26/Jan/2020 09:10:02] "GET /static/admin/js/core.js HTTP/1.1" 200 5723
[26/Jan/2020 09:10:02] "GET /static/admin/js/admin/RelatedObjectLookups.js HTTP/1.1" 200 6918
[26/Jan/2020 09:10:02] "GET /static/admin/css/fonts.css HTTP/1.1" 200 423
[26/Jan/2020 09:10:02] "GET /static/admin/js/vendor/jquery/jquery.js HTTP/1.1" 200 280364
[26/Jan/2020 09:10:02] "GET /static/admin/js/vendor/xregexp/xregexp.js HTTP/1.1" 200 128820
[26/Jan/2020 09:10:02] "GET /static/admin/img/tooltag-add.svg HTTP/1.1" 200 331
[26/Jan/2020 09:10:02] "GET /static/admin/img/sorting-icons.svg HTTP/1.1" 200 1097
Not Found: /favicon.ico
... 

Now I can try to run the command collectstatic.
When this command is executed, Django performs these operations:

• it looks for static files in all the directories listed in STATICFILES_DIRS;
• the static-files are then copied and saved in STATIC_ROOT directory;
• when the server is requested for static content, it will fetch a file from STATIC_ROOT;
• that file will have its URL modified with STATIC_URL.

These errors show us many informations about this process:

 (env) [mythcat@desk mysite]$ python3 manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings:

    /home/mythcat/projects/django/mysite

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
Found another file with the destination path 'admin/js/urlify.js'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
...
Found another file with the destination path 'django.png'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'favicon.ico'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'admin/js/urlify.js'. It will be ignored since only the first 
encountered file is collected. If this is not what you want, make sure every static file has a unique path.
...

Found another file with the destination path 'rest_framework/js/prettify-min.js'. 
It will be ignored since only the first encountered file is collected. If this is not what you want, make
 sure every static file has a unique path.
...
166 static files copied to '/home/mythcat/projects/django/mysite'. 

But, Django respects the order of your applications in settings.py file the INSTALLED_APPS area, when running collectstatic command.
If you have two installed apps that write the same static files then Django collectstatic command will write the static files for the app appearing first in the list.
In my case: django.contrib.admin , test001 and ... .
Also, is need to set this code source on urls.py to return the proper URL pattern for serving static files to your already defined pattern list.

from django.contrib.staticfiles.urls import staticfiles_urlpatterns
...
urlpatterns += staticfiles_urlpatterns()

Let's run the runserver:

 (env) [mythcat@desk mysite]$ python3 manage.py runserver
...
[26/Jan/2020 09:29:18] "GET /static/admin/css/fonts.css HTTP/1.1" 200 423
Not Found: /favicon.ico 

You can see the /favicon.ico is not found in the default path and need to move it at /home/mythcat/projects/django/mysite/test001.
After these changes let's fix all the problems I created with these learning steps on static issues.
Let's move the static folder into mysite folder.
Change the settings.py file for the static issue with this source of code:

# Static files (CSS, JavaScript, Images)

STATIC_URL = '/static/'

STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = [
   os.path.join(BASE_DIR, 'static'),
 ]

Run the (env) [mythcat@desk mysite]$ python3 manage.py collectstatic

(env) [mythcat@desk mysite]$ python3 manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings:

    /home/mythcat/projects/django/mysite/static

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes

163 static files copied to '/home/mythcat/projects/django/mysite/static', 2 unmodified.
(env) [mythcat@desk mysite]$ ls
db.sqlite3  favicon.ico  manage.py  mysite  rest_framework  static  test001
(env) [mythcat@desk mysite]$ cd static/
(env) [mythcat@desk static]$ ls
admin  django.png  favicon.ico  rest_framework

This is a simple base tutorial.
In this point you can select the way of your Django project.

Python 3.8.2 : Create a Django Calendar - part 002.

I was a bit busy with another project I am working on and this is the reason that led to the delay of this tutorial.

The big problem with Django tutorials is the volume of the source code.

Because of this, this tutorial will highlight changes to the source code and share it in my GitHub account.

Let's see the changes into my project tree and files:

└───test_calendar
    ├───first_calendar
    │   ├───migrations
    │   └───static
    │       └───first_calendar
    │           └───css
    ├───media
    ├───templates
    └───test_calendar

The settings.py file:

...
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'first_calendar.apps.FirstCalendarConfig',
]
...
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.0/howto/static-files/

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, '/static/')
]

STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATIC_URL = '/static/'

MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

The complete project is a basic calendar, which can be modified by adding events or linked to a Google calendar.

The source code can be found here.

Python 3.8.2 : Create a Django Calendar - part 001.

This tutorial show you how can use pipenv tool and set the Django project in order to create a calendar with Django project.

First, install the pipenv tool using the pip tool:

python -m pip install pipenv
Collecting pipenv
...
Successfully installed pipenv-2020.6.2 virtualenv-clone-0.5.4

You can see all options and features with this command:

pipenv
Usage: pipenv [OPTIONS] COMMAND [ARGS]...

Options:
  --where                         Output project home information.
  --venv                          Output virtualenv information.
  --py                            Output Python interpreter information.
  --envs                          Output Environment Variable options.
  --rm                            Remove the virtualenv.
  --bare                          Minimal output.
  --completion                    Output completion (to be executed by the
                                  shell).

  --man                           Display manpage.
  --support                       Output diagnostic information for use in
                                  GitHub issues.

  --site-packages / --no-site-packages
                                  Enable site-packages for the virtualenv.
                                  [env var: PIPENV_SITE_PACKAGES]

  --python TEXT                   Specify which version of Python virtualenv
                                  should use.

  --three / --two                 Use Python 3/2 when creating virtualenv.
  --clear                         Clears caches (pipenv, pip, and pip-tools).
                                  [env var: PIPENV_CLEAR]

  -v, --verbose                   Verbose mode.
  --pypi-mirror TEXT              Specify a PyPI mirror.
  --version                       Show the version and exit.
  -h, --help                      Show this message and exit.


Usage Examples:
   Create a new project using Python 3.7, specifically:
   $ pipenv --python 3.7

   Remove project virtualenv (inferred from current directory):
   $ pipenv --rm

   Install all dependencies for a project (including dev):
   $ pipenv install --dev

   Create a lockfile containing pre-releases:
   $ pipenv lock --pre

   Show a graph of your installed dependencies:
   $ pipenv graph

   Check your installed dependencies for security vulnerabilities:
   $ pipenv check

   Install a local setup.py into your virtual environment/Pipfile:
   $ pipenv install -e .

   Use a lower-level pip command:
   $ pipenv run pip freeze

Commands:
  check      Checks for PyUp Safety security vulnerabilities and against PEP
             508 markers provided in Pipfile.

  clean      Uninstalls all packages not specified in Pipfile.lock.
  graph      Displays currently-installed dependency graph information.
  install    Installs provided packages and adds them to Pipfile, or (if no
             packages are given), installs all packages from Pipfile.

  lock       Generates Pipfile.lock.
  open       View a given module in your editor.
  run        Spawns a command installed into the virtualenv.
  shell      Spawns a shell within the virtualenv.
  sync       Installs all packages specified in Pipfile.lock.
  uninstall  Uninstalls a provided package and removes it from Pipfile.
  update     Runs lock, then sync.

Create a folder for your project, I used this folder named django_test_002:

mkdir django_test_002

Create a virtualenv using the pipenv shell

django_test_002>pipenv shell
Creating a virtualenv for this project…
Pipfile: D:\Projects\Python\django_test_002\Pipfile
Using D:/Python38/python.exe (3.8.2) to create virtualenv…
...

Install Django python package:

(django_test_002-bp-hUvnN) D:\Projects\Python\django_test_002> pipenv install Django
Installing Django…
Adding Django to Pipfile's [packages]…
Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
 Locking...Building requirements...
Resolving dependencies...
Success!
Updated Pipfile.lock (a6086c)!
Installing dependencies from Pipfile.lock (a6086c)…
  ================================ 0/0 - 00:00:00

Let's test the pipenv shell tool with a simple example for activate, deactivate and exit:

D:\Projects\Python\django_test_002>pipenv shell
Launching subshell in virtual environment…
Microsoft Windows [Version 10.0.18363.900]
(c) 2019 Microsoft Corporation. All rights reserved.

(django_test_002-bp-hUvnN) D:\Projects\Python\django_test_002>activate

(django_test_002) (django_test_002-bp-hUvnN) D:\Projects\Python\django_test_002>deactivate
(django_test_002-bp-hUvnN) D:\Projects\Python\django_test_002>exit

Stores all the python packages you installed into a requirements.txt file

D:\Projects\Python\django_test_002>pipenv shell
Launching subshell in virtual environment…
Microsoft Windows [Version 10.0.18363.900]
(c) 2019 Microsoft Corporation. All rights reserved.

(django_test_002-bp-hUvnN) D:\Projects\Python\django_test_002>pip3 freeze > requirements.txt

Create django project named test_calendar:

(django_test_002-bp-hUvnN) D:\Projects\Python\django_test_002>django-admin startproject test_calendar

Start server to check that our project is running at localhost:8000.

(django_test_002-bp-hUvnN) D:\Projects\Python\django_test_002>cd test_calendar

(django_test_002-bp-hUvnN) D:\Projects\Python\django_test_002\test_calendar>python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 17 unapplied migration(s). Your project may not work properly until you apply the migrations 
for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
June 13, 2020 - 20:52:00
Django version 3.0.7, using settings 'test_calendar.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.

Now, you can open with your browser the default link http://127.0.0.1:8000/ and you will see the start page of Django framework.

Use Ctrl+C keys to stop it, and create an application.

I used the next command to create first_calendar application.

D:\Projects\Python\django_test_002\test_calendar>python manage.py startapp first_calendar

Into the folder application first_calendar make these changes to views.py from to create a index view:

from django.shortcuts import render
from django.http import HttpResponse

# Create your views here.

def index(request):
    return HttpResponse('hello')

Create a new file named urls.py in the folder first_calendar and add this source code:

from django.conf.urls import url
from . import views

app_name = 'first_calendar'
urlpatterns = [
    #url(r'^index/$', views.index, name='index'),
    url('', views.index,  name='index'),
]

Return to the project base folder and add the view to urls.py from the project test_calendar folder:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('first_calendar.urls')),
]

Into the project folder test_calendar add the application to the file settings.py.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'first_calendar',
]

Use migrate option to migrate the project:

(django_test_002-bp-hUvnN) D:\Projects\Python\django_test_002\test_calendar>python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying sessions.0001_initial... OK

Go back to default project folder. Now you can test using this command:

(django_test_002-bp-hUvnN) D:\Projects\Python\django_test_002\test_calendar>python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
June 13, 2020 - 21:52:42
Django version 3.0.7, using settings 'test_calendar.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.

You will see a simple word: hello. Let's fix the admin login by adding user:

python manage.py createsuperuser
Username (leave blank to use 'catal'): catafest
Email address: catafest@yahoo.com
Password:

I set user catafest and password admin76 and then use the next command to see the result on http://127.0.0.1:8000/admin:

python manage.py runserver

The next step is to create a Event class into models.py file from first_calendar:

from django.db import models

class Event(models.Model):
    title = models.CharField(max_length=200)
    description = models.TextField()
    start_time = models.DateTimeField()
    end_time = models.DateTimeField()

Then add this class into admin.py file:

from django.contrib import admin
from first_calendar.models import Event

admin.site.register(Event)

This tutorial set the one default Django project with the Django framework version 3.0.7 .

Python 3.8.3 : Using the fabric python module - part 002.

This is second tutorial about the fabric.

Using the old example, I will show you how can use it with some basic features.

The example cover these issues:

• show fabric version;
• use task decorator;
• import and use multiple hosts;

The documentation page come with this info about task:

Fabric 1.1 introduced the Task class to facilitate new features and enable some programming best practices, specifically:

  Object-oriented tasks. Inheritance and all that comes with it can make for much more sensible code reuse than passing around simple function objects. The classic style of task declaration didn’t entirely rule this out, but it also didn’t make it terribly easy.

  Namespaces. Having an explicit method of declaring tasks makes it easier to set up recursive namespaces without e.g. polluting your task list with the contents of Python’s os module (which would show up as valid “tasks” under the classic methodology.)

With the introduction of Task, there are two ways to set up new tasks:

 Decorate a regular module level function with @task, which transparently wraps the function in a Task subclass. The function name will be used as the task name when invoking.

 Subclass Task (Task itself is intended to be abstract), define a run method, and instantiate your subclass at module level. Instances’ name attributes are used as the task name; if omitted the instance’s variable name will be used instead.

Let's see the example:

import fabric
from fabric import Connection
print("fabric.__version__")
print(fabric.__version__)

print("fabric.__version_info__")
print(fabric.__version_info__)

print("dir(fabric)")
print(dir(fabric))

import getpass
host = "catafest@tty.sdf.org"
password = getpass.getpass('Password for SDF account:')
with Connection("catafest@tty.sdf.org", connect_kwargs={"password":password}) as con:
    print("I will run command: ls")
    con.run("ls")
    
# using the task decorator 
from fabric import task
@task    
def processes(con):
    return con.run("ps -aux", hide = True).stdout.strip()

# print processes
print(processes(con))
con.run("exit")
# use the import for multiple host with all modules

from fabric import Connection, Config, SerialGroup, ThreadingGroup, exceptions, runners
from fabric.exceptions import GroupException

hosts = "catafest@tty.sdf.org,catafest@tty.sdf.org"

def testHosts(hosts):
    # Get list of hosts from somewhere, and convert them to connections
    hosts = hosts.split(",")
    servers = [Connection(host=host,connect_kwargs={"password":password}) for host in hosts]
    thread_group = ThreadingGroup.from_connections(servers)
    results = thread_group.run("who -a", hide=True)
    for r in results:
        connection = results[r]
        return connection.stdout.strip() 
        
print(testHosts(hosts))

The result of this python script:

fabric.__version__
2.5.0
fabric.__version_info__
(2, 5, 0)
dir(fabric)
['Config', 'Connection', 'Executor', 'Group', 'GroupResult', 'Remote', 'Result', 'SerialGroup', 'Task', 
'ThreadingGroup', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', 
'__package__', '__path__', '__spec__', '__version__', '__version_info__', '_version', 'config', 'connection', 
'exceptions', 'executor', 'group', 'runners', 'task', 'tasks', 'transfer', 'tunnels', 'util']
Password for SDF account:
I will run command: ls
USER            PID %CPU %MEM    VSZ    RSS TTY     STAT STARTED     TIME COMMAND
...
catafest      10431  0.0  0.0  30496   3120 ?       S     5:50PM  0:00.00 sshd:
...
catafest      19243  0.0  0.0  12008   1096 ?       O     5:50PM  0:00.00 ps -a
...
user   pts/0    Jun  7 17:40 00:10         0  term=0 exit=0 sess=0 type=user process  ...

Python 3.8.3 : Using the fabric python module - part 001.

The tutorial for today is about fabric python module.

You can read about this python module on the official webpage.

The team comes with this intro:

Fabric is a high level Python (2.7, 3.4+) library designed to execute shell commands remotely over SSH, yielding useful Python objects in return

...

It builds on top of Invoke (subprocess command execution and command-line features) and Paramiko (SSH protocol implementation), extending their APIs to complement one another and provide additional functionality.

I used python version 3.8.3rc1 to install with the pip3 tool.

pip3 install fabric
Collecting fabric
...
Installing collected packages: invoke, fabric
Successfully installed fabric-2.5.0 invoke-1.4.1

from fabric import Connection
import getpass
host = "catafest@tty.sdf.org"
password = getpass.getpass('Password for SDF account:')
with Connection("catafest@tty.sdf.org", connect_kwargs={"password":password}) as con:
    print("I will run command: id -u -n")
    con.run("id -u -n")
    print("I will run command: df -h")
    con.run("df -h")
    print("I will run command for SDF help: help")
    con.run("help")
    out = con.run("free")
    print("output for dir(out)")
    print(dir(out))
    print("output for out.shell")
    print(out.shell)
    print("output for out.connection")
    print(out.connection)
    print("output for out.stdout")
    print(out.stdout)
    print("output for out.env")
    print(out.env)

This is a simple example tested on my SDF account.

About the SDF:

The Super Dimension Fortress is a networked community of free software
authors, teachers, librarians, students, researchers, hobbyists,
computer enthusiasts, the aural and visually impaired. It is operated
as a recognized non-profit 501(c)(7) and is supported by its members.

Our mission is to provide remotely accessible computing facilities for
the advancement of public education, cultural enrichment, scientific
research and recreation. Members can interact electronically with each
other regardless of their location using passive or interactive forums.
Further purposes include the recreational exchange of information
concerning the Liberal and Fine Arts.

Members have UNIX shell access to games, email, usenet, chat, bboard,
webspace, gopherspace, programming utilities, archivers, browsers, and
more. The SDF community is made up of caring, highly skilled people who
operate behind the scenes to maintain a non-commercial INTERNET.

For information about membership levels, click on 'join' above.

Let see my example:

python fabric_001.py
Password for SDF account:
I will run command: id -u -n
catafest
I will run command: df -h
Filesystem         Size       Used      Avail %Cap Mounted on
/dev/wd0a           35G       2.0G        31G   6% /
nol1:/sdf           23T       5.1T        17T  23% /sdf
ptyfs              1.0K       1.0K         0B 100% /dev/pts
I will run command for SDF help: help
tput: No terminal type specified and no TERM variable set in the environment.
SDF Help System - v8, 1993 - sdf!smj

 [a] Directory and File Utilities
 [b] Email Information
 [c] USENET Information
 [d] User and Process Information
 [e] Tutorials and Very Useful Commands
 [f] ARPA Services (internetworking)
 [g] Homepage, VHOST and MetaARPA Utilities

 [q] Quit SDF Help System

Your Choice? q
q
For more help, type 'faq'
For a list of basic UNIX commands, type 'unix'
For live help, type 'helpdesk'

             total       used       free    buffers
Mem:      16274580   15098664    1175916   13433820
Swap:      1049292          0    1049292
output for dir(out)
['__bool__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', 
'__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', 
'__ne__', '__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', 
'__str__', '__subclasshook__', '__weakref__', 'command', 'connection', 'encoding', 'env', 'exited', 'failed', 
'hide', 'ok', 'pty', 'return_code', 'shell', 'stderr', 'stdout', 'tail']
output for out.shell
C:\Windows\system32\cmd.exe
output for out.connection
<Connection host=tty.sdf.org user=catafest>
output for out.stdout
             total       used       free    buffers
Mem:      16274580   15098664    1175916   13433820
Swap:      1049292          0    1049292

output for out.env
{}

Python Qt5 : PyQt5 and PyGame compatibility with source code.

This tutorial tries to solve from the objectives related to solving and stabilizing compatibility errors between PyQt4 and PyQt5 and creating a common interface between PyQt5 and PyGame.

There is always the same problem in programming when the developer for some reason has to change classes, methods and functions and reusing the old code is no longer valid.

In this case, common or other errors occur, which leads to a waste of time.

I will present a simple way to solve these problems.

I really like to use the PyQt5 module to create interfaces for my python programs and scripts.

Like any programmer who hasn't fully used all A.P.I, I always use the old source code I programmed in the past.

What the developer says about the transition from PyQt4 to PyQt5 we can see on the official page.

Obviously, you will have to move on to things to know but it is quite difficult to always come back and read this content when you have programming errors.

Today, I wanted to make a simple drawing interface in PyGame that would be included in a PyQt5 interface.

I tried to use an old code created by me in PyQt4 but I realized that I had encountered errors before switching to the new PyQt5.

This compatibility problem generates errors and can be solved as follows: by knowing the exact solution and fixing errors in real time, studying the changes created by the developer or the classic search for errors.

My solution comes with the help of these solutions and requires a simple step using the commented source code.

To show you how simple it is to understand I will show you the source code for the interface I built that simply to solves the problem of understanding compatibility by reading the developer source code with simple and useful comments.

#the old import for PyQt4
#from PyQt4 import QtGui

#the new import for PyQt5
#from PyQt5 import QtCore, QtGui, QtWidgets
#class MainWindow(QtWidgets.QMainWindow, UI.MainUI.Ui_MainWindow):

from PyQt5 import QtGui
from PyQt5 import QtWidgets
import pygame
import sys

# old definition for PyQt4 for QWidget
#class ImageWidget(QtGui.QWidget):
class ImageWidget(QtWidgets.QWidget):   
    def __init__(self,surface,parent=None):
        super(ImageWidget,self).__init__(parent)
        w=surface.get_width()
        h=surface.get_height()
        self.data=surface.get_buffer().raw
        self.image=QtGui.QImage(self.data,w,h,QtGui.QImage.Format_RGB32)

    def paintEvent(self,event):
        my_paint=QtGui.QPainter()
        # the definitions for PyQt4 and PyQt5 use QtGui.QPainter()     
        my_paint.begin(self)
        my_paint.drawImage(0,0,self.image)
        my_paint.end()
        
# old definition for PyQt4 for QMainWindow
#class MainWindow(QtGui.QMainWindow):
class MainWindow(QtWidgets.QMainWindow):
    def __init__(self,surface,parent=None):
        super(MainWindow,self).__init__(parent)
        self.setFixedSize(640, 480)
        self.setCentralWidget(ImageWidget(surface))
# this part of source code need to be updated if you want to use animation        
# init PyGame 
pygame.init()
# define a surface 
my_surface=pygame.Surface((640,480))
# fill the surface, see https://www.pygame.org/docs/ref/surface.html#pygame.Surface.fill
my_surface.fill((0,0,255,176))
# draw circle see https://www.pygame.org/docs/ref/draw.html#pygame.draw.circle
pygame.draw.circle(my_surface,(0,0,127,255),(76,76),76)
# draw ellipse (surface, color(R,G,B), size (x,y,x+dx, y+y+dy) )
pygame.draw.ellipse(my_surface,(127,0,0,0),(0,0,12,76))
 
# this part of source code will show 
# the my_surface created with PyGame in PyQt5
# old definition for PyQt4
#app=QtGui.QApplication(sys.argv)
app=QtWidgets.QApplication(sys.argv)
my_window=MainWindow(my_surface)
my_window.show()
app.exec_()

Python 3.8.3 : A brief introduction to the Celery python package.

The development team tells us:

Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

To do that Celery uses tasks.

These task queues are used as a mechanism to distribute work across threads or machines using brokers.

A task a client puts a message on the queue and the broker then delivers the message to a worker.

The install of this python package is simple:

Celery can run on a single machine, on multiple machines, or even across datacenters.

pip3 install -U Celery
Collecting Celery
...
Installing collected packages: pytz, billiard, vine, amqp, kombu, Celery
Successfully installed Celery-4.4.2 amqp-2.5.2 billiard-3.6.3.0 kombu-4.6.8 pytz-2020.1 vine-1.3.0

The Celery library must be instantiated before use like an application.

First, I tested with a simple example application from the official website.

from celery import Celery
app = Celery()

@app.task
def add(x, y): return x + y

if __name__ == '__main__':
    app.worker_main()

When I run it this application the output is this:

python.exe .\example_tasks.py
 
 -------------- celery@DESKTOP-9DSLSMF v4.4.2 (cliffs)
--- ***** ----- 
-- ******* ---- Windows-10-10.0.18362-SP0 2020-05-24 18:30:04
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         __main__:0x1c697187940
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 6 (prefork)
                .> celery           exchange=celery(direct) key=celery


[2020-05-24 18:30:06,273: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//:
[WinError 10061] No connection could be made because the target machine actively refused it.
Trying again in 2.00 seconds... (1/100)

[2020-05-24 18:30:10,296: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//:
[WinError 10061] No connection could be made because the target machine actively refused it.
Trying again in 4.00 seconds... (2/100)

[2020-05-24 18:30:16,323: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//:
[WinError 10061] No connection could be made because the target machine actively refused it.
Trying again in 6.00 seconds... (3/100)


worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)

This error is show because I trying to connect to a local instance of default broker named RabbitMQ .

RabbitMQ is most popular open source message broker and can be download it from the official webpage.

If you want to install RabbitMQ, then you need to install Erlang, see screenshot:

This is not the only one option, you have many features like for example Redis.

The development team try experimental support for a myriad of other solutions, including using SQLite for local development.

A full list with all supported brokers can be found here.

Celery is easy to integrate with web frameworks, some of them even have integration packages:

• Pyramid  with pyramid_celery;
• Pylons with celery-pylons;
• Flask not needed;
• web2py with web2py-celery;
• Tornado with tornado-celery;
• Tryton with celery_tryton;
• and Django also no not needed because works well.
  

Maybe I will make another tutorial about this python module and how works with Redis or Django.

News : Microsoft Build digital event for Python users.

We in the Python team are excited to be joining Microsoft Build 2020, which is starting next Tuesday, May 19. During a global pandemic, Build has turned into a digital-only event, running for 48 continuous hours, that is open to anyone around the world, at no cost.

At Build, we’ll demo the new capabilities we’ve been building recently. We’ll also showcase the work we have been doing to enable Python developers to build applications and analyze data using Visual Studio Code. Lastly, we’ll show how you can run Python web apps and machine learning models on the cloud with Azure.

Our friends across Microsoft will also present services and tools that developers, including those working with Python, can leverage to remain productive and continue collaborating even in remote, distributed teams.

You can participate in a live Q&A stream session:

• Tuesday May 19 at 9pm
• Wednesday May 20 at 12:30pm
• Thursday May 21 at 3am

This event can be found at the official webpage.

Python 3.8.3 : Simple example to fix maximum recursion depth exceeded.

This short tutorial try to solve simple and easy the stack limit for recursion without using advanced programming techniques.

sys.setrecursionlimit(limit)
    Set the maximum depth of the Python interpreter stack to limit. 
This limit prevents infinite recursion from causing an overflow of the C stack 
and crashing Python.

This can be done in most cases with:

import sys
sys.setrecursionlimit(1976)

Let's test with an simple example for a given number.

input_number = 1234567890987654321

I create a script for the next function to print the result:

def split_input(n):
    return split_input(n // 10) + [n % 10]

The result of this function is:

python flow_001.py
Traceback (most recent call last):
  File "flow_001.py", line 12, in 
    input_list = split_input(input_number)
  File "flow_001.py", line 10, in split_input
    return split_input(n // 10) + [n % 10]
  File "flow_001.py", line 10, in split_input
    return split_input(n // 10) + [n % 10]
  File "flow_001.py", line 10, in split_input
    return split_input(n // 10) + [n % 10]
  [Previous line repeated 996 more times]
RecursionError: maximum recursion depth exceeded

You can see I got an error. Let fix this function with a proper source code:

def split_nr(n):
    if n < 1:
        return [n]
    else:
        return split_nr(n // 10) + [n % 10] 

This will solve the recursion split function.

python flow_001.py
[1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 9, 8, 7, 6, 5, 4, 3, 2, 1]

IronPython 2.7.7 : Intro on RevitPythonShell.

This is one of the several ways to use RevitPythonWrapper:

• pyRevit;
• RevitPythonShell;
• Dynamo;

This tool help you to use python with the Revit tool and can be found at the GitHub webpage.

They say:
The RevitPythonShell adds an IronPython interpreter to Autodesk Revit and Vasari. The RevitPythonShell (RPS) lets you to write plugins for Revit in Python, my favourite scripting language! But even better, it provides you with an interactive shell that lets you see the results of your code as you type it. This is great for exploring the Revit API while writing your Revit Addins - use this in combination with the RevitLookup database exploration tool to become a Revit API Ninja :)
You need to install your instaler according with the Revit version.
You can start it from Main Menu - Add-Ins and you see it.
Is very easy to use it, let's see one simple example:

import clr
clr.AddReference("RevitAPI")
from Autodesk.Revit.DB import BuiltInCategory as Bic
from Autodesk.Revit.DB import FilteredElementCollector as Fec
from Autodesk.Revit.DB import Transaction
# reference to the current open revit model is
doc = __revit__.ActiveUIDocument.Document

Let's see a simple example on Revit with an wall and a simple door.

IronPython 2.7.7 (2.7.7.0) on .NET 4.0.30319.42000 (64-bit)
Type "help", "copyright", "credits" or "license" for more information.
>>> a = b = 1 
>>> c = a + b + 1
>>> print(c)
3
>>> from Autodesk.Revit.DB import FilteredElementCollector as Fec
>>> from Autodesk.Revit.DB import BuiltInCategory as Bic
>>> doors = Fec(doc).OfCategory(Bic.OST_Doors).WhereElementIsNotElementType().ToElements()
>>> print(dir(doors))
['Add', 'AddRange', 'AsReadOnly', 'BinarySearch', 'Capacity', 'Clear', 'Contains', 'ConvertAll', 'CopyTo',
'Count', 'Enumerator', 'Equals', 'Exists', 'Find', 'FindAll', 'FindIndex', 'FindLast', 'FindLastIndex', 
'ForEach', 'GetEnumerator', 'GetHashCode', 'GetRange', 'GetType', 'IndexOf', 'Insert', 'InsertRange', 
'IsReadOnly', 'IsSynchronized', 'Item', 'LastIndexOf', 'MemberwiseClone', 'ReferenceEquals', 'Remove', 
'RemoveAll', 'RemoveAt', 'RemoveRange', 'Reverse', 'Sort', 'SyncRoot', 'ToArray', 'ToString', 'TrimExcess', 
'TrueForAll', '__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__format__', 
'__getattribute__', '__getitem__', '__hash__', '__init__', '__iter__', '__len__', '__new__', 
'__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__', 
'__str__', '__subclasshook__']
>>> for each_door in doors:
... 	print(each_door.Id)
... 	print(each_door.Mirrored)
... 	print(each_door.Symbol.LookupParameter("Type Name").ToString())
... 
840590
False
Autodesk.Revit.DB.Parameter
>>> 

Let's see the output of this code on Revit.