Python 3.12.1 : About multiprocessing performance.

Today I test a simple python script for Pool and ThreadPool python classes from multiprocessing python module.

The main goal was to test Python’s multiprocessing performance with my computer.

NumPy releases the GIL for many of its operations, which means you can use multiple CPU cores even with threads.

Processing large amounts of data with Pandas can be difficult, and with Polars dataframe library is a potential solution.

Sciagraph gives you both performance profiling and peak memory profiling information.

Let's teste only these class:

The multiprocessing.pool.Pool class provides a process pool in Python.

The multiprocessing.pool.ThreadPool class in Python provides a pool of reusable threads for executing spontaneous tasks.

This is the python script:

from time import time
import multiprocessing as mp
from multiprocessing.pool import ThreadPool
import numpy as np
import pickle

def main():
    arr = np.ones((1024, 1024, 1024), dtype=np.uint8)
    expected_sum = np.sum(arr)

    with ThreadPool(1) as threadpool:
        start = time()
        assert (
            threadpool.apply(np.sum, (arr,)) == expected_sum
        )
        print("Thread pool:", time() - start)

    with mp.get_context("spawn").Pool(1) as process_pool:
        start = time()
        assert (
            process_pool.apply(np.sum, (arr,))
            == expected_sum
        )
        print("Process pool:", time() - start)

if __name__ == "__main__":
    main()

This is the result:

python thread_process_pool_001.py
Thread pool: 1.6689703464508057
Process pool: 11.644825458526611

Python 2.7.12 : Django running on glitch website.

Django was design and developed by Lawrence journal world in 2003 and publicly released under BSD license in July 2005. Currently, DSF (Django Software Foundation) maintains its development and release cycle.

Django was released on 21, July 2005. Its current stable version is 2.2.5 which was released April 1, 2019.

It is maintained by the Django Software Foundation

Some time ago, the website bug allowed the use of Python language.

Today I looked at the core project in Django, fixed it and added a test user called test with the password password_test

You can test in the admin area.

I did not add new functionalities and pages because I wanted it to be a default start project.

Here's how I set permissions for this test user.

Python 3.12.1 : Read EXIF data with PIL.

This is a simple source code I created to read EXIF date from a photo.

import sys
import PIL
import PIL.Image as PILimage
from PIL import ImageDraw, ImageFont, ImageEnhance
from PIL.ExifTags import TAGS, GPSTAGS

class EXIF(object):
    def __init__(self, img):
        self.img = img
        self.exif_data = self.get_exif_data()
        self.lat = self.get_lat()
        self.lon = self.get_lon()
        self.date =self.get_date_time()
        super(Worker, self).__init__()

    @staticmethod
    def get_if_exist(data, key):
        if key in data:
            return data[key]
        return None

    def get_exif_data(self):
        """Returns a dictionary from the exif data of an PIL Image item. Also
        converts the GPS Tags"""
        exif_data = {}
        info = self.img._getexif()
        if info:
            for tag, value in info.items():
                decoded = TAGS.get(tag, tag)
                if decoded == "GPSInfo":
                    gps_data = {}
                    for t in value:
                        sub_decoded = GPSTAGS.get(t, t)
                        gps_data[sub_decoded] = value[t]

                    exif_data[decoded] = gps_data
                else:
                    exif_data[decoded] = value
        print('exif_data ===')
        print(exif_data)
        print('exif_data ===')
        return exif_data

if __name__ == '__main__':
    try:
        img = PILimage.open(sys.argv[1])
        image = EXIF(img)

    except Exception as e:
        print(e)

The result of running source code is this:

python detect_exif_data_001.py paint.jpg
exif_data ===
{'ResolutionUnit': 2, 'ExifOffset': 196, 'Make': 'Canon', 'Model': '', 'Orientation': 1, 
'DateTime': '2012:08:19 13:20:09', 'YCbCrPositioning': 1, 'XResolution': 180.0, 'YResolution': 180.0, 
'ExifVersion': b'0220', 'ComponentsConfiguration': b'\x01\x02\x03\x00', 'CompressedBitsPerPixel': 5.0, 
'DateTimeOriginal': '2012:08:19 13:20:09', 'DateTimeDigitized': '2012:08:19 13:20:09', 'ShutterSpeedValue': 4.3125,
'ApertureValue': 2.75, 'ExposureBiasValue': 0.0, 'MaxApertureValue': 2.75, 'MeteringMode': 5, 'Flash': 16, 
'FocalLength': 5.8, 'UserComment':
...
 'ColorSpace': 1, 'ExifImageWidth': 3072, 'FocalPlaneXResolution': 13653.333333333334, 'ExifImageHeight': 1728,
 'FocalPlaneYResolution': 10224.852071005917, 'FocalPlaneResolutionUnit': 2, 'SensingMethod': 2, 'FileSource': b'\x03',
 'ExposureTime': 0.05, 'ExifInteroperabilityOffset': 3334, 'FNumber': 2.6, 'CustomRendered': 0, 'ISOSpeedRatings': 80,
 'ExposureMode': 0, 'FlashPixVersion': b'0100', 'WhiteBalance': 0, 'DigitalZoomRatio': 1.0, 'SceneCaptureType': 1,
 'MakerNote' ...

Python 3.12.1 : PrettyTable

A simple Python library for easily displaying tabular data in a visually appealing ASCII table format, see this webpage.

Let's install with pip tool:

pip install prettytable
Collecting prettytable
...
Successfully installed prettytable-3.10.0 wcwidth-0.2.13

Let's test with the default example:

from prettytable import PrettyTable
table = PrettyTable()
table.field_names = ["City name", "Area", "Population", "Annual Rainfall"]
table.add_row(["Adelaide", 1295, 1158259, 600.5])
table.add_row(["Brisbane", 5905, 1857594, 1146.4])
table.add_row(["Darwin", 112, 120900, 1714.7])
table.add_row(["Hobart", 1357, 205556, 619.5])
table.add_row(["Sydney", 2058, 4336374, 1214.8])
table.add_row(["Melbourne", 1566, 3806092, 646.9])
table.add_row(["Perth", 5386, 1554769, 869.4])
print(table)

The result is this:

python test_001.py
+-----------+------+------------+-----------------+
| City name | Area | Population | Annual Rainfall |
+-----------+------+------------+-----------------+
|  Adelaide | 1295 |  1158259   |      600.5      |
|  Brisbane | 5905 |  1857594   |      1146.4     |
|   Darwin  | 112  |   120900   |      1714.7     |
|   Hobart  | 1357 |   205556   |      619.5      |
|   Sydney  | 2058 |  4336374   |      1214.8     |
| Melbourne | 1566 |  3806092   |      646.9      |
|   Perth   | 5386 |  1554769   |      869.4      |
+-----------+------+------------+-----------------+

Let's test with sqlite3 feature:

import sqlite3
from prettytable import from_db_cursor

try:
    # Connect to the database
    connection = sqlite3.connect("file_paths.db")
    cursor = connection.cursor()

    # Get the list of table names
    cursor.execute("SELECT name FROM sqlite_master WHERE type='table'")
    table_names = cursor.fetchall()
    # Print the table names
    for table in table_names:
        print(table[0])
        query = str('SELECT * FROM ' + table[0])
        cursor.execute(query)
        table_names = cursor.fetchall()
        print(table_names)
except sqlite3.Error as e:
    print(f"Error: {e}")

finally:
    # Close the connection
    connection.close()

The result for my sqlite3 file named file_paths.db and database named files is this:

files
[(1, 'C:/BACKUP/3D\\001.blend'), (2, 'C:/BACKUP/3D\\002.blend'), (3, 'C:/BACKUP/3D\\003.blend'), 
(4, 'C:/BACKUP/3D\\default_camera.blend')]

The result is according with the data from sqlite3 table.

Python 3.12.1 : From script to executable with pyinstaller.

Using scripts in or out of the environment is useful and accessible. Another version is to create an executable, but it will no longer allow the same access to the source code resource and the executable will not be very buildable.

It is debatable, especially for those who do not trust the compiled language.

Today, I use a Python packet called aaa that allows, through other options, the creation of an executable.

I installed this Python packet with the pip utility.

pip install PyInstaller

The source code I used is a simple example that displays some text.

import time 
print("Welcome catafest !")
time.sleep(100)

To convert the Python program file into a single standalone executable, I used this command:

pyinstaller --onefile --console exec_001.py
9538 INFO: PyInstaller: 6.5.0, contrib hooks: 2024.3
9539 INFO: Python: 3.12.1
9636 INFO: Platform: Windows-10-10.0.19045-SP0
9638 INFO: wrote C:\PythonProjects\executable_001\exec_001.spec
9711 INFO: Extending PYTHONPATH with paths
['C:\\PythonProjects\\executable_001']
18951 INFO: checking Analysis
18999 INFO: Building because C:\PythonProjects\executable_001\exec_001.py changed
18999 INFO: Initializing module dependency graph...
19031 INFO: Caching module graph hooks...
19392 INFO: Analyzing base_library.zip ...
26750 INFO: Loading module hook 'hook-encodings.py' from 'C:\\Python312\\Lib\\site-packages\\PyInstaller\\hooks'...
35164 INFO: Loading module hook 'hook-pickle.py' from 'C:\\Python312\\Lib\\site-packages\\PyInstaller\\hooks'...
40857 INFO: Loading module hook 'hook-heapq.py' from 'C:\\Python312\\Lib\\site-packages\\PyInstaller\\hooks'...
44873 INFO: Caching module dependency graph...
45194 INFO: Running Analysis Analysis-00.toc
45194 INFO: Looking for Python shared library...
45217 INFO: Using Python shared library: C:\Python312\python312.dll
45217 INFO: Analyzing C:\PythonProjects\executable_001\exec_001.py
45223 INFO: Processing module hooks...
45240 INFO: Performing binary vs. data reclassification (2 entries)
45243 INFO: Looking for ctypes DLLs
45259 INFO: Analyzing run-time hooks ...
45264 INFO: Including run-time hook 'C:\\Python312\\Lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_inspect.py'
45326 INFO: Looking for dynamic libraries
45708 INFO: Extra DLL search directories (AddDllDirectory): []
45709 INFO: Extra DLL search directories (PATH): []
46487 INFO: Warnings written to C:\PythonProjects\executable_001\build\exec_001\warn-exec_001.txt
46534 INFO: Graph cross-reference written to C:\PythonProjects\executable_001\build\exec_001\xref-exec_001.html
46696 INFO: checking PYZ
46797 INFO: checking PKG
46962 INFO: Building because C:\PythonProjects\executable_001\exec_001.py changed
46962 INFO: Building PKG (CArchive) exec_001.pkg
52220 INFO: Building PKG (CArchive) exec_001.pkg completed successfully.
52222 INFO: Bootloader C:\Python312\Lib\site-packages\PyInstaller\bootloader\Windows-64bit-intel\run.exe
52222 INFO: checking EXE
52225 INFO: Building EXE because EXE-00.toc is non existent
52225 INFO: Building EXE from EXE-00.toc
52251 INFO: Copying bootloader EXE to C:\PythonProjects\executable_001\dist\exec_001.exe
52334 INFO: Copying icon to EXE
52383 INFO: Copying 0 resources to EXE
52383 INFO: Embedding manifest in EXE
52416 INFO: Appending PKG archive to EXE
52457 INFO: Fixing EXE headers
52738 INFO: Building EXE from EXE-00.toc completed successfully.

News : All Books Bundle from Michael Driscoll

For $60 you can buy a lot of all the card decks from Michael Driscoll.

• Get a copy of all my self-published Python eBooks :
• Python 101 - 2nd Edition
• Python 201: Intermediate Python
• ReportLab: PDF Processing with Python
• Jupyter Notebook 101
• Creating GUI Applications with Python
• Pillow: Image Processing with Python
• Automating Excel with Python
• The Python Quiz Book

News : Website for Python users.

Here is a site called clcoding.com for those who use the python language with many simple examples and even a test fairy: python coding challenge day.

Python 3.12.1 : flaskcode - an web based code editor.

This is an web based code editor on python flask framework.

Let's install with pip tool:

pip install flaskcode
Collecting flaskcode
  Downloading flaskcode-0.0.8.tar.gz (14.5 MB)
...
Installing collected packages: flaskcode
Successfully installed flaskcode-0.0.8

The flaskcode can be integrated in to your own Flask app by configuring and registering flaskcode.blueprint with your app:

from flask import Flask
import flaskcode

app = Flask(__name__)
app.config.from_object(flaskcode.default_config)
app.config['FLASKCODE_RESOURCE_BASEPATH'] = '/path/to/resource/folder'
app.register_blueprint(flaskcode.blueprint, url_prefix='/flaskcode')

@app.route('/')
def hello():
    return "Hello World!"

if __name__ == '__main__':
    app.run()

You can run easy this online tool on web browser with this command:

flaskcode
FlaskCode CLI: C:\PythonProjects

 * Serving Flask app 'flaskcode.cli'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5001
Press CTRL+C to quit

Python Qt6 : Test application with unittest and QtTest.

In this simple example, I add a test class with unittest to sleep the application and QtTest to wait for the window to open after pressing the button.

To run the unittest you need to uncomment this row of source code:

#unittest.main()

Let's see the source code:

import sys

from PyQt6.QtWidgets import QApplication, QDialog, QMainWindow, QPushButton
from PyQt6 import QtCore, QtGui, QtWidgets
from PyQt6 import QtTest 

# Unitest area
import unittest
from time import sleep
def sleep_sec(sec):
    sleep(10*sec)
#
#define class for unittest
class Test(unittest.TestCase):
    def test_square(self):
        sleep_sec(5)

class Ui_Dialog(object):
    def setupUi(self, Dialog):
        Dialog.setObjectName("Dialog")
        Dialog.resize(640, 480)
        self.verticalLayout = QtWidgets.QVBoxLayout(Dialog)

    def retranslateUi(self, Dialog):
        _translate = QtCore.QCoreApplication.translate

class Window(QMainWindow):
    """Main window."""
    def __init__(self, parent=None):
        """Initializer."""
        super().__init__(parent)
        # Use a QPushButton for the central widget
        self.centralWidget = QPushButton("Test_Button")

        # Connect the .clicked() signal with the .onTest_BtnClicked() slot
        self.centralWidget.clicked.connect(self.onTest_BtnClicked)
        self.setCentralWidget(self.centralWidget)

    # Create a slot for launching the Test_ dialog
    def onTest_BtnClicked(self):
        """Launch the Test_ dialog."""
        dlg = Test_Dlg(self)
        # This will test with QtTest just for click 
        QtTest.QTest.qWait(2500)

        dlg.exec()

class Test_Dlg(QDialog):
    """Test dialog."""
    def __init__(self, parent=None):
        super().__init__(parent)
        # Create an instance of the GUI
        self.ui = Ui_Dialog()
        # Run the .setupUi() method to show the GUI
        self.ui.setupUi(self)

if __name__ == "__main__":
    # this test all run of application and show :
    # Ran 1 test in 50.001s
    # uncoment this
    #unittest.main()
    
    # Create the application
    app = QApplication(sys.argv)
    # Create and show the application's main window
    win = Window()
    win.show()
    # Run the application's main loop
    sys.exit(app.exec())

Python 3.12.1 : Bandit tool for security issues in Python code.

Bandit is a tool designed to find common security issues in Python code. To do this Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files it generates a report.

Bandit was originally developed within the OpenStack Security Project and later rehomed to PyCQA

Installation is simple with the pip tool:.

pip install bandit
Collecting bandit
  Downloading bandit-1.7.7-py3-none-any.whl.metadata (5.9 kB)
...
Installing collected packages: PyYAML, pygments, pbr, mdurl, stevedore, markdown-it-py, rich, bandit
Successfully installed PyYAML-6.0.1 bandit-1.7.7 markdown-it-py-3.0.0 mdurl-0.1.2 pbr-6.0.0 pygments-2.17.2 
rich-13.7.1 stevedore-5.2.0

You can see all features with this command:

bandit -h

I test on the script from the last tutorial with pypdf python module and the result is great:

bandit test_001.py
[main]  INFO    profile include tests: None
[main]  INFO    profile exclude tests: None
[main]  INFO    cli include tests: None
[main]  INFO    cli exclude tests: None
[main]  INFO    running on Python 3.12.1
Run started:2024-03-05 19:53:56.858212

Test results:
        No issues identified.

Code scanned:
        Total lines of code: 24
        Total lines skipped (#nosec): 0

Run metrics:
        Total issues (by severity):
                Undefined: 0
                Low: 0
                Medium: 0
                High: 0
        Total issues (by confidence):
                Undefined: 0
                Low: 0
                Medium: 0
                High: 0
Files skipped (0):

Python 3.12.1 : Testing pypdf - version 4.1.0 .

I tried to find a solution for processing PDF files.

The newer Python package called "unstructured" I tested was a disaster and a waste of time and resources.

Today I will show you tests with the Python package called pypdf with version: 4.1.0.

You can find it on the official page.

Installation is simple with the pip tool and you can also add options offered by crypto.

pip install pypdf[crypto]
Collecting pypdf[crypto]
  Downloading pypdf-4.1.0-py3-none-any.whl.metadata (7.4 kB)
...
Installing collected packages: pypdf
Successfully installed pypdf-4.1.0

Here is some information displayed with the show option.

python -m pip show pypdf
Name: pypdf
Version: 4.1.0
Summary: A pure-python PDF library capable of splitting, merging, cropping, and transforming PDF files
Home-page:
Author:
Author-email: Mathieu Fenniak <biziqe@mathieu.fenniak.net>
License:
Location: C:\Python312\Lib\site-packages
Requires:
Required-by:

I create a little script for testing:

import os
from pypdf import PdfReader  
from pypdf import PdfWriter
#PdfMerger is deprecated and will be removed in pypdf 5.0.0. Use PdfWriter instead.
#from pypdf import PdfMerger

pdf_file = PdfReader("invoice-001.pdf")
print("Size in pages : ",len(pdf_file.pages))
print("========")
page = pdf_file .pages[0]
print("Page : ", page)
print("========")
text = page.extract_text()
print("Page text : ", text)
print("========")
print("PDF Metadata : ", pdf_file.metadata)
print("PDF Metadata - Title: ", pdf_file.metadata.title)
print("========")
pdf_writer = PdfWriter("invoice-002.pdf")
page = pdf_writer.add_blank_page(width=8.27 * 72, height=11.7 * 72)
pdf_writer.write("invoice-002.pdf")

from pypdf import PdfWriter

merger = PdfWriter()

for pdf in ["invoice-001.pdf", "invoice-002.pdf"]:
    merger.append(pdf)

merger.write("invoice-003.pdf")
merger.close()

The result is this:

python test_001.py
Size in pages :  1
========
Page :  {'/Type': '/Page', '/Resources': {'/ProcSet': ['/PDF', '/Text', '/ImageB', '/ImageC', '/ImageI'], '/ExtGState': 
{'/G3': IndirectObject(3, 0, 2484091272080)}, '/XObject': {'/X4': IndirectObject(4, 0, 2484091272080)}, '/Font': {'/F7': 
IndirectObject(7, 0, 2484091272080), '/F8': IndirectObject(8, 0, 2484091272080)}}, '/MediaBox': [0, 0, 612, 792], 
'/Contents': IndirectObject(9, 0, 2484091272080), '/StructParents': 0, '/Parent': IndirectObject(10, 0, 2484091272080)}
========
Page text :  Dino Store
227 Cobblestone Road
30000 Bedrock, Cobblestone County
+555 7 789-1234
https://dinostore.bed | hello@dinostore.bedPayment details:
ACC:123006705
IBAN:US100000060345
SWIFT:BOA447
Bill to:
Slate Rock and Gravel Company
222 Rocky Way
30000 Bedrock, Cobblestone County
+555 7 123-5555
fred@slaterockgravel.bedInvoice No. 1
Invoice Date: 03.03.2024
Issue Date: 03.03.2024
Due Date: 02.04.2024
INVOICE
Item Quantity Price Discount Tax Linetotal
1 Test 001 1 50,00 € 1% 19% 49,50 €
2 Test 002 2 40,00 € 2% 19% 78,40 €
3 Frozen Brontosaurus Ribs 1 100,00 € 0% 19% 100,00 €
Subtotal: 227,90 €
Tax 19%: 43,30 €
Total: 271,20 €
Terms & Notes
Fred, thank you very much. We really appreciate your business.
Please send payments before the due date.
========
PDF Metadata :  {'/Creator': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0', '/Producer': 'Skia/PDF m122', '/CreationDate': 
"D:20240304221509+00'00'", '/ModDate': "D:20240304221509+00'00'"}
PDF Metadata - Title:  None
========

The run of the script will create a second blend PDF named invoice-002 then will merge with the first one will result a PDF named : invoice-003.pdf .