python-catalin

Tuesday, October 1, 2019

Python 3.7.4 : Install the protobuf from sources on Fedora distro.

Today I will show you how to build protobuf from sources using the Fedora distro.
The google team comes with this intro:
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler...
This google project comes with these tutorials.
The GitHub project can be found here.
To install the compiler, download the package.

[mythcat@desk ~]$ cd Downloads/
[mythcat@desk Downloads]$ cp protobuf-python-3.9.2.tar.gz ~/
[mythcat@desk Downloads]$ cd ..
[mythcat@desk ~]$ tar xvzf protobuf-python-3.9.2.tar.gz 
...
protobuf-3.9.2/aclocal.m4
protobuf-3.9.2/install-sh
protobuf-3.9.2/generate_descriptor_proto.sh
protobuf-3.9.2/CHANGES.txt
protobuf-3.9.2/configure.ac
protobuf-3.9.2/configure

Let's see the content:

[mythcat@desk ~]$ cd protobuf-3.9.2/
[mythcat@desk protobuf-3.9.2]$ ls
aclocal.m4                   config.sub                    ltmain.sh            README.md
ar-lib                       configure                     m4                   six.BUILD
autogen.sh                   configure.ac                  Makefile.am          src
benchmarks                   conformance                   Makefile.in          test-driver
BUILD                        CONTRIBUTORS.txt              missing              third_party
CHANGES.txt                  depcomp                       objectivec           update_file_lists.sh
cmake                        editors                       protobuf.bzl         util
compile                      examples                      protobuf_deps.bzl    WORKSPACE
compiler_config_setting.bzl  generate_descriptor_proto.sh  protobuf-lite.pc.in
config.guess                 install-sh                    protobuf.pc.in
config.h.in                  LICENSE                       python

[mythcat@desk protobuf-3.9.2]$ ./configure
...
checking how to run the C preprocessor... gcc -E
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/mythcat/protobuf-3.9.2':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details
...
[root@desk protobuf-3.9.2]# dnf install g++
...
Installed:
  gcc-c++-9.2.1-1.fc30.x86_64                                                                               

Complete![root@desk protobuf-3.9.2]# exit
exit

Let's build again:

[mythcat@desk protobuf-3.9.2]$ ./configure
checking whether to enable maintainer-specific portions of Makefiles... yes
checking build system type... x86_64-pc-linux-gnu
...
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/gmock-config
config.status: creating build-aux/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
[mythcat@desk protobuf-3.9.2]$ nproc --all
2
[mythcat@desk protobuf-3.9.2]$ make -j2
...
  CXXLD    protoc
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2'
[mythcat@desk protobuf-3.9.2]$ make check -j2
...PASS: protobuf-lazy-descriptor-test
PASS: protobuf-lite-test
PASS: google/protobuf/compiler/zip_output_unittest.sh
PASS: google/protobuf/io/gzip_stream_unittest.sh
PASS: protobuf-lite-arena-test
PASS: no-warning-test
PASS: protobuf-test
============================================================================
Testsuite summary for Protocol Buffers 3.9.2
============================================================================
# TOTAL: 7
# PASS:  7
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'

The last steps is for install:

[mythcat@desk protobuf-3.9.2]$ sudo make install
[sudo] password for mythcat: 
...
make[2]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
make[1]: Leaving directory '/home/mythcat/protobuf-3.9.2/src'
[mythcat@desk protobuf-3.9.2]$ sudo ldconfig

Let's test it:

[mythcat@desk protobuf-3.9.2]$ protoc --version
libprotoc 3.9.2

Now the next step comes for python module:

[mythcat@desk protobuf-3.9.2]$ cd python/
[mythcat@desk python]$ ls
google  MANIFEST.in  mox.py  README.md  release  release.sh  setup.cfg  setup.py  stubout.py  tox.ini
[mythcat@desk python]$ python setup.py build
...
testSerialize (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeMessageSetWireFormatUnknownExtension 
(google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testSerializeProto3 (google.protobuf.internal.unknown_fields_test.UnknownFieldsTest) ... ok
testByteSizeFunctions (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testPackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testUnpackTag (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagDecode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok
testZigZagEncode (google.protobuf.internal.wire_format_test.WireFormatTest) ... ok

----------------------------------------------------------------------
Ran 818 tests in 5.343s

OK (skipped=10)

Let's test the python module:

[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import google
>>> from google.protobuf import descriptor as _descriptor
>>> from google.protobuf import message as _message
>>> from google.protobuf import reflection as _reflection
>>> from google.protobuf import symbol_database as _symbol_database
>>> from google.protobuf import descriptor_pb2
>>> from google.protobuf import text_format
>>> dir(_descriptor)
['Descriptor', 'DescriptorBase', 'DescriptorMetaclass', 'EnumDescriptor', 'EnumValueDescriptor', 
'Error', 'FieldDescriptor', 'FileDescriptor', 'MakeDescriptor', 'MethodDescriptor', 'OneofDescriptor', 
'ServiceDescriptor', 'TypeTransformationError', '_Lock', '_NestedDescriptorBase', '_OptionsOrNone', 
'_ParseOptions', '_ToCamelCase', '_ToJsonName', '_USE_C_DESCRIPTORS', '__author__', '__builtins__', 
'__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_lock', 
'api_implementation', 'six', 'threading']

Now I can to define the structure for the data structured as messages
These message is a small logical record of information containing a series of name-value pairs called fields.
The protobuffers compiler (protoc) to generate the source code in the language you need (from the .proto file).
To generate a Python file, you need to execute:
protoc -I=$SRC_DIR --python_out=$DST_DIR $SRC_DIR/my_example.proto
Let's test with am example (you can use any proto file) in default folder (use . for default folder):

[mythcat@desk python]$ protoc -I=. --python_out=. my_example.proto
[mythcat@desk python]$ ls my_example*
my_example_pb2.py  my_example.proto
[mythcat@desk python]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_example_pb2
>>> dir(my_example_pb2)
['AddressBook', 'DESCRIPTOR', 'Person', '_ADDRESSBOOK', '_PERSON', '_PERSON_PHONENUMBER', '_PERSON_PHONETYPE',
 '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__',
 '_b', '_descriptor', '_message', '_reflection', '_sym_db', '_symbol_database', 'sys']

Saturday, September 28, 2019

The tensorflow python module - part 004.

If you using the tensorflow then you can get some warnings.
You can use warnings python package to manage all of this:

[mythcat@desk ~]$ $ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> import tensorflow as tf
/home/mythcat/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: 
Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be 
understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/mythcat/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: 
Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be 
understood as (type, (1,)) / '(1,)type'.
...

/home/mythcat/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: 
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of 
numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
>>>
>>> warnings.filterwarnings('ignore')
>>>
[7]+  Stopped                 python3

Use it in this way to filter these warnings:

[mythcat@desk ~]$ $ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> warnings.filterwarnings('ignore',category=FutureWarning)
>>> import tensorflow as tf
>>>

Wednesday, September 25, 2019

Python 3.7.4 : Print with random colors.

This is a simple example for custom print output.
The script detect the platform for color settings and then use print.
The first print will print with blue color the name of the script.
I used random to select a random color from colors array and used to print the -=RANDOM COLOR=- text.
The print (W+'') is used to set default white color for terminal

import sys
import random
if sys.platform == "linux" or sys.platform == "linux2":
        BB = "\033[34;1m" # Blue light
        YY = "\033[33;1m" # Yellow light
        GG = "\033[32;1m" # Green light
        WW = "\033[0;1m"  # White light
        RR = "\033[31;1m" # Red light
        CC = "\033[36;1m" # Cyan light
        B = "\033[34m"    # Blue
        Y = "\033[33m"    # Yellow
        G = "\033[32m"    # Green
        W = "\033[0m"     # White
        R = "\033[31m"    # Red
        C = "\033[36m"    # Cyan
colors = [BB,YY,GG,WW,RR,CC,B,Y,G,W,R,C]
print (B+"\033[2;2m "+sys.argv[0]+"\n"+B)

color=random.choice(colors)
print (color+"-=RANDOM COLOR=-"+color)
print (W+'')

For winodws platform you need to add this:

elif sys.platform == "win32":

 BB = '' # Blue light
 YY = '' # Yellow light
 GG = '' # Green light
 WW = '' # White light
 RR = '' # Red light
 CC = '' # Cyan light
 B = ''  # Blue
 Y = ''  # Yellow
 G = ''  # Green
 W = ''  # White
 R = ''  # Red
 C = ''  # Cyan
 P = ''  # Random color

Wednesday, September 11, 2019

Python 3.7.4 : Using the theano pakage.

If you want to test theano then you need to see this webpage.

[root@desk mythcat]# dnf search theano
======================== Name & Summary Matched: theano ========================
python-theano-doc.noarch : Theano documentation
============================= Name Matched: theano =============================
python3-theano.noarch : Mathematical expressions involving multidimensional
                      : arrays
=========================== Summary Matched: theano ============================
python3-lasagne.noarch : Lightweight library to build and train neural networks
                       : in Theano
[root@desk mythcat]# pip3 install Theano --user
WARNING: Running pip install with root privileges is generally not a good idea.

Try `pip3 install --user` instead.
Collecting Theano
...
  Running setup.py install for Theano ... done
Successfully installed Theano-1.0.4 scipy-1.3.1

Let's see first example:

[mythcat@desk ~]$ python3 
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
/home/mythcat/.local/lib/python3.7/site-packages/theano/configdefaults.py:560:

UserWarning: DeprecationWarning: there is no c++ compiler.This is deprecated and with

Theano 0.11 a c++ compiler will be mandatory
  warnings.warn("DeprecationWarning: there is no c++ compiler."
WARNING (theano.configdefaults): g++ not detected ! Theano will be unable to execute

optimized C-implementations (for both CPU and GPU) and will default to Python implementations.

 Performance will be severely degraded. To remove this warning, set Theano flags cxx to an empty

 string.
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
>>> import theano
>>> import theano.tensor as T
>>> x = T.dmatrix('x')
>>> s = 1 / (1 + T.exp(-x))
>>> logistic = theano.function([x], s)
>>> logistic([[0, 1], [-1, -2]])
array([[0.5       , 0.73105858],
       [0.26894142, 0.11920292]])
>>> ...

Monday, September 9, 2019

Python 3.7.4 : Using the sunpy - part 001.

I wrote about sunpy in the past on this website.
Now this package comes with new features, see the official webpage.
Let's install it.

[mythcat@desk ~]$ pip3 install sunpy --user
Collecting sunpy
...
Successfully installed aioftp-0.13.0 aiohttp-3.6.0 astropy-3.2.1 async-timeout-3.0.1 
multidict-4.5.2 parfive-1.0.0 scipy-1.3.1 sunpy-1.0.3 tqdm-4.35.0 yarl-1.3.0

If you search on web you can find example with spectra , but in the new package is not supported.

ModuleNotFoundError: No module named 'sunpy.spectra'