Getting Started with MySQL

SQL is a programming language used to communicate with a relational database (think of a programmatic way to interact with data in a spreadsheet-like form). Multiple database packages use SQL and the most commonly used application in the open source community is MySQL. To install on Ubuntu:


sudo apt-get install mysql-server

As a part of the installation process you will be required to set the root password for MySQL. Then you can start it form the command line:


mysql -u root -p

To start MySQL from the command line one must specify the user (so far we only have root) and specify -p to prompt for the password:


ilya@lin1:~$ mysql -u root -p

From the MySQL prompt we can create a database and a new user and then grant access this user access to the newly created database.


mysql> create database test1;
mysql> create user 'user1'@'localhost' identified by 'password';
mysql> grant all on test1.* to 'user1';

To access the database as a newly created user we need to log out and log in:


mysql> mysql -u use1 -p

Now we can create a table in the database:


mysql> use test1;
mysql> mysql> CREATE TABLE example (
         id INT,
         data VARCHAR(100)
       );

Linux Access from Mac OS

Both Ubuntu and Mac OS X come with pre-installed VNC solutions. The vino package in Ubuntu is a VNC server for GNOME. To enable screen sharing one can start Desktop Sharing Preferences and enable an option to allow other users to view desktop.

Screenshot from 2016-02-13 20-37-18

If you are using a Mac on the same network your Linux system should appear in the Shared tab in the Finder.

Screen Shot 2016-02-13 at 9.05.10 PM

Clicking on Share Screen will invoke Mac’s VNC utility and prompt for the password.

Screen Shot 2016-02-13 at 9.03.24 PM

If the connection fails

Screen Shot 2016-02-13 at 8.18.38 PM

you might need to disable encryption on the server (Ubuntu) side by running the following command.

ilya@lin1:~$ gsettings set org.gnome.Vino require-encryption false

There other ways to invoke Share Screen in Mac OS.  You can do it directly from the terminal with

$ open /System/Library/CoreServices/Applications/Screen\ Sharing.app

or you can open an XTerm and start a VNC session from it specifying the IP address and port of the target system.

$ open vnc://10.0.1.35:5900

If the connection is successful and you can view and control your Ubuntu desktop on the Mac you can add

/usr/lib/vino/vino-server

command to you Startup Applications.

Screenshot from 2016-02-13 20-35-05

Of course for pure command line access one can just ssh into the Linux system.   To allow ssh access on the default port 22:

$ sudo apt-get update
$ sudo apt-get install openssh-server
$ sudo ufw allow 22

To make the setup more secure one can follow recommendations of thepcspy.com to install fail2ban:

$ sudo apt-get install fail2ban

and change the port setting from the default to some value in the 10,000-64,000 range.   To do that change the line “Port 22” in /etc/ssh/sshd_config to the chosen value and run restart the server.

$ sudo /etc/init.d/ssh restart

Once that is done, you can connect to the right port with

$ ssh -p 12345 yourname@10.0.1.35

 

Building Numpy with MKL

Numpy and SciPy are very powerful Python extensions for numerical and scientific computing. One can easily install their generic versions via standard Ubuntu commands:

ilya@lin1:/tmp$ sudo pip install numpy 

The problem with generic versions is that they are not optimized for a particular platform and don’t use platform-specific libraries that you might have installed. If you do have optimized mathematical libraries, in particular Intel’s MKL, using them with Numpy/SciPy can make a huge difference in performance.

To illustrate it we can compare generic Numpy with the version that uses MKL using a very simple benchmark. The following program performs a multiplication of two dense matrices followed by an eigenvalue decomposition.

ilya@lin1:/tmp$ cat numpy_blas_time.py 
import time
import numpy as np

A = np.random.rand(2000,2000)
B = np.random.rand(2000,2000)

print('Matrix multiplication')
time1 = time.time()
clock1 = time.clock()
C = np.dot(A,B) 
clock2 = time.clock()
time2 = time.time()
print('  Elapsed time: %.02f sec.' % (time2-time1) )
print('  CPU time: %.02f sec.' % (clock2-clock1) )

print('Eigenvalue computation')
time1 = time.time()
clock1 = time.clock()
np.linalg.eig(A)
clock2 = time.clock()
time2 = time.time()
print('  Elapsed time: %.02f sec.' % (time2-time1) )
print('  CPU time: %.02f sec.' % (clock2-clock1) )

After installing Numpy I am getting the following run times for these two operations on a two-core Skylake system (Intel i5-6260U CPU @ 1.80GHz):

ilya@lin1:/tmp$ python numpy_blas_time.py
Matrix multiplication
  Elapsed time: 17.62 sec.
  CPU time: 17.62 sec.
Eigenvalue computation
  Elapsed time: 82.66 sec.
  CPU time: 82.67 sec.

Here the elapsed time is just the wall clock time and CPU time is the time the CPU cores are busy. CPU time can be lower than the elapsed time if cores go idle or it can exceed the elapsed time on a parallel system if the program uses multiple cores concurrently. Here the two measurements match meaning that Numpy keeps a core busy throughout the computation but executes it serially.

Now we can uninstall Numpy

ilya@lin1:/tmp$ sudo pip uninstall numpy 

then download the source version from http://sourceforge.net/projects/numpy/files/NumPy/ and build it locally with MKL. We untar the Numpy distribution file and create the site.cfg file using site.cfg.example as a template and pointing to the location of MKL libraries (following Intel Developer Zone suggestions):

ilya@lin1:~/Tools/numpy/numpy-1.10.4$ diff site.cfg site.cfg.example 
157,161d156
< [mkl]
< include_dirs = /opt/intel/compilers_and_libraries/linux/mkl/include
< library_dirs = /opt/intel/compilers_and_libraries/linux/mkl/lib/intel64
< mkl_libs = mkl_rt
< lapack_libs = 

Now it’s time to compile and install the optimized Numpy.

ilya@lin1:~/Tools/numpy/numpy-1.10.4$ export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries/linux/mkl/lib/intel64:${LD_LIBRARY_PATH}
ilya@lin1:~/Tools/numpy/numpy-1.10.4$ python setup.py config --compiler=intel build_clib --compiler=intel build_ext --compiler=intel build
ilya@lin1:~/Tools/numpy/numpy-1.10.4$ sudo python setup.py config --compiler=intel build_clib --compiler=intel build_ext --compiler=intel install

After the installation is complete we can rerun the same test program to see the effect of MKL.

ilya@lin1:/tmp$ python numpy_blas_time.py
Matrix multiplication
  Elapsed time: 1.86 sec.
  CPU time: 3.66 sec.
Eigenvalue computation
  Elapsed time: 11.46 sec.
  CPU time: 21.06 sec.

We can see almost a tenfold improvement in performance for both operations. We can also notice that the CPU time is almost twice the elapsed time which indicates that the computations are running in parallel now using both cores of the system.

Similarly we can download and build SciPy (http://sourceforge.net/projects/scipy/files/).

SciPy inherits library configuration from NumPy so we don’t need any additional steps to point it to MKL

ilya@lin1:~/Tools/scipy$ unzip scipy-0.16.1.zip 
ilya@lin1:~/Tools/scipy$ cd scipy-0.16.1/
ilya@lin1:~/Tools/scipy/scipy-0.16.1$ python setup.py config --compiler=intelem --fcompiler=intelem build_clib --compiler=intelem --fcompiler=intelem build_ext --compiler=intelem --fcompiler=intelem build 
ilya@lin1:~/Tools/scipy/scipy-0.16.1$ sudo python setup.py config --compiler=intelem --fcompiler=intelem build_clib --compiler=intelem --fcompiler=intelem build_ext --compiler=intelem --fcompiler=intelem install

Once we are at it, we can install scikit-learn as well

$ sudo pip uninstall -U scikit-learn

and a few other packages

$ sudo apt-get install libfreetype6-dev
$ sudo pip install matplotlib
$ sudo pip install nltk

Note that if you have multiple Python installations (e.g. 2.7 and 3.5), you need to build and install optimized Numpy, SciPy and other packages for all of them.

Installing Intel SDE on Ubuntu

Intel Software Development Emulator (SDE) is a very handy tool for dynamic program analysis.  The binary version of the tool is available at https://software.intel.com/en-us/articles/intel-software-development-emulator

Once you untar the distribution you’ll find the sde executable:

ilya@lin1:~/Tools/SDE$ tar -xvf sde-external-7.39.0-2016-01-18-lin.tar.bz2 
ilya@lin1:~/Tools/SDE$ tar -xf sde-external-7.39.0-2016-01-18-lin.tar.bz2
ilya@lin1:~/Tools/SDE$ ln -s sde-external-7.39.0-2016-01-18-lin sde
ilya@lin1:~/Tools/SDE$ cd sde/
ilya@lin1:~/Tools/SDE/sde$ ls -l sde
-rwxr-xr-x 1 ilya ilya 52272 Jan 17 18:47 sde

But when you try to run it you might get a somewhat puzzling error message:

ilya@lin1:~/Tools/SDE/sde$ ./sde
bash: ./sde: No such file or directory

Of course there is the executable is there, the problem is that it’s a 32-binary

ilya@lin1:~/Tools/SDE/sde$ file sde
sde: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.4, not stripped

and we are trying to run in a 64-bit environment:

ilya@lin1:~/Tools/SDE/sde$ uname -a
Linux lin1 4.2.0-27-generic #32-Ubuntu SMP Fri Jan 22 04:49:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

To fix this problem we need to install 32-bit libraries:

ilya@lin1:~/Tools/SDE/sde$ sudo apt-get install lib32gcc1 libc6-i386 lib32z1 lib32stdc++6

Now we can make a simple SDE test run

ilya@lin1:~/Tools/SDE/sde$ ./sde -mix -- ls
ia32	 LICENSE.txt  README.txt  sde64		   src	xed64
intel64  misc	      sde	  sde-mix-out.txt  xed

This command generates file sde-mix-out.txt with a histogram of basic blocks executed by ls.

Ubuntu: Enabling Hibernation

Out of the box Ubuntu shows the “Suspend” option in the system menu (rightmost option on the menu bar) but it doesn’t offer an option to hibernate the system.

Screenshot from 2016-02-04 19-02-32

The difference is that a suspended system enters a low power state but still needs energy to keep the content of DRAM. If you unplug a suspended system the unsaved state will be lost and it will boot afresh when it’s turned back on.

In hibernation the system saves its memory content to disk and can be powered down completely without losing any application state. When it’s powered back on application windows open and they resume their operation.

Before enabling the hibernation option in the menu we can test if this functionality works. The following command will put the system in a hibernated state and turn it off.

ilya@lin1:~$ sudo pm-hibernate

When powered on the system should keep all the windows/applications open.

If this test was successful we can enable “Hibernate” option in the system menu by creating the following file.

root@lin1:~# cat /var/lib/polkit-1/localauthority/50-local.d/com.ubuntu.enable-hibernate.pkla
[Re-enable hibernate by default in upower]
Identity=unix-user:*
Action=org.freedesktop.upower.hibernate
ResultActive=yes

[Re-enable hibernate by default in logind]
Identity=unix-user:*
Action=org.freedesktop.login1.hibernate
ResultActive=yes

After rebooting the new option should be visible in the menu.

Screenshot from 2016-02-04 19-08-51

Source: How to Enable Hibernation in Ubuntu 14.04

Installing Developer Tools on Ubuntu

GNU Compilers

By default Ubuntu installs some development tools but some key components might be missing.  For example I didn’t see g++, GNU’s C++ compiler.  Luckily it’s easy to fix by installing it directly:

ilya@ilya-ubuntu:~$ sudo apt-get install g++

If you plan to use the system for development it makes sense to just the build bundle:

ilya@ilya-ubuntu:~$ sudo apt-get install build-essential

A good practice is to issue update and upgrade commands prior to installing any packages.

ilya@ilya-ubuntu:~$ sudo apt-get update
ilya@ilya-ubuntu:~$ sudo apt-get upgrade

Intel Compilers

Most Intel developments tools are commercial products and you need a valid license to use them.  If you have it, the tools are very easy to install.  Htere’s the installation process for the Intel Parallel Studio XE 2016.  Untar the distribution file and run the installation script:

ilya@ilya-ubuntu:~$ tar -xvf parallel_studio_xe_2016_update1.tgz
ilya@ilya-ubuntu:~$ cd parallel_studio_xe_2016_update1/ 
ilya@ilya-ubuntu:~$ ./install.sh

Before using Intel compilers one needs to set environment variables:

ilya@ilya-ubuntu:~$ source /opt/intel/bin/compilervars.sh intel64

A corresponding record in the shell rc file will ensure that the variables are set every time the user logs in .

ilya@ilya-ubuntu:~$ diff ~/.bashrc ~/.bashrc.orig
115d114
< source /opt/intel/bin/compilervars.sh intel64

Now we can test the compiler with a simple hello world example:

ilya@ilya-ubuntu:/tmp$ cat hello.c
#include<stdio.h>
int main(){
printf("hello world\n");
return(0);}
ilya@ilya-ubuntu:/tmp$ icc hello.c -o hello_icc
ilya@ilya-ubuntu:/tmp$ ./hello_icc
hello world
ilya@ilya-ubuntu:/tmp$ gcc hello.c -o hello_gcc
ilya@ilya-ubuntu:/tmp$ ./hello_gcc
hello world

Other Tools

There are many other development tools and packages that might get hand.  Here’s a small sample:

ilya@ilya-ubuntu:~$ sudo apt-get install cmake git libgtk2.0-dev pkg-config 
ilya@ilya-ubuntu:~$ sudo apt-get install python-dev python-numpy
ilya@ilya-ubuntu:~$ sudo apt-get install libtbb2 libtbb-dev

 

Ubuntu Customization

Appearance

The default look of Ubuntu is too dark for my taste, but luckily the UI is easily customizable.  I must be not the only one who finds Mac OS X colors more pleasing because there is a package Macbuntu that transforms Ubuntu desktop to look and feel like Mac OS.   To me it feels like a step too far: I want to feel that I am on Linux, but I want my color palette and keyboard shortcuts to be familiar.

A simple adjustment makes the look more uplifting.  In System Settings under Appearance change the theme from the dark default Ambiance to Radiance.  Also, while you are at it, you can change the desktop wallpaper to something brighter.

Screenshot from 2016-01-24 16:56:56

General Keyboard Shortcuts

Some shortcuts can be found in the Keyboard tab of System Settings, but for real customization we need to issue terminal commands or use ComizConfig Settings Manager.

sudo apt-get install compiz compizconfig-settings-manager

After invoking CompizConfig Settings Manager select Ubuntu Unity Plugin under the Desktop category.  To make the <command>-space combination bring the search form similar to that in Mac OS, select the Launcher tab and change Key to start the Launcher Application Switcher to <Super>space. <Super> is Ubuntu speak for what Apple calls <command> key also known as Windows key on PC keyboards.

To make the application switcher respond to <command><tab> and <shift><command><tab> combinations familiar to Mac users, select the Switcher tab change Key to start the Switcher and Key to switch to the previous window in the Switcher to <Super>Tab and <Shift><Super>Tab respectively.

Screenshot from 2016-01-25 23:08:47

To map the window picker to F3, Mac OS-style, navigate from the top level to
Advanced Search and then select Scale and Bindings, then set Initiate Window Picker to F3 although you’ll see it as XF86LaunchA.

Terminal Appearance and Shortcuts

Terminal is the application that I likely use the most so I prefer to adjust its look and feel to the settings I am used to on Mac OS.  To change the font and background colors, launch the terminal and navigate to the Profile Preferences menu under Edit and change the settings to black-on-white.   Then in Keyboard Shortcuts menu of Edit change shortcuts for cut, paste, and new window to familiar settings.

Screenshot from 2016-01-25 23:12:58

Finally, for those who use terminal often, it makes sense to pin it to the launcher by right clicking on its icon and selecting the corresponding option.

Screenshot from 2016-01-25 23:17:36

Mouse Scrolling Direction

To reverse the mouse scroll direction (disable natural scrolling) put the following line

pointer = 1 2 3 5 4 6 7 8 9 10 11 12

in the ~/.Xmodmap file and re-login.

 

Installing Ubuntu

Prepare Bootable Media

Installing Ubuntu on a new computer is straightforward. First, prepare a bootable USB stick or SD card. On another machine running Ubuntu download the latest OS image and save it to a local file. Then run Startup Disk Creator. Click on the Other button on the opening screen to navigate to the image.

Screenshot from 2016-01-24 18_22_03

Select target USB device and click Make Startup Disk.

Installation

I suggest booting Ubuntu from a USB device prior to installation to make sure that all system components are visible to the operating system.  In particular, make sure that storage devices can be used.  Run Disk application to check if the target device is visible.

Screenshot from 2016-01-24 22_38_27

Run Install Ubuntu application.  By default it will create three partitions on target disk drive: EFIboot, Linux filesystem, and Swap.

Screenshot from 2016-01-24 22:55:55

Default breakdown will work fine.  On my system with 16GB of RAM and 240GB SSD the Linux partition takes 222GB, Swap takes 17GB and EFIboot is less than a gigabyte.

Screenshot from 2016-01-24 15:04:42

Once installation completes, reboot the system and it is ready to go.

The user account created during installation has superuser privileges but it’s also handy to have access to the root account. To set the password for root just run passwd under sudo.

ilya@ilya-ubuntu:~$ sudo passwd

If you want to keep the PATH environment settings from your user account when you use sudo you can edit the settings in /etc/sudoers:

root@lin1:~# diff /etc/sudoers /etc/sudoers.orig
9,11c9,11
< #Defaults	env_reset
< #Defaults	mail_badpass
< #Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" --- > Defaults	env_reset
> Defaults	mail_badpass
> Defaults	secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

To run sudo commands with the same LD_LIBRARY_PATH as the user account one can explicitly pass it as a parameter:

$ sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH pip install -U scikit-learn

or include an alias in the the ~/.bashrc file:

alias sudo='sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH'

In either case one should ensure that PATH and LD_LIBRARY_PATH don’t point to directories with malicious code as it wreak havoc on the system if executed with superuser privileges.