Python: Fastest way to check key in dictionary

The problem

When you want access value by it’s key in Python dictionary instance, your program will break on KeyError if the accessed key is not contained in a dictionary.

So it’s necessary to check the presence of a key before access it.

Here is the most obvious way to check it:

def k_in_d(d, k):
  return k in d.keys()

Is this way most efficient?

It might be done better:

def k_in_d(d, k):
  return k in d

But the following one is faster:

def k_in_d(d, k):
  try:
    v = d[k]
  except KeyError:
    return False
  return True

Performance checking

By the following code it is possible to check performance and make conclusion:

import datetime as dt
import random

# *********************************************************
# dict_size = 100
# num_iterations = 1000000
dict_size = 1000000
num_iterations = 100

d = {i: i for i in range(dict_size)}

# ---------------------------------------------------------
def k_in_d_object():
    k = random.randint(0,dict_size-1)
    return k in d
# ---------------------------------------------------------
def k_in_d_keys():
    k = random.randint(0,dict_size-1)
    return k in d.keys()
# ---------------------------------------------------------
def k_in_d_except():
    k = random.randint(0,dict_size-1)
    try:
    	v = d[k]
    except KeyError:
    	return False
    return True
# ---------------------------------------------------------
def test(func):
    t = dt.datetime.utcnow()
    for i in range(num_iterations):
        func()
    print ("%s --> %1.6f sec" % 
           (func, (dt.datetime.utcnow()-t).total_seconds()))

# *********************************************************
test(k_in_d_keys)
test(k_in_d_object)
test(k_in_d_except)

Statistics on different versions of Python

The code has been tested on computer from 2015 with i5 CPU.

Python 2.7.15

dict_size = 1000000 num_iterations = 100

<function k_in_d_keys at 0x10d7f50c8> --> 2.251373 sec
<function k_in_d_object at 0x10d7f5050> --> 0.000535 sec
<function k_in_d_except at 0x10d7f5140> --> 0.000242 sec

dict_size = 100 num_iterations = 1000000

<function k_in_d_keys at 0x1014fe0c8> --> 2.959747 sec
<function k_in_d_object at 0x1014fe050> --> 2.479454 sec
<function k_in_d_except at 0x1014fe140> --> 1.402201 sec

Python 3.7.2

dict_size = 1000000 num_iterations = 100

<function k_in_d_keys at 0x10ec81598> --> 0.000792 sec
<function k_in_d_object at 0x10ec81620> --> 0.001852 sec
<function k_in_d_except at 0x10ec81510> --> 0.000638 sec

dict_size = 100 num_iterations = 1000000

<function k_in_d_keys at 0x105209598> --> 1.761158 sec
<function k_in_d_object at 0x105209620> --> 2.526102 sec
<function k_in_d_except at 0x105209510> --> 1.677804 sec

Python 3.7.4

dict_size = 1000000 num_iterations = 100

<function k_in_d_keys at 0x102a94290> --> 0.002660 sec
<function k_in_d_object at 0x102a13e60> --> 0.002838 sec 
<function k_in_d_except at 0x102a94200> --> 0.000991 sec

dict_size = 100 num_iterations = 1000000

<function k_in_d_keys at 0x103cdb290> --> 1.739541 sec 
<function k_in_d_object at 0x103c5ae60> --> 1.649423 sec 
<function k_in_d_except at 0x103cdb200> --> 1.644620 sec

Conclusion

With a time performance of Python 3 become better.

Using exceptions is the fastest way to check key existence in a dictionary.

References

Stack Overflow: Python is ‘key in dict’ different/faster than ‘key in dict.keys()’

Install Debian 9.4.0 on HP Zbook 15 G3

First of all BIOS

Set display setting in BIOS to: Discrete

The regular installation from media

CD-ROM

The Installation from USB Flash Drive

Prepare the bootable USB flash drive:

sudo dd if=~/Download/debian-live-9.4.0-amd64-gnome.iso of=/dev/sdd bs=4M; sync

Got staked on first boot

“Ctrl+Alt+F3” to login as root.

Add stretch-backports to the list of APT Sources

/etc/apt/sources.list file
deb http://debian.org/debian/ stretch main contrib non-free
deb-src http://debian.org/debian/ stretch main contrib non-free

deb http://debian.org/debian/ stretch-backports main contrib non-free
deb-src http://debian.org/debian/ stretch-backports main contrib non-free

deb http://security.debian.org/debian-security stretch/updates main contrib non-free
deb-src http://security.debian.org/debian-security stretch/updates main contrib non-free

# stretch-updates, previously known as 'volatile'
deb http://debian.org/debian/ stretch-updates main contrib non-free
deb-src http://debian.org/debian/ stretch-updates main contrib non-free

Errors with firmware

apt install -t stretch-backports firmware-misc-nonfree
apt install -t stretch-backports firmware-iwlwifi

Install NVIDIA drivers

Thanks to allaboutlinux.eu for they article: link

Disable the nouveau driver for NVIDIA graphical card

apt remove nvidia*
apt autoremove
apt update
apt install dkms build-essential linux-headers-$(uname -r)
echo options nouveau modeset=0 | tee -a /etc/modprobe.d/nouveau-kms.conf
update-initramfs -u

blacklist the nouveau driver

nano /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
blacklist lbm-nouveau
options nouveau modeset=0
alias nouveau off
alias lbm-nouveau off

Install backports kernel

Look for the latest kernel:

apt-cache search -t stretch-backports linux-image-4.
linux-headers-4.17.0-0.bpo.1-amd64 - Header files for Linux 4.17.0-0.bpo.1-amd64
linux-image-4.17.0-0.bpo.1-amd64 - Linux 4.17 for 64-bit PCs
apt install -t stretch-backports linux-image-4.17.0-0.bpo.1-amd64
apt install -t stretch-backports linux-headers-4.17.0-0.bpo.1-amd64

Install the backports drivers for NVIDIA card

Stop the X-Server

/etc/init.d/gdm3 stop

Install the driver

apt install -t stretch-backports nvidia-driver
reboot

 

Linux file operations

Useful file operation commands

Linux Search Files

Find all files modified at date:
find ./ -type f —name “*.php” newermt “2017-10-14”
Find all files modified at date and time:
find ./ -type f —name “*.php” -newermt “2017-10-14 10:21:00”
Find all modified files between dates:
find ./ -type f —name “*.php” -newermt “2017-10-14 10:21:00” \! -newermt “2017-10-14 10:40:00”
Delete all files modified between dates:
find ./ -type f -newermt “2017-10-14 10:21:00” \! -newermt “2017-10-14 10:40:00” -exec rm {} +
Backup root directory excluding home, tmp, etc:
sudo tar --exclude='/proc' --exclude='/home' --exclude='/tmp' --exclude='/var/web-cache' --exclude='/boot' -zcvf yyyy-mm-dd_root.tgz /
Archive by 7z excluding directories
sudo 7z a -r 2017_11_14-root.7z /* '-xr!/boot' '-xr!/proc' '-xr!/tmp' '-xr!/var/web-cache' '-xr!/home' -spf

rsync – a fast, versatile, remote (and local) file-copying tool

Copy files by rsync over SSH from server to local machine:

rsync -avz -e “ssh -o StrictHostKeyChecking=no -o UserKnownHostFile=/dev/null” —progress [email protected]:/var/cache/apt/archives/*.deb /home/username/Downloads

Debian 9 (stretch) update kernel & Nvidia drivers

Updating kernel of Debian 9 (stretch) and Nvidia drivers to solve freeze of NVIDIA video adapter

Why to update the kernel?

After several years of using Debian 8 (jessie) I decided to upgrade the system to Debian 9 (stretch).

The distribution upgrade went well as described in many of good articles, and ended with useless system.

So it was necessary to perform a clean installation of new system. Since I have a separate partition dedicated to /home directory it was easy to install and almost all settings for different software didn’t changed.

After moving the cache of browsers to tmpfs, as you can read in that article LINK, and playing with Tweak Tool for pleasant interface, I got a working system.

But, here is reason of this article, the system become freeze each time I used the nice Vivaldi browser with many of open web pages each in it’s own tab. Suspicions was on drivers for video adapter which is “NVIDIA Corporation GK107GL [Quadro K2000] (rev a1)”.

Investigation of /var/log/kern.log showed a problem with that driver:

Thoughts about installation of native NVIDIA drivers came to my mind and here is a nice article about it: “How to install Steam and Nvidia drivers on Debian Jessie” thanks to Eike Sauer.

But updating the kernel to new version fixed the problem.

Below are instructions for kernel update.

Update the list of sources

Add to apt sources the support for non free and backports packages as following:

You can use the http://ftp.debian.org/debian/ in the /etc/apt/sources.list file:

#
# deb cdrom:[Debian GNU/Linux 9.2.1 _Stretch_ - Official amd64 NETINST 20171013-13:07]/ stretch main
# deb cdrom:[Debian GNU/Linux 9.2.1 _Stretch_ - Official amd64 NETINST 20171013-13:07]/ stretch main

deb http://debian.co.il/debian/ stretch main contrib non-free
deb-src http://debian.co.il/debian/ stretch main contrib non-free

deb http://security.debian.org/debian-security stretch/updates main
deb-src http://security.debian.org/debian-security stretch/updates main

# stretch-updates, previously known as 'volatile'
deb http://debian.co.il/debian/ stretch-updates main contrib non-free
deb-src http://debian.co.il/debian/ stretch-updates main contrib non-free

deb http://debian.co.il/debian/ stretch-backports main contrib non-free

The updating of apt repositories is necessary:

sudo apt update
sudo apt upgrade

Look for the desired kernel image

sudo apt-cache search linux-image

In my case the interesting packages was:

linux-headers-4.13.0-0.bpo.1-amd64 - Header files for Linux 4.13.0-0.bpo.1-amd64
linux-image-4.13.0-0.bpo.1-amd64 - Linux 4.13 for 64-bit PCs

Installation of two kernel related packages:

sudo apt install linux-image-4.13.0-0.bpo.1-amd64 linux-headers-4.13.0-0.bpo.1-amd64
reboot

uname -a
Linux lab 4.13.0-0.bpo.1-amd64 #1 SMP Debian 4.13.4-2~bpo9+1 (2017-10-17) x86_64 GNU/Linux

The system is stable and didn’t suffered from crashes at least till time of writing this article.

Update (15-11-2017):

System freeze, not so often like with previous kernel, but freeze.

Updated the Nvidia driver not to default proprietary 340xx version but to one from the backports 375.82 (Jul 2017).

Since the stretch-backports repository is configured, what is necessary, to install the backport driver such as:

sudo apt-get -t stretch-backports install nvidia-driver
reboot

No freezes!

tmpfs as cache location for internet browsers

Linux administration

How set the tmpfs under Linux Debian

fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>

tmpfs /tmp tmpfs defaults,noatime,mode=1777,size=20G 0 0
#
tmpfs /var/web-cache/vivaldi/username tmpfs defaults,noatime,uid=username,gid=usergroup,nosuid,size=500M 0 0
tmpfs /var/web-cache/firefox/username tmpfs defaults,noatime,uid=username,gid=usergroup,nosuid,size=500M 0 0
tmpfs /var/web-cache/opera/username tmpfs defaults,noatime,uid=username,gid=usergroup,nosuid,size=500M 0 0

Manage cache directories

Under root permissions create following directories:

# sudo mkdir -p /var/web-cache/vivaldi/username
# sudo mkdir -p /var/web-cache/firefox/username
# sudo mkdir -p /var/web-cache/opera/username

And change user’s permissions:

# sudo chown username:usergroup /var/web-cache/vivaldi/username
# sudo chown username:usergroup /var/web-cache/firefox/username
# sudo chown username:usergroup /var/web-cache/opera/username

Setting cache location in browsers

After the preparation on the OS side it is necessary to prepare the browser side, the location of the cache directory should be changed in the settings of a browser.

Firefox

In the Firefox browser it is very easy to change the location of cache directory inside the configuration settings, just enter the following URL in the address bar of a browser:

about:config

You will see a page like that:

Just accept and go further to configuration settings where you will find the records related to browser’s cache, two of those records are interesting:

browser.cache.disk.capacity

browser.cache.disk.parent_directory

You can set those records to your own values according to directories in /var that has been prepared earlier.

The sample of such configuration is below:

Chromium based browsers Vivaldi, Opera, Chrome etc.

Command line argument passed to the executable of the browser:

--disk-cache-dir="/var/web-cache/vivaldi/username" 
--disk-cache-size=104857600

Cache directory

/home/username/.cache/vivaldi
/home/username/.cache/vivaldi-snapshot
Change location:
# cd ~/.cache/
# rm -fr vivaldi-snapshot/
# ln -s /var/web-cache/vivaldi/username/ vivaldi-snapshot
Sample of output from ls -ls command after links are applied

Configuration directory

/home/username/.config/vivaldi
/home/username/.config/vivaldi-snapshot

Additional information

How to speed up the Vivaldi web browser