Python standard library includes about 220 modules and 89 built-in functions (Python 3.9). Each of these modules designed for specific purposes and includes several functions (methods) and most of the users might never use many of them (use help('modules')
to see list of the modules). On the other hand, built-in functions are very limited and it is not hard to learn and apply most of them.
In this tutorial we will learn more about Python built-in functions and some useful modules for general Python users.
Sources:
You might also like these related articles:
In general we can categorize built-in functions to:
abs
, divmod
, max
, min
, pow
, round
, sum
all
, any
, isinstance
, issubclass
bool
, bytes
, bytearray
, complex
, dict
, float
, frozenset
, int
, list
, set
, str
, type
, tuple
exec
, eval
, filter
, len
, map
, reversed
, sorted
, slice
, zip
enumerate
, iter
, next
, range
input
, open
, print
ascii
, bin
, chr
, format
, hex
, oct
, ord
, repr
dir
, globals
, id
, locals
, vars
callable
, delattr
, getattr
, hasattr
, setattr
breakpoint
, classmethod
, compile
, memoryview
, property
, staticmethod
, super
, …Note that functions require parentheses, for instance abs(-7)
or range(3)
. Some of the built-in functions are part of a module that can add the module’s methods to the objects, for instance open
function from module io has read
, write
, seek
, close
and more methods (see examples in the below).
The most common operators are:
+
, -
, *
, /
, **
, //
, %
, @
[
:
=
, +=
, -=
, *=
, /=
, **=
, //=
, %=
, @=
<
, >
, <=
, >=
, ==
, !=
The following identifiers are used as reserved words, or keywords of the language, and cannot be used as ordinary identifier:
False await else import pass
None break except in raise
True class finally is return
and continue for lambda try
as def from nonlocal while
assert del global not with
async elif if or yield
Next are some examples for the above functions:
divmod(6,4)
## (1, 2) # (Quotient,Remainder)
(6 // 4, 6 % 4)
## (1, 2)
all([i < 5 for i in [1,2,3]])
## True
all([i < 5 for i in [1,2,10]])
## False
[isinstance(i,str) for i in ['a','b',3]]
## [True, True, False]
list(map(pow, a, [2]*len(a))) # or
list(map(lambda x: x**2, a)) # or
[x**2 for x in a]
## [1, 9, 25]
list(filter(lambda x: x < 5, a)) # or
[x for x in a if x < 5]
## [1, 3]
a = [1,3,5]
b = [2,4,6]
list(zip(a,b)) # or
list(map(lambda x,y: (x,y), a, b))
## [(1, 2), (3, 4), (5, 6)]
[x for x in enumerate(b)]
## [(0, 2), (1, 4), (2, 6)]
[i for i,j in enumerate(b)]
## [0, 1, 2]
[j for i,j in enumerate(b)]
## [2, 4, 6]
x = 4
eval('2 * x**2 + 6')
## 38
exec('z = 40')
z
## 40
locals()/globals()
## Return a dictionary containing the current scope's local/global variables
# For example the following add x_0 = 0, x_1 = 1 and x_2 = 2 to the locals dictionary:
for i in range(3):
locals()['x_%s' % i] = i # or exec('x_%s = %s' % (i,i))
x_0,x_1,x_2
## (0, 1, 2)
hex(id(x)) # this is the address of the object x in memory
## '0x1048d7f10'
with open('new_file.txt', 'w') as fw:
fw.write('The first line\nThe second line\n')
my_file = open('new_file.txt', 'r')
my_file.read() ## read the openned file from the begining to the end
## 'The first line\nThe second line\n'
my_file.read() ## since we read the file, the cursor is at the end
## ''
my_file.seek(0) ## move the cursor to the begining
my_file.read()
## 'The first line\nThe second line\n'
my_file.seek(0) ## move the cursor to the begining
my_file.readline() ## read line by line
## 'The first line\n'
my_file.readline()
## 'The second line\n'
my_file.seek(0) ## move the cursor to the begining
my_file.readlines() ## read all lines as a list
['The first line\n', 'The second line\n']
my_file.seek(0) ## move the cursor to the begining
for line in my_file:
print(line, end = '')
## The first line
## The second line
my_file.close() ## we should close the file
Python includes a very extensive standard library that offering a wide range of facilities. We can categorize the below modules as follows:
cmath
, decimal
, math
, random
, statistics
csv
, json
argparse
, ctypes
, os
, time
sys
glob
, shutil
dbm
, pickle
, sqlite3
functools
, itertools
, operator
re
, readline
, srting
collections
, datetime
venv
concurrent.futures
The following are some of applications of the above modules.
Miscellaneous operating system interfaces (os
) module provides a portable way of using operating system dependent functionality.
import os
# Run OS commands
os.system("""
echo $(date) $(hostname) > date_hname.txt
""")
os.popen("""
echo $(date) $(hostname)
""").read().strip()
## 'Fri Feb 21 18:19:11 CST 2020 UserHost.local'
os.popen("""
echo $(date) $(hostname)
echo $HOME
""").readlines()
## ['Fri Feb 21 18:19:11 CST 2020 UserHost.local\n', '/home/user\n']
os.popen("""
echo $(date) $(hostname)
echo $HOME
""").read().strip().split('\n')
## ['Fri Feb 21 18:19:11 CST 2020 UserHost.local', '/home/user']
# Make directory and navigation
os.getcwd()
## '/home/user'
os.mkdir('./new_directory')
os.chdir('./new_directory')
os.getcwd()
## '/home/user/new_directory'
# Rename (mv) and remove
os.chdir('../') # move one directory up
os.rename('./new_directory', './renamed_dir')
os.removedirs('./renamed_dir')
os.remove('<file_name>')
# Env variables
os.getenv('HOME')
## '/home/user'
os.environ ## returns all the envs as ENV:PATH
Unix style pathname pattern expansion (glob
) module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell, although results are returned in arbitrary order.
System-specific parameters and functions (sys
) module help to pass arguments and standard inputs. Let’s create the following Python script called test-sys.py
:
import sys
filename = sys.argv[0]
usr = sys.argv[1]
host = sys.stdin
print('File name is "%s"' % filename)
print('User "%s" is in %s' % (usr,host.read()))
Now, open a Unix Shell and run:
hostname | python3 test-sys.py buzz
## File name is "test-sys.py"
## User "buzz" is in hostname.local
When argv[0]
is file name, argv[1]
is user name (buzz) and host name comes from the Shell pipe as standard input (stdin
). Sys module has many methods that you may find them in the Python docs.
Parser for command-line options, arguments and sub-commands (argparse
) module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse
will figure out how to parse those out of sys.argv
. Lets create a script (reverse-file.py
) that read text files and print in reverse order (bottom to top):
import argparse
import sys
parser = argparse.ArgumentParser(description = 'Read a file in reverse')
parser.add_argument('filename', help = 'the file to read')
parser.add_argument('-v', '--version', action = 'version', version = '%(prog)s 1.0', help = 'show program version and exit')
parser.add_argument('-l', '--limit', type = int, help = 'the number of lines to read')
args = parser.parse_args()
try:
f = open(args.filename)
except FileNotFoundError as err:
print("Error:", err)
sys.exit(2)
else:
with open(args.filename, 'r') as f:
lines = f.readlines()
lines.reverse()
if args.limit:
lines = lines[:args.limit]
for line in lines:
print(line.strip())
Now we can use the script to read files in reverse. We can use -h
or -v
options to see help and version:
usage: reverse-file.py [-h] [-v] [-l LIMIT] filename
Read a file in reverse
positional arguments:
filename the file to read
optional arguments:
-h, --help show this help message and exit
-v, --version show program version and exit
-l LIMIT, --limit LIMIT the number of lines to read
To read last 2 lines of a file and print in reverse order we can run:
JSON encoder and decoder (json
) module read and write JSON files.
import json
# Read
with open('./input.json', 'r') as jsf:
input_json = json.load(jsf)
# Write
list_dict = [{'title': 'Monty Python and the Holy Grail', 'year': [1975, 'March 14']}]
with open('output.json', 'w') as jso:
json.dump(list_dict, jso)
# Serialize to a JSON formatted str
js = json.dumps(list_dict)
Note that we read files by system arguments (sys.argv
) by using with open(sys.argv[1], 'r') as jsf
. Also, we can read use standard inputs (sys.stdin
) to read the files. For instance the following script, jread.py
, read the output.json
(from the above example) and print titles:
Now, open a Unix Shell and run:
Python object serialization (pickle
) module implements binary protocols for serializing and de-serializing a Python object structure. You might prefer JSON to pickle for many reasons such as security and human readability. Learn more about pickle in here.
import pickle
# Read
data = with open('input.pickle','rb') as pkl:
input_pkl = pickle.load(pkl)
# Write
dict_data = {'title': 'Monty Python and the Holy Grail', 'year': [1975, 'March 14']}
with open('output.pickle', 'wb') as pkl:
pickle.dump(dict_data, pkl)
# Serialize to a pickle formatted str
pkl = pickle.dumps(dict_data)
print(pkl)
## b'\x80\x04\x95H\x00\x00\x00\x00\x00\x00\x00}\x94(\x8c\x05title\x94\x8c\x1fMonty Python and the Holy Grail\x94\x8c\x04year\x94]\x94(M\xb7\x07\x8c\x08March 14\x94eu.'
CSV file reading and writing (csv
) module read and write CSV files.
import csv
# Read to list
with open('./input.csv', 'r') as fl:
csv_list = list(csv.reader(fl))
print(csv_list)
## [['a', 'b', 'c', 'd'], ['22', 'yes', '5', '0'], ['34', 'no', '7', '8']]
# Write from list
with open('output1.csv', 'w') as nfl:
csv_writer = csv.writer(nfl)
csv_writer.writerows(csv_list)
# Read to dict
with open('./input.csv', 'r') as fl:
csv_dict = list(csv.DictReader(fl))
print(csv_dict)
## [{'a': '22', 'b': 'yes', 'c': '5', 'd': '0'}, {'a': '34', 'b': 'no', 'c': '7', 'd': '8'}]
# Write from dict
with open('output2.csv', 'w') as nfl:
csv_fl = csv.DictWriter(nfl, fieldnames = csv_dict[0].keys())
csv_fl.writeheader()
csv_fl.writerows(csv_dict)
We can also read CSV files as standard inputs (sys.stdin
) or as system arguments (sys.argv
). For instance the following script, csvread.py
, read the output1.csv
(from the above example) by sys.stdin
:
Now, open a Unix Shell and run:
Interface for SQLite (sqlite3
) module provides a SQL interface compliant. Example from Python documentation.
import sqlite3
conn = sqlite3.connect('example.db')
c = conn.cursor()
# Create table
c.execute("""CREATE TABLE stocks
(date text, trans text, symbol text, qty real, price real)""")
# Insert a row of data
c.execute("INSERT INTO stocks VALUES ('2006-01-05','BUY','RHAT',100,35.14)")
# Save (commit) the changes
conn.commit()
# We can also close the connection if we are done with it
# just be sure any changes have been committed or they will be lost
conn.close()
Container datatypes (collections
) module implements specialized container datatypes providing alternatives to Python’s general purpose built-in containers, dict
, list
, set
, and tuple
.
import collections
# Example 1
dict_list = collections.defaultdict(list)
for i in list(range(2))*3:
dict_list[i].append(1)
dict(dict_list)
## {0: [1, 1, 1], 1: [1, 1, 1]}
for k in dict_list:
dict_list[k] = sum(dict_list[k])
dict(dict_list)
## {0: 3, 1: 3}
# Example 2
s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
d = collections.defaultdict(list)
for k,v in s:
d[k].append(v)
sorted(d.items())
## [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
dict(sorted(d.items()))
## {'blue': [2, 4], 'red': [1], 'yellow': [1, 3]}
Time access and conversions (time
) module provides various time-related functions.
import time
local_time = time.localtime() # a time tuple expressing local time
local_time
## time.struct_time(tm_year=2021, tm_mon=5, tm_mday=6, tm_hour=22, tm_min=3, tm_sec=32, tm_wday=3, tm_yday=126, tm_isdst=1)
local_time.tm_year
## 2021
time.strftime("%X", local_time) # convert a time tuple to a string according to a format specification
## '22:03:32'
time.strftime("%Y-%m-%d %H:%M:%S") # the default tuple is localtime()
## '2021-05-06 22:10:03'
time.time() # return the current time in seconds since the Epoch
## 1620356510.8557692
time.mktime(local_time) # convert a time tuple in local time to seconds since the Epoch
## 1620356612.0
local_time2 = time.localtime()
difference = time.mktime(local_time2) - time.mktime(local_time) # time difference in sec
difference
## 452.0
time.gmtime(difference) # convert seconds since the Epoch to a time tuple
## time.struct_time(tm_year=1970, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=1, tm_isdst=0)
time.strptime("30 Nov 20", "%d %b %y") # parse a string to a time tuple according to a format specification
## time.struct_time(tm_year=2020, tm_mon=11, tm_mday=30, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=0, tm_yday=335, tm_isdst=-1)
Commonly used time format codes:
%Y
Year with century as a decimal number.%m
Month as a decimal number [01,12].%d
Day of the month as a decimal number [01,31].%H
Hour (24-hour clock) as a decimal number [00,23].%M
Minute as a decimal number [00,59].%S
Second as a decimal number [00,61].%z
Time zone offset from UTC.%a
Locale’s abbreviated weekday name.%A
Locale’s full weekday name.%b
Locale’s abbreviated month name.%B
Locale’s full month name.%c
Locale’s appropriate date and time representation.%x
Locale’s appropriate date representation.%X
Locale’s appropriate time representation.%p
Locale’s equivalent of either AM or PM.%I
Hour (12-hour clock) as a decimal number [01,12].Functions creating iterators for efficient looping (itertools
) module implements a number of iterator building blocks.
import itertools
# Combinatoric iterators
list(itertools.combinations('ABC', 2))
## [('A', 'B'), ('A', 'C'), ('B', 'C')]
list(itertools.combinations_with_replacement('ABC', 2))
## [('A', 'A'), ('A', 'B'), ('A', 'C'), ('B', 'B'), ('B', 'C'), ('C', 'C')]
list(itertools.permutations('ABC', 2))
## [('A', 'B'), ('A', 'C'), ('B', 'A'), ('B', 'C'), ('C', 'A'), ('C', 'B')]
list(itertools.product('ABC', repeat = 2))
## [('A', 'A'), ('A', 'B'), ('A', 'C'), ('B', 'A'), ('B', 'B'), ('B', 'C'), ('C', 'A'), ('C', 'B'), ('C', 'C')]
a = [1,3]
b = [2,4]
list(zip(a,b))
## [(1, 2), (3, 4)]
list(itertools.permutations(a+b,2))
## [(1, 3), (1, 2), (1, 4), (3, 1), (3, 2), (3, 4), (2, 1), (2, 3), (2, 4), (4, 1), (4, 3), (4, 2)]
# Accumulate
mylist = [1,3,5,7,9,11,13]
list(itertools.accumulate(mylist))
## [1, 4, 9, 16, 25, 36, 49]
In Regular expression operations (re
) module, we specify the rules for the set of possible strings that you want to match; this set might contain English sentences, or e-mail addresses, or TeX commands, or anything you like (from Python HOWTOs).
import re
re.findall('begin', 'begin with this example for beginning') # Find all 'begin's in text
## ['begin', 'begin']
re.findall('^begin', 'begin with this example for beginning') # Find 'begin' only at the beginnig (^) of text
['begin']
re.findall('begin$', 'this is another begin') # Search only last word ($)
## ['begin']
re.findall('.*begin', 'this is another begin') # Search 'begin' and everything before (.*)
## ['this is another begin']
re.findall('goo?al','goal vs goooooal') # Zero or one (?) 'o' character
## ['goal']
re.findall('goo*al','goal vs goooooal') # Zero or more (*) 'o' character
## ['goal', 'goooooal']
re.findall('goo+al','goal vs goooooal') # One or more (*) 'o' character
## ['goooooal']
re.findall('\d', 'today is Oct 10') # Find digits (\d)
## ['1', '0']
re.findall('\d{2}', 'today is Oct 10') # Find two digits (\d{2})
## ['10']
re.findall('\w', 'today is Oct 10') # Find any words (\w)
## ['t', 'o', 'd', 'a', 'y', 'i', 's', 'O', 'c', 't', '1', '0']
re.findall('\w+', 'yesterday was October 9') # Find any word with one or more (+) characters
## ['yesterday', 'was', 'October', '9']
re.findall('\w{4}\w*', 'yesterday was October 9') # Find any word with 4 letters or more
## ['yesterday', 'October']
re.findall('[A-Z]..', 'yesterday was October 9') # Find any capital word ([A-Z]) and two characters after (..)
## ['Oct']
re.findall('([A-Z][a-z]* \d+)', 'yesterday was October 9') # Find any capital letter ([A-Z]) followed by two small letters ([a-z]{2}) and a space ( ) and two digits (\d{2})
## ['October 9']
re.findall('(?<=; )[\w ]*', 'I want everything after; this part is important') # After (\w = [a-zA-Z0-9_])
## ['this part is important']
re.findall('[\w ]+(?=;)', 'I want everything before; this part is not important') # Before
## ['I want everything before']
pattern = re.compile('[\w ]*(?=;)') # We can keep the pattern in this way
mm = re.match(pattern, 'I want everything before; this part is not important') # Before
mm.group(0)
## 'I want everything before'
Beside the standard library, we can add numerous packages to the Python toolkit. The following are some of the most famous Python packages that can empower your toolbox.
Review here to learn about virtual environments and package manager systems for Python.