Python Pandas Intro

8/18/2019 Python Pandas Intro

1/15

8/26/15, 6:5andas-Intro

Page 1ttps://newclasses.nyu.edu/access/lessonbuilder/item/16037308/…44f8-963b-8160554a987f/Python%20Lab/Week%207/Pandas-Intro.html

Pandas

Pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aim

be the fundamental high-level building block for doing practical, real world data analysis in Python. Pandas provides high-performance, easy-to-use data structures and data

analysis tools for the Python programming language. To get started with pandas, you will need to get comfortable with its two workhorse data structures: Series and DataFram

Series

Pandas Series is a one-dimensional array-like object that has index and value just like Numpy. Infact if you view the type of the values of series object, you will see that it ind

is numpy.ndarray.

You can assign name to pandas Series.

In [1]: import pandas as pd

import numpy as np

% matplotlib inline

In [2]: ob = pd.Series([8,7,6,5], name='test_data')

print 'Name: ',ob.name

print 'Data:\n',ob

print 'Type of Object: ',type(ob)

print 'Type of elements:',type(ob.values)

You can also use your numpy array and convert them to Series.

In [3]: ob = pd.Series(np.linspace(5, 8, num=4, dtype=int)[::-1]) # np.linspace(5,8,num=4,dtype=int) = Evenly spaced integers

# between 5 to 8 (reversed)

print ob

print type(ob)

You can also provide custom index to the values and just like in Numpy, access them with the index.

In [4]: ob = pd.Series([8,7,6,5], index=['a','b','c','d'])

print ob['b']

Pandas Series is more like an fixed size dictionary whose mapping of index-value is preserved when array operations are applied to them. For example,

In [5]: print ob[(ob>4) & (ob


2/15



This also means that if you have a dictionary, you can easily convert that into pandas series.

In [6]: states_dict = {'State1': 'Alabama', 'State2': 'California', 'State3': 'New Jersey', 'State4': 'New York'}

ob = pd.Series(states_dict)

print ob

print type(ob)

Just like dictionaries, you can also change the index..

In [7]: ob.index = ['AL','CA','NJ','NY']

print ob

or use dictionary's method to get the label..

In [8]: ob.get('CA', np.nan)

Dataframe

Dataframe is something like spreadsheet or a sql table. I t is basically a 2 dimensional labelled data structure with columns of potentially di!erent datatype. Like Series, DataFra

accepts many di!erent kinds of input:

Dict of 1D ndarrays, lists, dicts, or Series

2-D numpy.ndarray

Structured or record ndarray (http://docs.scipy.org/doc/numpy/user/basics.rec.html)

A Series

Another DataFrame

Compared with other such DataFrame-like structures you may have used before ( like R’s data.frame ), row- oriented and column-oriented operations in DataFrame are treat

roughly symmetrically. Under the hood, the data is stored as one or more two-dimensional blocks rather than a list, dict, or some other collection of one-dimensional arrays.

Creating Dataframes from dictionaries

In [9]: data = {'one' : pd.Series([1., 2., 3.], index=['a', 'b', 'c']),

'two' : pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}

In [10]: df = pd.DataFrame(data)

print 'Dataframe:\n',df

print 'Type of Object:',type(df)

print 'Type of elements:',type(df.values)

State1 Alabama

State2 California

State3 New Jersey

State4 New York

dtype: object

AL Alabama

CA California

NJ New Jersey

NY New York

dtype: object

Out[8]: 'California'

Dataframe:

one two

a 1 1

b 2 2

c 3 3

d NaN 4

Type of Object:

Type of elements:

http://docs.scipy.org/doc/numpy/user/basics.rec.html


3/15



Another way to construct dataframe from dictionaries is by using DataFrame.from_dict function. DataFrame.from_dict takes a dict of dicts or a dict of

array-like sequences and returns a DataFrame. It operates like the DataFrame constructor except for the orient parameter which is 'columns' by default, but which

can be set to 'index' in order to use the dict keys as row labels.

Just like Series, you can access index, values and also columns.

In [11]: print 'Index: ',df.index

print 'Columns: ',df.columns

print 'Values of Column one: ',df['one'].valuesprint 'Values of Column two: ',df['two'].values

Creating dataframe from list of dictionaries

As with Series, if you pass a column that isn’t contained in data, it will appear with NaN values in the result

In [12]: df2 = pd.DataFrame([{'a': 1, 'b': 2, 'c':3, 'd':None}, {'a': 2, 'b': 2, 'c': 3, 'd': 4}],

index=['one', 'two'])

print 'Dataframe: \n',df2

# Ofcourse you can also transpose the result:

print '\nTransposed Dataframe: \n',df2.T

Assigning a column that doesn’t exist will create a new column.

In [13]: df['three'] = None

print 'Added third column: \n',df

# The del keyword will delete columns as with a dict:

del df['three']

print '\nDeleted third column: \n',df

Index: Index([u'a', u'b', u'c', u'd'], dtype='object')

Columns: Index([u'one', u'two'], dtype='object')

Values of Column one: [ 1. 2. 3. nan]

Values of Column two: [ 1. 2. 3. 4.]

Dataframe:

a b c d

one 1 2 3 NaN

two 2 2 3 4

Transposed Dataframe:

one two

a 1 2

b 2 2

c 3 3

d NaN 4

Added third column:

one two three

a 1 1 None

b 2 2 None

c 3 3 None

d NaN 4 None

Deleted third column:

one two

a 1 1b 2 2

c 3 3

d NaN 4


4/15



Each Index has a number of methods and properties for set logic and answering other common questions about the data it contains.

Method Description

append Concatenate with additional Index objects, producing a new Index

diff Compute set di!erence as an Index

intersection Compute set intersection

union Compute set union

isin Compute boolean array indicating whether each value is contained in the passed collection

delete Compute new Index with element at index i deleted

drop Compute new index by deleting passed values

insert Compute new Index by inserting element at index i

is_monotonic Returns True if each element is greater than or equal to the previous element

is_unique Returns True if the Index has no duplicate values

unique Compute the array of unique values in the Index

for example:

In [14]: print 1 in df.one.values

print 'one' in df.columns

Reindexing

A critical method on pandas objects is reindex, which means to create a new object with the data conformed to a new index.

In [15]: data = {'one' : pd.Series([1., 2., 3.], index=['a', 'b', 'c']),

'two' : pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}

df = pd.DataFrame(data)

print df

In [16]: print df.reindex(['d','c','b','a']) # Reindex in descending order.

If you reindex with more number of rows than in the dataframe, it will return the dataframe with new row whose values are NaN.

In [17]: print df.reindex(['a','b','c','d','e'])

Reindexing is also useful when you want to introduce any missing values. For example in our case, look at column one and row d

TrueTrue

one two

a 1 1

b 2 2

c 3 3

d NaN 4

one two

d NaN 4

c 3 3

b 2 2

a 1 1

one two

a 1 1

b 2 2c 3 3

d NaN 4

e NaN NaN


5/15



In [18]: df.reindex(['a','b','c','d','e'], fill_value=0)

# Guess why the df['one']['d'] was not filled with 0 ?

For ordered data like time series, it may be desirable to do some interpolation or filling of values when reindexing. The method option allows us to do this, using a method s

as ffill which forward fills the values:

In [19]: df.reindex(['a','b','c','d','e'], method='ffill')

There are basically two di!erent types of method (interpolation) options:

Method Description

ffill or pad Fill (or carry) values forward

bfill or backfill Fill (or carry) values backward

Reindexing has following arguments:

Argument Description

index New sequence to use as index. Can be Index instance or any other sequence-like Python data structure. An Index will be used exactly as is without any copy

method Interpolation (fill) method, see above table for options.

fill_value Substitute value to use when introducing missing data by reindexing.

limit When forward- or backfilling, maximum size gap to fill

level Match simple Index on level of MultiIndex, otherwise select subset of

copy Do not copy underlying data if new index is equivalent to old index. True by default (i.e. always copy data)

Dropping Entries

Dropping one or more entries from an axis is easy if you have an index array or list without those entries.

In [20]: # Drop row c and row a

df.drop(['c', 'a'])

Out[18]:one two

a 1 1

b 2 2

c 3 3

d NaN 4

e 0 0

Out[19]: one two

a 1 1

b 2 2

c 3 3

d NaN 4

e NaN 4

Out[20]:one two

b 2 2

d NaN 4


6/15



In [21]: # Drop column two

df.drop(['two'], axis=1)

Indexing, selection, Sorting and filtering

Series indexing works analogously to NumPy array indexing, except you can use the Series’s index values instead of only integers.

In [22]: print df

# Slicing and selecting only row 0 and row 4

df['one'][['a', 'd']]

In [23]: # Slicing df from row b to row 4

df['one']['b':'d']

If you observe the above command (and the one above it), you will see that slicing with labels behaves di!erently than normal Python slicing in that the endpoint is inclusive.

For DataFrame label-indexing on the rows, there is a special indexing field ix. It enables you to select a subset of the rows and columns from a DataFrame with NumPy- like

notation plus axis labels. It is a less verbose way to do the reindexing.

In [24]: df.ix[['a','c'],['one']]

In [25]: df.ix[df.one > 1]

Out[21]:one

a 1

b 2

c 3

d NaN

one two

a 1 1

b 2 2

c 3 3

d NaN 4

Out[22]: a 1

d NaNName: one, dtype: float64

Out[23]: b 2

c 3

d NaN

Name: one, dtype: float64

Out[24]: one

a 1

c 3

Out[25]:one two

b 2 2

c 3 3


7/15



There are many ways to select and rearrange the data contained in a pandas object. Some indexing options can be seen in below table:

Indexing Type Description

df[val]Select single column or sequence of columns from the DataFrame. Special case con- veniences: boolean array (filter rows), slice (slice rows), or boole

DataFrame (set values based on some criterion).

df .ix[ val] Selects single row of subset of rows from the DataFrame.

df .ix[ :, val] Selects single column of subset of columns.

df.ix[val1, val2] Select both rows and columns.

reindex method Conform one or more axes to new indexes.

xs method Select single row or column as a Series by label.

icol, irowmethods Select single column or row, respectively, as a Series by integer location.

get_value, set_value

methodsSelect single value by row and column label.

You can sort a data f rame or series (by some criteria) using the built-in functions. To sort lexicographically by row or column index, use the sort_index method, which returns a

new, sorted object:

In [26]: dt = pd.Series(np.random.randint(3, 10, size=7), index=['g','c','a','b','e','d','f'])

print 'Original Data: \n', dt

print 'Sorted by Index: \n',dt.sort_index()

Data alignment and arithmetic

Data alignment between DataFrame objects automatically align on both the columns and the index (row labels). The resulting object will have the union of the column and row

labels.

Original Data:

g 6

c 9

a 9

b 5

e 3

d 8

f 7

dtype: int64

Sorted by Index:

a 9

b 5

c 9

d 8

e 3

f 7

g 6

dtype: int64


8/15



In [27]: df1 = pd.DataFrame(np.random.randn(10, 4), columns=['A', 'B', 'C', 'D'])

df2 = pd.DataFrame(np.random.randn(7, 3), columns=['A', 'B', 'C'])

print 'df1:\n',df1

print 'df2:\n',df2

print 'Sum:\n',df1.add(df2)

Note that in arithmetic operations between di!erently-indexed objects, you might want to fill with a special value, like 0, when an axis label is found in one object but not the ot

In [28]: print 'Sum:\n',df1.add(df2, fill_value=0)

Similarly you can perform subtracion, multiplication and division.

When doing an operation between DataFrame and Series, the default behavior is to align the Series index on the DataFrame columns, thus broadcasting (just like in numpy) ro

wise.

df1:

A B C D

0 -1.869235 0.114255 0.816411 -0.297434

1 0.112815 0.660802 1.037941 0.576426

2 1.041494 -0.078062 -0.972924 -0.568679

3 -2.785414 1.578352 0.924656 0.226743

4 -0.429171 0.321302 0.183773 0.8509855 -0.536632 0.500795 1.429295 -1.099967

6 0.592204 0.392437 0.174914 -0.009833

7 0.425151 0.453137 -1.347765 1.300194

8 0.081314 -0.324954 0.347301 1.892119

9 1.738767 1.396856 0.326706 -0.741861

df2:

A B C

0 -0.074048 0.530960 -1.013815

1 0.709423 -0.953860 -0.270428

2 0.215185 1.276945 -1.479264

3 -1.376585 -0.417693 0.039363

4 0.305415 0.403303 1.495533

5 1.983297 -0.363862 1.657616

6 0.673487 1.211236 -0.347881

Sum:

A B C D

0 -1.943283 0.645215 -0.197403 NaN

1 0.822238 -0.293058 0.767512 NaN2 1.256679 1.198883 -2.452188 NaN

3 -4.161999 1.160658 0.964019 NaN

4 -0.123756 0.724605 1.679306 NaN

5 1.446665 0.136932 3.086912 NaN

6 1.265692 1.603673 -0.172967 NaN

7 NaN NaN NaN NaN

8 NaN NaN NaN NaN

9 NaN NaN NaN NaN

Sum:

A B C D

0 -1.943283 0.645215 -0.197403 -0.2974341 0.822238 -0.293058 0.767512 0.576426

2 1.256679 1.198883 -2.452188 -0.568679

3 -4.161999 1.160658 0.964019 0.226743

4 -0.123756 0.724605 1.679306 0.850985

5 1.446665 0.136932 3.086912 -1.099967

6 1.265692 1.603673 -0.172967 -0.009833

7 0.425151 0.453137 -1.347765 1.300194

8 0.081314 -0.324954 0.347301 1.892119

9 1.738767 1.396856 0.326706 -0.741861


9/15



In [29]: print df1.loc[0]

print 'Sum: \n',df1.sub(df1.loc[0])

In the special case of working with time series data, and the DataFrame index also contains dates, the broadcasting will be column-wise:

In [30]: ind1 = pd.date_range('08/1/2015', periods=10)

df1.set_index(ind1)

Using Numpy functions on DataFrame

Elementwise NumPy ufuncs like log, exp, sqrt, ... and various other NumPy functions can be used on DataFrame

In [31]: np.abs(df1)

A -1.869235

B 0.114255

C 0.816411

D -0.297434

Name: 0, dtype: float64

Sum:

A B C D

0 0.000000 0.000000 0.000000 0.000000

1 1.982050 0.546547 0.221530 0.873859

2 2.910729 -0.192316 -1.789335 -0.2712453 -0.916179 1.464097 0.108245 0.524177

4 1.440064 0.207047 -0.632639 1.148418

5 1.332603 0.386540 0.612884 -0.802533

6 2.461440 0.278182 -0.641497 0.287601

7 2.294386 0.338882 -2.164176 1.597627

8 1.950549 -0.439209 -0.469110 2.189553

9 3.608003 1.282602 -0.489706 -0.444427

Out[30]: A B C D

2015-08-01 -1.869235 0.114255 0.816411 -0.297434

2015-08-02 0.112815 0.660802 1.037941 0.576426

2015-08-03 1.041494 -0.078062 -0.972924 -0.568679

2015-08-04 -2.785414 1.578352 0.924656 0.226743

2015-08-05 -0.429171 0.321302 0.183773 0.850985

2015-08-06 -0.536632 0.500795 1.429295 -1.099967

2015-08-07 0.592204 0.392437 0.174914 -0.009833

2015-08-08 0.425151 0.453137 -1.347765 1.300194

2015-08-09 0.081314 -0.324954 0.347301 1.892119

2015-08-10 1.738767 1.396856 0.326706 -0.741861

Out[31]: A B C D

0 1.869235 0.114255 0.816411 0.297434

1 0.112815 0.660802 1.037941 0.576426

2 1.041494 0.078062 0.972924 0.568679

3 2.785414 1.578352 0.924656 0.226743

4 0.429171 0.321302 0.183773 0.850985

5 0.536632 0.500795 1.429295 1.099967

6 0.592204 0.392437 0.174914 0.009833

7 0.425151 0.453137 1.347765 1.300194

8 0.081314 0.324954 0.347301 1.892119

9 1.738767 1.396856 0.326706 0.741861


10/15



In [32]: np.asarray(df1) # Convert input to numpy array

Another frequent operation is applying a function on 1D arrays to each column or row. DataFrame’s apply method does exactly this:

In [33]: def fn(x):

return pd.Series([x.min(), x.max()], index=['min', 'max']) # Get max and min of the columns

#fn = lambda x: x - x.min() # Subtract the minimum of the column from each element of that column

df1.apply(fn)

Element-wise Python functions can be used, too. Suppose you wanted to format the dataframe elements in floating point format with accuracy of only 3 decimal places. You cdo this with applymap:

In [34]: fmt = lambda x: "{:.3f}".format(x)

df1.applymap(fmt)

The reason for the name applymap is that Series has a map method for applying an element-wise function

Out[32]: array([[-1.86923522, 0.11425481, 0.81641128, -0.29743373],

[ 0.11281503, 0.66080205, 1.03794085, 0.57642562],

[ 1.04149359, -0.07806151, -0.97292403, -0.56867919],

[-2.78541399, 1.57835165, 0.92465601, 0.22674327],

[-0.4291715 , 0.32130162, 0.18377266, 0.8509845 ],

[-0.53663223, 0.5007948 , 1.42929534, -1.09996685],

[ 0.59220433, 0.39243689, 0.17491424, -0.00983318],

[ 0.42515075, 0.4531367 , -1.34776521, 1.30019367],

[ 0.08131366, -0.32495414, 0.34730131, 1.89211945],

[ 1.73876733, 1.39685642, 0.32670562, -0.74186091]])

Out[33]: A B C D

min -2.785414 -0.324954 -1.347765 -1.099967

max 1.738767 1.578352 1.429295 1.892119

Out[34]: A B C D

0 -1.869 0.114 0.816 -0.297

1 0.113 0.661 1.038 0.576

2 1.041 -0.078 -0.973 -0.569

3 -2.785 1.578 0.925 0.227

4 -0.429 0.321 0.184 0.851

5 -0.537 0.501 1.429 -1.100

6 0.592 0.392 0.175 -0.010

7 0.425 0.453 -1.348 1.300

8 0.081 -0.325 0.347 1.892

9 1.739 1.397 0.327 -0.742


11/15



Loading Data

You can read data from a CSV file using the read_csv function. By default, it assumes that the fields are comma-separated. Pandas supports following file formats:

Function Description

read_csv Load delimited data from a file, URL, or file-like object. Use comma as default delimiter

read_table Load delimited data from a file, URL, or file-like object. Use tab ('\t') as default delimiter

read_fwf Read data in fixed-width column format (that is, no delimiters)

read_clipboard Version of read_table that reads data from the clipboard. Useful for converting tables from web pages.

Let's try loading some citibike data that you used for your challenge #2 using pandas. (We will also use the same technique later on for loading big file like the one you had to u

for Core Challenge). If you do not have the csv file from Challenge #2, you can download it again from here: Dec-2week-2014.csv (http://sharmamohit.com/misc_files/dec-

2week-2014.csv)

In [35]: dec = pd.read_csv('dec-2week-2014.csv')

dec.describe()

Out[35]:tripduration start station id

start station

latitude

start station

longitudeend station id

end station

latitude

end station

longitudebikeid birth year

count 192260.000000 192260.000000 192260.000000 192260.000000 192260.000000 192260.000000 192260.000000 192260.000000 187314.00000

mean 746.854666 436.637116 40.735708 -73.990421 437.083829 40.735578 -73.990647 18141.270124 1975.495451

std 2997.200035 318.126922 0.018599 0.011611 321.761738 0.018638 0.011726 2061.113390 11.737892

min 60.000000 72.000000 40.680342 -74.017134 72.000000 40.680342 -74.017134 14529.000000 1899.000000

25% 348.000000 307.000000 40.724055 -73.998393 307.000000 40.723627 -73.999061 16387.000000 1967.000000

50% 529.000000 417.000000 40.737262 -73.990617 414.000000 40.737050 -73.990741 18135.000000 1977.000000

75% 816.000000 491.000000 40.750380 -73.981948 490.000000 40.750200 -73.981948 19911.000000 1985.000000

max 732149.000000 3002.000000 40.771522 -73.950048 3002.000000 40.771522 -73.950048 21690.000000 1998.000000

http://sharmamohit.com/misc_files/dec-2week-2014.csv


12/15



As we can see, the describe() method produces some very useful statistics about the csv data that we loaded.

The parser functions have many additional arguments to help you handle the wide variety of exception file formats that occur

Argument Description

path String indicating filesystem location, URL, or file-like object

sep or

delimiterCharacter sequence or regular expression to use to split fields in each row

header Row number to use as column names. Defaults to 0 (first row), but should be None if there is no header row

index_col Column numbers or names to use as the row index in the result. Can be a single name/number or a list of them for a hierarchical index

names List of column names for result, combine with header=None

skiprows Number of rows at beginning of file to ignore or list of row numbers (starting from 0) to skip

na_values Sequence of values to replace with NA

comment Character or characters to split comments o! the end of lines

parse_dates Attempt to parse data to datetime; False by default. If True, will attempt to parse all columns. Otherwise can specify a list of column numbers or name to

parse. If element of list is tuple or list, will combine multiple columns together and parse to date (for example if date/time split across two columns)

keep_date_col If joining columns to parse date, drop the joined columns. Default True

converters Dict containing column number of name mapping to functions. For example {' foo': f} would apply the function f to all values in the 'foo' column

dayfirst When parsing potentially ambiguous dates, treat as international format (e.g. 7/6/2012 -> June 7, 2012). Default False

date_parser Function to use to parse dates

nrows Number of rows to read from beginning of file

iterator Return a TextParser object for reading file piecemeal

chunksize For iteration, size of file chunks

skip_footer Number of lines to ignore at end of file

verbose Print various parser output information, like the number of missing values placed in non-numeric columns

encoding Text encoding for unicode. For example 'utf-8' for UTF-8 encoded text

squeeze If the parsed data only contains one column return a Series

thousands Separator for thousands, e.g. ',' or '.'

If you have a file that is comparatively huge in size and you see that pandas or numpy(genfromtxt or loadfromtxt) is struggling to load it then pandas

provide an iterator that can be used. The arguments with pd.read_csv() would be something like (along with any other arguments as required):

data_iter = pd.read_csv(infile, iterator=True, chunksize=1000, ) # This returns iterator with chunk of 1000 rows.

data = pd.concat(data_iter)

In [36]: dec[:3]

In [37]: type(dec['starttime'].values[0])

Out[36]:

tripduration starttime stoptime

start

station

id

start

station

name

start

station

latitude

start

station

longitude

end

station

id

end

station

name

end

station

latitude

end

station

longitude

bikeid usertypebirth

yeargen

0 125712/1/2014

00:00:28

12/1/2014

00:21:25475

E 16 St &

Irving Pl40.735243 -73.987586 521

8 Ave

& W 31

St

40.750450 -73.994811 16047 Customer NaN 0

1 27512/1/2014

00:00:43

12/1/2014

00:05:18498

Broadway

& W 32 St40.748549 -73.988084 546

E 30 St& Park

Ave S

40.744449 -73.983035 18472 Subscriber 1988 2

2 45012/1/2014

00:01:22

12/1/2014

00:08:52444

Broadway

& W 24 St40.742354 -73.989151 434

9 Ave

& W 18

St

40.743174 -74.003664 19589 Subscriber 1983 1

Out[37]: str


13/15



From above example, we can see that the starttime column is parsed as a string. We need to parse the dates as a datetime object so we can perform some datetime relate

computation.

Pandas provide an excellent and easy way to parse the column with date and/or time as a datetime object. To do that, you simply need to proide the read_csv function with

parse_dates with column name that has date (and/or time).

In [38]: dec = pd.read_csv('dec-2week-2014.csv', parse_dates=['starttime'])

type(dec['starttime'].values[0])

The above option works perfectly fine and as we can see the starttime column now has numpy.datetime64 objects. You have to provide parse_date with the column th

has the date (and/or time) information. This uses Pandas dateutil.parser.parser to do the conversion.

Pandas will try to call date_parser in three di!erent ways, advancing to the next if an exception occurs:

1. Pass one or more arrays (as defined by parse_dates ) as arguments.

2. Concatenate (row-wise) the string values from the columns defined by parse_dates into a single array and pass that;

3. Call date_parser once for each row using one or more strings (corresponding to the columns defined by parse_dates ) as arguments.

Now this works fine but it consumes (comparatively) quite a lot of time. If you know the format of your date and is consistent then you can create a function to do the conversi

and pass it to date_parser. date_parser will basically pass every element of the column specified in parse_dates to the function and let your function manually convert

datetime object. This reduces the computation time. (This is a good time to check it for yourself. use the ipython's magic function %timeit )

Once you start parsing huge files for dates, you might have to write your own cython functions. Do not worry about cython for now. But for the curious heads,

check how to improve performance of pandas.. http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html(http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html)

In [39]: from datetime import datetime

from matplotlib import dates

dt_parse = lambda x: datetime.strptime(x, '%m/%d/%Y %H:%M:%S')

dec = pd.read_csv('dec-2week-2014.csv', parse_dates=['starttime'], date_parser=dt_parse, index_col='starttime')

In [40]: ax = dec['tripduration'].plot(kind='area', stacked=False, figsize=(12, 8), color='#3F5D7D')

Out[38]: numpy.datetime64

http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html


14/15



In the above example, I have also used the starttime as my index column. Also plot() function returns matplotlib.axes._subplots.AxesSubplot so

you can play around with the plot before showing it. Refer to our matplotlib notes to use some ways to plot it better.

A quick example:

dt = pd.date_range(start=dec.index[0], end=dec.index[-1], freq='D')

ax = dec['tripduration'].plot(kind='area', stacked=False, figsize=(12, 8), xticks=dt)

ax.xaxis.set_minor_locator(dates.HourLocator(interval=12))

ax.xaxis.grid(True, which="major", linestyle='--')ax.xaxis.grid(True, which="minor")

ax.yaxis.grid(True, which="major")

ax.xaxis.set_major_formatter(dates.DateFormatter('%b %d'))

Pandas makes it really easy to select a subset of the columns: just index with list of columns you want.

In [41]: dec[['start station id', 'end station id']][:5]

Another very common question that can be asked is.. just of curiosity, which bike was used the most in these 15days.. and the answer is..

In [42]: dec['bikeid'].value_counts()[:5] # Top 5 bikes by id

Also, just for fun, lets plot this!

In [43]: famous_bikes = dec['bikeid'].value_counts()

famous_bikes[:10][::-1].plot(kind='barh', alpha=0.5, color='#3F5D7D')

Out[41]:start station id end station id

starttime

2014-12-01 00:00:28 475 521

2014-12-01 00:00:43 498 546

2014-12-01 00:01:22 444 434

2014-12-01 00:02:17 475 521

2014-12-01 00:02:21 519 527

Out[42]: 18440 118

19977 115

19846 110

19757 108

19494 105

dtype: int64

Out[43]:


15/15



End Note

Remember, this is just the tip of the iceberg of what functions Pandas provide. Pandas combined with Numpy and Matplotlib gives you an ultimate tool for almost all your Data

Analysis needs.

Because of the high majority of the votes to not introduce Pandas, I have created this concise version of otherwise what would have been a 3 part course.

It is highly recommended to check out some tutorials below for more information on Pandas:

Pandas own 10 minute to Pandas (http://pandas.pydata.org/pandas-docs/stable/10min.html#min)

Hernan Rojas's Learn Pandas (https://bitbucket.org/hrojas/learn-pandas)

Pandas Cookbook (http://pandas.pydata.org/pandas-docs/stable/cookbook.html#cookbook)

Brandon Rhodes's Exercise and Solutions (https://github.com/brandon-rhodes/pycon-pandas-tutorial)

Greg Reda's Blog (http://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/)

You can also find many PyCon talks:

PyCon 2015:

Brandon Rhodes's Pandas from Ground up (https://www.youtube.com/watch?v=5JnMutdy6Fw)

PyVideo Videos:

Some Videos from pyvideo.org on Pandas (ht tp://pyvideo.org/search?q=pandas)

In [ ]:
http://pyvideo.org/search?q=pandashttps://www.youtube.com/watch?v=5JnMutdy6Fwhttp://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/https://github.com/brandon-rhodes/pycon-pandas-tutorialhttp://pandas.pydata.org/pandas-docs/stable/cookbook.html#cookbookhttps://bitbucket.org/hrojas/learn-pandashttp://pandas.pydata.org/pandas-docs/stable/10min.html#min

Python Pandas Intro

Documents

Transcript of Python Pandas Intro