Pandas - https://pandas.pydata.org/¶
From: https://en.wikipedia.org/wiki/Pandas_%28software%29
Pandas is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series. It is free software released under the three-clause BSD license. The name is derived from the term “panel data”, an econometrics term for data sets that include observations over multiple time periods for the same individuals.
Pandas is mainly used for Machine Learning in form of dataframes. Pandas allow importing data of various file formats such as csv, excel etc. Pandas allows various data manipulation operations such as groupby, join, merge, melt, concatenation as well as data cleaning features such as filling, replacing or imputing null values.
Key features:
DataFrame object for data manipulation with integrated indexing.
Tools for reading and writing data between in-memory data structures and different file formats.
Data alignment and integrated handling of missing data.
Reshaping and pivoting of data sets.
Label-based slicing, fancy indexing, and subsetting of large data sets.
Data structure column insertion and deletion.
Group by engine allowing split-apply-combine operations on data sets.
Data set merging and joining.
Hierarchical axis indexing to work with high-dimensional data in a lower-dimensional data structure.
Time series-functionality: Date range generation[4] and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging.
Provides data filtration.
The library is highly optimized for performance, with critical code paths written in Cython or C.
Bottom line - if you are doing anything related to with spreadsheets you should be using Pandas.
Pandas tutorial¶
If you are interested in data science and machine learning then you need to know about Kaggle. You will use it a few times throughout the year.
https://youtu.be/TNzDMOg_zsw
Because pandas is such a big deal in data science and machine learning Kaggle are put together a great tutorial on how to use it. That is what we are going to focus on today:
https://www.kaggle.com/learn/pandas
Today’s lecture will be presented by Ruari Dunne from BP.
Ruairi Dunne is a Data Science graduate at bp with an educational background at Imperial College specialising in Geology (BSc) and Petroleum Geoscience (MSc). With over 2 years’ experience at bp, Ruairi has led Time Series, Natural Language Processing (NLP) and Computer Vision projects, applying data science to tackle industry problems. The multi-disciplinary nature of data science, capability development, and leveraging machine and deep learning for traditional geoscience and engineering workflows, is what Ruairi is most passionate about and his favourite libraries are Pytorch-Lightning and SHAP.