Python Data Science Handbook

Interactive computing environment

IPython (Interactive Python) provides an enhanced interactive shell that has become the default computing environment for data science. Jupyter Notebooks extend this concept to the browser, creating "computational narratives" that interleave code, results, visualizations, and explanatory text in a single shareable document.

Key IPython features include: tab completion (explore objects and modules without documentation), magic commands (%timeit for benchmarking, %matplotlib for inline plots, %run for executing scripts, %%bash for shell commands), rich display system (rendering HTML, images, LaTeX, and interactive widgets), and integrated debugging with %debug.

JupyterLab, the next-generation interface, provides a full IDE-like environment with multiple notebooks, terminals, file browsers, and extensions in a single browser window. JupyterHub enables multi-user deployment for classrooms and research teams. Google Colab offers free cloud-based Jupyter environments with GPU access.

• Jupyter Notebooks are the standard tool for reproducible data analysis
• Magic commands dramatically accelerate interactive development
• JupyterLab provides an IDE-like experience for notebook-based workflows

• Create a Jupyter Notebook with code cells, markdown cells, and inline visualizations
• Use %timeit to compare list comprehension vs for loop performance
• Export a notebook as HTML and PDF for sharing

Arrays, broadcasting, ufuncs, fancy indexing

NumPy (Numerical Python) is the foundation of the entire Python data science ecosystem. Its core data structure — the ndarray (n-dimensional array) — stores homogeneous data in contiguous memory, enabling vectorized operations that are 10-100x faster than equivalent Python loops. Understanding NumPy is essential because Pandas, Matplotlib, scikit-learn, and TensorFlow all build upon it.

Vectorized operations replace explicit loops with array-level expressions: instead of looping through elements to compute squares, you write arr**2. This works because NumPy delegates computation to optimized C/Fortran libraries (BLAS, LAPACK) that process entire arrays in compiled code. The performance difference is dramatic — a 1-million element operation that takes 1 second in a Python loop completes in 3 milliseconds with NumPy.

Broadcasting is NumPy's mechanism for performing operations on arrays of different shapes. When operating on two arrays, NumPy compares their shapes element-wise from the trailing dimensions: dimensions are compatible when they are equal, or when one of them is 1. This allows adding a 1D array (column means) to every row of a 2D array without explicit looping or tiling.

Fancy indexing uses arrays of indices or boolean masks to select subsets of data. Boolean indexing (arr[arr > 0]) selects elements meeting a condition. Integer indexing (arr[[0, 3, 7]]) selects specific elements by position. Combined with broadcasting, fancy indexing enables powerful one-line data filtering, replacement, and transformation operations.

Universal functions (ufuncs) are NumPy's element-wise operations: arithmetic (+, -, *, /), comparison (>, <, ==), trigonometric (sin, cos), logarithmic (log, log10), and statistical (sum, mean, std, min, max). Aggregation ufuncs accept an axis parameter for dimension-specific operations — arr.mean(axis=0) computes column means, arr.mean(axis=1) computes row means.

• NumPy arrays are 10-100x faster than Python lists for numerical operations
• Vectorization replaces loops with array-level expressions for massive speedups
• Broadcasting enables operations on arrays of different shapes
• Fancy indexing provides powerful data selection and filtering

• Create a 1000x1000 random matrix and compute column means, row sums, and overall statistics
• Time a Python loop vs NumPy vectorized operation on 1 million elements
• Use boolean indexing to replace all negative values in an array with zero
• Implement matrix multiplication using @ operator and compare to np.dot

• Harris, C.R. et al. (2020). Array programming with NumPy. Nature 585
• NumPy Documentation: https://numpy.org/doc/stable/

DataFrames, indexing, merging, groupby, time series

Pandas provides the DataFrame — a labeled, column-oriented data structure that has become the lingua franca of data manipulation in Python. Think of a DataFrame as a spreadsheet with superpowers: labeled rows and columns, mixed data types, missing value handling, and an expressive query language. Pandas is built on NumPy, inheriting its performance while adding the high-level abstractions that make tabular data manipulation intuitive.

Indexing in Pandas uses two primary accessors: .loc[] for label-based access and .iloc[] for integer-position access. This distinction prevents the ambiguity that arises when index labels are integers. Chained indexing (df["col"][0]) should be avoided because it creates copies rather than views, leading to the infamous SettingWithCopyWarning.

The split-apply-combine pattern, implemented through groupby(), is Pandas' most powerful analytical operation. Split the data by category (df.groupby("region")), apply a function to each group (.agg({"sales": "sum", "profit": "mean"})), and combine the results into a new DataFrame. This single operation replaces what would require nested loops, temporary variables, and manual aggregation in other languages.

Merging and joining combine DataFrames using shared keys, analogous to SQL JOIN operations. pd.merge() supports inner, left, right, and outer joins on one or more columns. pd.concat() stacks DataFrames vertically or horizontally. Understanding when to use merge vs concat vs join is essential for combining data from multiple sources without data loss or duplication.

Time series functionality is a Pandas strength. DatetimeIndex enables powerful slicing (df["2024-01":"2024-06"]), resampling (daily to monthly aggregation with .resample("M").mean()), and rolling window calculations (.rolling(30).mean() for 30-day moving averages). For financial data, Pandas handles business day calendars, period arithmetic, and timezone conversions.

• DataFrame is the central data structure for tabular data manipulation
• Use .loc[] for label-based and .iloc[] for position-based indexing
• groupby() implements the powerful split-apply-combine analytical pattern
• Pandas excels at time series analysis with resampling and rolling windows

• Load a CSV dataset and perform a complete EDA: describe, value_counts, groupby, pivot_table
• Merge two DataFrames on multiple keys and handle missing values from outer join
• Create a time series analysis with resampling, rolling average, and trend visualization
• Use groupby with multiple aggregation functions to produce a summary report

• Using chained indexing instead of .loc[]/.iloc[]
• Forgetting that most Pandas operations return new DataFrames (immutable by default)
• Not handling missing values (NaN) before aggregation
• Using apply() when vectorized operations are available

Line plots, scatter, histograms, 3D, customization

Matplotlib is Python's foundational visualization library, providing publication-quality figures in a wide variety of formats. While newer libraries like Seaborn, Plotly, and Altair offer higher-level APIs, understanding Matplotlib's object model is essential because all these libraries are built on top of it.

The object-oriented interface (fig, ax = plt.subplots()) provides explicit control over every element. The Figure contains one or more Axes (individual plots). Each Axes has an x-axis, y-axis, title, labels, and the plotted data. Multi-panel figures use plt.subplots(nrows, ncols) to create grids of Axes, enabling complex dashboard-like layouts in a single figure.

Customization ranges from simple (font sizes, colors, line styles) to complex (custom colormaps, annotations with arrows, inset axes, twin axes for dual y-scales). Style sheets (plt.style.use("seaborn-v0_8-whitegrid")) provide consistent aesthetics across an entire project. For publication, rcParams control global defaults for fonts, figure size, DPI, and line widths.

• Object-oriented interface provides full control over figure elements
• Style sheets ensure consistent aesthetics across projects
• Understanding Matplotlib is prerequisite for all Python visualization libraries

• Create a 2x2 subplot grid with line, scatter, bar, and histogram plots
• Customize a publication-quality figure with proper fonts, labels, and legend
• Create a dual y-axis plot comparing temperature and rainfall

Classification, regression, clustering, dimensionality reduction

Scikit-learn provides a consistent, well-documented API for the entire machine learning workflow: data preprocessing, model selection, training, evaluation, and prediction. Every estimator follows the same pattern: instantiate with hyperparameters, fit to training data (.fit(X_train, y_train)), and predict on new data (.predict(X_test)).

Supervised learning — learning from labeled examples — includes classification (predicting categories: spam vs not-spam, disease vs healthy) and regression (predicting continuous values: house prices, temperature). Key algorithms covered include: k-Nearest Neighbors (simple, intuitive baseline), Decision Trees (interpretable, handles mixed data types), Random Forests (ensemble of trees, robust and accurate), Support Vector Machines (optimal margin classifiers), and Gradient Boosting (state-of-the-art for tabular data).

Unsupervised learning — finding structure in unlabeled data — includes clustering (grouping similar observations: k-means, DBSCAN, hierarchical) and dimensionality reduction (PCA for visualization and noise reduction, t-SNE and UMAP for 2D embedding of high-dimensional data). These techniques are essential for exploratory analysis and feature engineering.

Model evaluation prevents overfitting through train/test splitting, cross-validation (k-fold CV rotates through training and validation splits), and metrics appropriate to the problem (accuracy, precision, recall, F1 for classification; MSE, MAE, R² for regression). The bias-variance tradeoff is the central tension — too simple models underfit, too complex models overfit.

Feature engineering — creating informative features from raw data — is often more impactful than algorithm selection. Techniques include: encoding categorical variables (one-hot, ordinal, target encoding), handling missing values (imputation, indicator variables), scaling numerical features (standardization, min-max), and creating interaction features. Scikit-learn Pipelines chain preprocessing and modeling into reproducible workflows.

• Scikit-learn provides a consistent API: fit → predict for all algorithms
• Model selection must account for bias-variance tradeoff through cross-validation
• Feature engineering often matters more than algorithm choice
• Pipelines ensure reproducible, leak-free ML workflows

• Build a complete classification pipeline: load data → preprocess → train → evaluate → visualize confusion matrix
• Compare 5 algorithms on the same dataset using cross-validation
• Apply PCA and t-SNE to visualize a high-dimensional dataset in 2D
• Create a Pipeline with StandardScaler, feature selection, and Random Forest