Python for Data Analysis: Essential Libraries Guide

🏗️ Array Creation and Basic Operations

import numpy as np

# Creating arrays
arr_1d = np.array([1, 2, 3, 4, 5])
arr_2d = np.array([[1, 2, 3], [4, 5, 6]])
zeros = np.zeros((3, 4))
ones = np.ones((2, 3))
range_arr = np.arange(0, 10, 2)  # [0, 2, 4, 6, 8]
linear_space = np.linspace(0, 1, 5)  # 5 equally spaced points

• np.array(): Convert lists/tuples to NumPy arrays
• np.zeros(): Create arrays filled with zeros
• np.ones(): Create arrays filled with ones
• np.arange(): Create sequences with step size
• np.linspace(): Create evenly spaced numbers

🔧 Array Manipulation and Indexing

# Array manipulation
arr = np.array([1, 2, 3, 4, 5, 6])
reshaped = arr.reshape(2, 3)  # Convert to 2x3 matrix
flattened = reshaped.flatten()  # Convert back to 1D

# Indexing and slicing
first_element = arr[0]  # First element
last_three = arr[-3:]   # Last three elements
subset = arr[arr > 3]   # Elements greater than 3

# Boolean indexing
data = np.array([1, 2, 3, 4, 5])
mask = data > 2
filtered_data = data[mask]  # [3, 4, 5]

• reshape(): Change array dimensions without copying data
• flatten(): Convert multi-dimensional array to 1D
• Boolean indexing: Filter arrays based on conditions
• Slicing: Extract portions of arrays efficiently

⚡ Mathematical Operations

# Element-wise operations
a = np.array([1, 2, 3, 4])
b = np.array([2, 3, 4, 5])

addition = a + b      # [3, 5, 7, 9]
multiplication = a * b # [2, 6, 12, 20]
power = a ** 2        # [1, 4, 9, 16]

# Statistical operations
mean_val = np.mean(a)     # 2.5
std_val = np.std(a)       # 1.29
sum_val = np.sum(a)       # 10
max_val = np.max(a)       # 4
min_val = np.min(a)       # 1

# Linear algebra
matrix_a = np.array([[1, 2], [3, 4]])
matrix_b = np.array([[5, 6], [7, 8]])
dot_product = np.dot(matrix_a, matrix_b)  # Matrix multiplication

• Vectorized operations: Apply operations to entire arrays at once
• Statistical functions: Built-in functions for descriptive statistics
• Linear algebra: Matrix operations and decompositions
• Broadcasting: Perform operations on arrays of different shapes

📊 DataFrames and Series

import pandas as pd

# Creating DataFrames
data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],
    'Age': [25, 30, 35, 28],
    'City': ['New York', 'London', 'Tokyo', 'Paris'],
    'Salary': [70000, 80000, 90000, 75000]
}
df = pd.DataFrame(data)

# Creating Series
ages = pd.Series([25, 30, 35, 28], name='Age')

# Reading from files
df_csv = pd.read_csv('data.csv')
df_excel = pd.read_excel('data.xlsx')
df_json = pd.read_json('data.json')

• DataFrame: 2D labeled data structure with columns of different types
• Series: 1D labeled array, similar to a column in a spreadsheet
• Multiple formats: Read from CSV, Excel, JSON, SQL, and more

🔍 Data Exploration and Inspection

# Basic information about the dataset
df.head()          # First 5 rows
df.tail()          # Last 5 rows
df.info()          # Data types and non-null counts
df.describe()      # Summary statistics
df.shape           # (rows, columns)
df.columns         # Column names
df.dtypes          # Data types of each column

# Checking for missing values
df.isnull().sum()          # Count of null values per column
df.duplicated().sum()      # Count of duplicate rows

# Unique values and value counts
df['City'].unique()        # Unique values in City column
df['City'].value_counts()  # Count of each unique value

• Quick overview: Understand your data structure and quality
• Missing data detection: Identify data quality issues
• Statistical summary: Get descriptive statistics instantly

⚙️ Data Manipulation and Cleaning

# Selecting data
df['Name']                    # Select single column
df[['Name', 'Age']]          # Select multiple columns
df[df['Age'] > 30]           # Filter rows based on condition
df.loc[0:2, 'Name':'Age']    # Select by label
df.iloc[0:3, 0:2]            # Select by position

# Adding and modifying columns
df['Age_Group'] = df['Age'].apply(lambda x: 'Young' if x < 30 else 'Senior')
df['Salary_K'] = df['Salary'] / 1000

# Handling missing values
df.dropna()                  # Remove rows with missing values
df.fillna(df.mean())         # Fill missing values with mean
df['Age'].fillna(df['Age'].median(), inplace=True)

# Grouping and aggregation
grouped = df.groupby('City')['Salary'].mean()
df.groupby('Age_Group').agg({'Salary': ['mean', 'max', 'min']})

• Flexible selection: Multiple ways to slice and filter data
• Data transformation: Create new columns and modify existing ones
• Grouping operations: Aggregate data by categories
• Missing data handling: Various strategies for incomplete data

📊 Basic Plot Types

import matplotlib.pyplot as plt
import numpy as np

# Line plot
x = np.linspace(0, 10, 100)
y = np.sin(x)
plt.plot(x, y, label='sin(x)', linewidth=2)
plt.xlabel('X axis')
plt.ylabel('Y axis')
plt.title('Sine Wave')
plt.legend()
plt.grid(True)
plt.show()

# Bar plot
categories = ['A', 'B', 'C', 'D']
values = [23, 45, 56, 78]
plt.bar(categories, values, color='skyblue')
plt.title('Bar Chart Example')

# Histogram
data = np.random.normal(100, 15, 1000)
plt.hist(data, bins=30, alpha=0.7, color='green')
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.title('Normal Distribution Histogram')

• Line plots: Perfect for time series and continuous data
• Bar charts: Compare categorical data effectively
• Histograms: Visualize data distributions
• Customization: Control colors, labels, and styling

🎨 Advanced Visualizations

# Scatter plot with customization
plt.figure(figsize=(10, 6))
plt.scatter(df['Age'], df['Salary'], 
           c=df['Salary'], cmap='viridis', 
           s=100, alpha=0.7)
plt.colorbar(label='Salary')
plt.xlabel('Age')
plt.ylabel('Salary')
plt.title('Age vs Salary')

# Subplots
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
axes[0,0].plot(x, y)
axes[0,1].bar(categories, values)
axes[1,0].hist(data, bins=20)
axes[1,1].scatter(df['Age'], df['Salary'])
plt.tight_layout()

# Customizing appearance
plt.style.use('seaborn-v0_8')  # Apply seaborn style
plt.rcParams['figure.figsize'] = (10, 6)
plt.rcParams['font.size'] = 12

• Scatter plots: Explore relationships between variables
• Subplots: Create multiple plots in one figure
• Styling: Use built-in styles or create custom themes
• Color maps: Add dimensional information through color

# Complete workflow example: Sales data analysis
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# 1. Data Loading and Exploration
df = pd.read_csv('sales_data.csv')
print(f"Dataset shape: {df.shape}")
print(f"Missing values: {df.isnull().sum().sum()}")

# 2. Data Cleaning
df['Date'] = pd.to_datetime(df['Date'])
df = df.dropna()
df['Month'] = df['Date'].dt.month
df['Revenue'] = df['Quantity'] * df['Price']

# 3. Data Analysis with NumPy
monthly_revenue = df.groupby('Month')['Revenue'].sum()
growth_rate = np.diff(monthly_revenue) / monthly_revenue[:-1] * 100

# 4. Advanced Pandas Operations
top_products = df.groupby('Product').agg({
    'Quantity': 'sum',
    'Revenue': 'sum',
    'Price': 'mean'
}).sort_values('Revenue', ascending=False)

# 5. Visualization
fig, axes = plt.subplots(2, 2, figsize=(15, 10))

# Monthly revenue trend
axes[0,0].plot(monthly_revenue.index, monthly_revenue.values, 
               marker='o', linewidth=2)
axes[0,0].set_title('Monthly Revenue Trend')
axes[0,0].set_xlabel('Month')
axes[0,0].set_ylabel('Revenue ($)')

# Top products by revenue
top_5_products = top_products.head(5)
axes[0,1].bar(range(len(top_5_products)), top_5_products['Revenue'])
axes[0,1].set_title('Top 5 Products by Revenue')
axes[0,1].set_xticks(range(len(top_5_products)))
axes[0,1].set_xticklabels(top_5_products.index, rotation=45)

# Revenue distribution
axes[1,0].hist(df['Revenue'], bins=30, alpha=0.7, color='green')
axes[1,0].set_title('Revenue Distribution')
axes[1,0].set_xlabel('Revenue ($)')
axes[1,0].set_ylabel('Frequency')

# Quantity vs Revenue scatter
axes[1,1].scatter(df['Quantity'], df['Revenue'], alpha=0.6)
axes[1,1].set_title('Quantity vs Revenue')
axes[1,1].set_xlabel('Quantity')
axes[1,1].set_ylabel('Revenue ($)')

plt.tight_layout()
plt.show()

# 6. Summary Statistics
print(f"Total Revenue: ${df['Revenue'].sum():,.2f}")
print(f"Average Order Value: ${df['Revenue'].mean():.2f}")
print(f"Best Month: {monthly_revenue.idxmax()} with ${monthly_revenue.max():,.2f}")

⚡ Performance Optimization

• Vectorization: Use NumPy operations instead of loops
• Memory efficiency: Use appropriate data types
• Chunking: Process large datasets in chunks
• Index optimization: Set appropriate indexes in Pandas
• Avoid copying: Use views when possible

✅ Best Practices

• Data validation: Always check data quality first
• Reproducible analysis: Set random seeds
• Clear documentation: Comment your analysis steps
• Version control: Track changes to your analysis
• Modular code: Break analysis into functions

Memory Issues with Large Datasets

Loading entire datasets into memory can cause crashes.

Solution: Use pd.read_csv(chunksize=1000) to process data in chunks, or consider using Dask for larger-than-memory datasets.

Chain Assignment Warnings

Using chained assignments like df[df['A'] > 0]['B'] = 1 can be unreliable.

Solution: Use df.loc[df['A'] > 0, 'B'] = 1 for explicit indexing.

Inefficient Looping

Using Python loops on DataFrames is extremely slow.

Solution: Use vectorized operations: df['new_col'] = df['col1'] + df['col2'] instead of looping through rows.

Keep Practicing

Work on real datasets to reinforce learning

Build Projects

Apply these skills to solve actual problems

Expand Toolkit

Learn additional libraries as needed