In this blog post, we will explore in depth the memory management and garbage collection (GC) mechanisms in Python. Understanding these concepts is crucial for writing efficient and reliable Python code, especially when dealing with large amounts of data and high-load applications. We will start with the basics of Python memory management, delve into the workings of garbage collection, and demonstrate through practical code examples how to optimize memory usage in Python programs.

## 1. Python's Memory Management Mechanisms

As a high-level scripting language that combines interpretive, compiled, interactive, and object-oriented aspects, Python is unique among programming languages. In Python, variables do not need to be declared in advance, nor do they require a specified type. Programmers do not need to manage memory, as the Python interpreter automates this with garbage collection. Developers need not be overly concerned with the memory management mechanisms, as these are fully handled by the Python memory manager.

Python's memory management can be broken down into several key components:

- **Memory Allocator**: Python uses its own internal memory allocator to manage the allocation of objects in heap memory.
- **Memory Pool**: Python provides a memory pool for small objects (especially integers and short strings), which are frequently created and destroyed. By reusing these objects, Python significantly reduces the overhead of memory allocations.

### 1. Why Introduce a Memory Pool?

One of the design goals of Python, a high-level programming language, is to simplify memory management. The introduction of memory pools primarily addresses performance issues caused by the frequent allocation and deallocation of small memory blocks. Without a memory pool, each allocation and recycling of a small object would require a system call involving the operating system's memory manager, which is very time-consuming. The use of memory pools offers several advantages:

- **Improved Memory Allocation Efficiency**: By reusing already allocated memory blocks, the number of system calls is reduced, thereby accelerating the speed of memory allocation.
- **Reduced Memory Fragmentation**: Frequent allocation and deallocation of small objects can easily lead to memory fragmentation. Memory pools mitigate this issue by managing a continuous area of memory.
- **Lowered Memory Management Overhead**: Centralized management of the allocation and release of a group of small objects reduces memory management overhead and improves the overall performance of the program.

### 2. How Does the Memory Pool Work?

The implementation of the memory pool in Python is managed through a system called "blocks." The memory pool in Python is primarily implemented by Python's interpreter-level memory manager, such as Pymalloc, which works as follows:

- **Creation of Memory Pools**: When Python starts, it pre-allocates a large block of memory, which is divided into multiple fixed-size blocks. Each block is typically a multiple of the memory required by small objects.
- **Allocation of Objects**: When a program creates a small object, the interpreter checks if there are enough free blocks in the memory pool. If so, it directly allocates a block to this object from the pool, instead of requesting new memory from the operating system.
- **Memory Recycling and Reuse**: When an object is no longer used, its occupied memory block is not immediately returned to the operating system but is marked as available and retained in the memory pool. This block can then be reallocated to new objects.

Python's memory management is layered, with different layers using different memory allocation mechanisms. Let's take a look at the memory architecture of CPython:

![python-memory-architecture](https://api.liamli.com/media/editor/python-memory-architechture_20240428214841329297.jpg)

Python's object management mainly occurs from Level+1 to Level+3.

Level+3: For Python's built-in objects (such as int, dict, etc.), each has a separate private memory pool, and the memory pools between objects are not shared, meaning the memory released by an int will not be allocated to a float.

Level+2: When the requested memory size is less than 256KB, memory allocation is mainly carried out by Python’s object allocator.

Level+1: When the requested memory size is greater than 256KB, allocation is carried out by Python's native memory allocator, essentially calling functions like malloc/realloc from the C standard library.

Let's look at some examples. For instance, Python uses a special optimization strategy for small integers and short strings. These objects are often created and destroyed frequently, so by using memory pool technology, Python pre-creates a range of these objects and continually reuses them, thereby avoiding frequent memory allocations and recycling.

### Example: Observing Reuse of Small Integer Objects

```python
# Python pre-creates and stores integers between -5 and 256 in the memory pool
a = 100
b = 100
print(a is b)  # Outputs True, indicating that both variables point to

 the same object in the memory pool

c = 300
d = 300
print(c is d)  # Outputs False, integers outside this range do not use the memory pool and are independently created
```

## 2. Python's Garbage Collection

Python's garbage collection (GC) is a key internal mechanism used to automatically free memory that is no longer in use. This mechanism primarily relies on reference counting to manage memory, supplemented by a more complex method—Generational Garbage Collection—to handle circular references and other issues that are difficult to resolve through reference counting alone. Below, we will detail these two mechanisms and their working principles.

### 1. Reference Counting

The most basic garbage collection technique in Python is reference counting. This is a simple and intuitive method of memory management, where:

- **Reference Counting Mechanism**: Each object has a reference counter that increases when the object is referenced and decreases when a reference is removed.
- **Memory Release**: When an object's reference count drops to zero, meaning no references are pointing to it, Python automatically frees the memory occupied by that object.

#### Flaws of Reference Counting

Despite its efficiency, reference counting has clear drawbacks. Its major problem is that it cannot handle circular references:

```python
a = []
b = [a]
a.append(b)

# At this point, a and b reference each other. Even if they are no longer referenced by other objects, their reference counts will not drop to zero.
```

To address the issue of circular references, Python introduced a second-tier garbage collection mechanism: Generational Recycling.

### 2. Generational Recycling

Generational recycling is an advanced garbage collection strategy based on the observation that in most programs, the lifetimes of objects vary greatly. Some objects are quickly no longer needed, while others may persist throughout the runtime of the program. Python's generational recycling places objects into three "generations" (commonly referred to as generation 0, 1, and 2), with newly created objects initially placed in generation 0.

- **Promotion of Objects**: Each time garbage collection occurs in a generation, surviving objects are moved to the next generation. Each generation has a lower frequency of garbage collection than the previous one, reducing the inspection frequency for long-lived objects and thereby increasing efficiency.
- **Detection of Circular References**: Python uses a "mark-sweep" algorithm to detect and recycle objects with circular references. During garbage collection, the garbage collector marks all objects that can be reached from the root set (such as global namespaces, call stacks, etc.). Unreachable objects are considered garbage.

#### Example Code: Manually Triggering Garbage Collection

```python
import gc

# Create objects
a = {}
b = {}
a['b'] = b
b['a'] = a

# Remove references
del a
del b

# Manually trigger garbage collection
collected = gc.collect()
print(f"Number of objects collected: {collected}")
print(f"Garbage collector: {gc.garbage}")
```

Discover Python's memory management and garbage collection strategies to optimize your Python applications

Python's Memory Management and Garbage Collection Mechanisms

When developing **IoT devices that require system data monitoring**, or **performing system maintenance tasks on Linux** and other operating systems, Python scripts are often used to simplify routine operations. This is particularly crucial in operational maintenance and development, where monitoring various hardware metrics of the system and detecting anomalies is of paramount importance. In Linux, there are many system commands, such as ps, top, free, etc., that allow for constant monitoring of the system's running state. To retrieve this information, Python can execute these commands using the subprocess module and parse the results. However, this approach can be cumbersome, especially when a lot of parsing code is needed.

In Python, there's an easier way to access system operational data: the ``psutil` module. Psutil stands for “process and system utilities”. It's a cross-platform library supporting Linux/UNIX/OSX/Windows and is an indispensable module for IoT development and operational work, enabling system monitoring with minimal code.

## Installing psutil

To install psutil on various operating systems, use the following command:

```shell
pip install psutil
```

(If you encounter a Permission Denied error, please retry with sudo.)

## Accessing Various System Information with Psutil

Psutil allows us to easily access various system information. Here are some examples of its common features:

### 1. Retrieving CPU Information

Using psutil to monitor CPU usage, core count, and other vital metrics is helpful.

```python
import psutil

# Retrieve complete CPU information
print(psutil.cpu_times())

# Get the number of logical CPUs
print(psutil.cpu_count())

# Monitor CPU usage over three seconds
# The interval=1 parameter indicates once per second
for i in range(3):
    print(psutil.cpu_percent(interval=1))
```

### 2. Retrieving Memory Information

Monitoring memory is crucial for ensuring efficient program execution.

```python
import psutil

# Retrieve complete memory information
print(psutil.virtual_memory())

# Get available memory specifically
print(psutil.virtual_memory().available)

# Retrieve swap memory information
print(psutil.swap_memory())
```

### 3. Retrieving Disk Information

Monitoring disk information helps prevent issues with insufficient storage space.

```python
import psutil

# Retrieve disk partition information
print(psutil.disk_partitions())

# Monitor disk usage
print(psutil.disk_usage('/'))

# Retrieve disk I/O
print(psutil.disk_io_counters())
```

### 4. Retrieving Network Information

Monitoring network information is useful for understanding data transmission and network status.

```python
import psutil

# Retrieve total network I/O information
print(psutil.net_io_counters())

# Get information for each network interface
print(psutil.net_if_addrs())

# Monitor network interface status
print(psutil.net_if_stats())
```

### 5. Retrieving System User Information

Understanding who is currently logged into the system.

```python
import psutil

# Retrieve user login information
print(psutil.users())
```

## Other Uses

Psutil is not limited to retrieving system information; it can also handle more complex system management tasks. Here are some detailed introductions and code examples of its advanced uses:

### Process Management

Psutil can list and manage processes within the system. It allows us to retrieve process information, terminate processes, or create new ones.

```python
import psutil

# List all process IDs
for proc in psutil.process_iter(['pid', 'name']):
    print(proc.info)

# Terminate a specific process (by process ID)
pid = 12345  # An assumed process ID
if psutil.pid_exists(pid):
    p = psutil.Process(pid)
    p.terminate()  # or use p.kill()
```

### System Reboot and Shutdown

Psutil can also be used to programmatically control system reboot or shutdown, which is especially useful in remote management of servers or IoT devices.

```python
import psutil

# Reboot the system
psutil.reboot()

# Shutdown the system
psutil.shutdown()
```

### Battery Information and Power Plug Status

For portable devices, monitoring battery status and power plug status is an important feature. Psutil can provide this information.

```python
import psutil

# Retrieve battery status
battery = psutil.sensors_battery()
print(f"Battery remaining: {battery.percent}%")
print(f"Power plugged in: {'Yes' if battery.power_plugged else 'No'}")
```

### Monitoring File System Changes

Psutil can be used to monitor changes in the file system, such as space usage and changes in mount points.

```python
import psutil

# Check disk usage
disk_usage = psutil.disk_usage('/')
print(f"Total space: {disk_usage.total / (1024**3):.2f} GB")
print(f"Used space: {disk_usage.used / (1024**3):.2f} GB")
print(f"Free space: {disk_usage.free / (1024**3):.2f} GB")
```

### Monitoring Network Connections

Psutil can also be used to monitor network connections, including current network connection information.

```python
import psutil

# Retrieve network connection information
connections = psutil.net_connections()
for conn in connections:
    print(conn)
```

> For more usage details and API documentation, please refer to the psutil official website: [psutil.readthedocs.io](https://psutil.readthedocs.io/en/latest/).

## Script Examples

1. **Routine Linux Maintenance Monitoring and Alerts**

```python
import psutil
import time
import smtplib
from email.mime.text import MIMEText
from email.header import Header
import logging

# Email sending settings
smtp_server = "smtp.example.com"  # SMTP server address
smtp_port = 587  # SMTP server port number
email_user = "your_email@example.com"  # Email username
email_pass = "your_password"  # Email password
receiver_email = "receiver@example.com"  # Receiver's email

# Logging configuration
logging.basicConfig(filename='system_monitor.log', level=logging.INFO,
                    format='%(asctime)s:%(levelname)s:%(message)s')

def send_email(subject, body):
    """ Send an email """
    message = MIMEText(body, 'plain', 'utf-8')
    message['From'] = Header("System Monitor", 'utf-8')
    message['To'] = Header("Admin", 'utf-8')
    message['Subject'] = Header(subject, 'utf-8')

    try:
        smtp = smtplib.SMTP(smtp_server, smtp_port)
        smtp.starttls()  # Enable secure transport mode
        smtp.login(email_user, email_pass)
        smtp.sendmail(email_user, receiver_email, message.as_string())
        smtp.quit()
        logging.info("Email sent successfully")
    except smtplib.SMTPException as e:
        logging.error("Email sending failed: " + str(e))

def check_system():
    """ Check system status and send warnings if necessary """
    if check_cpu_usage():
        warning = "Warning: High CPU usage!"
        logging.warning(warning)
        send_email("CPU Warning", warning)

    if check_memory_usage():
        warning = "Warning: High memory usage!"
        logging.warning(warning)
        send_email("Memory Warning", warning)

    if check_disk_usage():
        warning = "Warning: High disk usage!"
        logging.warning(warning)
        send_email("Disk Warning", warning)

def main():
    print("Starting system monitoring...")
    while True:
        check_system()
        network_data = check_network_activity()
        print(f"Current network activity: {network_data} bytes")
        time.sleep(5)

if __name__ == "__main__":
    main()

```

2.**IoT Device System Data Upload**

   If you're an IoT device developer needing to monitor system operation data for Linux-based IoT devices (like Raspberry Pi), regularly monitoring and uploading CPU and memory usage data is crucial. This script is ideal for scenarios where remote monitoring of hardware performance is needed, such as tracking the health or performance bottlenecks of IoT projects. It helps maintenance teams or developers promptly understand the device's operational status and make data-driven optimization or maintenance decisions.

   First, install the necessary libraries:

```shell
pip install psutil schedule requests
```

Then, here is the example script:

```python
import psutil
import schedule
import time
import requests
import logging
import json

# Logging configuration
logging.basicConfig(filename='iot_monitor.log', level=logging.INFO,
                    format='%(asctime)s:%(levelname)s:%(message)s')

# API setup (assumed API address and endpoint)
api_url = "http://example.com/api/upload"

def get_system_stats():
    """ Retrieve system's CPU and memory usage information """
    cpu_usage = psutil.cpu_percent()
    memory_usage = psutil.virtual_memory().percent
    return {"cpu": cpu_usage, "memory": memory_usage}

def upload_data():
    """ Upload collected data to the server """
    data = get_system_stats()
    try:
        response = requests.post(api_url, json=data)
        if response.status_code == 200:
            logging.info("Data uploaded successfully")
        else:
            logging.error(f"Data upload failed: {response.status_code}")
    except requests.exceptions.RequestException as e:
        logging.error(f"Request exception: {e}")

def job():
    """ Task to be executed periodically """
    logging.info("Starting data collection")
    upload_data()

def main():
    # Define a scheduled task: to be executed every 10 minutes
    schedule.every(10).minutes.do(job)

    while True:
        schedule.run_pending()
        time.sleep(1)

if __name__ == "__main__":
    main()

```

Uncover the power of Python's psutil for system and IoT device monitoring, complete with code examples and advanced uses.

Python

Python's Memory Management and Garbage Collection Mechanisms

Psutil - Python System Monitoring and Maintenance -