Back to Glossary Index


Convert data into a linear format for efficient storage and processing.

Data fragmentation definition:

Data fragmentation refers to the breaking down of data into smaller chunks or fragments for storage and management purposes. In modern data engineering, data fragmentation is often used to optimize data processing and storage, as well as to improve data availability and scalability.

Fragmentation can be applied to various types of data, such as files, databases, or objects, and can be done in various ways, such as horizontal fragmentation (splitting data by rows), vertical fragmentation (splitting data by columns), or hybrid fragmentation (a combination of horizontal and vertical fragmentation).

Data fragmentation example using Python:

Please note that you need to have the Pandas library installed in your Python environment to run the following code examples.

Here's a self-contained Python example that demonstrates horizontal fragmentation of a pandas DataFrame:

import pandas as pd

# Create a sample DataFrame with 10 rows and 3 columns
df = pd.DataFrame({'A': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
                   'B': ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'],
                   'C': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]})

# Split the DataFrame into two fragments: one with rows 0 to 4 and one with rows 5 to 9
df1 = df.iloc[:5]
df2 = df.iloc[5:]

# Print the original DataFrame and the two fragments
print('Original DataFrame:')
print('Fragment 1:')
print('Fragment 2:')

This will yield the following output:

Original DataFrame:
    A  B    C
0   1  a  0.1
1   2  b  0.2
2   3  c  0.3
3   4  d  0.4
4   5  e  0.5
5   6  f  0.6
6   7  g  0.7
7   8  h  0.8
8   9  i  0.9
9  10  j  1.0
Fragment 1:
   A  B    C
0  1  a  0.1
1  2  b  0.2
2  3  c  0.3
3  4  d  0.4
4  5  e  0.5
Fragment 2:
    A  B    C
5   6  f  0.6
6   7  g  0.7
7   8  h  0.8
8   9  i  0.9
9  10  j  1.0

In this example, we create a sample DataFrame with 10 rows and 3 columns. We then split the DataFrame into two fragments using the iloc method of pandas: df1 contains rows 0 to 4, and df2 contains rows 5 to 9. Finally, we print the original DataFrame and the two fragments to verify that the data has been correctly fragmented. This example demonstrates horizontal fragmentation, as we split the data by rows.

In practice, data fragmentation can be used in a variety of data engineering scenarios, such as distributed computing, parallel processing, or database sharding, to name a few. By breaking down data into smaller, more manageable fragments, we can distribute data processing and storage more efficiently, reduce data access times, and improve overall system performance and scalability.

Other data engineering terms related to
Data Transformation:


Aligning data can mean one of three things: aligning datasets, meeting business rules or arranging data elements in memory.

Big Data Processing

Process large volumes of data in parallel and distributed computing environments to improve performance.

Clean or Cleanse

Remove invalid or inconsistent data values, such as empty fields or outliers.


Group data points based on similarities or patterns to facilitate analysis and modeling.


Remove noise or artifacts from data to improve its accuracy and quality.


Optimize data for faster read access by reducing the number of joins needed to retrieve related data.


Transform continuous data into discrete categories or bins to simplify analysis.


Extract, transform, and load data between different systems.


Extract a subset of data based on specific criteria or conditions.


Fill in missing data values with estimated or imputed values to facilitate analysis.


See 'wrangle'.


Standardize data values to facilitate comparison and analysis. organize data into a consistent format.


Convert a large set of data into a smaller, more manageable form without significant loss of information.


Change the structure of data to better fit specific analysis or modeling requirements.


Convert data into a linear format for efficient storage and processing.


Break down large datasets into smaller, more manageable pieces for easier processing and analysis.


An imbalance in the distribution or representation of data.


Transform data to a common unit or format to facilitate comparison and analysis.


Convert data into tokens or smaller units to simplify analysis or processing.


Convert data from one format or structure to another.


Convert unstructured data into a structured format.