# Repartition

> Redistribute data across multiple partitions for improved parallelism and performance.

Source: https://dagster.io/glossary/data-repartitioning

## Data repartitioning definition:

Data repartitioning is the process of redistributing data across a cluster to achieve a more optimal data partitioning. It is a critical operation for optimizing performance in distributed computing environments such as Apache Spark. Repartitioning can help improve data locality, reduce data skew, and balance computational loads across worker nodes.

Note that repartitioning can be an expensive operation, as it involves shuffling data across the network. It should be used judiciously and only when necessary for performance optimization.

## Data repartitioning example using Python and PySpark:

Please note that you need to [have the necessary Python libraries installed in your Python environment](https://dagster.io/glossary) to run this code.

In Apache Spark, data repartitioning can be achieved using the `repartition()` or `coalesce()` method on a DataFrame or RDD. The `repartition()` method shuffles the data and creates a new set of partitions with a specified number. The `coalesce()` method reduces the number of partitions without shuffling the data.

Here's an example of using `repartition()` to increase the number of partitions of a DataFrame in Apache Spark:

```python
from pyspark.sql import SparkSession

# create SparkSession
spark = SparkSession.builder.appName("RepartitioningExample").getOrCreate()

# read input data
df = spark.read.csv("input_data.csv", header=True, inferSchema=True)

# check current number of partitions
print("Current number of partitions:", df.rdd.getNumPartitions())

# repartition to 4 partitions
df = df.repartition(4)

# check new number of partitions
print("New number of partitions:", df.rdd.getNumPartitions())

# write output data
df.write.csv("output_data", mode="overwrite", header=True)
```

In this example, the input data is read from a CSV file, and the `getNumPartitions()` method is used to check the current number of partitions. The `repartition()` method is then used to create a new DataFrame with 4 partitions, and the `getNumPartitions()` method is used again to check the new number of partitions. Finally, the output data is written to a CSV file.

It will print to the terminal:

```plaintext
Current number of partitions: 1
New number of partitions: 4
```

And will save the results to a folder called `` `output_data ``.\`

When working with data pipelines at scale, you will be defining partitioning rules in code, and letting the orchestrator manage the partitions and backfills.  You can find the details on Dagster’s [partitioning API here](https://docs.dagster.io/api/dagster/partitions).