Repolode
Submit
mrpowers-io

mrpowers-io/quinn

PySpark helper functions for DataFrame validation, column transformations, and schema utilities to speed up Spark development.

View on GitHubWebsite
Stars
691
Forks
95
Open issues
25
Last commit
13 days ago

What is it?

Quinn is a Python library of helper methods for PySpark. It adds DataFrame validation functions that check for required or forbidden columns and schema fields, along with column functions for cleaning strings, handling null and truthy values, and computing week start and end dates.

The library also includes DataFrame transformations such as converting column names to snake_case, sorting columns alphabetically, and renaming many columns at once. Helpers convert columns or whole DataFrames into Python lists and dictionaries, and schema utilities read a schema from a CSV file or print an existing schema as Python code.

Quinn is installed from PyPI with pip and is documented on a project website. It is aimed at data engineers and developers who write PySpark jobs and want to avoid repeating common boilerplate.

Key features

  • 01Validates DataFrames for required columns, forbidden columns, and schema fields.
  • 02Provides string column functions such as whitespace removal and non-word character stripping.
  • 03Offers null-safe checks like is_null_or_blank and null_between.
  • 04Converts column names to snake_case and sorts columns alphabetically.
  • 05Builds PySpark schemas from CSV files and prints schemas as Python code.

Use cases

  • Validating incoming DataFrames before running a Spark transformation pipeline.
  • Standardizing messy column names and string values in data cleaning jobs.
  • Joining DataFrames with overlapping column names by renaming them with a shared suffix.
  • Defining Spark schemas in CSV files that non-programmers can edit.

Quick start

  1. 1

    Install Quinn from PyPI

    pip install quinn
  2. 2

    Import the library in Python

    import quinn

Things to know

  • Requires PySpark, so it is only useful for Spark-based projects.

Topics

#pyspark#apache-spark#data-validation#data-engineering#python#etl

FAQ

What is quinn?
PySpark helper functions for DataFrame validation, column transformations, and schema utilities to speed up Spark development. Quinn is a Python library of helper methods for PySpark. It adds DataFrame validation functions that check for required or forbidden columns and schema fields, along with column functions for cleaning strings, handling null and truthy values, and computing week start and end dates.
Is quinn free to use? What license does it have?
quinn is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is quinn still maintained?
Actively maintained. The last commit to quinn was on Oct 25, 2026.
What can I use quinn for?
Validating incoming DataFrames before running a Spark transformation pipeline. Standardizing messy column names and string values in data cleaning jobs. Joining DataFrames with overlapping column names by renaming them with a shared suffix. Defining Spark schemas in CSV files that non-programmers can edit.
How do I install quinn?
Install Quinn from PyPI: pip install quinn Import the library in Python: import quinn
Where can I get quinn?
The source code and installation instructions are on GitHub: https://github.com/mrpowers-io/quinn

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects