Abstract

Processing input data for AI models is challenging. DupliPy aims to simplify this process by providing a quick and easy-to-use package for text formatting, data augmentation tasks for NLP, and image augmentation tasks in Python.

Methodology

DupliPy offers a range of functions to streamline data preprocessing and augmentation. For NLP tasks, methods are included for formatting text, replication, sentiment analysis, and similarity calculations. Additionally, it features random cropping, rotations, and flipping for image augmentation.

Key features include:

• Text Formatting: Remove special characters, normalize whitespace, and standardize text.
• Text Replication: Augment entire text datasets using random synonym replacement and paraphrasing.
• Sentiment Analysis: Analyze the sentiment within sentences.
• Similarity Calculation: Calculate similarities between text.
• Image Augmentation: Perform random crop, rotation, and flipping tasks.
• Profanity Removal: Remove profanity or hate speech from input text using predefined sets, and AI models.

Results

DupliPy has proven to be an effective tool for NLP and image augmentation tasks. It supports multiple Python versions and integrates seamlessly with other Python packages. With over 15k downloads from PyPI in the last year, DupliPy has proven to be an effective tool for streamlining development workflows.

For more details, check out DupliPy's GitHub repository, or the package on PyPI.