Introduction

This project analyzes Twitter data to uncover distinctions between how younger and older generations engage with the platform. The task requires participants to conduct preliminary data analysis, apply feature engineering, and utilize natural language processing (NLP) methods to examine differences in follower counts, discussion topics, sentiment, and language use across various age groups.

Dataset Description

The dataset consists of two main components:

Tweets: This file includes the following fields:

• user_id: ID of the user who posted the tweet.
• tweet_timestamp: Timestamp of when the tweet was posted.
• tweet_text: Text content of the tweet.

Users: This file contains the following fields:

• user_id: ID of the user.
• username: Username of the user on Twitter.
• user_age: Age of the user.
• followers: List of user IDs who follow this user.

Objectives

Data Exploration

• Examine the structure and contents of the JSON file.
• Analyze statistics such as the distribution of follower counts, tweet frequency, and demographics (age groups).

Feature Engineering

• Extract relevant features from the Twitter data, including:
  Follower count
  Tweet content
  Timestamps

Natural Language Processing (NLP) Analysis

Sentiment Analysis

• Utilize TextBlob for straightforward sentiment analysis.

Topic Modeling

• Use Latent Dirichlet Allocation (LDA) to identify common topics across tweets.
• Apply N-gram models to capture the context of words and improve sentiment analysis accuracy.
• Identify common topics or keywords used by users in each age group.

Visualization

• Create visualizations to illustrate findings and insights derived from the data analysis.
• Visualize trends in follower counts, sentiment, and topic distribution across different age groups.

Tools and Libraries Used

Pandas: For data manipulation and analysis.
NumPy: For numerical data processing.
Matplotlib: For creating static, animated, and interactive visualizations in Python.
Seaborn: For making statistical graphics.
NLTK: For natural language processing tasks.
Scikit-learn: For machine learning and predictive data analysis.
TextBlob: For text processing tasks and sentiment analysis.

Results and Discussion

Age Distribution of Tweets (Pie Chart):

A pie chart showing the percentage distribution of tweets based on age groups. The chart is divided into two sections:

• Young: 57.7%
• Old: 42.3%
  

Occurrences by Day of the Month (Bar Chart):

A bar chart representing the count of tweets occurring on different days of the month. It shows high activity on days like 1st, 5th, 7th, 15th, and 31st, with relatively lower activity on other days such as the 10th, 11th, and 25th.

Sentiment Distribution (Pie Chart):

This pie chart displays the distribution of sentiments in the tweets:

• Positive: 43.6%
• Neutral: 35.6%
• Negative: 20.8%

Sentiment Distribution for Young and Old Age Groups (Stacked Bar Chart):

A stacked bar chart showing the sentiment distribution (Positive, Neutral, Negative) across two age groups (young and old).
The young group shows more tweets with positive sentiment compared to the old group.
Both groups have a comparable distribution of neutral and negative sentiments, though the young group has higher counts overall.

Top 40 4-grams in Text Data (Bar Chart):

This chart lists the top 40 most frequent 4-grams (phrases of 4 consecutive words) used in the tweets. Examples of frequent 4-grams include:
"watch mtv music award"
"tonight show conan brien"
"michael john idol new"
The frequencies are displayed as bars, with some phrases being more frequent than others.

Top 40 4-grams in Text Data (Bar Chart):

Another bar chart representing the top 40 4-grams, but with different content compared to the previous one. Examples include:
"love spending time family"
"baby cousin graduation party"
"happy family never love"
The emphasis here is more on family and relationships.

These visualizations provide insights into how different age groups interact on Twitter, their sentiment distribution, and common topics of discussion based on word combinations.

Conclusion

This project provides a comprehensive analysis of Twitter data, revealing how user behavior, sentiments, and interests vary across different age groups. The insights from this project can help understand social media dynamics and can be useful for targeted marketing, sentiment analysis, and user engagement studies.