![homepage](https://raw.githubusercontent.com/hericlibong/datascope_project/main/medias/home_picture.png) (_Tip_: This would show the "Analyze Article" landing page to give users a visual entry point.) --- --DIVIDER--# 📊 DataScope Detect the Hidden "Data Potential" in News Articles — NLP meets LLM ## Project Overview **DataScope** is a lightweight web application designed to help journalists and media organizations analyze the factual richness of a news article and identify opportunities for data-driven storytelling. **⚡ This is a Minimum Viable Product (MVP)** version, built with Flask, combining local NLP (spaCy) and Generative AI (OpenAI) to deliver an accessible tool even for non-technical users. It focuses on making advanced text analysis intuitive for small newsrooms, freelance journalists, and journalism schools. --- --DIVIDER--## How it works ```mermaid flowchart TD; A[User Input:
Paste or Upload .txt, .docx, .pdf] --> B[Language Selection:
English 🇬🇧 or French 🇫🇷]; B --> C[NLP Analysis with spaCy:
• Named Entities
• Numbers/Units
• Dates
• Strong Verbs]; C --> D[Datafication Scoring:
• Score out of 10
• Editorial Profile]; D --> E[Prompt Generation:
• Build Custom Prompt for LLM]; E --> F[LLM Call OpenAI API:
• GPT-3.5/4]; F --> G1[Editorial Angles:
• 3 Suggestions]; F --> G2[Suggested Datasets:
• 5 Data Sources]; G1 --> H[Flask Server:
Display Results via Jinja2 Template]; G2 --> H; H --> I[Optional:
Export Analysis to Markdown .md]; ``` -----DIVIDER-- ## Key Features - **Smart Entity Extraction**: Names, numbers, dates, strong action verbs. - **Datafication Scoring**: Quantifies the “data structure density” of the article. - **AI-Powered Suggestions**: Editorial angles and useful datasets. - **Multilingual Support**: Full English 🇬🇧 and French 🇫🇷 interface. - **Markdown Export**: Download complete analysis in markdown format. - **Feedback System**: Collect user insights for continuous improvement. - **Mini Admin Interface**: Manage feedbacks and users easily. -----DIVIDER-- ## Tech Stack | Component | Usage | |:-----------------|:------------------------| | **Python 3.12** | Backend logic | | **Flask** | Web framework | | **spaCy** | NLP entity extraction | | **OpenAI API** | Generative editorial suggestions | | **Langdetect** | Language detection | | **Docker** | Optional containerization | | **Render** | Cloud deployment | -----DIVIDER--## Dataset Source and Collection DataScope does not rely on a pre-existing dataset. Instead, it analyzes user-submitted articles in `.txt`, `.pdf`, or `.docx` formats. All entity extraction and analysis are performed locally on the provided text. This ensures maximum flexibility and respects the originality of each input without requiring dataset collection or storage.. -----DIVIDER--## Current State Gap Identification While Natural Language Processing (NLP) techniques are widely used for large-scale media analysis or structured data mining, most existing tools require significant technical expertise or are primarily designed to process datasets, not stories. In the field of data journalism, traditional workflows typically start with datasets and then seek stories ("data first"). However, many journalistic stories — especially local, investigative, or narrative articles — are not initially structured as data. DataScope reverses this approach: it starts with unstructured texts and helps identify their "data potential." By combining lightweight NLP entity extraction and LLM-powered editorial suggestions, it assists journalists in exploring whether a story can be expanded, quantified, or supported with open data sources. This makes DataScope particularly valuable for small newsrooms, freelance journalists, and educational contexts, where resources for technical data analysis are often limited. It aims to democratize advanced text exploration without requiring deep technical expertise. -----DIVIDER--## Evaluation Summary Key Evaluation Criteria for DataScope: - Entity Extraction Relevance: Manual review of named entities and key elements extracted. - Editorial Angle Usefulness: User feedback on the relevance of suggested editorial angles (rating system). - Open Data Accuracy: Verification of the relevance of suggested datasets. - Global User Satisfaction: Optional feedback form (1 to 5 stars) after each analysis. Validation Approach: Cross-check outputs with expert manual analyses on sample articles and collect real user feedback to measure utility and accuracy. -----DIVIDER-- ## Current Limitations - **Language Detection Simplicity**: Language detection relies on lightweight models and may occasionally misclassify ambiguous texts. - **LLM Variability**: Editorial angle suggestions depend on LLM behavior, which may vary slightly across requests. - **Entity Extraction Coverage**: Entity extraction is based on basic SpaCy models (`en_core_web_sm`, `fr_core_news_sm`), without fine-tuning on specialized journalistic corpora. --- --DIVIDER-- ## Future Development Directions DataScope is currently a **Minimal Viable Product (MVP)** focused on journalistic usability. Several improvements are envisioned to strengthen its capabilities: - **Enhanced Multilingual Support**: Extending analysis beyond French and English, integrating more languages with adapted NLP pipelines. - **Editorial Dashboard**: Allowing users to manage multiple analyses, track article scores, and visualize progression over time. - **Data Visualization Layer**: Generating automatic charts (e.g., entity frequency, date timelines) from NLP outputs to support data storytelling. - **Input Metadata Enrichment**: Adding fields such as **Article Title**, **Author Name**, and **Publication Date** for better contextualization of analyses. - **Improved LLM Suggestion Precision**: Refining prompt engineering techniques to generate more targeted and differentiated editorial angles. **Global Characterization**: _"Toward a complete lightweight data storytelling assistant for journalists."_ --- --DIVIDER--## Industry Context With the growing importance of **data journalism** and **AI-assisted writing** in media organizations, there is a strong demand for **accessible, lightweight tools** that empower journalists to extract, interpret, and enhance data-driven narratives. DataScope positions itself at the intersection of NLP democratization and editorial innovation. -----DIVIDER-- ## Target Audience - Journalists and media outlets - Investigative reporters - Journalism schools - NGOs working with open data - Freelance writers -----DIVIDER-- ## Live Demo & Installation - 🔗 **App URL**: [https://datascope.onrender.com/](https://datascope.onrender.com/) - 🔗 **GitHub Repo**: [https://github.com/hericlibong/datascope_project](https://github.com/hericlibong/datascope_project) -----DIVIDER--## 📜 License MIT License — open for collaboration and improvements. "MIT License — open for public collaboration and commercial adaptation with proper attribution." -----DIVIDER-- # Why Share on Ready Tensor? - Practical application of **NLP + LLM** for non-technical users. - Bridge between **journalism expertise** and **AI capabilities**. - Showcase an **accessible, real-world use case** for AI in media. -----DIVIDER-- ## ✉️ Notes > "DataScope provides suggestions and inspiration but leaves final editorial judgment to human journalists."