Building Skywalker: Automated QA via Classification & Clustering

A Fully Automated QA Analysis Engine Using Classification & Clustering

Origin Story — Built as a passion project while working as a Business Analyst leading a QA-intensive project for McGraw Hill's ALEKS e-learning platform. Skywalker was born out of the need to transform raw QA feedback into actionable intelligence — long before Neural Networks and LLMs were available to the general public.

[!NOTE] This repository serves as documentation only. No source code is included to respect intellectual property boundaries from a previous organization.

---

📖 Table of Contents

• Background & Motivation
• What is ALEKS?
• The Problem
• The Solution — Skywalker
• System Architecture
• Process Flowchart
• Core Features
• Dashboard Views
• Tech Stack
• Impact & Results
• Lessons Learned

---

🌌 Background & Motivation

As a Business Analyst leading a cross-functional project team, I was deeply involved in the end-to-end lifecycle of interactive e-learning content. The QA process — while critical — was manual, fragmented, and lacked structured analytics. Testers would identify issues, but the way those issues were communicated, categorized, and actioned varied wildly across teams.

I wanted a tool that could:

• Standardize QA outputs into meaningful categories
• Surface patterns in defect types across question banks
• Empower leadership with data-driven visibility into QA health
• Free up QA engineers to focus on testing, not on formatting reports

So I built Skywalker — a classification and clustering engine wrapped in an interactive dashboard — for fun, and because the problem was too interesting to leave unsolved.

---

📘 What is ALEKS?

ALEKS (Assessment and Learning in Knowledge Spaces) is an AI-powered adaptive learning platform developed by McGraw Hill. It serves over 50 million students across:

Subject	Description
Mathematics	K–12 through college-level courses
Chemistry	General and introductory chemistry
Statistics	Introductory statistics courses
Accounting	Foundational accounting concepts

ALEKS uses Knowledge Space Theory — a mathematical cognitive science framework — to map each student's knowledge state and deliver personalized learning paths. The platform avoids traditional multiple-choice formats, instead using sophisticated input tools that require authentic problem-solving.

Content at ALEKS

The content team at ALEKS was responsible for designing:

• 🎯 Dynamic questions — algorithmically generated with variable parameters
• 📚 Interactive question banks — curated pools for adaptive assessments
• 🧩 Interactive learning topics — step-by-step guided instruction with embedded practice

Each piece of content required rigorous QA to ensure mathematical correctness, pedagogical soundness, accessibility compliance, and seamless user experience.

---

🔴 The Problem

The QA process had several pain points:

┌─────────────────────────────────────────────────────────────────────┐
│                        QA Pain Points                               │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  1. INCONSISTENT CATEGORIZATION                                     │
│     → QA engineers described the same defect types differently      │
│                                                                     │
│  2. NO AGGREGATED VISIBILITY                                        │
│     → Leadership had no way to see trends across question banks     │
│                                                                     │
│  3. DEVELOPER FRICTION                                              │
│     → Developers struggled to interpret freeform QA feedback        │
│                                                                     │
│  4. QA TIME WASTED ON REPORTING                                     │
│     → Testers spent more time formatting reports than testing       │
│                                                                     │
│  5. NO FEEDBACK LOOP                                                │
│     → No systematic way to measure QA effectiveness over time       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

---

💡 The Solution — Skywalker

Skywalker is a QA Analysis Engine that ingests manually QA'd topic data, applies text classification and clustering algorithms to categorize defect findings, and presents the results through an interactive Plotly Dash dashboard.

How It Works (High Level)

1. QA engineers perform manual testing on ALEKS topics and log their findings
2. Skywalker ingests the raw QA data and preprocesses it (cleaning, tokenization, normalization)
3. Classification models categorize each QA finding into predefined defect types
4. Clustering algorithms discover emerging patterns and group similar issues
5. Interactive dashboards present insights to both leadership and development teams

---

🏗 System Architecture

graph TB
    subgraph Data_Sources["📥 Data Sources"]
        QA_Input["🧪 Manual QA Findings<br/>(Spreadsheets / Forms)"]
        Topic_DB["📚 ALEKS Topic Metadata<br/>(Question Banks, Subjects)"]
    end

    subgraph Ingestion_Layer["⚙️ Ingestion & Preprocessing"]
        Parser["📄 Data Parser<br/>(CSV / Excel Ingestion)"]
        Cleaner["🧹 Text Preprocessor<br/>(Tokenization, Stopwords,<br/>Lemmatization)"]
        Feature_Eng["🔧 Feature Engineering<br/>(TF-IDF Vectorization,<br/>N-gram Extraction)"]
    end

    subgraph ML_Engine["🤖 Skywalker ML Engine"]
        Classifier["📊 Classification Module<br/>(Naive Bayes / SVM /<br/>Random Forest)"]
        Clusterer["🔬 Clustering Module<br/>(K-Means / DBSCAN /<br/>Hierarchical)"]
        Taxonomy["🏷️ Defect Taxonomy<br/>(Category Mapping &<br/>Confidence Scoring)"]
    end

    subgraph Application_Layer["🖥️ Application Layer (Flask)"]
        Flask_App["🌐 Flask Web Server<br/>(REST API + Routing)"]
        Auth["🔐 Authentication<br/>(Role-Based Access)"]
        API["📡 API Endpoints<br/>(Data Retrieval &<br/>Filter Queries)"]
    end

    subgraph Dashboard_Layer["📈 Dashboard Layer (Plotly Dash)"]
        Lead_Dash["👔 Leadership Dashboard<br/>(Executive Summary,<br/>Trend Analysis)"]
        Dev_Dash["👨‍💻 Developer Dashboard<br/>(Defect Details,<br/>Priority Queue)"]
        QA_Dash["🧪 QA Dashboard<br/>(Coverage Metrics,<br/>Efficiency Stats)"]
    end

    subgraph Consumers["👥 End Users"]
        Leadership["👔 Leadership Team"]
        Developers["👨‍💻 Content Developers"]
        QA_Team["🧪 QA Engineers"]
    end

    QA_Input --> Parser
    Topic_DB --> Parser
    Parser --> Cleaner
    Cleaner --> Feature_Eng
    Feature_Eng --> Classifier
    Feature_Eng --> Clusterer
    Classifier --> Taxonomy
    Clusterer --> Taxonomy
    Taxonomy --> Flask_App
    Flask_App --> Auth
    Auth --> API
    API --> Lead_Dash
    API --> Dev_Dash
    API --> QA_Dash
    Lead_Dash --> Leadership
    Dev_Dash --> Developers
    QA_Dash --> QA_Team

    style Data_Sources fill:#1a1a2e,stroke:#e94560,color:#eee
    style Ingestion_Layer fill:#16213e,stroke:#0f3460,color:#eee
    style ML_Engine fill:#0f3460,stroke:#533483,color:#eee
    style Application_Layer fill:#533483,stroke:#e94560,color:#eee
    style Dashboard_Layer fill:#e94560,stroke:#f5c542,color:#fff
    style Consumers fill:#1a1a2e,stroke:#e94560,color:#eee

Architecture Components Explained

Layer	Component	Purpose
Data Sources	Manual QA Findings	Raw defect reports from QA engineers testing ALEKS topics
Data Sources	Topic Metadata	Subject area, question type, difficulty level, and content structure
Ingestion	Data Parser	Reads and normalizes input from CSVs, Excel files, or form submissions
Ingestion	Text Preprocessor	NLP pipeline — tokenization, stopword removal, lemmatization
Ingestion	Feature Engineering	Converts text to numerical features using TF-IDF and n-gram extraction
ML Engine	Classification Module	Assigns predefined defect categories using supervised learning
ML Engine	Clustering Module	Discovers emergent defect patterns using unsupervised learning
ML Engine	Defect Taxonomy	Maps ML outputs to a structured category hierarchy with confidence scores
Application	Flask Web Server	Core application server handling routing and business logic
Application	Authentication	Role-based access control for different user personas
Application	API Endpoints	RESTful endpoints for data retrieval, filtering, and aggregation
Dashboard	Leadership Dashboard	High-level KPIs, trend lines, and executive summaries
Dashboard	Developer Dashboard	Granular defect details, priority queues, and fix recommendations
Dashboard	QA Dashboard	Coverage metrics, efficiency tracking, and workload distribution

---

🔄 Process Flowchart

flowchart TD
    A([🧪 QA Engineer Tests<br/>ALEKS Topic]) --> B[📝 Log QA Findings<br/>in Standardized Form]
    B --> C{📥 Data Ingestion<br/>Triggered?}
    C -- Manual Upload --> D[📄 Parse CSV/Excel<br/>Input Files]
    C -- Scheduled Batch --> D
    D --> E[🧹 Text Preprocessing]

    E --> E1[Tokenization]
    E --> E2[Stopword Removal]
    E --> E3[Lemmatization]

    E1 --> F[🔧 Feature Engineering<br/>TF-IDF Vectorization]
    E2 --> F
    E3 --> F

    F --> G{🤖 ML Processing}
    G --> H[📊 Classification<br/>Assign Defect Category]
    G --> I[🔬 Clustering<br/>Discover Patterns]

    H --> J[🏷️ Apply Defect<br/>Taxonomy Labels]
    I --> J

    J --> K[💾 Store Categorized<br/>Results in Database]
    K --> L{👤 Who is<br/>Viewing?}

    L -- Leadership --> M[👔 Executive Dashboard]
    L -- Developer --> N[👨‍💻 Developer Dashboard]
    L -- QA Engineer --> O[🧪 QA Dashboard]

    M --> M1[📈 QA Health Trends]
    M --> M2[📊 Defect Distribution]
    M --> M3[⏱️ Cycle Time Analytics]

    N --> N1[🎯 Priority Defect Queue]
    N --> N2[🔍 Root Cause Clusters]
    N --> N3[📋 Actionable Fix List]

    O --> O1[✅ Coverage Metrics]
    O --> O2[📊 Efficiency Score]
    O --> O3[📝 Simplified Logging]

    M1 --> P([📊 Data-Driven<br/>Decision Making])
    M2 --> P
    M3 --> P
    N1 --> Q([🛠️ Faster<br/>Bug Resolution])
    N2 --> Q
    N3 --> Q
    O1 --> R([🚀 Improved<br/>QA Focus])
    O2 --> R
    O3 --> R

    style A fill:#e94560,stroke:#1a1a2e,color:#fff
    style G fill:#533483,stroke:#0f3460,color:#fff
    style J fill:#0f3460,stroke:#16213e,color:#fff
    style L fill:#f5c542,stroke:#1a1a2e,color:#1a1a2e
    style P fill:#2ecc71,stroke:#1a1a2e,color:#fff
    style Q fill:#2ecc71,stroke:#1a1a2e,color:#fff
    style R fill:#2ecc71,stroke:#1a1a2e,color:#fff

---

⚡ Core Features

1. Automated Defect Classification

Manually QA'd findings are automatically classified into structured defect categories using supervised ML models trained on historical QA data.

Defect Categories Include:

Category	Description	Example
MATH_ERROR	Incorrect mathematical computation or formula	Wrong coefficient in quadratic equation
DISPLAY_BUG	Rendering or visual presentation issues	Graph axes mislabeled
INPUT_TOOL	Student input tool malfunction	Drag-and-drop not responding
CONTENT_GAP	Missing or incomplete instructional content	Explanation step skipped
ACCESSIBILITY	ADA/WCAG compliance failures	Missing alt text on diagrams
LOGIC_FLAW	Incorrect branching or adaptive logic	Wrong prerequisite mapping
LOCALIZATION	Language or formatting inconsistencies	Decimal vs comma notation

2. Pattern Discovery via Clustering

Unsupervised clustering identifies emerging defect patterns that don't fit predefined categories — surfacing systemic issues before they become widespread.

3. Role-Based Interactive Dashboards

Dashboard	Audience	Key Metrics
Executive	Leadership	QA health score, defect trends over time, release readiness
Developer	Content Devs	Priority queue, root cause analysis, fix suggestions
QA Ops	QA Engineers	Coverage %, efficiency score, workload balance

4. Standardized QA Taxonomy

Eliminates inconsistency by mapping freeform QA notes to a controlled vocabulary — ensuring everyone speaks the same language about quality.

---

📊 Dashboard Views

Leadership Dashboard

┌──────────────────────────────────────────────────────────────────┐
│  SKYWALKER — Executive QA Overview                    🔄 Live    │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌────────────┐ │
│  │ QA Health  │  │   Topics   │  │  Avg Fix   │  │  Release   │ │
│  │   Score    │  │   Tested   │  │   Time     │  │ Readiness  │ │
│  │   94.2%    │  │   1,247    │  │  2.3 days  │  │    87%     │ │
│  │   ▲ +3.1%  │  │   ▲ +156   │  │   ▼ -0.8d  │  │   ▲ +5%   │ │
│  └────────────┘  └────────────┘  └────────────┘  └────────────┘ │
│                                                                  │
│  📈 Defect Trend (Last 6 Months)    📊 Category Distribution    │
│  ┌───────────────────────────┐      ┌──────────────────────────┐│
│  │    ╱╲                     │      │ ████████░░  Math Error   ││
│  │   ╱  ╲    ╱╲             │      │ ██████░░░░  Display Bug  ││
│  │  ╱    ╲  ╱  ╲  ╱╲       │      │ ████░░░░░░  Input Tool   ││
│  │ ╱      ╲╱    ╲╱  ╲──    │      │ ███░░░░░░░  Content Gap  ││
│  └───────────────────────────┘      │ ██░░░░░░░░  Accessibility││
│                                      └──────────────────────────┘│
└──────────────────────────────────────────────────────────────────┘

Developer Dashboard

┌──────────────────────────────────────────────────────────────────┐
│  SKYWALKER — Developer Focus View                     🔄 Live    │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  🎯 Priority Defect Queue                                        │
│  ┌──────────────────────────────────────────────────────────────┐│
│  │ #  │ Topic ID  │ Category     │ Cluster  │ Priority │ Age   ││
│  │ 1  │ ALG-2041  │ MATH_ERROR   │ Quad-Eq  │ 🔴 HIGH  │ 3d    ││
│  │ 2  │ CHM-0892  │ LOGIC_FLAW   │ Mol-Bal  │ 🔴 HIGH  │ 2d    ││
│  │ 3  │ ALG-1587  │ DISPLAY_BUG  │ Graph-Ax │ 🟡 MED   │ 5d    ││
│  │ 4  │ GEO-0341  │ INPUT_TOOL   │ DnD-Set  │ 🟡 MED   │ 4d    ││
│  │ 5  │ STA-0156  │ CONTENT_GAP  │ Prob-Dis │ 🟢 LOW   │ 7d    ││
│  └──────────────────────────────────────────────────────────────┘│
│                                                                  │
│  🔬 Emerging Clusters (Auto-Discovered)                          │
│  ┌──────────────────────────────────────────────────────────────┐│
│  │ • "Quadratic Coefficient" cluster — 12 related issues        ││
│  │ • "Graph Axis Labeling" cluster — 8 related issues           ││
│  │ • "Molecular Balance Input" cluster — 6 related issues       ││
│  └──────────────────────────────────────────────────────────────┘│
└──────────────────────────────────────────────────────────────────┘

---

🛠 Tech Stack

graph LR
    subgraph Frontend["🎨 Frontend"]
        Dash["Plotly Dash<br/>(Interactive Charts)"]
        HTML["HTML/CSS<br/>(Layout & Styling)"]
    end

    subgraph Backend["⚙️ Backend"]
        Flask["Flask<br/>(Web Framework)"]
        REST["REST API<br/>(Data Endpoints)"]
    end

    subgraph ML_Stack["🤖 ML / NLP"]
        Sklearn["scikit-learn<br/>(Classification &<br/>Clustering)"]
        NLTK["NLTK / spaCy<br/>(Text Processing)"]
        TFIDF["TF-IDF<br/>(Feature Extraction)"]
    end

    subgraph Data["💾 Data"]
        Pandas["Pandas<br/>(Data Wrangling)"]
        DB["SQLite / CSV<br/>(Storage)"]
    end

    Dash --> Flask
    HTML --> Flask
    Flask --> REST
    REST --> Sklearn
    REST --> Pandas
    Sklearn --> NLTK
    Sklearn --> TFIDF
    Pandas --> DB

    style Frontend fill:#e94560,stroke:#1a1a2e,color:#fff
    style Backend fill:#533483,stroke:#1a1a2e,color:#fff
    style ML_Stack fill:#0f3460,stroke:#1a1a2e,color:#fff
    style Data fill:#16213e,stroke:#1a1a2e,color:#eee

Layer	Technology	Role
Web Framework	Flask (Python)	Application server, routing, API endpoints
Dashboard	Plotly Dash	Interactive, real-time data visualizations
ML — Classification	scikit-learn (Naive Bayes, SVM, Random Forest)	Supervised defect categorization
ML — Clustering	scikit-learn (K-Means, DBSCAN)	Unsupervised pattern discovery
NLP	NLTK / spaCy	Tokenization, lemmatization, stopword removal
Feature Extraction	TF-IDF, N-grams	Text-to-numeric feature conversion
Data Processing	Pandas, NumPy	Data wrangling and transformation
Storage	SQLite / CSV	Lightweight data persistence
Visualization	Plotly, Matplotlib	Charts, graphs, and visual analytics

---

📈 Impact & Results

Metric	Before Skywalker	After Skywalker	Improvement
QA report formatting time	~40% of QA cycle	~5% of QA cycle	87.5% reduction
Defect categorization consistency	Variable (manual)	Standardized (ML)	Unified taxonomy
Leadership visibility	Ad-hoc email reports	Real-time dashboards	Instant access
Developer defect comprehension	Read raw QA notes	Structured priority queue	Faster resolution
Pattern detection	Post-mortem only	Real-time clustering	Proactive identification

Key Outcomes

• ✅ Reduced QA cycle time — QA engineers focused on testing, not report formatting
• ✅ Improved quality — Systematic categorization revealed blind spots in coverage
• ✅ Better developer focus — Priority queues directed developers to highest-impact fixes
• ✅ Leadership empowerment — Data-driven dashboards replaced anecdotal status updates
• ✅ Emerging pattern detection — Clustering surfaced systemic issues before they scaled

---

🧠 Lessons Learned

1. Pre-LLM NLP is powerful — TF-IDF + classical ML models delivered production-quality text classification without any deep learning infrastructure.

2. Domain expertise matters — The defect taxonomy was designed with input from QA leads, developers, and subject matter experts — not just ML metrics.

3. Dashboards change behavior — When leadership could see QA health in real-time, resource allocation and prioritization decisions improved dramatically.

4. QA is a data problem — Treating QA feedback as structured data (not just freeform notes) unlocked analytics that were previously impossible.

5. Build for all stakeholders — A single tool serving leadership, developers, and QA with role-appropriate views created alignment across teams.

---

📜 Disclaimer

This project was designed and conceptualized during employment at a previous organization. No proprietary source code, data, or trade secrets are included in this repository. All content here represents conceptual architecture, methodology documentation, and general technical approaches that reflect the author's individual skills and problem-solving abilities.

---

Built with ❤️ and curiosity — because the best tools are born from real problems.

_{Skywalker — QA Analysis Engine | Classification & Clustering | Flask + Plotly Dash}