readme docs

2025-12-05 18:46:44 -05:00 · 2025-12-02 07:21:23 -05:00
parent 7fb24fdde3
commit d7ae362c23
3 changed files with 403 additions and 57 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,67 @@
+# Ballistic Backend  
+### Internal Engine for the Builder Ecosystem
+
+The Ballistic Backend is the operational backbone behind the Shadown System Armory and its admin tools. It ingests merchant feeds, normalizes product data, manages categories, synchronizes prices, and powers the compatibility, pricing, and product logic behind the consumer-facing Builder.
+
+It’s built for reliability, longevity, and clean extensibility — the kind of foundation you want when scaling from a small beta to a fully public platform.
+
+---
+
+## What This Backend Does
+
+### **Merchant Feed Ingestion**
+- Pulls AvantLink feeds (CSV or TSV)
+- Automatically detects delimiters
+- Normalizes raw merchant fields
+- Creates or updates product records
+- Upserts price and stock offers
+- Tracks first-seen / last-seen timestamps
+- Safely handles malformed or incomplete rows  
+- Ensures repeat imports never duplicate offers
+
+### **Category Mapping Engine**
+- Identifies every unique raw category coming from each merchant feed  
+- Exposes *unmapped* categories in the admin UI  
+- Allows you to assign:  
+  - Part Role  
+  - Product Configuration (Stripped, Complete, Kit, etc.)
+- Applies mappings automatically on future imports  
+- Respects manual overrides such as `platform_locked`
+
+### **Builder Support**
+The frontend Builder depends on this backend for:
+
+- Loading parts grouped by role  
+- Offering compatible options  
+- Calculating build cost  
+- Comparing offers across merchants  
+- Providing product metadata, imagery, and offer data  
+
+**Future expansions include:** price history, compatibility engine, build exports, public build pages, multi-merchant aggregation, and automated anomaly detection.
+
+---
+
+## Tech Stack
+
+- **Spring Boot 3.x**  
+- **Java 17**  
+- **PostgreSQL**  
+- **Hibernate (JPA)**  
+- **HikariCP**  
+- **Apache Commons CSV**  
+- **Maven**  
+- **REST API**
+
+---
+
+## Local Development
+
+### Requirements
+- Java 17 or newer  
+- PostgreSQL running locally  
+- Port 8080 open (default backend port)
+
+### Run Development Server
+
+```bash
+./mvnw spring-boot:run
--- a/importLogic.md
+++ b/importLogic.md
@@ -0,0 +1,213 @@
+# Ballistic Import Pipeline  
+A high-level overview of how merchant data flows through the Spring ETL system.
+
+---
+
+## Purpose
+
+This document explains how the Ballistic backend:
+
+1. Fetches merchant product feeds (CSV/TSV)
+2. Normalizes raw data into structured entities  
+3. Updates products and offers in an idempotent way  
+4. Supports two sync modes:
+   - Full Import
+   - Offer-Only Sync
+
+---
+
+# 1. High-Level Flow
+
+## ASCII Diagram
+
+```
+                         ┌──────────────────────────┐
+                         │   /admin/imports/{id}     │
+                         │  (Full Import Trigger)    │
+                         └─────────────┬────────────┘
+                                       │
+                                       ▼
+                        ┌──────────────────────────────┐
+                        │ importMerchantFeed(merchantId)│
+                        └─────────────┬────────────────┘
+                                      │
+                                      ▼
+           ┌────────────────────────────────────────────────────────┐
+           │ readFeedRowsForMerchant()                               │
+           │  - auto-detect delimiter                               │
+           │  - parse CSV/TSV → MerchantFeedRow objects             │
+           └─────────────────┬──────────────────────────────────────┘
+                             │ List<MerchantFeedRow>
+                             ▼
+          ┌──────────────────────────────────────┐
+          │ For each MerchantFeedRow row:        │
+          │  resolveBrand()                      │
+          │  upsertProduct()                     │
+          │    - find existing via brand+mpn/upc │
+          │    - update fields (mapped partRole) │
+          │  upsertOfferFromRow()                │
+          └──────────────────────────────────────┘
+```
+
+---
+
+# 2. Full Import Explained
+
+Triggered by:
+
+```
+POST /admin/imports/{merchantId}
+```
+
+### Step 1 — Load merchant  
+Using `merchantRepository.findById()`.
+
+### Step 2 — Parse feed rows  
+`readFeedRowsForMerchant()`:
+- Auto-detects delimiter (`\t`, `,`, `;`)
+- Validates required headers
+- Parses each row into `MerchantFeedRow`
+
+### Step 3 — Process each row
+
+For each parsed row:
+
+#### a. resolveBrand()
+- Finds or creates brand
+- Defaults to “Aero Precision” if missing
+
+#### b. upsertProduct()
+Dedupes by:
+
+1. Brand + MPN  
+2. Brand + UPC (currently SKU placeholder)  
+
+If no match → create new product.
+
+Then applies:
+- Name + slug
+- Descriptions
+- Images
+- MPN/identifiers
+- Platform inference
+- Category mapping
+- Part role inference
+
+#### c. upsertOfferFromRow()
+Creates or updates a ProductOffer:
+- Prices
+- Stock
+- Buy URL
+- lastSeenAt
+- firstSeenAt when newly created
+
+Idempotent — does not duplicate offers.
+
+---
+
+# 3. Offer-Only Sync
+
+Triggered by:
+
+```
+POST /admin/imports/{merchantId}/offers-only
+```
+
+Does NOT:
+- Create products  
+- Update product fields  
+
+It only updates:
+- price
+- originalPrice
+- inStock
+- buyUrl
+- lastSeenAt
+
+If the offer does not exist, it is skipped.
+
+---
+
+# 4. Auto-Detecting CSV/TSV Parser
+
+The parser:
+
+- Attempts multiple delimiters
+- Validates headers
+- Handles malformed or short rows
+- Never throws on missing columns
+- Returns clean MerchantFeedRow objects
+
+Designed for messy merchant feeds.
+
+---
+
+# 5. Entities Updated During Import
+
+### Product
+- name  
+- slug  
+- short/long description  
+- main image  
+- mpn  
+- upc (future)  
+- platform  
+- rawCategoryKey  
+- partRole  
+
+### ProductOffer
+- merchant
+- product
+- avantlinkProductId (SKU placeholder)
+- price
+- originalPrice
+- inStock
+- buyUrl
+- lastSeenAt
+- firstSeenAt
+
+### Merchant
+- lastFullImportAt
+- lastOfferSyncAt
+
+---
+
+# 6. Extension Points
+
+You can extend the import pipeline in these areas:
+
+- Add per-merchant column mapping  
+- Add true UPC parsing  
+- Support multi-platform parts  
+- Improve partRole inference  
+- Implement global deduplication across merchants  
+
+---
+
+# 7. Quick Reference: Main Methods
+
+| Method | Purpose |
+|--------|---------|
+| importMerchantFeed | Full product + offer import |
+| readFeedRowsForMerchant | Detect delimiter + parse feed |
+| resolveBrand | Normalize brand names |
+| upsertProduct | Idempotent product write |
+| updateProductFromRow | Apply product fields |
+| upsertOfferFromRow | Idempotent offer write |
+| syncOffersOnly | Offer-only sync |
+| upsertOfferOnlyFromRow | Update existing offers |
+| detectCsvFormat | Auto-detect delimiter |
+| fetchFeedRows | Simpler parser for offers |
+
+---
+
+# 8. Summary
+
+The Ballistic importer is:
+
+- Robust against bad data  
+- Idempotent and safe  
+- Flexible for multiple merchants  
+- Extensible for long-term scaling  
+
+This pipeline powers the product catalog and offer data for the Ballistic ecosystem.
--- a/src/main/java/group/goforward/ballistic/services/impl/MerchantFeedImportServiceImpl.java
+++ b/src/main/java/group/goforward/ballistic/services/impl/MerchantFeedImportServiceImpl.java
@@ -320,6 +320,77 @@ public class MerchantFeedImportServiceImpl implements MerchantFeedImportService
    //  Feed reading + brand resolution
    // ---------------------------------------------------------------------

+    /**
+     * Open a Reader for either an HTTP(S) URL or a local file path.
+     */
+    private Reader openFeedReader(String feedUrl) throws java.io.IOException {
+        if (feedUrl.startsWith("http://") || feedUrl.startsWith("https://")) {
+            return new InputStreamReader(new URL(feedUrl).openStream(), StandardCharsets.UTF_8);
+        } else {
+            return java.nio.file.Files.newBufferedReader(
+                    java.nio.file.Paths.get(feedUrl),
+                    StandardCharsets.UTF_8
+            );
+        }
+    }
+
+    /**
+     * Try a few common delimiters (tab, comma, semicolon) and pick the one
+     * that yields the expected AvantLink-style header set.
+     */
+    private CSVFormat detectCsvFormat(String feedUrl) throws Exception {
+        char[] delimiters = new char[]{'\t', ',', ';'};
+        java.util.List<String> requiredHeaders =
+                java.util.Arrays.asList("SKU", "Manufacturer Id", "Brand Name", "Product Name");
+
+        Exception lastException = null;
+
+        for (char delimiter : delimiters) {
+            try (Reader reader = openFeedReader(feedUrl);
+                 CSVParser parser = CSVFormat.DEFAULT.builder()
+                         .setDelimiter(delimiter)
+                         .setHeader()
+                         .setSkipHeaderRecord(true)
+                         .setIgnoreSurroundingSpaces(true)
+                         .setTrim(true)
+                         .build()
+                         .parse(reader)) {
+
+                Map<String, Integer> headerMap = parser.getHeaderMap();
+                if (headerMap != null && headerMap.keySet().containsAll(requiredHeaders)) {
+                    System.out.println(
+                            "IMPORT >>> detected delimiter '" +
+                                    (delimiter == '\t' ? "\\t" : String.valueOf(delimiter)) +
+                                    "' for feed: " + feedUrl
+                    );
+
+                    return CSVFormat.DEFAULT.builder()
+                            .setDelimiter(delimiter)
+                            .setHeader()
+                            .setSkipHeaderRecord(true)
+                            .setIgnoreSurroundingSpaces(true)
+                            .setTrim(true)
+                            .build();
+                } else if (headerMap != null) {
+                    System.out.println(
+                            "IMPORT !!! delimiter '" +
+                                    (delimiter == '\t' ? "\\t" : String.valueOf(delimiter)) +
+                                    "' produced headers: " + headerMap.keySet()
+                    );
+                }
+            } catch (Exception ex) {
+                lastException = ex;
+                System.out.println("IMPORT !!! error probing delimiter '" + delimiter +
+                        "' for " + feedUrl + ": " + ex.getMessage());
+            }
+        }
+
+        if (lastException != null) {
+            throw lastException;
+        }
+        throw new RuntimeException("Could not auto-detect delimiter for feed: " + feedUrl);
+    }
+
    private List<MerchantFeedRow> readFeedRowsForMerchant(Merchant merchant) {
        String rawFeedUrl = merchant.getFeedUrl();
        if (rawFeedUrl == null || rawFeedUrl.isBlank()) {
@@ -331,68 +402,41 @@ public class MerchantFeedImportServiceImpl implements MerchantFeedImportService

        List<MerchantFeedRow> rows = new ArrayList<>();

-        try (Reader baseReader = (feedUrl.startsWith("http://") || feedUrl.startsWith("https://"))
-                ? new InputStreamReader(new URL(feedUrl).openStream(), StandardCharsets.UTF_8)
-                : java.nio.file.Files.newBufferedReader(java.nio.file.Paths.get(feedUrl), StandardCharsets.UTF_8);
-             BufferedReader reader = new BufferedReader(baseReader)) {
+        try {
+            // Auto-detect delimiter (TSV/CSV/semicolon) based on header row
+            CSVFormat format = detectCsvFormat(feedUrl);

-            // --- Step 1: peek at the first line to detect delimiter ---
-            reader.mark(10_000);
-            String firstLine = reader.readLine();
-            reader.reset();
+            try (Reader reader = openFeedReader(feedUrl);
+                 CSVParser parser = new CSVParser(reader, format)) {

-            if (firstLine == null || firstLine.isEmpty()) {
-                throw new RuntimeException("Empty feed received from " + feedUrl);
-            }
+                System.out.println("IMPORT >>> detected headers: " + parser.getHeaderMap().keySet());

-            // --- Step 2: detect delimiter (TSV vs CSV) ---
-            char delimiter;
-            if (firstLine.contains("\t")) {
-                delimiter = '\t'; // TSV (AvantLink-style)
-            } else if (firstLine.contains(",")) {
-                delimiter = ',';  // CSV
-            } else {
-                // Fallback: default to comma
-                delimiter = ',';
-            }
-
-            // --- Step 3: build CSVFormat with detected delimiter ---
-            CSVFormat format = CSVFormat.DEFAULT.builder()
-                    .setDelimiter(delimiter)
-                    .setHeader()
-                    .setSkipHeaderRecord(true)
-                    .setIgnoreSurroundingSpaces(true)
-                    .setTrim(true)
-                    .build();
-
-            // --- Step 4: parse the rows into MerchantFeedRow records ---
-            try (CSVParser parser = new CSVParser(reader, format)) {
                for (CSVRecord rec : parser) {
                    MerchantFeedRow row = new MerchantFeedRow(
-                            rec.get("SKU"),
-                            rec.get("Manufacturer Id"),
-                            rec.get("Brand Name"),
-                            rec.get("Product Name"),
-                            rec.get("Long Description"),
-                            rec.get("Short Description"),
-                            rec.get("Department"),
-                            rec.get("Category"),
-                            rec.get("SubCategory"),
-                            rec.get("Thumb URL"),
-                            rec.get("Image URL"),
-                            rec.get("Buy Link"),
-                            rec.get("Keywords"),
-                            rec.get("Reviews"),
-                            parseBigDecimal(rec.get("Retail Price")),
-                            parseBigDecimal(rec.get("Sale Price")),
-                            rec.get("Brand Page Link"),
-                            rec.get("Brand Logo Image"),
-                            rec.get("Product Page View Tracking"),
-                            rec.get("Variants XML"),
-                            rec.get("Medium Image URL"),
-                            rec.get("Product Content Widget"),
-                            rec.get("Google Categorization"),
-                            rec.get("Item Based Commission")
+                            getCsvValue(rec, "SKU"),
+                            getCsvValue(rec, "Manufacturer Id"),
+                            getCsvValue(rec, "Brand Name"),
+                            getCsvValue(rec, "Product Name"),
+                            getCsvValue(rec, "Long Description"),
+                            getCsvValue(rec, "Short Description"),
+                            getCsvValue(rec, "Department"),
+                            getCsvValue(rec, "Category"),
+                            getCsvValue(rec, "SubCategory"),
+                            getCsvValue(rec, "Thumb URL"),
+                            getCsvValue(rec, "Image URL"),
+                            getCsvValue(rec, "Buy Link"),
+                            getCsvValue(rec, "Keywords"),
+                            getCsvValue(rec, "Reviews"),
+                            parseBigDecimal(getCsvValue(rec, "Retail Price")),
+                            parseBigDecimal(getCsvValue(rec, "Sale Price")),
+                            getCsvValue(rec, "Brand Page Link"),
+                            getCsvValue(rec, "Brand Logo Image"),
+                            getCsvValue(rec, "Product Page View Tracking"),
+                            null,
+                            getCsvValue(rec, "Medium Image URL"),
+                            getCsvValue(rec, "Product Content Widget"),
+                            getCsvValue(rec, "Google Categorization"),
+                            getCsvValue(rec, "Item Based Commission")
                    );

                    rows.add(row);
@@ -435,6 +479,28 @@ public class MerchantFeedImportServiceImpl implements MerchantFeedImportService
        }
    }

+    /**
+     * Safely get a column value by header name. If the record is "short"
+     * (fewer values than headers) or the header is missing, return null
+     * instead of throwing IllegalArgumentException.
+     */
+    private String getCsvValue(CSVRecord rec, String header) {
+        if (rec == null || header == null) {
+            return null;
+        }
+        if (!rec.isMapped(header)) {
+            // Header not present at all
+            return null;
+        }
+        try {
+            return rec.get(header);
+        } catch (IllegalArgumentException ex) {
+            System.out.println("IMPORT !!! short record #" + rec.getRecordNumber()
+                    + " missing column '" + header + "', treating as null");
+            return null;
+        }
+    }
+
    // ---------------------------------------------------------------------
    //  Misc helpers
    // ---------------------------------------------------------------------