readme docs

README.md

@@ -0,0 +1,67 @@
+# Ballistic Backend  
+### Internal Engine for the Builder Ecosystem
+
+The Ballistic Backend is the operational backbone behind the Shadown System Armory and its admin tools. It ingests merchant feeds, normalizes product data, manages categories, synchronizes prices, and powers the compatibility, pricing, and product logic behind the consumer-facing Builder.
+
+It’s built for reliability, longevity, and clean extensibility — the kind of foundation you want when scaling from a small beta to a fully public platform.
+
+---
+
+## What This Backend Does
+
+### **Merchant Feed Ingestion**
+- Pulls AvantLink feeds (CSV or TSV)
+- Automatically detects delimiters
+- Normalizes raw merchant fields
+- Creates or updates product records
+- Upserts price and stock offers
+- Tracks first-seen / last-seen timestamps
+- Safely handles malformed or incomplete rows  
+- Ensures repeat imports never duplicate offers
+
+### **Category Mapping Engine**
+- Identifies every unique raw category coming from each merchant feed  
+- Exposes *unmapped* categories in the admin UI  
+- Allows you to assign:  
+  - Part Role  
+  - Product Configuration (Stripped, Complete, Kit, etc.)
+- Applies mappings automatically on future imports  
+- Respects manual overrides such as `platform_locked`
+
+### **Builder Support**
+The frontend Builder depends on this backend for:
+
+- Loading parts grouped by role  
+- Offering compatible options  
+- Calculating build cost  
+- Comparing offers across merchants  
+- Providing product metadata, imagery, and offer data  
+
+**Future expansions include:** price history, compatibility engine, build exports, public build pages, multi-merchant aggregation, and automated anomaly detection.
+
+---
+
+## Tech Stack
+
+- **Spring Boot 3.x**  
+- **Java 17**  
+- **PostgreSQL**  
+- **Hibernate (JPA)**  
+- **HikariCP**  
+- **Apache Commons CSV**  
+- **Maven**  
+- **REST API**
+
+---
+
+## Local Development
+
+### Requirements
+- Java 17 or newer  
+- PostgreSQL running locally  
+- Port 8080 open (default backend port)
+
+### Run Development Server
+
+```bash
+./mvnw spring-boot:run

importLogic.md

@@ -0,0 +1,213 @@
+# Ballistic Import Pipeline  
+A high-level overview of how merchant data flows through the Spring ETL system.
+
+---
+
+## Purpose
+
+This document explains how the Ballistic backend:
+
+1. Fetches merchant product feeds (CSV/TSV)
+2. Normalizes raw data into structured entities  
+3. Updates products and offers in an idempotent way  
+4. Supports two sync modes:
+   - Full Import
+   - Offer-Only Sync
+
+---
+
+# 1. High-Level Flow
+
+## ASCII Diagram
+
+```
+                         ┌──────────────────────────┐
+                         │   /admin/imports/{id}     │
+                         │  (Full Import Trigger)    │
+                         └─────────────┬────────────┘
+                                       │
+                                       ▼
+                        ┌──────────────────────────────┐
+                        │ importMerchantFeed(merchantId)│
+                        └─────────────┬────────────────┘
+                                      │
+                                      ▼
+           ┌────────────────────────────────────────────────────────┐
+           │ readFeedRowsForMerchant()                               │
+           │  - auto-detect delimiter                               │
+           │  - parse CSV/TSV → MerchantFeedRow objects             │
+           └─────────────────┬──────────────────────────────────────┘
+                             │ List<MerchantFeedRow>
+                             ▼
+          ┌──────────────────────────────────────┐
+          │ For each MerchantFeedRow row:        │
+          │  resolveBrand()                      │
+          │  upsertProduct()                     │
+          │    - find existing via brand+mpn/upc │
+          │    - update fields (mapped partRole) │
+          │  upsertOfferFromRow()                │
+          └──────────────────────────────────────┘
+```
+
+---
+
+# 2. Full Import Explained
+
+Triggered by:
+
+```
+POST /admin/imports/{merchantId}
+```
+
+### Step 1 — Load merchant  
+Using `merchantRepository.findById()`.
+
+### Step 2 — Parse feed rows  
+`readFeedRowsForMerchant()`:
+- Auto-detects delimiter (`\t`, `,`, `;`)
+- Validates required headers
+- Parses each row into `MerchantFeedRow`
+
+### Step 3 — Process each row
+
+For each parsed row:
+
+#### a. resolveBrand()
+- Finds or creates brand
+- Defaults to “Aero Precision” if missing
+
+#### b. upsertProduct()
+Dedupes by:
+
+1. Brand + MPN  
+2. Brand + UPC (currently SKU placeholder)  
+
+If no match → create new product.
+
+Then applies:
+- Name + slug
+- Descriptions
+- Images
+- MPN/identifiers
+- Platform inference
+- Category mapping
+- Part role inference
+
+#### c. upsertOfferFromRow()
+Creates or updates a ProductOffer:
+- Prices
+- Stock
+- Buy URL
+- lastSeenAt
+- firstSeenAt when newly created
+
+Idempotent — does not duplicate offers.
+
+---
+
+# 3. Offer-Only Sync
+
+Triggered by:
+
+```
+POST /admin/imports/{merchantId}/offers-only
+```
+
+Does NOT:
+- Create products  
+- Update product fields  
+
+It only updates:
+- price
+- originalPrice
+- inStock
+- buyUrl
+- lastSeenAt
+
+If the offer does not exist, it is skipped.
+
+---
+
+# 4. Auto-Detecting CSV/TSV Parser
+
+The parser:
+
+- Attempts multiple delimiters
+- Validates headers
+- Handles malformed or short rows
+- Never throws on missing columns
+- Returns clean MerchantFeedRow objects
+
+Designed for messy merchant feeds.
+
+---
+
+# 5. Entities Updated During Import
+
+### Product
+- name  
+- slug  
+- short/long description  
+- main image  
+- mpn  
+- upc (future)  
+- platform  
+- rawCategoryKey  
+- partRole  
+
+### ProductOffer
+- merchant
+- product
+- avantlinkProductId (SKU placeholder)
+- price
+- originalPrice
+- inStock
+- buyUrl
+- lastSeenAt
+- firstSeenAt
+
+### Merchant
+- lastFullImportAt
+- lastOfferSyncAt
+
+---
+
+# 6. Extension Points
+
+You can extend the import pipeline in these areas:
+
+- Add per-merchant column mapping  
+- Add true UPC parsing  
+- Support multi-platform parts  
+- Improve partRole inference  
+- Implement global deduplication across merchants  
+
+---
+
+# 7. Quick Reference: Main Methods
+
+| Method | Purpose |
+|--------|---------|
+| importMerchantFeed | Full product + offer import |
+| readFeedRowsForMerchant | Detect delimiter + parse feed |
+| resolveBrand | Normalize brand names |
+| upsertProduct | Idempotent product write |
+| updateProductFromRow | Apply product fields |
+| upsertOfferFromRow | Idempotent offer write |
+| syncOffersOnly | Offer-only sync |
+| upsertOfferOnlyFromRow | Update existing offers |
+| detectCsvFormat | Auto-detect delimiter |
+| fetchFeedRows | Simpler parser for offers |
+
+---
+
+# 8. Summary
+
+The Ballistic importer is:
+
+- Robust against bad data  
+- Idempotent and safe  
+- Flexible for multiple merchants  
+- Extensible for long-term scaling  
+
+This pipeline powers the product catalog and offer data for the Ballistic ecosystem.

src/main/java/group/goforward/ballistic/services/impl/MerchantFeedImportServiceImpl.java

@@ -7,6 +7,7 @@ import java.util.List;
 import java.util.Map;
 import java.util.HashMap;
 import java.io.Reader;
+import java.io.BufferedReader;
 import java.io.InputStreamReader;
 import java.net.URL;
 import java.nio.charset.StandardCharsets;
@@ -319,6 +320,77 @@ public class MerchantFeedImportServiceImpl implements MerchantFeedImportService
     //  Feed reading + brand resolution
     // ---------------------------------------------------------------------
 
+    /**
+     * Open a Reader for either an HTTP(S) URL or a local file path.
+     */
+    private Reader openFeedReader(String feedUrl) throws java.io.IOException {
+        if (feedUrl.startsWith("http://") || feedUrl.startsWith("https://")) {
+            return new InputStreamReader(new URL(feedUrl).openStream(), StandardCharsets.UTF_8);
+        } else {
+            return java.nio.file.Files.newBufferedReader(
+                    java.nio.file.Paths.get(feedUrl),
+                    StandardCharsets.UTF_8
+            );
+        }
+    }
+
+    /**
+     * Try a few common delimiters (tab, comma, semicolon) and pick the one
+     * that yields the expected AvantLink-style header set.
+     */
+    private CSVFormat detectCsvFormat(String feedUrl) throws Exception {
+        char[] delimiters = new char[]{'\t', ',', ';'};
+        java.util.List<String> requiredHeaders =
+                java.util.Arrays.asList("SKU", "Manufacturer Id", "Brand Name", "Product Name");
+
+        Exception lastException = null;
+
+        for (char delimiter : delimiters) {
+            try (Reader reader = openFeedReader(feedUrl);
+                 CSVParser parser = CSVFormat.DEFAULT.builder()
+                         .setDelimiter(delimiter)
+                         .setHeader()
+                         .setSkipHeaderRecord(true)
+                         .setIgnoreSurroundingSpaces(true)
+                         .setTrim(true)
+                         .build()
+                         .parse(reader)) {
+
+                Map<String, Integer> headerMap = parser.getHeaderMap();
+                if (headerMap != null && headerMap.keySet().containsAll(requiredHeaders)) {
+                    System.out.println(
+                            "IMPORT >>> detected delimiter '" +
+                                    (delimiter == '\t' ? "\\t" : String.valueOf(delimiter)) +
+                                    "' for feed: " + feedUrl
+                    );
+
+                    return CSVFormat.DEFAULT.builder()
+                            .setDelimiter(delimiter)
+                            .setHeader()
+                            .setSkipHeaderRecord(true)
+                            .setIgnoreSurroundingSpaces(true)
+                            .setTrim(true)
+                            .build();
+                } else if (headerMap != null) {
+                    System.out.println(
+                            "IMPORT !!! delimiter '" +
+                                    (delimiter == '\t' ? "\\t" : String.valueOf(delimiter)) +
+                                    "' produced headers: " + headerMap.keySet()
+                    );
+                }
+            } catch (Exception ex) {
+                lastException = ex;
+                System.out.println("IMPORT !!! error probing delimiter '" + delimiter +
+                        "' for " + feedUrl + ": " + ex.getMessage());
+            }
+        }
+
+        if (lastException != null) {
+            throw lastException;
+        }
+        throw new RuntimeException("Could not auto-detect delimiter for feed: " + feedUrl);
+    }
+
     private List<MerchantFeedRow> readFeedRowsForMerchant(Merchant merchant) {
         String rawFeedUrl = merchant.getFeedUrl();
         if (rawFeedUrl == null || rawFeedUrl.isBlank()) {
@@ -330,45 +402,46 @@ public class MerchantFeedImportServiceImpl implements MerchantFeedImportService
 
         List<MerchantFeedRow> rows = new ArrayList<>();
 
-        try (Reader reader = (feedUrl.startsWith("http://") || feedUrl.startsWith("https://"))
+        try {
-                ? new InputStreamReader(new URL(feedUrl).openStream(), StandardCharsets.UTF_8)
+            // Auto-detect delimiter (TSV/CSV/semicolon) based on header row
-                : java.nio.file.Files.newBufferedReader(java.nio.file.Paths.get(feedUrl), StandardCharsets.UTF_8);
+            CSVFormat format = detectCsvFormat(feedUrl);
-             CSVParser parser = CSVFormat.DEFAULT
+
-                     .withFirstRecordAsHeader()
+            try (Reader reader = openFeedReader(feedUrl);
-                     .withIgnoreSurroundingSpaces()
+                 CSVParser parser = new CSVParser(reader, format)) {
-                     .withTrim()
+
-                     .parse(reader)) {
+                System.out.println("IMPORT >>> detected headers: " + parser.getHeaderMap().keySet());
 
                 for (CSVRecord rec : parser) {
                     MerchantFeedRow row = new MerchantFeedRow(
-                        rec.get("SKU"),
+                            getCsvValue(rec, "SKU"),
-                        rec.get("Manufacturer Id"),
+                            getCsvValue(rec, "Manufacturer Id"),
-                        rec.get("Brand Name"),
+                            getCsvValue(rec, "Brand Name"),
-                        rec.get("Product Name"),
+                            getCsvValue(rec, "Product Name"),
-                        rec.get("Long Description"),
+                            getCsvValue(rec, "Long Description"),
-                        rec.get("Short Description"),
+                            getCsvValue(rec, "Short Description"),
-                        rec.get("Department"),
+                            getCsvValue(rec, "Department"),
-                        rec.get("Category"),
+                            getCsvValue(rec, "Category"),
-                        rec.get("SubCategory"),
+                            getCsvValue(rec, "SubCategory"),
-                        rec.get("Thumb URL"),
+                            getCsvValue(rec, "Thumb URL"),
-                        rec.get("Image URL"),
+                            getCsvValue(rec, "Image URL"),
-                        rec.get("Buy Link"),
+                            getCsvValue(rec, "Buy Link"),
-                        rec.get("Keywords"),
+                            getCsvValue(rec, "Keywords"),
-                        rec.get("Reviews"),
+                            getCsvValue(rec, "Reviews"),
-                        parseBigDecimal(rec.get("Retail Price")),
+                            parseBigDecimal(getCsvValue(rec, "Retail Price")),
-                        parseBigDecimal(rec.get("Sale Price")),
+                            parseBigDecimal(getCsvValue(rec, "Sale Price")),
-                        rec.get("Brand Page Link"),
+                            getCsvValue(rec, "Brand Page Link"),
-                        rec.get("Brand Logo Image"),
+                            getCsvValue(rec, "Brand Logo Image"),
-                        rec.get("Product Page View Tracking"),
+                            getCsvValue(rec, "Product Page View Tracking"),
-                        rec.get("Variants XML"),
+                            null,
-                        rec.get("Medium Image URL"),
+                            getCsvValue(rec, "Medium Image URL"),
-                        rec.get("Product Content Widget"),
+                            getCsvValue(rec, "Product Content Widget"),
-                        rec.get("Google Categorization"),
+                            getCsvValue(rec, "Google Categorization"),
-                        rec.get("Item Based Commission")
+                            getCsvValue(rec, "Item Based Commission")
                     );
 
                     rows.add(row);
                 }
+            }
         } catch (Exception ex) {
             throw new RuntimeException("Failed to read feed for merchant "
                     + merchant.getName() + " from " + feedUrl, ex);
@@ -406,6 +479,28 @@ public class MerchantFeedImportServiceImpl implements MerchantFeedImportService
         }
     }
 
+    /**
+     * Safely get a column value by header name. If the record is "short"
+     * (fewer values than headers) or the header is missing, return null
+     * instead of throwing IllegalArgumentException.
+     */
+    private String getCsvValue(CSVRecord rec, String header) {
+        if (rec == null || header == null) {
+            return null;
+        }
+        if (!rec.isMapped(header)) {
+            // Header not present at all
+            return null;
+        }
+        try {
+            return rec.get(header);
+        } catch (IllegalArgumentException ex) {
+            System.out.println("IMPORT !!! short record #" + rec.getRecordNumber()
+                    + " missing column '" + header + "', treating as null");
+            return null;
+        }
+    }
+
     // ---------------------------------------------------------------------
     //  Misc helpers
     // ---------------------------------------------------------------------