groceries/SOFT_DELETE_IMPLEMENTATION.md

Soft Delete Implementation with Historical Tracking

Overview

This implementation extends the existing temporal tables system to support soft deletes with proper historical tracking. When a product is "deleted", it's not actually removed from the database but marked as deleted with a timestamp, while maintaining full historical data.

Key Features

1. Soft Delete Mechanism

• Products are marked as deleted = true instead of being physically removed
• Deletion creates a historical record of the product's state before deletion
• Deleted products get a new valid_from date set to the deletion date
• All historical versions remain intact for audit purposes

2. UI Enhancements

• Product List: "Show deleted" toggle next to "Add New Product" button
• Visual Indicators: Deleted products shown with:
  • Red background tint and reduced opacity
  • Strikethrough text
  • 🗑️ emoji indicator
  • Disabled edit/duplicate/delete actions
• Shopping Events: Products deleted before/on shopping date are automatically filtered out

3. API Behavior

• Default: Deleted products are hidden from all product listings
• Optional: show_deleted=true parameter shows all products including deleted ones
• Shopping Events: New endpoint /products/available-for-shopping/{date} filters products based on deletion status at specific date

Database Schema Changes

Products Table

ALTER TABLE products 
ADD COLUMN deleted BOOLEAN DEFAULT FALSE NOT NULL;

Products History Table

-- Already included in products_history table
deleted BOOLEAN DEFAULT FALSE NOT NULL

Updated Trigger

The products_versioning_trigger now consistently handles ALL historization:

• UPDATE operations: Creates history records for both automatic and manual versioning
• DELETE operations: Creates history records when products are deleted
• Smart versioning: Automatically detects manual vs automatic versioning based on valid_from changes
• Centralized logic: All temporal logic is in the database trigger, not split between trigger and application

Trigger Benefits

1. Consistency: All versioning operations follow the same pattern
2. Reliability: Database-level enforcement prevents inconsistencies
3. Simplicity: Application code just sets fields, trigger handles the rest
4. Performance: Single database operation handles both data update and history creation

API Endpoints

Modified Endpoints

GET /products/

• New Parameter: show_deleted: bool = False
• Behavior: Filters out deleted products by default
• Usage: GET /products/?show_deleted=true to include deleted products

PUT /products/{id} & DELETE /products/{id}

• Enhanced: Now properly handles soft delete with historical tracking
• Validation: Prevents operations on already-deleted products

New Endpoints

GET /products/available-for-shopping/{shopping_date}

• Purpose: Get products that were available (not deleted) on a specific shopping date
• Logic: Returns products where:
  • deleted = false (never deleted), OR
  • deleted = true AND valid_from > shopping_date (deleted after shopping date)
• Usage: Used by shopping event modals to filter product lists

Frontend Implementation

ProductList Component

// New state for toggle
const [showDeleted, setShowDeleted] = useState(false);

// Updated API call
const response = await productApi.getAll(showDeleted);

// Visual styling for deleted products
className={`hover:bg-gray-50 ${product.deleted ? 'bg-red-50 opacity-75' : ''}`}

AddShoppingEventModal Component

// Dynamic product fetching based on shopping date
const response = formData.date 
  ? await productApi.getAvailableForShopping(formData.date)
  : await productApi.getAll(false);

// Refetch products when date changes
useEffect(() => {
  if (isOpen && formData.date) {
    fetchProducts();
  }
}, [formData.date, isOpen]);

Deletion Process Flow

1. User Initiates Delete

• User clicks "Delete" button on a product
• Confirmation modal appears

2. Backend Processing

# Simple application code - trigger handles the complexity
product.deleted = True
product.valid_from = date.today()  # Manual versioning date
product.updated_at = func.now()

# Trigger automatically:
# 1. Detects the change (deleted field + valid_from change)
# 2. Creates history record with old data (deleted=False, valid_to=today)
# 3. Ensures new record has valid_to='9999-12-31'

Trigger Logic (Automatic):

• Detects manual versioning because valid_from changed
• Uses the new valid_from as the cutoff date for history
• Creates history record: {...old_data, valid_to: new_valid_from, operation: 'U'}
• No additional application logic needed

3. Frontend Updates

• Product list refreshes
• Deleted product appears with visual indicators (if "Show deleted" is enabled)
• Product becomes unavailable for new shopping events

Historical Data Integrity

Shopping Events

• Guarantee: Shopping events always show products exactly as they existed when purchased
• Implementation: Uses /products/{id}/at/{date} endpoint to fetch historical product state
• Benefit: Even if a product is deleted later, historical shopping events remain accurate

Audit Trail

• Complete History: All product versions are preserved in products_history
• Deletion Tracking: History records show when and why products were deleted
• Temporal Queries: Can reconstruct product state at any point in time

Usage Examples

1. View All Products (Default)

GET /products/
# Returns only non-deleted products

2. View All Products Including Deleted

GET /products/?show_deleted=true
# Returns all products with deleted status

3. Get Products Available for Shopping on Specific Date

GET /products/available-for-shopping/2024-01-15
# Returns products that were not deleted on 2024-01-15

4. View Historical Shopping Event

GET /shopping-events/123/products-as-purchased
# Returns products exactly as they were when purchased, regardless of current deletion status

Benefits

1. Data Preservation: No data is ever lost
2. Audit Compliance: Complete audit trail of all changes
3. Historical Accuracy: Shopping events remain accurate over time
4. User Experience: Clean interface with optional deleted product visibility
5. Flexibility: Easy to "undelete" products if needed (future enhancement)

Future Enhancements

1. Undelete Functionality: Add ability to restore deleted products
2. Bulk Operations: Delete/restore multiple products at once
3. Deletion Reasons: Add optional reason field for deletions
4. Advanced Filtering: Filter by deletion date, reason, etc.
5. Reporting: Generate reports on deleted products and their impact

Migration Instructions

1. Run Migration: Execute temporal_migration.sql to add deleted column
2. Deploy Backend: Update backend with new API endpoints and logic
3. Deploy Frontend: Update frontend with new UI components and API calls
4. Test: Verify soft delete functionality and historical data integrity

Testing Scenarios

1. Basic Deletion: Delete a product and verify it's hidden from default view
2. Show Deleted Toggle: Enable "Show deleted" and verify deleted products appear with proper styling
3. Shopping Event Filtering: Create shopping event and verify deleted products don't appear in product list
4. Historical Accuracy: Delete a product that was in a past shopping event, verify the shopping event still shows correct historical data
5. Date-based Filtering: Test /products/available-for-shopping/{date} with various dates before/after product deletions

Database Initialization

⚠️ Important: Trigger Creation

The temporal triggers are essential for the soft delete functionality. They must be created in all environments:

Method 1: Automatic (Recommended)

The triggers are now automatically created when using SQLAlchemy's create_all():

# This now creates both tables AND triggers
models.Base.metadata.create_all(bind=engine)

How it works:

• SQLAlchemy event listener detects when ProductHistory table is created
• Automatically executes trigger creation SQL
• Works for fresh dev, test, and production databases

Method 2: Manual Database Script

For explicit control, use the initialization script:

# Run this for fresh database setup
python backend/database_init.py

Features:

• Creates all tables
• Creates all triggers
• Checks for existing triggers (safe to run multiple times)
• Provides detailed feedback

Method 3: Migration File

For existing databases, run the migration:

-- Execute temporal_migration.sql
\i temporal_migration.sql

Environment Setup Guide

Development (Fresh DB)

# Option A: Automatic (when starting app)
python backend/main.py
# ✅ Tables + triggers created automatically

# Option B: Explicit setup
python backend/database_init.py
python backend/main.py

Production (Fresh DB)

# Recommended: Explicit initialization
python backend/database_init.py
# Then start the application

Existing Database (Migration)

# Apply migration to add soft delete functionality
docker-compose exec db psql -U postgres -d groceries -f /tmp/temporal_migration.sql