groceries/SOFT_DELETE_IMPLEMENTATION.md

# Soft Delete Implementation with Historical Tracking

## Overview

This implementation extends the existing temporal tables system to support soft deletes with proper historical tracking. When a product is "deleted", it's not actually removed from the database but marked as deleted with a timestamp, while maintaining full historical data.

## Key Features

### 1. Soft Delete Mechanism
- Products are marked as `deleted = true` instead of being physically removed
- Deletion creates a historical record of the product's state before deletion
- Deleted products get a new `valid_from` date set to the deletion date
- All historical versions remain intact for audit purposes

### 2. UI Enhancements
- **Product List**: "Show deleted" toggle next to "Add New Product" button
- **Visual Indicators**: Deleted products shown with:
  - Red background tint and reduced opacity
  - Strikethrough text
  - 🗑️ emoji indicator
  - Disabled edit/duplicate/delete actions
- **Shopping Events**: Products deleted before/on shopping date are automatically filtered out

### 3. API Behavior
- **Default**: Deleted products are hidden from all product listings
- **Optional**: `show_deleted=true` parameter shows all products including deleted ones
- **Shopping Events**: New endpoint `/products/available-for-shopping/{date}` filters products based on deletion status at specific date

## Database Schema Changes

### Products Table
```sql
ALTER TABLE products 
ADD COLUMN deleted BOOLEAN DEFAULT FALSE NOT NULL;
```

### Products History Table
```sql
-- Already included in products_history table
deleted BOOLEAN DEFAULT FALSE NOT NULL
```

### Updated Trigger
The `products_versioning_trigger` now **consistently handles ALL historization**:
- **UPDATE operations**: Creates history records for both automatic and manual versioning
- **DELETE operations**: Creates history records when products are deleted
- **Smart versioning**: Automatically detects manual vs automatic versioning based on `valid_from` changes
- **Centralized logic**: All temporal logic is in the database trigger, not split between trigger and application

### Trigger Benefits
1. **Consistency**: All versioning operations follow the same pattern
2. **Reliability**: Database-level enforcement prevents inconsistencies  
3. **Simplicity**: Application code just sets fields, trigger handles the rest
4. **Performance**: Single database operation handles both data update and history creation

## API Endpoints

### Modified Endpoints

#### `GET /products/`
- **New Parameter**: `show_deleted: bool = False`
- **Behavior**: Filters out deleted products by default
- **Usage**: `GET /products/?show_deleted=true` to include deleted products

#### `PUT /products/{id}` & `DELETE /products/{id}`
- **Enhanced**: Now properly handles soft delete with historical tracking
- **Validation**: Prevents operations on already-deleted products

### New Endpoints

#### `GET /products/available-for-shopping/{shopping_date}`
- **Purpose**: Get products that were available (not deleted) on a specific shopping date
- **Logic**: Returns products where:
  - `deleted = false` (never deleted), OR
  - `deleted = true` AND `valid_from > shopping_date` (deleted after shopping date)
- **Usage**: Used by shopping event modals to filter product lists

## Frontend Implementation

### ProductList Component
```typescript
// New state for toggle
const [showDeleted, setShowDeleted] = useState(false);

// Updated API call
const response = await productApi.getAll(showDeleted);

// Visual styling for deleted products
className={`hover:bg-gray-50 ${product.deleted ? 'bg-red-50 opacity-75' : ''}`}
```

### AddShoppingEventModal Component
```typescript
// Dynamic product fetching based on shopping date
const response = formData.date 
  ? await productApi.getAvailableForShopping(formData.date)
  : await productApi.getAll(false);

// Refetch products when date changes
useEffect(() => {
  if (isOpen && formData.date) {
    fetchProducts();
  }
}, [formData.date, isOpen]);
```

## Deletion Process Flow

### 1. User Initiates Delete
- User clicks "Delete" button on a product
- Confirmation modal appears

### 2. Backend Processing
```python
# Simple application code - trigger handles the complexity
product.deleted = True
product.valid_from = date.today()  # Manual versioning date
product.updated_at = func.now()

# Trigger automatically:
# 1. Detects the change (deleted field + valid_from change)
# 2. Creates history record with old data (deleted=False, valid_to=today)
# 3. Ensures new record has valid_to='9999-12-31'
```

**Trigger Logic (Automatic)**:
- Detects manual versioning because `valid_from` changed
- Uses the new `valid_from` as the cutoff date for history
- Creates history record: `{...old_data, valid_to: new_valid_from, operation: 'U'}`
- No additional application logic needed

### 3. Frontend Updates
- Product list refreshes
- Deleted product appears with visual indicators (if "Show deleted" is enabled)
- Product becomes unavailable for new shopping events

## Historical Data Integrity

### Shopping Events
- **Guarantee**: Shopping events always show products exactly as they existed when purchased
- **Implementation**: Uses `/products/{id}/at/{date}` endpoint to fetch historical product state
- **Benefit**: Even if a product is deleted later, historical shopping events remain accurate

### Audit Trail
- **Complete History**: All product versions are preserved in `products_history`
- **Deletion Tracking**: History records show when and why products were deleted
- **Temporal Queries**: Can reconstruct product state at any point in time

## Usage Examples

### 1. View All Products (Default)
```bash
GET /products/
# Returns only non-deleted products
```

### 2. View All Products Including Deleted
```bash
GET /products/?show_deleted=true
# Returns all products with deleted status
```

### 3. Get Products Available for Shopping on Specific Date
```bash
GET /products/available-for-shopping/2024-01-15
# Returns products that were not deleted on 2024-01-15
```

### 4. View Historical Shopping Event
```bash
GET /shopping-events/123/products-as-purchased
# Returns products exactly as they were when purchased, regardless of current deletion status
```

## Benefits

1. **Data Preservation**: No data is ever lost
2. **Audit Compliance**: Complete audit trail of all changes
3. **Historical Accuracy**: Shopping events remain accurate over time
4. **User Experience**: Clean interface with optional deleted product visibility
5. **Flexibility**: Easy to "undelete" products if needed (future enhancement)

## Future Enhancements

1. **Undelete Functionality**: Add ability to restore deleted products
2. **Bulk Operations**: Delete/restore multiple products at once
3. **Deletion Reasons**: Add optional reason field for deletions
4. **Advanced Filtering**: Filter by deletion date, reason, etc.
5. **Reporting**: Generate reports on deleted products and their impact

## Migration Instructions

1. **Run Migration**: Execute `temporal_migration.sql` to add `deleted` column
2. **Deploy Backend**: Update backend with new API endpoints and logic
3. **Deploy Frontend**: Update frontend with new UI components and API calls
4. **Test**: Verify soft delete functionality and historical data integrity

## Testing Scenarios

1. **Basic Deletion**: Delete a product and verify it's hidden from default view
2. **Show Deleted Toggle**: Enable "Show deleted" and verify deleted products appear with proper styling
3. **Shopping Event Filtering**: Create shopping event and verify deleted products don't appear in product list
4. **Historical Accuracy**: Delete a product that was in a past shopping event, verify the shopping event still shows correct historical data
5. **Date-based Filtering**: Test `/products/available-for-shopping/{date}` with various dates before/after product deletions 

## Database Initialization

### ⚠️ Important: Trigger Creation

The temporal triggers are essential for the soft delete functionality. They must be created in **all environments**:

### **Method 1: Automatic (Recommended)**
The triggers are now automatically created when using SQLAlchemy's `create_all()`:

```python
# This now creates both tables AND triggers
models.Base.metadata.create_all(bind=engine)
```

**How it works**:
- SQLAlchemy event listener detects when `ProductHistory` table is created
- Automatically executes trigger creation SQL
- Works for fresh dev, test, and production databases

### **Method 2: Manual Database Script**
For explicit control, use the initialization script:

```bash
# Run this for fresh database setup
python backend/database_init.py
```

**Features**:
- Creates all tables
- Creates all triggers
- Checks for existing triggers (safe to run multiple times)
- Provides detailed feedback

### **Method 3: Migration File**
For existing databases, run the migration:

```sql
-- Execute temporal_migration.sql
\i temporal_migration.sql
```

## Environment Setup Guide

### **Development (Fresh DB)**
```bash
# Option A: Automatic (when starting app)
python backend/main.py
# ✅ Tables + triggers created automatically

# Option B: Explicit setup
python backend/database_init.py
python backend/main.py
```

### **Production (Fresh DB)**
```bash
# Recommended: Explicit initialization
python backend/database_init.py
# Then start the application
```

### **Existing Database (Migration)**
```bash
# Apply migration to add soft delete functionality
docker-compose exec db psql -U postgres -d groceries -f /tmp/temporal_migration.sql
```