create script for test data
This commit is contained in:
222
backend/TEST_DATA_README.md
Normal file
222
backend/TEST_DATA_README.md
Normal file
@@ -0,0 +1,222 @@
|
||||
# Test Data Scripts Documentation
|
||||
|
||||
This directory contains scripts for creating and managing test data for the Grocery Tracker application.
|
||||
|
||||
## Scripts Overview
|
||||
|
||||
### 1. `create_test_data.py` - Comprehensive Test Data Generator
|
||||
|
||||
Creates realistic test data including shops, groceries, and shopping events.
|
||||
|
||||
#### Basic Usage
|
||||
|
||||
```bash
|
||||
# Create all test data (default: 30 events over 90 days)
|
||||
python create_test_data.py
|
||||
|
||||
# Create with custom parameters
|
||||
python create_test_data.py --events 50 --days 120
|
||||
|
||||
# Verbose output
|
||||
python create_test_data.py --verbose
|
||||
|
||||
# Dry run (see what would be created without creating it)
|
||||
python create_test_data.py --dry-run
|
||||
```
|
||||
|
||||
#### Command Line Options
|
||||
|
||||
| Option | Description | Default |
|
||||
|--------|-------------|---------|
|
||||
| `--events N` | Number of shopping events to create | 30 |
|
||||
| `--days N` | Number of days back to generate events | 90 |
|
||||
| `--url URL` | API base URL | http://localhost:8000 |
|
||||
| `--shops-only` | Create only shops | False |
|
||||
| `--groceries-only` | Create only groceries | False |
|
||||
| `--events-only` | Create only shopping events (requires existing data) | False |
|
||||
| `--verbose`, `-v` | Verbose output with detailed progress | False |
|
||||
| `--dry-run` | Show what would be created without creating it | False |
|
||||
|
||||
#### Examples
|
||||
|
||||
```bash
|
||||
# Create only shops
|
||||
python create_test_data.py --shops-only
|
||||
|
||||
# Create only groceries
|
||||
python create_test_data.py --groceries-only
|
||||
|
||||
# Create 100 shopping events using existing shops and groceries
|
||||
python create_test_data.py --events-only --events 100
|
||||
|
||||
# Create test data for the past 6 months with verbose output
|
||||
python create_test_data.py --events 60 --days 180 --verbose
|
||||
|
||||
# Preview what would be created without actually creating it
|
||||
python create_test_data.py --dry-run
|
||||
|
||||
# Use a different API URL
|
||||
python create_test_data.py --url http://localhost:3000
|
||||
```
|
||||
|
||||
### 2. `cleanup_test_data.py` - Data Cleanup Script
|
||||
|
||||
Safely removes all test data with confirmation prompts.
|
||||
|
||||
```bash
|
||||
python cleanup_test_data.py
|
||||
```
|
||||
|
||||
## Data Structure
|
||||
|
||||
### Shops (10 total)
|
||||
- **Whole Foods Market** (San Francisco)
|
||||
- **Safeway** (San Francisco)
|
||||
- **Trader Joe's** (Berkeley)
|
||||
- **Berkeley Bowl** (Berkeley)
|
||||
- **Rainbow Grocery** (San Francisco)
|
||||
- **Mollie Stone's Market** (Palo Alto)
|
||||
- **Costco Wholesale** (San Mateo)
|
||||
- **Target** (Mountain View)
|
||||
- **Sprouts Farmers Market** (Sunnyvale)
|
||||
- **Lucky Supermarket** (San Jose)
|
||||
|
||||
### Groceries (50+ items across 8 categories)
|
||||
|
||||
| Category | Items | Organic Options |
|
||||
|----------|-------|-----------------|
|
||||
| **Fruits** | 10 items | 5 organic |
|
||||
| **Vegetables** | 10 items | 5 organic |
|
||||
| **Dairy** | 7 items | 4 organic |
|
||||
| **Meat & Seafood** | 6 items | 3 organic |
|
||||
| **Pantry** | 10 items | 5 organic |
|
||||
| **Beverages** | 6 items | 3 organic |
|
||||
| **Frozen** | 5 items | 2 organic |
|
||||
| **Snacks** | 5 items | 3 organic |
|
||||
|
||||
### Shopping Events
|
||||
- **Realistic dates**: Distributed over specified time period (default: 90 days)
|
||||
- **Smart quantities**: Appropriate amounts based on item type
|
||||
- **Category-based pricing**: Realistic price ranges per category
|
||||
- **Organic premiums**: 20-50% higher prices for organic items
|
||||
- **Random notes**: 30% of events include descriptive notes
|
||||
- **Varied trip sizes**: 2-8 items per shopping trip
|
||||
|
||||
## Features
|
||||
|
||||
### Smart Data Generation
|
||||
- **Realistic pricing**: Category-based price ranges with organic premiums
|
||||
- **Appropriate quantities**: Items sold by piece, weight, or volume as appropriate
|
||||
- **Temporal distribution**: Events spread realistically over time
|
||||
- **Shopping patterns**: Varied trip sizes and frequencies
|
||||
|
||||
### Error Handling
|
||||
- **Graceful failures**: Script continues even if some items fail
|
||||
- **Network timeouts**: Reasonable timeout values for API calls
|
||||
- **Progress tracking**: Clear feedback on creation progress
|
||||
- **Connection testing**: Verifies API availability before starting
|
||||
|
||||
### Flexible Options
|
||||
- **Partial creation**: Create only specific data types
|
||||
- **Custom parameters**: Adjust event count and date range
|
||||
- **Dry run mode**: Preview without creating data
|
||||
- **Verbose output**: Detailed progress information
|
||||
- **Custom API URLs**: Support for different backend configurations
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Connection Error**
|
||||
```
|
||||
❌ Cannot connect to the API server at http://localhost:8000
|
||||
```
|
||||
**Solution**: Make sure the backend server is running:
|
||||
```bash
|
||||
cd backend
|
||||
uvicorn main:app --reload
|
||||
```
|
||||
|
||||
2. **Module Not Found Error**
|
||||
```
|
||||
ModuleNotFoundError: No module named 'requests'
|
||||
```
|
||||
**Solution**: Install dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. **Database Connection Error**
|
||||
**Solution**: Ensure PostgreSQL is running and database exists:
|
||||
```bash
|
||||
# Check if PostgreSQL is running
|
||||
brew services list | grep postgresql
|
||||
|
||||
# Start PostgreSQL if needed
|
||||
brew services start postgresql
|
||||
```
|
||||
|
||||
4. **Partial Data Creation**
|
||||
If some items fail to create, the script will continue and report the final count.
|
||||
Use `--verbose` to see detailed error messages.
|
||||
|
||||
### Performance Tips
|
||||
|
||||
- **Large datasets**: For creating many events (100+), consider running in smaller batches
|
||||
- **Network issues**: Use `--verbose` to identify specific failures
|
||||
- **Database performance**: Ensure your database has adequate resources for bulk operations
|
||||
|
||||
## Data Cleanup
|
||||
|
||||
To remove all test data:
|
||||
|
||||
```bash
|
||||
python cleanup_test_data.py
|
||||
```
|
||||
|
||||
This script will:
|
||||
1. Show what data exists
|
||||
2. Ask for confirmation before deletion
|
||||
3. Handle foreign key constraints properly
|
||||
4. Provide progress feedback
|
||||
|
||||
## Integration with Application
|
||||
|
||||
After running the test data scripts:
|
||||
|
||||
1. **Frontend**: Refresh the application to see new data
|
||||
2. **API**: All endpoints will return the test data
|
||||
3. **Database**: Data is persisted and will survive server restarts
|
||||
|
||||
The test data is designed to showcase all application features:
|
||||
- Multiple shops and locations
|
||||
- Diverse grocery categories
|
||||
- Realistic shopping patterns
|
||||
- Price variations and organic options
|
||||
- Historical data for analytics
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Custom Data Sets
|
||||
|
||||
You can modify the data arrays in `create_test_data.py` to create custom test scenarios:
|
||||
|
||||
- Add more shops for specific regions
|
||||
- Include specialty grocery categories
|
||||
- Adjust price ranges for different markets
|
||||
- Create seasonal shopping patterns
|
||||
|
||||
### Automated Testing
|
||||
|
||||
The scripts can be integrated into automated testing workflows:
|
||||
|
||||
```bash
|
||||
# Setup test environment
|
||||
python create_test_data.py --events 10 --days 30
|
||||
|
||||
# Run tests
|
||||
pytest
|
||||
|
||||
# Cleanup
|
||||
python cleanup_test_data.py --force # (if you add a --force option)
|
||||
```
|
||||
Reference in New Issue
Block a user