# Data Migration Scripts

This directory contains scripts for migrating filament data from Confluence to DynamoDB.

## Prerequisites

1. AWS credentials configured (either via AWS CLI or environment variables)
2. DynamoDB table created via Terraform
3. Confluence API credentials (if migrating from Confluence)

## Setup

```bash
cd scripts
npm install
```

## Configuration

Create a `.env.local` file in the project root with:

```env
# AWS Configuration
AWS_REGION=eu-central-1
DYNAMODB_TABLE_NAME=filamenteka-filaments

# Confluence Configuration (optional)
CONFLUENCE_API_URL=https://your-domain.atlassian.net
CONFLUENCE_TOKEN=your-email:your-api-token
CONFLUENCE_PAGE_ID=your-page-id
```

## Usage

### Migrate from local data (data.json)

```bash
npm run migrate
```

### Clear existing data and migrate

```bash
npm run migrate:clear
```

### Manual execution

```bash
# Migrate without clearing
node migrate-with-parser.js

# Clear existing data first
node migrate-with-parser.js --clear
```

## What the script does

1. **Checks for Confluence credentials**
   - If found: Fetches data from Confluence page
   - If not found: Uses local `public/data.json` file

2. **Parses the data**
   - Extracts filament information from HTML table (Confluence)
   - Or reads JSON directly (local file)

3. **Prepares data for DynamoDB**
   - Generates unique IDs for each filament
   - Adds timestamps (createdAt, updatedAt)

4. **Writes to DynamoDB**
   - Writes in batches of 25 items (DynamoDB limit)
   - Shows progress during migration

5. **Verifies the migration**
   - Counts total items in DynamoDB
   - Shows a sample item for verification

## Troubleshooting

- **Table not found**: Make sure you've run `terraform apply` first
- **Access denied**: Check your AWS credentials and permissions
- **Confluence errors**: Verify your API token and page ID
- **Empty migration**: Check that the Confluence page has a table with the expected format