Filamenteka/scripts

Data Migration Scripts

This directory contains scripts for migrating filament data from Confluence to DynamoDB.

Prerequisites

1. AWS credentials configured (either via AWS CLI or environment variables)
2. DynamoDB table created via Terraform
3. Confluence API credentials (if migrating from Confluence)

Setup

cd scripts
npm install

Configuration

Create a .env.local file in the project root with:

# AWS Configuration
AWS_REGION=eu-central-1
DYNAMODB_TABLE_NAME=filamenteka-filaments

# Confluence Configuration (optional)
CONFLUENCE_API_URL=https://your-domain.atlassian.net
CONFLUENCE_TOKEN=your-email:your-api-token
CONFLUENCE_PAGE_ID=your-page-id

Usage

Migrate from local data (data.json)

npm run migrate

Clear existing data and migrate

npm run migrate:clear

Manual execution

# Migrate without clearing
node migrate-with-parser.js

# Clear existing data first
node migrate-with-parser.js --clear

What the script does

1. Checks for Confluence credentials

   • If found: Fetches data from Confluence page
   • If not found: Uses local public/data.json file

2. Parses the data

   • Extracts filament information from HTML table (Confluence)
   • Or reads JSON directly (local file)

3. Prepares data for DynamoDB

   • Generates unique IDs for each filament
   • Adds timestamps (createdAt, updatedAt)

4. Writes to DynamoDB

   • Writes in batches of 25 items (DynamoDB limit)
   • Shows progress during migration

5. Verifies the migration

   • Counts total items in DynamoDB
   • Shows a sample item for verification

Troubleshooting

• Table not found: Make sure you've run terraform apply first
• Access denied: Check your AWS credentials and permissions
• Confluence errors: Verify your API token and page ID
• Empty migration: Check that the Confluence page has a table with the expected format