Storage Unit [deprecated]

1. File Chunking Service

• Functionality:

  • Split files into chunks.

  • Generate unique chunk IDs.

  • Return metadata for each chunk (ID, size, order).

• Testing:

  • Handle large files and edge cases (e.g., empty files).

  • Ensure chunk IDs are unique.

  • Test performance with files of varying sizes.

2. Node Allocation Service

• Functionality:

  • Use consistent hashing for chunk-to-node mapping.

  • Enable dynamic addition and removal of nodes.

  • Support replication with fault tolerance.

• Testing:

  • Verify the balanced distribution of chunks.

  • Test node addition/removal without significant redistribution.

  • Ensure chunks are allocated to multiple nodes for redundancy.

3. Metadata Management

• Functionality:

  • Store and manage mappings:

    • File-to-chunks.

    • Chunk-to-nodes.

  • Persist metadata in PostgreSQL.

• Testing:

  • Validate metadata integrity.

  • Test query performance for large datasets.

  • Ensure compatibility with chunking and node allocation service