How to bulk ingest from AWS S3

  • Updated on Oct 14, 2024
  • Published on Apr 18, 2024
  • 1 minute(s) read

Applications this article relates to: Curator Connect

Curator version supported from this article: 3.0

This guide details how to create placeholder assets and ingest a large number of files from one AWS S3 bucket (folder). Bulk ingesting is useful as to ensure you can ingest all of of your assets collectively and do not want to start ingesting until you have everything compiled ready for ingest to one location. This is especially useful for Camera cards and multi-file assets.

Learn more how to Prepare for AWS S3 for bulk ingest

Ingesting the Manifest File

  1. From the Dashboard, open the S3 Bulk Ingest application.
    NOTE: If you do not see the application in your dashboard, please contact IPV support.

image.png

  1. Submit the information required including mandatory and non-mandatory metadata. The other non-required metadata can be filled out as you wish. The metadata will be applied to all of the assets created. If you want to revise the metadata fields you can submit data for to tag to the assets on S3 Bulk Ingest, learn more about Managing asset metadata

S3 Bulk Ingest mandatory metadata:

  • S3 Bucket: Enter the bucket in which your manifest file was placed earlier.
  • S3 Manifest File Path: Enter the file name and extension of your Manifest file.

image.png

  1. When you’re ready, select Submit.
  2. Your placeholder assets will now be created.

Process Monitoring S3 Bulk Ingest

  1. Open the Process Engine application from the Dashboard.
  2. Navigate to the Processes tab.
  3. Search for the Plug-in - Send to GenericV4 workflow.
  4. Find the workflow that corresponds to your placeholder. Expanding the parent process using the drop-down arrows will display the child processes.

image.png

  1. Ensure that this matches the time you selected the Submit button.
  2. The process status will turn from yellow to green when complete.
  3. Once complete, navigate to the Clip Link application from the Dashboard. You should be able to see your assets in a placeholder state.
  4. Once a placeholder is created, a scavenge should gather the placeholder assets to process for the remaining ingest steps such as Proxy, Mediainfo and