Workstream Overview

  1. A source feed exists that we want to set up as an HTTP Live Stream (HLS) for ingest into the aiWARE platform for cognitive processing and data insights
  2. The source feed is fed into an encoder which is configured to generates a HLS and compatible with AWS MediaStore that is transmitted to a defined AWS MediaStore container
  3. HLS manifest and .ts segments are sent over an internet connection to an AWS MediaStore bucket as the stream grows
  4. AWS MediaStore will store the manifest and segments and act as an access point for aiWARE to process from even after the source has been terminated. This adds redundancy to go back and reprocess streams that have been archived.
  5. The .m3u8 manifest on AWS MediaStore will be used to generate a duplicate HLS that originates from AWS MediaStore and will be sent over an internet connection to aiWARE platform for ingest
  6. A Veritone adapter, specific to television and radio ingestion, receives the HLS stream from MediaStore and starts cognitive processing

Requirements

  • Encoder: An encoder that is compatible with AWS MediaLive will be used to push content to AWS MediaStore (https://aws.amazon.com/media-services/resources/compatibility/)
  • AWS account: Compatible with an existing client account or Veritone can create a managed account on behalf of the client 
  • AWS MediaStore container: This is used to receive and store streams sent from the encoder and eventually routed to Veritone
  • Veritone aiWARE Organization: A Veritone organization is required to ingest the stream and assign processing as it comes in from AWS MediaStore 

Setup Steps

  1. Setup an AWS Media Store
  2. Setup Encoder
  3. Start a Live Event
  4. Setup an aiWARE Ingestion Job

1) Setup an AWS Media Store

Setup AWS Users

You will need to create a user group for each AWS Elemental MediaStore policy and assign users to a group rather than attaching individual policies to each user. Using the following procedure you can create a group for the user we will use to set up the encoder using the AWSElementalMediaStoreFullAccess policy and one for read-only access using the AWSElementalMediaStoreReadOnly policy, which are preexisting AWS managed policies.

Create User Groups

  1. In the navigation pane of the IAM console, choose Groups, and then choose Create New Group.
  2. On the Groups page, choose Create New Group, and then create an administrator group using the AWSElementalMediaStoreFullAccess policy.
  • On the Set Group Name page, enter a name for the group such as MediaStoreAdmins 
  • Choose Next Step
  • On the Attach Policy page, for Filter, choose AWS Managed, and then enter mediastore . In the policy list, choose the AWSElementalMediaStoreFullAccess policy.
  • Choose Next Step
  • On the Review page, choose Create Group

3. On the Groups page, choose Create New Group, and then create a read-only group using the AWSElementalMediaStoreReadOnly policy.

  • On the Set Group Name page, enter a name for the group such as MediaStoreReaders
  • Choose Next Step
  • On the Attach Policy page, for Filter, choose AWS Managed, and then enter mediastore
  • In the policy list, choose the AWSElementalMediaStoreReadOnly policy. Choose Next Step. On the Review page, choose Create Group

Create Users
Create IAM users for the individuals who require access to AWS Elemental MediaStore, and add each user to the appropriate user group to ensure that they have the right level of permissions.

  1. In the navigation pane of the IAM console, choose Users, and then choose Add user
  2. For User name, enter the name that the user will use to sign in to MediaStore.
  3. Select the check box next to Programmatic access
  4. Choose Next: Permissions
  5. On the Set permissions for user page, choose Add user to group
  6. In the group list, choose the group with the appropriate attached policy. Remember that permissions levels are as follows:
  • The MediaStoreAdmins group has permissions that allow all actions on all resources in MediaStore. (Used for the encoder)
  • The MediaStoreReaders group has permissions that allow read-only rights for all resources in MediaStore.

7. Choose Next: Tags
8. Choose Next: Review to see the list of group memberships that will be added to the new user
9. When you are ready to proceed, choose Create user
10. Download .csv with credentials

  • Credentials will be used when setting up the encoder to give it access to write to the AWS MediaStore container you will set up in the next section

Creating a MediaStore Container

Now that you have created a group and users you are ready to create a MediaStore container to hold the output of the stream that will come from the encoder.

Access AWS Elemental MediaStore

  1. Log into AWS account
  2. Under services search for MediaStore and select service (https://console.aws.amazon.com/mediastore/)

Create a Container

  1. Click "Create container" in upper right
  2. Enter "Container name" (Rules for Container Names)
  3. Click "Create container"    
  • It can take up to 30 seconds for the container to be created at which point it is unavailable, when successfully created you will receive a message

2) Setup Encoder

Now that you have a MediaStore container set up you are ready to set up the encoder to send a stream to the MediaStore. These steps are specific to an Elemental Live Encoder, which is our preferred encoder because it is specifically designed to work with AWS Media Services services, but there are plenty of other encoders that can be used with similar compatibility (MediaStore Compatible Encoders).

Security Best Practices

  1. Before you begin the process of setting up your new appliance, be sure to review security best practices. These include:
  • Consider the physical security of your deployment
  • Deploy on private, secure networks. These appliances are not intended to be deployed on the public internet
  • Determine how you want to handle authentication and configure your appliance accordingly
  • Ensure that proper monitoring is enabled
  • Enable HTTPS during installation
  • For a complete list of security best practices including product-specific guidance, refer to AWS Elemental Security Practices & Policies (AWS_Elemental_Security_Practices_20181015.pdf)

3) Start the Live Event

  1. Every encoder will be unique into how you can set up a Live Event HLS stream to be sent to an AWS Media Store endpoint. Please make sure your encoder is a MediaStore Compatible Encoder and see the documentation on your specific encoder to understand how to set this up.

General Overview

  • Set up your encoder to take incoming source and output to a defined HLS encoding profile
  • Setup encoder to output the Media Store container that you have setup using the containers unique Data Endpoint
  • Start the Live Event Stream so it begins pushing to AWS MediaStore Container

4) Setup an aiWARE Ingestion Job

  1. Log into Veritone CMS (https://cms.veritone.com/#/)
  2. Click on the "Ingestion Jobs" link in the left panel
  3. Click "New Ingestion Job"
  4. Click on "Streams" tab at the top
  5. Click "Radio & TV"
  6. Click "Next"

Define Ingestion Job Configuration

  1. If you have already set up a source with the required configurations then you can reuse that as your source. Otherwise, you will need to create a new source by clicking "Create New Source"

Fill out the "Source Configuration" specs

  1. Source Name (required): Give your source a name to help uniquely identify it
  2. Band (required): Band Associated with your stream (i.e. FM/AM/TV)
  3. Live Time Zone (required): Time zone that your source originates in
  4. Stream URL (required): This is the Object Name from MediaStore for your .m3u8 manifest for the Stream URL field in your configuration and can be found in AWS MediaStore
  5. Fill in any other applicable optional information
  6. Click "Create"

Example: How to find object name for your "Stream Uri"

  • Go to your MediaStore container
  • find the .m3u8 object associated with your live stream
  • Copy the object name and paste into Stream URI field in aiWare

Finish defining configuration settings

  1. Recommended: Leave cluster and segment duration as default
  2. Click "next"

Define Program Information

Information in this section is optional but is where you can add more detail about the specific program associated with your source.

  • Images: Here is where you can add a logo/image for your source
  • Program Website: URL directing to a programs homepage
  • Program Format: The format associated with your program

Click Next

Define Schedule

Recurring: Capture your stream at a repeating cadence

Continuous: Source will be captured continuously for the defined period of the job and will not repeat

On Demand: Source will be captured on demand

Click Next

Define Cognition Processing for Ingestion Job

Simple Cognitive Workflow
Here you can select a simple cognitive workflow in the cognitive categories of your choosing

Advanced Cognitive Workflow

Allows for defining more advanced cognitive workflows to tailor workflows to more unique use cases.

Click Next

Define a Content Template

Add a Content template if you would like to associate some predefined Structured Data to the Ingestion Job

Customize

  1. Add an Ingestion Job Name
  2. Add any tags to be applied to Ingestion Job
  3. Save

And that's it -- you're all finished! 

From here you'll want to validate that the HLS feed is successfully routed from the encoder to the MediaStore to Veritone's aiWARE. In Veritone's CMS you can monitor the status of you ingestion job by selecting 'Processing Status'.

For any additional assistance, you can reach our team via the live support channel in the bottom right corner of your screen or contact [email protected]

Did this answer your question?