Important Note: This article is written for a technical audience. The information in this article is intended for use by DevOps/IT staff or consultants affiliated with Veritone customers.
Creating a Cluster is an essential step in setting up an aiWare for Xcellis account. A Cluster defines a grouping of nodes, networks, storage, and resources for increased cognitive processing performance, capacity, and reliability. Once a Cluster has been created, your organization's media content can be ingested and processed with cognitive analytics technology.
Memory & Storage Size
Cluster settings allow you to specify sizes for per-task memory and overall storage to meet your specific needs. The Cluster Size, which determines the amount of per-task disk space, auto-populates with a default value of 10 GB and can be adjusted up to 1,000 GB. Because some engines require an amount of space equal to the file size for processing large video files, it's recommended to adjust the Cluster Size to exceed the individual file size. For example, processing 100 GB of video with a 5 GB engine would require a minimum Cluster Size of 105 GB.
Add a Device
Within a Cluster, a device can serve as an Ingestor, Task Runner, or both ("dual-role").
- Ingestor: Discovers and catalogs local media and creates jobs and tasks.
- Task Runner: Executes engine processing for jobs.
- A dual-role device serves as both an Ingestor and Task Runner simultaneously.
Important Device Notes:
- For maximum versatility and capability, all devices are pre-configured for dual-role functionality by default. The Task Runner setting can be disabled on the Devices tab when creating new Cluster or modifying an existing one.
- Although a Cluster can contain multiple devices, a device can only be associated with one Cluster.
The final step in setting up a Cluster is selecting the engines that will be made available for processing media. The Engine table is organized by function and includes engines that can run locally on-prem or in Veritone's cloud. To keep all engine processing within your local environment, simply enable only the network-isolated engines. Currently, Veritone offers the following network-isolated engines:
- Transcription: Golgi and Callosum
- Object Detection: Nissi
- Text Recognition: Fossa
If non-network-isolated engines are selected, media may be processed in the Veritone cloud by the engine provider.
To Create a Cluster
Click the App Picker icon in the upper right of the window and select Admin from the drop-down list. The Admin application opens.
Click the aiWARE tile. The aiWARE page opens to the Clusters tab.
Click Add Cluster in the upper right of the window. The aiWARE Cluster opens to the General tab.
1. By default, a generic Cluster name is provided. Enter a new name to identify the
2. Cluster Size sets the maximum disk space allowed per task. By default, the Cluster
Size auto-populates to 10 GB. Accept the default value or enter a custom size up to
3. Cluster Memory is the amount of space that will be set aside per processing job,
leaving the remainder for concurrency. By default, the Cluster Memory defaults to
4 GB. Accept the default value or click the drop-down arrow and select a different
size from the list.
4. Click Next in the bottom right of the window. The Devices tab opens.
5. Click Add Device in the upper right of the window. The Enter Unique ID window
6. Enter one or more aiWARE Node IDs separated by commas.
7. Click ADD when all Node IDs have been entered. The added Nodes display on the
8. Click the pencil icon to change the name of the Node. (optional)
9. The Task Runner is enabled by default. Click the toggle button to enable Online
Browsing that allows users to access directories even when they're not connected
to the network.
10. Click Save in the lower right corner of the window. The Processing tab opens to
the Engine table.
Add Engines to the Cluster
11. Click an Engine Category on the left menu to view available Engine options.
12. Click the Enabled toggle button to add an Engine to the Cluster. Active Engines
are toggled on in blue.
Note: Network-isolated Engines are easily identified by the blue and white shield.
13. To quickly enable or disable all Engines in an Engine Category, click the vertical
ellipsis icon and select an option from the drop-down.
14. Click Save in the bottom right corner of the window when all Engines are added to
the Cluster. The Engine table closes and the new Cluster displays at the top of the
list on the aiWARE Cluster page.