We are customer-centric and provide customized or one-stop full-stack solutions to empower all industries.
xx new media produces a large number of video and audio media programs every day, providing multi-content, full-service, high-quality program services to more than one million users. With the increasing accumulation of video content and the expansion of content distribution, higher requirements are placed on production and storage systems. Therefore, establishing a complete and highly secure material storage system is one of the most important tasks today.
The construction of the new project will provide a high-performance data storage platform for the storage of the new media assets library of xx station. While satisfying the storage space, the storage system is also required to be stable and reliable, high-performance, easy to expand, and easy to maintain. The detailed analysis of the storage requirements of the new media assets library of xx station is as follows:
1. Based on the current and recent data capacity growth estimates, the storage system needs to provide nearly 500TB of effective storage space.
2. The storage system has good scalability to cope with the larger storage scale requirements after the media asset system is expanded
3. The business system requires the storage system to provide a stable bandwidth of 3 GB/s.
4. Highly available storage: the system has complete autonomy to ensure that data is not lost and storage services are not shut down.
5. Multi-protocol support makes it easy for various servers and workstations to access and obtain storage resources.
6. Simple configuration, easy maintenance, convenient use and later management.
According to user needs and combined with user’s actual application scenarios, the PL-XDFS distributed file storage system is used in this project to build a massive, efficient and scalable distributed storage system.
The PL-XDFS distributed storage system provides high-performance, dynamic and highly scalable CIFS/FTP storage services for the media asset system through its rich protocol support, powerful horizontal expansion capabilities, and ultra-high throughput performance. At the same time, combined with its powerful dynamic expansion capabilities, it can quickly achieve dynamic expansion of performance or capacity as business develops, greatly simplifying the difficulty of system planning and reducing initial investment.
In this solution, five PL-XDFS nodes are configured, each with 24 8TB hard disks, with a total raw capacity of 960TB. With the 4+1 erasure code redundancy strategy, considering the large capacity of a single disk, the long time it takes to recover from a failure, and the large impact on the load of the business system, RAID protection is configured inside each node, and two groups of 12 RAID5+HotSpare are configured for every 24 disks. Therefore, the final capacity utilization is 960TB*20/24*0.8*0.91=582TB. Each node provides 600MB/s of bandwidth output, and the total output bandwidth can reach more than 3GB/s. Both capacity and performance can meet business needs.
Each PL-XDFS is configured with 2*10Gbe+2*1Gb network cards. The five PL-XDFS nodes are interconnected by a 10G network. Streaming media and new media workstations use 10G network cards to access the 10G network. Other business servers use 1G network cards and the 1G network ports of each PL-XDFS to achieve business networking.
PL-XDFS supports POSIX/NFS/CIFS/FTP protocols at the same time. It can realize multi-client cross-platform access without installing any client, meeting the needs of different host platforms of the media platform to share and access media resources.
In addition to erasure code protection, PL-XDFS also supports replica protection. In the future, for some data that requires higher reliability, replicas can be used for enhanced protection.
The core of PL-XDFS is the XDFS distributed file system. XDSF is a massive distributed storage system launched by PowerLeader for unstructured data application environments based on the industry-leading and widely used GlusterFS distributed file system, combined with the technical team’s many years of engineering and system optimization experience. XDFS can achieve dynamic capacity and performance expansion and excellent system high availability; with the help of virtualized and parallelized volume groups and file system resource management, a large-scale, high-performance, dynamically scalable shared storage architecture can be built. XDFS can achieve a massive storage scale of 192PB and high performance that can be linearly expanded.
1. Distributed storage cluster, decentralized architecture, no need for independent metadata server
2. Intelligent data positioning based on DHT algorithm effectively improves the efficiency of media resource retrieval
3. All-Ethernet single network architecture, no need for dedicated Fibre Channel equipment (FC-SAN)
4. Comprehensive use of RAID and erasure code data protection effectively realizes composite data within and between nodes
Protect and reasonably control the system resource usage and network usage of data repair
5. Standard client design, no need to install any client on the business host, allowing the business host to focus on the application
6. Expandable copy protection to increase the availability of critical data
7. High-density equipment, using 4U/24bay equipment, effectively reducing space and energy consumption, and reasonably controlling costs
8. The system has no single point of failure, data is highly available, and storage services do not stop.
9. The system supports data tiering. SSD or SAS can be added in the future to achieve automatic tiered storage according to data hotspots.
PL-XDFS adopts a completely decentralized distributed file storage architecture, without an independent metadata server design, eliminating the metadata information interaction bottleneck; Scale-out dynamic expansion, rapid online addition and deletion of nodes, up to 1024 node clusters can be achieved, and nearly 100PB of massive data can be managed.
PL-XDFS provides distributed file read, write and share services. The 5-node cluster configured this time can achieve 3GB performance and manage massive files. In the future, it can be expanded through clusters to achieve 100GB bandwidth. It supports 10G network cards and Infiniband networks, and optimizes concurrent read and write performance through RDMA. It can also achieve data tiering or small file performance optimization through SSD.
PL-XDFS provides two data/cluster protection modes: replication and erasure coding. RAID protection is also optional within the node. Users can choose the configuration mode based on application type, data importance, performance, and cost balance to achieve high-availability protection for data and storage services. XDFS can achieve online expansion without downtime, greatly reducing or eliminating planned downtime.
