ACCELERATING KX® KDB+ WITH DDN SFA® AND INTEL® ENTERPRISE EDITION LUSTRE*
Kx kdb+ is a high-performance column-store database with a built-in expressive query and programming language, q. Used as a central repository to store time-series data within an enterprise, kdb+ supports real-time analysis of billions of records and fast access to terabytes of historical data. With extreme scalability and high performance, kdb+ is widely adopted by financial institutions around the world.
Parallelism Delivers Better Strategies, Faster
By using fast, scalable, external disk systems with massively parallel access to data, researchers can perform analysis against much larger datasets than they can by batching large data sets through memory. Visionary hedge funds, proprietary trading firms, and other financial institutions have been changing their infrastructure to take advantage of parallelism so as to analyze more positions faster and to develop more effective trading and risk management strategies that can be deployed in less time.
Advantages of Running Kdb+ over Lustre
A parallel file system offers several advantages over a single, direct-attached file system. When it is used in conjunction with the in-memory database kdb+, the advantages are:
- Significantly decreases operational latency per kdb+ query, especially when running queries that search through significant amounts of historical market information, by balancing content around multiple file system servers
- Allows a user to treat any data workload independently from other data workloads because of parallelization of kdb+ query “threads” in a single shared namespace
- Supports simultaneous read/write operations on a single namespace for the entire database and for any number of kdb+ clients (i.e., end-of-day data consolidations into an hdb instance)
- Allows for sharing of data among different independent hdb/rdb instances: many instances of kdb can view the same data, meaning that strategies for data sharing and private data segments may be consolidated onto the same space, thereby avoiding the need for kdb+ administrators to copy data physically around the network or disks, as automatic space allocation balancing is built-in to the parallel FS
- “Stripes” kdb+ context around all FS servers or allocates in a round-robin fashion against each server (striping allows the opportunity for some files to attain maximal I/O rates for a single kdb+ “object”)
A recent STAC-M3 benchmark analysis for DDN, Intel, and Kx showed the combined solution provided up to 500% faster results than any other storage solution on the market, including SSD-direct attached storage or all flash arrays; 50% better latency characteristics on IO intensive workloads; and unmatched bandwidth, latency, and runtime for large scale workflows.
- Scaling storage IO performance linearly or near-linearly as Kx servers are added
- More than 3x improvement in algorithm development speeds
- Shared access to large amounts of data over multiple, internal teams
- Reduced time to completion of key metrics by more than 80%