Modern scientific instruments operate at such high rates that high-performance computers are needed to extract knowledge from vast amounts of data. But traditional file-based data movement and analysis methods cannot keep up with such data rates; as a consequence, either instruments are operated at less than maximum rates, or significant portions of collected data are discarded. In either case, valuable opportunities for scientific discovery are missed. To overcome this problem, the SciStream project will develop software tools to enable memory-to-memory  streaming of data directly from scientific instruments to remote high-speed computers. 

High-speed memory-to-memory data movement is challenging because data producers (e.g., data acquisition applications on scientific instruments, simulations on supercomputers) and consumers (e.g., data analysis applications) may be in different security domains (and thus require bridging of those domains) and may, further, lack external network connectivity (and thus require traffic forwarding proxies). SciStream establishes necessary bridging and end-to-end authentication between source and destination, while providing efficient memory-to-memory data streaming. By exploring architectural and design choices and addressing issues of control protocols and security, SciStream will advance understanding of the challenges in supporting high speed memory-to-memory data streaming between remote instruments in federated science environments.

SciStream will benefit all scientific applications that require memory-to-memory data streaming between scientific instruments and computer systems. Recent trends suggest that this is an important and growing requirement for many scientific applications. SciStream will help significantly reduce the time to solution for these applications, resulting in improved scientific productivity and thus far-reaching benefits for society. Key design choices such as application-agnostic streaming and support for best-effort streaming will make SciStream appealing to a broader science community. SciStream will engage with domain scientists, campus computing centers, and scientific user facilities to reach a wider audience. Through on-campus programs at UChicago, SciStream will train under-represented students in computer networking.