ITC’s team wins “Most innovative for HPC uses” award at Data Mover Challenge 2023
The Information Technology Center’s research team consisting of Professors Ryo Nakamura, Yohei Kuga, and Kunio Akashi has been selected as the winner of the “Most innovative for HPC uses” award at the Data Mover Challenge.
The team participated in the competition by developing a software “multi-threaded scp” (mscp). MSCP leverages SSH (Secure Shell), a protocol widely used for logging into remote computers over a network, to establish multiple concurrent connections, enabling high-speed file transfers. Although other high-speed file transfer tools, such as GridFTP, exist, they require installation of dedicated software and a complex set-up adapted to the network environment on both the sending and receiving computers due to their use of proprietary protocols.
Contrarily, mscp takes advantage of SSH, commonly installed on many computers already, thus eliminating the need for users to install any special server software. As long as the environment allows remote login via SSH, users can immediately begin using mscp.
Using mscp is as simple as using the scp command, which is available on a vast number of computers. In the actual transfer test during Data Mover Challenge 2023, mscp proved up to 240 times faster than the regular scp, as demonstrated by transferring files from Australia to Japan.
The award ceremony took place on Feb. 22nd in the conference Supercomputing Asia 2024, held in Sydney, Australia.
Upon receiving this award, Associate Professor Ryo Nakamura expressed, “We will continue to develop and enhance mscp, encouraged by this award, so that it can be widely used and contribute to the community.”
The mscp software is openly available through the following URL as an open source software: https://github.com/upa/mscp
An academic paper on mscp has been accepted at the international conference, Practice and Experience in Advanced Research Computing 2023 (PEARC 23), held in Portland, Oregon, in June 2023. The paper is open access and anyone can read it at: https://dl.acm.org/doi/10.1145/3569951.3597582
The team participated in the competition by developing a software “multi-threaded scp” (mscp). MSCP leverages SSH (Secure Shell), a protocol widely used for logging into remote computers over a network, to establish multiple concurrent connections, enabling high-speed file transfers. Although other high-speed file transfer tools, such as GridFTP, exist, they require installation of dedicated software and a complex set-up adapted to the network environment on both the sending and receiving computers due to their use of proprietary protocols.
Contrarily, mscp takes advantage of SSH, commonly installed on many computers already, thus eliminating the need for users to install any special server software. As long as the environment allows remote login via SSH, users can immediately begin using mscp.
Using mscp is as simple as using the scp command, which is available on a vast number of computers. In the actual transfer test during Data Mover Challenge 2023, mscp proved up to 240 times faster than the regular scp, as demonstrated by transferring files from Australia to Japan.
Relationship between the number of simultaneous connections and transfer speed using mscp. Under the conditions of this test, it can be observed that the transfer speed increases up to eight connections.
The award ceremony took place on Feb. 22nd in the conference Supercomputing Asia 2024, held in Sydney, Australia.
Upon receiving this award, Associate Professor Ryo Nakamura expressed, “We will continue to develop and enhance mscp, encouraged by this award, so that it can be widely used and contribute to the community.”
The mscp software is openly available through the following URL as an open source software: https://github.com/upa/mscp
An academic paper on mscp has been accepted at the international conference, Practice and Experience in Advanced Research Computing 2023 (PEARC 23), held in Portland, Oregon, in June 2023. The paper is open access and anyone can read it at: https://dl.acm.org/doi/10.1145/3569951.3597582
