Abstract
Cache coherent non-uniform memory access (CC-NUMA) machines provide significant performance benefits for scientific, engineering and commercial applications. They employ caches to reduce remote memory access overheads and provide shared memory programming. Thus, high system performance and fast program development are achieved. The most critical component in a CC-NUMA multiprocessor is the communication system, because it impacts memory access performance directly. The thesis thus focus on the design of the communication system in a CC-NUMA multiprocessor.We first study techniques to reduce the load overhead in accessing remote memories. The concept of directory hint is proposed and one such implementation is examined. Directory hints help a node in a CC-NUMA multiprocessor to keep track where valid copies of a memory block may reside. With this information the node can fetch the block directly from those nodes on a read miss. Our simulation results show that the directory hint scheme can effectively reduce the read stalltime. Also its performance is very competitive compared with a more expensive implementation which uses a large level-three cache. A drawback of the scheme is that it will introduce extra network traffic.We next investigate an enhanced CC-NUMA architecture, called {\it Depot-NUMA}, for reducing the load overhead. Depot-NUMA relies on fast interconnects and views the congregation of the private caches in all the nodes as a large remote access cache. Through fast interconnects, a memory block can be fetched from the private caches of other sharing nodes instead of from the home node. A novel multicast routing scheme, called {\it multi-hop}, is proposed to communicate between the potential sharers and fetch a missing block from their private caches. From the application-driven study, we find that Depot-NUMA can reduce the read stall time up to 41\% and is competitive compared with a CC-NUMA with a large local cache.For techniques that reduce the overhead of write operations in CC-NUMA multiprocessors, we study the use of multicast messages to invalidate the sharing nodes. We evaluate two multicast-based invalidation schemes (dual-path~\cite{lin94b} and pruning~\cite{malu96}) from architecture and application perspectives. We conclude that multicast-based invalidation reduces invalidation traffic and improve the overall execution time for four of the six evaluated applications. Dual-path outperforms pruning for inducing a smaller delay routing. Additionally, cache size has a large impact on system performance because fewer sharing blocks can be cached in a machine with small local caches. Finally network bandwidth is not essential for improving invalidation traffic in a CC-NUMA multiprocessor.We finally get to the core of the communication system of CC-NUMA multiprocessors --- the {\it node controller}. Node controllers perform protocol processing to transmit messages with other nodes in the system. We study the design of a thread-based node controller, in which protocol engines have a multithreaded architecture. Four important design parameters are examined: (1) the number of thread context storages, (2) the number of protocol operation units, (3) the scheduling policy and (4) the thread allocation scheme. From the application-driven simulation on six representative applications, we conclude that the number of thread contexts and protocol operation units have a great impact on the overall system performance. An appropriate thread allocation scheme for invalidation traffic is needed, and prioritizing a thread and scheduling it accordingly are also important for the system performance.