Botnets are increasingly being held responsible for most of the cybercrimes that occur nowadays. They are used to carry out malicious activities like banking credential theft and Distributed Denial of Service (DDoS) attacks to generate profit for their owner, the botmaster. Traditional botnets utilized centralized and decentralized Command-and-Control Servers (C2s). However, recent botnets have been observed to prefer P2P-based architectures to overcome some of the drawbacks of the earlier architectures.

A P2P architecture allows botnets to become more resilient and robust against random node failures and targeted attacks. However, the distributed nature of such botnets requires the defenders, i.e., researchers and law enforcement agencies, to use specialized tools such as crawlers and sensor nodes to monitor them. In return to such monitoring, botmasters have introduced various countermeasures to impede botnet monitoring, e.g., automated blacklisting mechanisms.

The presence of anti-monitoring mechanisms not only render any gathered monitoring data to be inaccurate or incomplete, it may also adversely affect the success rate of botnet takedown attempts that rely upon such data. Most of the existing monitoring mechanisms identified from the related works only attempt to tolerate anti-monitoring mechanisms as much as possible, e.g., crawling bots with lower frequency. However, this might also introduce noise into the gathered data, e.g., due to the longer delay for crawling the botnet. This in turn may also reduce the quality of the data.

This dissertation addresses most of the major issues associated with monitoring in P2P botnets as described above. Specifically, it analyzes the anti-monitoring mechanisms of three existing P2P botnets: 1) GameOver Zeus, 2)Sality, and 3) ZeroAccess, and proposes countermeasures to circumvent some of them. In addition, this dissertation also proposes several advanced anti-monitoring mechanisms from the perspective of a botmaster to anticipate future advancement of the botnets. This includes a set of lightweight crawler detection mechanisms as well as several novel mechanisms to detect sensor nodes deployed in P2P botnets. To ensure that the defenders do not loose this arms race, this dissertation also includes countermeasures to circumvent the proposed anti-monitoring mechanisms. Finally, this dissertation also investigates if the presence of third party monitoring mechanisms, e.g., sensors, in botnets influences the overall churn measurements. In addition, churn models for Sality and ZeroAccess are also derived using fine-granularity churn measurements.

The works proposed in this dissertation have been evaluated using either real-world botnet datasets, i.e., that were gathered using crawlers and sensor nodes, or simulated datasets. Evaluation results indicate that most of the anti-monitoring mechanisms implemented by existing botnets can either be circumvented or tolerated to obtain monitoring data with a better quality. However, many crawlers and sensor nodes in existing botnets are found vulnerable to the antimonitoring mechanisms that are proposed from the perspective of a botmaster in this dissertation. Analysis of the fine-grained churn measurements for Sality and ZeroAccess indicate that churn in these botnets are similar to that of regular P2P file-sharing networks like Gnutella and Bittorent. In addition, the presence of highly responsive sensor nodes in the botnets are found not influencing the overall churn measurements. This is mainly due to low number of sensor nodes currently deployed in the botnets. Existing and future botnet monitoring mechanisms should apply the findings of this dissertation to ensure high quality monitoring data, and to remain undetected from the bots or the botmasters.