This study (dataset is detailed in our paper ) focuses on reporting of Internet malicious activity (or mal-activity in short) by public blacklists with the objective of providing a systematic characterization of what has been reported over the years, and more importantly, the evolution of reported activities. Using an initial seed of 22 blacklists, covering the period from January 2007 to June 2017, we collect more than 51 million mal-activity reports involving 662K unique IP addresses worldwide. Leveraging the Wayback Machine, antivirus (AV) tool reports and several additional public datasets (e.g., BGP Route Views and Internet registries) we enrich the data with historical meta-information including geo-locations (countries), autonomous system (AS) numbers and types of mal-activity. Furthermore, we use the initially labelled dataset of approx. 1.57 million mal-activities (obtained from public blacklists) to train a machine learning classifier to classify the remaining unlabeled dataset of approx. 44 million mal-activities obtained through additional sources.
We make our unique collected dataset (and scripts used) publicly available for further research.
The main contributions of the study (dataset is detailed in our paper ) are a novel means of report collection, with a machine learning approach to classify reported activities, characterization of the dataset and, most importantly, temporal analysis of mal-activity reporting behavior. Inspired by P2P behavior modeling, our analysis shows that some classes of mal-activities (e.g., phishing) and a small number of mal-activity sources are persistent, suggesting that either blacklist-based prevention systems are ineffective or have unreasonably long update periods. Our analysis also indicates that resources can be better utilized by focusing on heavy mal-activity contributors, which constitute the bulk of mal-activities.
Our paper has been presented at the 14th ACM ASIA Conference on Computer and Communications Security (ACM AsiaCCS 2019).
Our scripts used for the publication can be found in our Github repo.
Benjamin Zi Hao Zhao, Muhammad Ikram, Hassan Jameel Asghar, Mohamed Ali Kaafar, Abdelberi Chaabane, and Kanchana Thilakarathna. 2019. A Decade of Mal-Activity Reporting: A Retrospective Analysis of Internet Malicious Activity Blacklists. In Proceedings of the 2019 ACM Asia Conference on Computer and Communications Security (Asia CCS '19). ACM, New York, NY, USA, 193-205. DOI: https://doi.org/10.1145/3321705.3329834
For industry: Please send us an email from official email account and briefly introduce yourself and your company. Please also attach a justification letter (PDF) in official letterhead. The justification letter should clearly stat the reasons why the data is being requested.
Historical malware dataset for this study is based on Optus Macquarie University Cybersecurity Hub's Internet malware analysis project.