OSINT TOOLKIT: SPIDERFOOT, A CYBER-FOCUSED GITHUB TOOL THAT AUTOMATES DATA COLLECTION AND MAPS DIGITAL FOOTPRINTS
- Senior Editor
- 11 minutes ago
- 7 min read
Priscilla Alves Pereira, Dominic Bianco, Martina Elena Nitti, Christian Collins, OSINT-RDT Team
James Raggio, Editor; Jennifer Loy, Chief Editor
April 15, 2025
Industry: Online Investigations, Threat and Social Media Intelligence, Cybersecurity
(The Open Source Intelligence [OSINT] Toolkit is a report to help teach about various OSINT tools that can be used by Threat, Security, Intelligence, and Investigative Professionals [TSIIPs])
Investigating Digital Footprints[1]
What is the BLUF about the OSINT Tool?
SpiderFoot is a free OSINT tool that automates the collection and analysis of publicly available data related to IPs, domains, emails, and more. TSIIPs can leverage this tool to map digital footprints, identify potential vulnerabilities, and connect entities across data sources, streamlining investigations and enhancing situational awareness in cyber-focused intelligence operations.
What is the name of the OSINT Tool?
SpiderFoot
URL:
Who makes this tool?
Steve Micallef[2]
What country is this tool based out of?
Switzerland[3]
What is the purpose of the OSINT Tool?
SpiderFoot is an OSINT tool written in Python that is available on GitHub. It gathers and analyzes data about IP addresses, domain names, usernames, and other online assets. Designed to automate the process of open-source intelligence gathering, SpiderFoot enables users to map a target’s digital footprint. TSIIPs should utilize SpiderFoot to identify potential vulnerabilities, data leaks, digital footprints, and other security risks associated with a target.
What is the reason TSIIPs should use this OSINT Tool?
SpiderFoot is a free, open-source tool with a user-friendly interface that will likely help TSIIPs access its results more easily. SpiderFoot returns results from different data inputs, such as domains, email addresses, phone numbers, IP addresses, and more, providing TSIIPs with a comprehensive investigative tool. SpiderFoot does not need a login or subscription, enhancing operational and personal security (OPSEC/PERSEC). SpiderFoot includes different modes of investigation, such as “All,” “Footprint,” “Investigate,” and “Passive, facilitating TSIIPs in acquiring results based on their needs. SpiderFoot also connects results, enhances reconnaissance, and collates information in one place, streamlining online investigations on persons of interest (POI).
How should TSIIPs use this OSINT Tool?
TSIIPs should use SpiderFoot for preliminary surveillance and information collection regarding persons and groups of interest during OSINT and social media intelligence (SOCMINT) investigations. Additionally, TSIIPs can use this tool to track and monitor both social media presence and the physical movements of targets. TSIIPs should use this tool to gather reconnaissance, analyze data to validate threats, and keep tabs on persons or groups who may display traits and actions associated with the pathway to violence. Users directly hunting clear and present dangers should consider utilizing this tool to proactively identify social media accounts and domains advocating radical beliefs and displaying violent content.
What results will TSIIPs receive from the use of this OSINT Tool?
TSIIPs can use SpiderFoot to scan and get results on:
Domain and Subdomain Names: Information about accounts on external sites, the company’s information, the domain itself, IPs, emails, and all information related to that domain, including blacklisting data and affiliations.
IPv4 and IPv6 Addresses: Information on ownership, country, domain, owner email, server, and creation date. It shows complaints and email addresses linked to them. It also returns all IPs related to a domain and their risk assessment, physical location, and usernames.
Subnet: Provides similar results to the IPs for domain, owner, country, owner, and email address.
Bitcoin Addresses: Information linked to that Bitcoin address.
Email Addresses: Information on username, social media account, or website it might have registered on.
Phone Number: Information about that number, such as country name and provider.
Human Name: Username available for that name and accounts on social media platforms.
Username: Details where the username can be found across social media platforms.
Network ASN: Provides information on accounts on external sites, affiliated email addresses, blacklists, company name, country of origin, human names, malicious IP addresses, phone numbers, and usernames.
How will this OSINT Tool help TSIIPs protect a person or organization?
SpiderFoot will very likely aid TSIIPs in identifying and thwarting instances of radicalism, propaganda, disinformation, and misinformation spread via social media posts, memes, websites, blogs, and other online assets. The tool allows TSIIPs to access and assess the personally identifiable information (PII) of groups and individuals connected to hate, prejudice, and bias. TSIIPs can likely leverage findings to uncover the virtual and physical whereabouts of those producing hate, attempting to attack threats at the root.
Instructions on using this OSINT Tool:
Installation process:
Access the tool using the link provided above.
Download the zip file from the GitHub page by clicking “Code” and then “Download zip.” Users should then extract the folder and save it on the desktop as “spiderfoot.”
To use the tool, users need to install Python. If not provided, users can download it at https://www.python.org/downloads/, selecting the appropriate choice for their device’s system.
Go to Google Cloud Shell Editor using the link https://shell.cloud.google.com/?pli=1&show=ide%2Cterminal to use the tool.
Once signed into a Google account, users must paste the code “git clone https://github.com/smicallef/spiderfoot.git” to paste the SpiderFoot repository.
Paste “cd spiderfoot” to locate the elements of the SpiderFoot folder.
Users need to install the dependencies by pasting the codes (separately):
“sudo apt-get update”
“sudo apt-get install -y python3-pip python3-dev”
“pip3 install -r requirements.txt”
Paste “python3 sf.py -l 127.0.0.1:5001”
Users should click on the number sequence to open a local web interface and use the tool more easily. Users should not close the Cloud Shell Editor tab when using the web interface.
Use process:
To start a new search, users should click “New Scan” at the top left of the interface's main page.
Users can select the “Scan Name” and “Scan Target” based on whether they are looking for an email address, IP address, username, phone number, human name, domain name, bitcoin address, network autonomous system number (ASN), etc.
Users can filter the research:
By use case: “All,” “Footprint,” “Investigate,” and “Passive” based on their needs.
By required data.
By module.
Click “Run Scan.”
Users can consult the results divided by data types and click “Correlations” to visualize detected risks.
Click “Browse” to consult every type of result detected during the scan, the unique and total data number, and the last data element.
Users can click “Scan Settings” to visualize “Meta information” and “Global Setting.”
To access a list of previous scans, click “Scans” at the top left of the main page of the web interface and download a scans’ list by clicking the download icon at the top right of the scans’ list.
Example of this OSINT Tool in use by a TSIIP?
Consider a scenario where TSIIPs receive an anonymous tip of a trending thread on X where users are spreading anti-Trump administration rhetoric and memes in response to rising prices due to recent tariffs. For the past two weeks, the thread has slowly gained traction, beginning with casual, distasteful comments to full-blown hate speech and death threats. Likes, comments, and reposts saw their biggest increase after an unknown actor under the username “MAGA4REAL” entered the scene, sharing detailed AI-generated memes depicting President Trump and other members of his administration meeting a detailed and violent demise. This content is followed by tens of millions of likes, reposts, and comments, with some supporters offering monetary pledges inquiries asking how to “#JoinTheFight.” The social media account claims its memes are created “in-house” and has seen a large increase in followers, mentions, and views since its initial post in the trending thread. Recent posts focus on instructional videos on how to logistically fund and facilitate a lone wolf attack in the most popular cities in the US, with how-tos catering to specific locations, impact, methodology, and weapon type. TSIIPs are tasked with uncovering the identity, physical location, and at least the IP Address associated with the unknown threat actor. The procedure would be as follows:
After gaining access to SpiderFoot, TSIIPs begin their investigation by inputting the username “MAGA4REAL” with the associated code, attempting to pull all known data linked to the account.
The tool outputs information containing an email, phone number, and IP address associated with the account in addition to other social media accounts tied to the outputted data.
Cross-referencing all of the information, TSIIPs denote the threat actor's name, appearance, and general geographical location.
After additional digging, TSIIPs eventually pinpoint the individual’s area of residency and nationality.
Accumulating the evidence, TSIIPs present the actionable intelligence to the local police department, prompting an official investigation.
After serving the warrant, police discover various receipts for a pressure cooker, pool chemicals, and other supplies that are hazardous when combined and purchased with the threat actor’s credit card.
The suspect is put under arrest for a short period before being transferred into federal custody, where the FBI begins an indictment.
What other tools should be used with this OSINT Tool?
TSIIPs should use this tool with other tools tracking data breaches and personal information leakages, such as Have I Been Pwned (HIBP), to verify information obtained from SpiderFoot and identify leaked websites. TSIIPs can access a comprehensive report on HIBP by the OSINT-RDT Team on the CTG website.[4] TSIIPs should use Maltego to help visualize relationships between people, domains, IP addresses, and other data and get a comprehensive map of relationships and data. TSIIPs can consult a thorough report on Maltego by the OSINT-RDT team on the CTG website.[5] TSIIPs can investigate usernames returned by SpiderFoot on SOCMINT tools, such as Sherlock, Webmii, or Analyst Research Tools, to verify the accuracy of these results and deepen the investigation. Users can read detailed reports on these tools by the OSINT-RDT Team by accessing the CTG website.[6] Users can also use Google Dorks and Boolean operators to research specific usernames on Google Advanced.
Are there any concerns that TSIIPs should have about using this OSINT tool?
TSIIPs should note that SpiderFoot is only available through GitHub. TSIIPs should refrain from downloading the tools on their devices, as it could compromise PERSEC/OPSEC in the case of malware and other cyber risks. To maximize security, TSIIPs should always use this tool through a virtual environment (VE) such as Google Cloud Shell Editor. TSIIPs should note that the installation process of this tool requires moderate technical expertise in VE and coding. TSIIPs should be aware that some data processing, such as domain, could require longer wait times to produce results, slowing the investigation process. TSIIPs should also note that SpiderFoot can produce false negatives, and although it connects multiple data sources, its data collection capabilities might still be limited, increasing uncertainty and potentially requiring additional cross-referencing and justification.
[1] Digital footprint, generated by a third party database
[2] Steve Micallef, Github, https://github.com/smicallef
[3] Steve Micallef, Linkedin, https://www.linkedin.com/in/stevemicallef/?originalSubdomain=ch
[4] OSINT TOOLKIT: HAVE I BEEN PWNED, A USER-FRIENDLY SEARCH ENGINE THAT TRACKS DATA BREACHES AND AIDS IN PROTECTING PERSONAL INFORMATION by Lewis Li, Martina Elena Nitti, Dalton Riedlbauer; OSINT-RDT Team
[5] OSINT TOOLKIT: MALTEGO, AN INTEGRATED PLATFORM THAT STREAMLINES CYBER INVESTIGATIONS AND HELPS ANALYZE RELATIONSHIPS WITHIN COMPLEX DATASETS by Dalton Riedlbauer, Martina Elena Nitti, Dominic Bianco, OSINT-RDT Team
[6] OSINT, Counter Threat Center, https://www.counterthreatcenter.com/reports/categories/osint