Collect All Functions
Contents
Collect All Functions#
This project comes with functions that collect all STIX objects from all ATT&CK Matrices at once. These functions help collect more with less API call requests.
Import ATTACK API Client#
from attackcti import attack_client
Import Extra Libraries#
from pandas import *
import json
import logging
logging.getLogger('taxii2client').setLevel(logging.CRITICAL)
pandas.__version__
'1.3.5'
Initialize ATT&CK Client Variable#
lift = attack_client()
Get All Techniques#
We can extract all STIX objects of type attack-pattern (technique) across all ATT&CK matrices.
By default, this function removes
deprecatedandrevokedtechniques. If you want to keep those techniques in the results, you can run the function with the parameterskip_revoked_deprecated=False.By default, this function also includes all techniques and sub-techniques. If you want to only get techniques that are not sub-techniques, you can run the function with the parameter
include_subtechniques=False.
techniques = lift.get_techniques()
print("Number of Techniques in ATT&CK")
len(techniques)
Number of Techniques in ATT&CK
736
By default, the data returned by the available functions in the attackcti library is of type stix2. However, if you want to interact with libraries such as Pandas, it needs to be of type dict
all_techniques = []
for t in techniques:
all_techniques.append(json.loads(t.serialize()))
df = pandas.json_normalize(all_techniques)
df.reindex(['created','name', 'x_mitre_data_sources', 'x_mitre_platforms'], axis=1)[0:5]
| created | name | x_mitre_data_sources | x_mitre_platforms | |
|---|---|---|---|---|
| 0 | 2021-10-12T20:02:31.866Z | Resource Forking | [File: File Creation, Process: Process Creatio... | [macOS] |
| 1 | 2021-10-08T14:06:28.212Z | Downgrade Attack | [Command: Command Execution, Process: Process ... | [Windows, Linux, macOS] |
| 2 | 2021-10-05T21:26:15.081Z | Login Items | [Process: Process Creation, File: File Modific... | [macOS] |
| 3 | 2021-10-05T01:15:06.293Z | Reflective Code Loading | [Script: Script Execution, Process: OS API Exe... | [macOS, Linux, Windows] |
| 4 | 2021-10-01T17:58:26.445Z | Cloud Storage Object Discovery | [Cloud Storage: Cloud Storage Enumeration, Clo... | [IaaS] |
We can now access the schema of the dataframe
list(df)
['type',
'id',
'created_by_ref',
'created',
'modified',
'name',
'description',
'kill_chain_phases',
'external_references',
'object_marking_refs',
'x_mitre_contributors',
'x_mitre_data_sources',
'x_mitre_defense_bypassed',
'x_mitre_detection',
'x_mitre_is_subtechnique',
'x_mitre_permissions_required',
'x_mitre_platforms',
'x_mitre_version',
'x_mitre_remote_support',
'x_mitre_system_requirements',
'x_mitre_network_requirements',
'x_mitre_effective_permissions',
'x_mitre_impact_type',
'x_mitre_tactic_type',
'x_mitre_old_attack_id']
Showing one technique example:
techniques[0]
AttackPattern(type='attack-pattern', id='attack-pattern--b22e5153-ac28-4cc6-865c-2054e36285cb', created_by_ref='identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5', created='2021-10-12T20:02:31.866Z', modified='2021-10-16T01:50:40.276Z', name='Resource Forking', description='Adversaries may abuse resource forks to hide malicious code or executables to evade detection and bypass security applications. A resource fork provides applications a structured way to store resources such as thumbnail images, menu definitions, icons, dialog boxes, and code.(Citation: macOS Hierarchical File System Overview) Usage of a resource fork is identifiable when displaying a file’s extended attributes, using <code>ls -l@</code> or <code>xattr -l</code> commands. Resource forks have been deprecated and replaced with the application bundle structure. Non-localized resources are placed at the top level directory of an application bundle, while localized resources are placed in the <code>/Resources</code> folder.(Citation: Resource and Data Forks)(Citation: ELC Extended Attributes)\n\nAdversaries can use resource forks to hide malicious data that may otherwise be stored directly in files. Adversaries can execute content with an attached resource fork, at a specified offset, that is moved to an executable location then invoked. Resource fork content may also be obfuscated/encrypted until execution.(Citation: sentinellabs resource named fork 2020)(Citation: tau bundlore erika noerenberg 2020)', kill_chain_phases=[KillChainPhase(kill_chain_name='mitre-attack', phase_name='defense-evasion')], revoked=False, external_references=[ExternalReference(source_name='mitre-attack', url='https://attack.mitre.org/techniques/T1564/009', external_id='T1564.009'), ExternalReference(source_name='macOS Hierarchical File System Overview', description='Tenon. (n.d.). Retrieved October 12, 2021.', url='http://tenon.com/products/codebuilder/User_Guide/6_File_Systems.html#anchor520553'), ExternalReference(source_name='Resource and Data Forks', description='Flylib. (n.d.). Identifying Resource and Data Forks. Retrieved October 12, 2021.', url='https://flylib.com/books/en/4.395.1.192/1/'), ExternalReference(source_name='ELC Extended Attributes', description="Howard Oakley. (2020, October 24). There's more to files than data: Extended Attributes. Retrieved October 12, 2021.", url='https://eclecticlight.co/2020/10/24/theres-more-to-files-than-data-extended-attributes/'), ExternalReference(source_name='sentinellabs resource named fork 2020', description='Phil Stokes. (2020, November 5). Resourceful macOS Malware Hides in Named Fork. Retrieved October 12, 2021.', url='https://www.sentinelone.com/labs/resourceful-macos-malware-hides-in-named-fork/'), ExternalReference(source_name='tau bundlore erika noerenberg 2020', description='Erika Noerenberg. (2020, June 29). TAU Threat Analysis: Bundlore (macOS) mm-install-macos. Retrieved October 12, 2021.', url='https://blogs.vmware.com/security/2020/06/tau-threat-analysis-bundlore-macos-mm-install-macos.html')], object_marking_refs=['marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168'], x_mitre_contributors=['Jaron Bradley @jbradley89', 'Ivan Sinyakov'], x_mitre_data_sources=['File: File Creation', 'Process: Process Creation', 'File: File Metadata', 'Command: Command Execution'], x_mitre_defense_bypassed=['Notarization; Gatekeeper'], x_mitre_detection='Identify files with the <code>com.apple.ResourceFork</code> extended attribute and large data amounts stored in resource forks. \n\nMonitor command-line activity leveraging the use of resource forks, especially those immediately followed by potentially malicious activity such as creating network connections. ', x_mitre_is_subtechnique=True, x_mitre_permissions_required=['User'], x_mitre_platforms=['macOS'], x_mitre_version='1.0')
Enrich Techniques Data Sources#
As you might already know, the ATT&CK data model now represents data sources as objects. However, when retrieving techniques from ATT&CK TAXII server, their data sources section only includes data sources and data components names. Therefore, we created a parameter that you can use with the get_techniques() function to enrich the data sources section of each technique. The parameter enrich_data_sources is set to False by default.
techniques = lift.get_techniques(enrich_data_sources=True)
This function returns a list of techniques with the x_mitre_data_sources attribute as a list of STIX objects representing data sources with their respective data components depending on the technique’s detection context.
for ds in techniques[0]['x_mitre_data_sources']:
for dc in ds['data_components']:
print(ds['name'], '-', dc['name'])
Command - Command Execution
File - File Metadata
File - File Creation
Process - Process Creation
techniques[0]['x_mitre_data_sources'][0]
{'object_marking_refs': ['marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168'],
'modified': '2021-11-10T09:30:48.694Z',
'name': 'Command',
'created_by_ref': 'identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5',
'type': 'x-mitre-data-source',
'id': 'x-mitre-data-source--73691708-ffb5-4e29-906d-f485f6fa7089',
'description': 'A directive given to a computer program, acting as an interpreter of some kind, in order to perform a specific task(Citation: Confluence Linux Command Line)(Citation: Audit OSX)',
'created': '2021-10-20T15:05:19.273Z',
'external_references': [{'url': 'https://attack.mitre.org/datasources/DS0017',
'external_id': 'DS0017',
'source_name': 'mitre-attack'},
{'url': 'https://confluence.atlassian.com/confkb/how-to-enable-command-line-audit-logging-in-linux-956166545.html',
'description': 'Confluence Support. (2021, September 8). How to enable command line audit logging in linux. Retrieved September 23, 2021.',
'source_name': 'Confluence Linux Command Line'},
{'url': 'https://www.scip.ch/en/?labs.20150108',
'description': 'Gagliardi, R. (n.d.). Audit in a OS X System. Retrieved September 23, 2021.',
'source_name': 'Audit OSX'}],
'x_mitre_version': '1.0',
'x_mitre_platforms': ['Windows', 'Linux', 'macOS', 'Network', 'Containers'],
'x_mitre_collection_layers': ['Host', 'Container'],
'x_mitre_contributors': ['Austin Clark',
'Center for Threat-Informed Defense (CTID)'],
'data_components': [{'object_marking_refs': ['marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168'],
'modified': '2021-10-20T15:05:19.273Z',
'id': 'x-mitre-data-component--685f917a-e95e-4ba0-ade1-c7d354dae6e0',
'description': 'Invoking a computer program directive to perform a specific task (ex: Windows EID 4688 of cmd.exe showing command-line parameters, ~/.bash_history, or ~/.zsh_history)',
'created_by_ref': 'identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5',
'name': 'Command Execution',
'created': '2021-10-20T15:05:19.273Z',
'type': 'x-mitre-data-component',
'x_mitre_version': '1.0',
'x_mitre_data_source_ref': 'x-mitre-data-source--73691708-ffb5-4e29-906d-f485f6fa7089'}]}
Get All Groups#
We can also extract all the available groups across all ATT&CK matrices at once.
groups = lift.get_groups()
print("Number of Groups in ATT&CK")
len(groups)
Number of Groups in ATT&CK
131
groups_list = []
for t in groups:
groups_list.append(json.loads(t.serialize()))
df = pandas.json_normalize(groups_list)
df[0:4]
| type | id | created_by_ref | created | modified | name | description | aliases | external_references | object_marking_refs | x_mitre_contributors | x_mitre_version | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | intrusion-set | intrusion-set--35d1b3be-49d4-42f1-aaa6-ef159c8... | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-10-01T01:57:31.229Z | 2021-10-15T18:47:18.824Z | TeamTNT | [TeamTNT](https://attack.mitre.org/groups/G013... | [TeamTNT] | [{'source_name': 'mitre-attack', 'url': 'https... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | [Will Thomas, Cyjax] | 1.0 |
| 1 | intrusion-set | intrusion-set--39d6890e-7f23-4474-b8ef-e7b0343... | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-09-29T15:10:19.236Z | 2021-10-15T15:16:47.329Z | Andariel | [Andariel](https://attack.mitre.org/groups/G01... | [Andariel, Silent Chollima] | [{'source_name': 'mitre-attack', 'url': 'https... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | [Kyoung-ju Kwak (S2W)] | 1.0 |
| 2 | intrusion-set | intrusion-set--6566aac9-dad8-4332-ae73-20c23ba... | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-09-28T17:41:12.950Z | 2021-10-25T14:28:10.337Z | Ferocious Kitten | [Ferocious Kitten](https://attack.mitre.org/gr... | [Ferocious Kitten] | [{'source_name': 'mitre-attack', 'url': 'https... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | [Pooja Natarajan, NEC Corporation India, Manik... | 1.0 |
| 3 | intrusion-set | intrusion-set--e5603ea8-4c36-40e7-b7af-a077d24... | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-09-24T21:41:34.797Z | 2021-10-16T02:06:06.404Z | IndigoZebra | [IndigoZebra](https://attack.mitre.org/groups/... | [IndigoZebra] | [{'source_name': 'mitre-attack', 'url': 'https... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | [Pooja Natarajan, NEC Corporation India, Yoshi... | 1.0 |
Get All Software#
We can extract all Enterprise, Mobile and ICS (Software Malware & Tools)
software = lift.get_software()
print("Number of Software in ATT&CK")
len(software)
Number of Software in ATT&CK
641
software_list = []
for t in software:
software_list.append(json.loads(t.serialize()))
df = pandas.json_normalize(software_list)
df[0:4]
| type | id | created_by_ref | created | modified | name | description | labels | external_references | object_marking_refs | x_mitre_aliases | x_mitre_contributors | x_mitre_platforms | x_mitre_version | x_mitre_old_attack_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | tool | tool--f91162cc-1686-4ff8-8115-bf3f61a4cc7a | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-09-14T21:45:30.280Z | 2021-09-21T18:03:13.205Z | Wevtutil | [Wevtutil](https://attack.mitre.org/software/S... | [tool] | [{'source_name': 'mitre-attack', 'url': 'https... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | [Wevtutil] | [Viren Chaudhari, Qualys, Harshal Tupsamudre, ... | [Windows] | 1.0 | NaN |
| 1 | tool | tool--11f8d7eb-1927-4806-9267-3a11d4d4d6be | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-07-30T15:43:17.770Z | 2021-10-15T15:49:25.284Z | Sliver | [Sliver](https://attack.mitre.org/software/S06... | [tool] | [{'source_name': 'mitre-attack', 'url': 'https... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | [Sliver] | [Achute Sharma, Keysight, Ayan Saha, Keysight] | [Windows, Linux, macOS] | 1.0 | NaN |
| 2 | tool | tool--80c815bb-b24a-4b9c-9d73-ff4c075a278d | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-03-19T13:11:50.666Z | 2021-04-26T22:35:19.315Z | Out1 | [Out1](https://attack.mitre.org/software/S0594... | [tool] | [{'source_name': 'mitre-attack', 'url': 'https... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | [Out1] | NaN | [Windows] | 1.0 | NaN |
| 3 | tool | tool--03c6e0ea-96d3-4b23-9afb-05055663cf4b | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-03-18T14:57:34.628Z | 2021-04-25T23:30:38.375Z | RemoteUtilities | [RemoteUtilities](https://attack.mitre.org/sof... | [tool] | [{'source_name': 'mitre-attack', 'url': 'https... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | [RemoteUtilities] | NaN | [Windows] | 1.0 | NaN |
Get All Relationships#
We can also get all relationships from all ATT&CK matrices with one API request.
relationships = lift.get_relationships()
print("Number of Relationships in ATT&CK")
len(relationships)
Number of Relationships in ATT&CK
15752
relations_list = []
for t in relationships:
relations_list.append(json.loads(t.serialize()))
df = pandas.json_normalize(relations_list)
df[0:4]
| type | id | created_by_ref | created | modified | relationship_type | source_ref | target_ref | object_marking_refs | description | external_references | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | relationship | relationship--9567076b-2a77-43e4-befd-19556def... | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-11-10T09:30:48.753Z | 2021-11-10T09:30:48.753Z | detects | x-mitre-data-component--3d20385b-24ef-40e1-9f5... | attack-pattern--910906dd-8c0a-475a-9cc1-5e029e... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | NaN | NaN |
| 1 | relationship | relationship--79fa693d-38b2-4730-8602-1f72eef5... | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-11-10T09:30:48.753Z | 2021-11-10T09:30:48.753Z | detects | x-mitre-data-component--9ce98c86-8d30-4043-ba5... | attack-pattern--60d0c01d-e2bf-49dd-a453-f8a9c9... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | NaN | NaN |
| 2 | relationship | relationship--ed1c4fff-998f-499d-8a00-cfdee554... | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-11-10T09:30:48.753Z | 2021-11-10T09:30:48.753Z | detects | x-mitre-data-component--9bde2f9d-a695-4344-bfa... | attack-pattern--2959d63f-73fd-46a1-abd2-109d7d... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | NaN | NaN |
| 3 | relationship | relationship--41c0352d-b377-4fe9-8c3a-67b78a9a... | identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5 | 2021-11-10T09:30:48.753Z | 2021-11-10T09:30:48.753Z | detects | x-mitre-data-component--c0a4a086-cc20-4e1e-b7c... | attack-pattern--6836813e-8ec8-4375-b459-abb388... | [marking-definition--fa42a846-8d90-4e51-bc29-7... | NaN | NaN |
Get All Data Sources#
Now that data sources are part of the ATT&CK data model as objects, we can retrieve all that information at once.
data_sources = lift.get_data_sources()
print("Number of Data Sources in ATT&CK")
len(data_sources)
Number of Data Sources in ATT&CK
38
data_sources[0]
{'object_marking_refs': ['marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168'],
'modified': '2021-10-20T15:05:19.275Z',
'name': 'Internet Scan',
'created_by_ref': 'identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5',
'type': 'x-mitre-data-source',
'id': 'x-mitre-data-source--38fe306c-bdec-4f3d-8521-b72dd32dbd17',
'description': 'Information obtained (commonly via active network traffic probes or web crawling) regarding various types of resources and servers connected to the public Internet',
'created': '2021-10-20T15:05:19.275Z',
'external_references': [{'url': 'https://attack.mitre.org/datasources/DS0035',
'external_id': 'DS0035',
'source_name': 'mitre-attack'}],
'x_mitre_version': '1.0',
'x_mitre_platforms': ['PRE'],
'x_mitre_collection_layers': ['OSINT'],
'x_mitre_contributors': []}
Get All Data Components#
Now that data components are also part of the ATT&CK data model as objects, we can retrieve all that information at once.
data_components = lift.get_data_components()
print("Number of data components in ATT&CK")
len(data_components)
Number of data components in ATT&CK
109
data_components[0]
{'object_marking_refs': ['marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168'],
'modified': '2021-10-20T15:05:19.275Z',
'id': 'x-mitre-data-component--cc150ad8-ecfa-4340-9aaa-d21165873bd4',
'description': 'Logged domain name system (DNS) data highlighting timelines of domain to IP address resolutions (ex: passive DNS)',
'created_by_ref': 'identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5',
'name': 'Passive DNS',
'created': '2021-10-20T15:05:19.275Z',
'type': 'x-mitre-data-component',
'x_mitre_version': '1.0',
'x_mitre_data_source_ref': 'x-mitre-data-source--dd75f457-8dc0-4a24-9ae5-4b61c33af866'}