CIRCL hashlookup (hashlookup.circl.lu)

CIRCL hash lookup is a public API to lookup hash values against known database of files. NSRL RDS database is included and many others are also included. The API is accessible via HTTP ReST API and the API is also described as an OpenAPI. The service is free and served as a best-effort basis.

Sources included in CIRCL hashlookup

Common Windows 10 and Windows 11 build (French, Dutch, German, UK, US)
NIST NSRL - All RDS hash sets including current, modern, android, iOS and legacy + SHA256 mapping.
Ubuntu packages distribution
CentOS core OS distribution
Fedora project EPEL repository
Kali linux packages distribution
OpenSUSE distribution packages
OpenBSD binary tar.gz distribution
CDNJS
Snap public repository

Is it a database of malicious or non-malicious hash of files?

CIRCL hashlookup service only gives details about known files appearing in specific database(s). This gives you context and information about file hashes which can be discovered during investigation or digital forensic analysis.

hashlookup:trust

A trust level is included all responses from hashlookup with the field name: hashlookup:trust.

The scale of the trust level is between 0 and 100. 50 means that we don’t have any opinion on the file. If it’s below 50, we have less trust in the legitimacy of the file. If it’s above 50, it appears in multiple sources and have an improved trust.

API Usage

Get information about the hash lookup database (via ReST)

curl -X 'GET' \
  'https://hashlookup.circl.lu/info' \
  -H 'accept: application/json'

 1{
 2  "nsrl-version": "March 2022",
 3  "stat:hashlookup_total_keys": 5077811007,
 4  "stat:nsrl_modern_rds": "192677749",
 5  "stat:nsrl_legacy": "113737918",
 6  "stat:nsrl_ios": "931242",
 7  "stat:nsrl_android": "41589780",
 8  "hashlookup-version": "1.2"
 9}

Perform an MD5 hash lookup

curl -X 'GET' \
  'https://hashlookup.circl.lu/lookup/md5/8ED4B4ED952526D89899E723F3488DE4' \
  -H 'accept: application/json'

 1{
 2  "CRC32": "7A5407CA",
 3  "FileName": "wow64_microsoft-windows-i..timezones.resources_31bf3856ad364e35_10.0.16299.579_de-de_f24979c73226184d.manifest",
 4  "FileSize": "2520",
 5  "MD5": "8ED4B4ED952526D89899E723F3488DE4",
 6  "OpSystemCode": {
 7    "MfgCode": "1006",
 8    "OpSystemCode": "362",
 9    "OpSystemName": "TBD",
10    "OpSystemVersion": "none"
11  },
12  "ProductCode": {
13    "ApplicationType": "Security",
14    "Language": "Multilanguage",
15    "MfgCode": "608",
16    "OpSystemCode": "868",
17    "ProductCode": "190742",
18    "ProductName": "Cumulative Update for Windows Server 2016 for x64 (KB4338817)",
19    "ProductVersion": "1709"
20  },
21  "SHA-1": "00000079FD7AAC9B2F9C988C50750E1F50B27EB5",
22  "SpecialCode": "",
23  "db": "nsrl_modern_rds",
24  "insert-timestamp": "1630942434.8964827",
25  "source": "NSRL"
26}

Perform an SHA-1 hash lookup

curl -X 'GET'   'https://hashlookup.circl.lu/lookup/sha1/FFFFFDAC1B1B4C513896C805C2C698D9688BE69F'   -H 'accept: application/json' | jq .

 1{
 2  "CRC32": "CBD64CD9",
 3  "FileName": ".rela.dyn",
 4  "FileSize": "240",
 5  "MD5": "131312A96CAD4ACAA7E2631A34A0D47C",
 6  "OpSystemCode": {
 7    "MfgCode": "1006",
 8    "OpSystemCode": "362",
 9    "OpSystemName": "TBD",
10    "OpSystemVersion": "none"
11  },
12  "ProductCode": {
13    "ApplicationType": "Operating System",
14    "Language": "English",
15    "MfgCode": "1722",
16    "OpSystemCode": "599",
17    "ProductCode": "163709",
18    "ProductName": "BlackArch Linux",
19    "ProductVersion": "2017.03.01"
20  },
21  "SHA-1": "FFFFFDAC1B1B4C513896C805C2C698D9688BE69F",
22  "SpecialCode": "",
23  "db": "nsrl_modern_rds",
24  "insert-timestamp": "1631011386.4436111",
25  "source": "NSRL"
26}

Perform an SHA-256 lookup

curl -s -X 'GET'   'https://hashlookup.circl.lu/lookup/sha256/301c9ec7a9aadee4d745e8fd4fa659dafbbcc6b75b9ff491d14cbbdd840814e9'   -H 'accept: application/json' | jq

 1{
 2  "FileName": "./usr/bin/openssl",
 3  "FileSize": "723944",
 4  "MD5": "34D827A288FA51B93297EF2A8A43B769",
 5  "SHA-1": "72F104BF11A12511154267328F069FE0541E841E",
 6  "SHA-256": "301C9EC7A9AADEE4D745E8FD4FA659DAFBBCC6B75B9FF491D14CBBDD840814E9",
 7  "SHA-512": "2533D682DB224F0D3BEA043A8A986DC1D341FBEFFD158CB97CD360190BE091F43CC6DBF07E6E985CC0DCE17ADC207A61AC9831BE91099202093ACFED584602D1",
 8  "SSDEEP": "12288:g7LKf6QceJ83r69SOPdxouwUnSysbLY+YR2L7b+3l7E71rb/t:gsceJ83rESOlxJwUZsbLY+YR2Xa3l7E7",
 9  "TLSH": "T150F4281AE64719BDC8B2C230455B50327A31B945F332BF6B26C196311E42B1EA73FBE5",
10  "insert-timestamp": "1636385379.0646722",
11  "source": "snap:BbsqA1how7wjAmzvZEBaOXf5L7I9NBHe_31",
12  "hashlookup:parent-total": 124,
13  "parents": [
14    {
15      "SHA-1": "0006E05A9FC1F165A94713131592E4269DCB0B5D"
16    },
17    {
18      "SHA-1": "027EC67FDB1BCB3CA236FEAC0A47334ECE3F5BB0"
19    },
20    {
21      "FileSize": "613848",
22      "MD5": "124A707963928961F17F873921B0DF13",
23      "PackageDescription": "Secure Sockets Layer toolkit - cryptographic utility\n This package is part of the OpenSSL project's implementation of the SSL\n and TLS cryptographic protocols for secure communication over the\n Internet.\n .\n It contains the general-purpose command line binary /usr/bin/openssl,\n useful for cryptographic operations such as:\n  * creating RSA, DH, and DSA key parameters;\n  * creating X.509 certificates, CSRs, and CRLs;\n  * calculating message digests;\n  * encrypting and decrypting with ciphers;\n  * testing SSL/TLS clients and servers;\n  * handling S/MIME signed or encrypted mail.",
24      "PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>",
25      "PackageName": "openssl",
26      "PackageSection": "utils",
27      "PackageVersion": "1.1.1-1ubuntu2.1~18.04.13",
28      "SHA-1": "02ADDB9985B9F21F42072CEA4A3C1A97448C67AC",
29      "SHA-256": "E8E123812167819F0D1AD572C85094F13369413A6E3D1127E4A786CC0A31FD0D"
30    },
31    {
32      "SHA-1": "05EAE0930E00C981FB9EE08BBA153CA6C310CB62"
33    },
34    {
35      "SHA-1": "06DFA4B0BA4E3E6A9CD72455A5F4B0D5F6D579C4"
36    },
37    {
38      "SHA-1": "0721FF5DB7675EEF9627EC9D664F6494A4DB651A"
39    },
40    {
41      "SHA-1": "08797034F4F2681C861EB210B7A0CFE1BE608E00"
42    },
43    {
44      "SHA-1": "088A0984F19981D1B3523C1B11752D19907C61D0"
45    },
46    {
47      "SHA-1": "0A879A1E2214A51D3101FA3406F885C93F0269CD"
48    },
49    {
50      "SHA-1": "0A8F5BBF8826329A0F4C7A062204B7F4BC901414"
51    }
52  ]
53}

Bulk search of MD5 hashes

curl -X 'POST'   'https://hashlookup.circl.lu/bulk/md5' -H "Content-Type: application/json"  -d "{\"hashes\": [\"6E2F8616A01725DCB37BED0A2495AEB2\", \"8ED4B4ED952526D89899E723F3488DE4\", \"344428FA4BA313712E4CA9B16D089AC4\"]}" | jq .

 1[
 2  {
 3    "CRC32": "E774FD92",
 4    "FileName": "network",
 5    "FileSize": "7279",
 6    "MD5": "6E2F8616A01725DCB37BED0A2495AEB2",
 7    "OpSystemCode": "362",
 8    "ProductCode": "8321",
 9    "SHA-1": "00000903319A8CE18A03DFA22C07C6CA43602061",
10    "SpecialCode": "",
11    "db": "nsrl_legacy",
12    "insert-timestamp": "1631050497.0385447",
13    "source": "NSRL"
14  },
15  {
16    "CRC32": "7A5407CA",
17    "FileName": "wow64_microsoft-windows-i..timezones.resources_31bf3856ad364e35_10.0.16299.579_de-de_f24979c73226184d.manifest",
18    "FileSize": "2520",
19    "MD5": "8ED4B4ED952526D89899E723F3488DE4",
20    "OpSystemCode": "362",
21    "ProductCode": "190742",
22    "SHA-1": "00000079FD7AAC9B2F9C988C50750E1F50B27EB5",
23    "SpecialCode": "",
24    "db": "nsrl_modern_rds",
25    "insert-timestamp": "1630942434.8964827",
26    "source": "NSRL"
27  },
28  {
29    "CRC32": "7516A25F",
30    "FileName": ".text._ZNSt14overflow_errorC1ERKSs",
31    "FileSize": "33",
32    "MD5": "344428FA4BA313712E4CA9B16D089AC4",
33    "OpSystemCode": "362",
34    "ProductCode": "219181",
35    "SHA-1": "0000001FFEF4BE312BAB534ECA7AEAA3E4684D85",
36    "SpecialCode": "",
37    "db": "nsrl_modern_rds",
38    "insert-timestamp": "1630942434.8922813",
39    "source": "NSRL"
40  }
41]

Bulk search of SHA-1 hashes

curl -X 'POST'   'https://hashlookup.circl.lu/bulk/sha1' -H "Content-Type: application/json"  -d "{\"hashes\": [\"FFFFFDAC1B1B4C513896C805C2C698D9688BE69F\", \"FFFFFF4DB8282D002893A9BAF00E9E9D4BA45E65\", \"FFFFFE4C92E3F7282C7502F1734B243FA52326FB\"]}" | jq .

 1[
 2  {
 3    "CRC32": "CBD64CD9",
 4    "FileName": ".rela.dyn",
 5    "FileSize": "240",
 6    "MD5": "131312A96CAD4ACAA7E2631A34A0D47C",
 7    "OpSystemCode": "362",
 8    "ProductCode": "163709",
 9    "SHA-1": "FFFFFDAC1B1B4C513896C805C2C698D9688BE69F",
10    "SpecialCode": "",
11    "db": "nsrl_modern_rds",
12    "insert-timestamp": "1631011386.4436111",
13    "source": "NSRL"
14  },
15  {
16    "CRC32": "8654F11A",
17    "FileName": "s_copypix.c",
18    "FileSize": "19541",
19    "MD5": "559D049F44942683093A91BA19D0AF54",
20    "OpSystemCode": "362",
21    "ProductCode": "223222",
22    "SHA-1": "FFFFFF4DB8282D002893A9BAF00E9E9D4BA45E65",
23    "SpecialCode": "",
24    "db": "nsrl_modern_rds",
25    "insert-timestamp": "1631011386.4556186",
26    "source": "NSRL"
27  },
28  {
29    "CRC32": "8E51A269",
30    "FileName": "358.git2-msvstfs.dll",
31    "FileSize": "65",
32    "MD5": "9E4C165089CBA3653484C3F23F1CBC67",
33    "OpSystemCode": "362",
34    "ProductCode": "201317",
35    "SHA-1": "FFFFFE4C92E3F7282C7502F1734B243FA52326FB",
36    "SpecialCode": "",
37    "db": "nsrl_modern_rds",
38    "insert-timestamp": "1631011386.44553",
39    "source": "NSRL"
40  }
41]

API and HTTP return codes

HTTP return code	Description and Interpretation
200	200 means the searched hash is present in at least one of the database
404	404 means the searched hash is not present in the any of the database
400	400 means the input used for the hash is in an incorrect format

Querying the hashlookup database via DNS

The domain to query is <query>.dns.hashlookup.circl.lu. The query can be info or an MD5 or SHA-1 value.

Info of the hashlookup database

dig +short -t TXT info.dns.hashlookup.circl.lu | jq -r . | jq .

 1{
 2  "nsrl-version": "RDS Verion 2.73.1 - July 2021",
 3  "nsrl-NSRL-items": "165968856",
 4  "nsrl-Android-items": "33419323",
 5  "nsrl-iOS-items": "46447082",
 6  "nsrl-NSRLMfg": "543004",
 7  "nsrl-NSRLOS": "6414",
 8  "nsrl-NSRLProd": "333546",
 9  "hashlookup-version": "0.1"
10}

Query of a hash

dig +short -t TXT 931606baaa7a2b4ef61198406f8fc3f4.dns.hashlookup.circl.lu | jq -r . | jq .

 1{
 2  "CRC32": "13C49389",
 3  "FileName": "ls",
 4  "FileSize": "133792",
 5  "MD5": "931606BAAA7A2B4EF61198406F8FC3F4",
 6  "OpSystemCode": "362",
 7  "ProductCode": "217853",
 8  "SHA-1": "D3A21675A8F19518D8B8F3CEF0F6A21DE1DA6CC7",
 9  "SpecialCode": ""
10}

Sample use-cases

How to quickly check a set of files in a local directory?

sha1sum * | cut -f1 -d" " | parallel 'curl  -s https://hashlookup.circl.lu/lookup/sha1/{}' | jq .

Negative output (hash not existing in the database) can be excluded with the -f option of curl.

sha1sum * | cut -f1 -d" " | parallel 'curl -f -s https://hashlookup.circl.lu/lookup/sha1/{}' | jq .

Querying hashlookup without online queries

If you don’t want to share your lookups online towards CIRCL, hashlookup provides a Bloom filter to download.

A Bloom filter (a compact representation of the dataset) is available at https://cra.circl.lu/hashlookup/hashlookup-full.bloom (~700MB) with all the SHA-1 value known in hashlookup. The format of the Bloom filter is the DCSO bloom library and cli. The bloom filter is updated on a monthly basis.

How to use locally the Bloom filter (just don’t forget to install the DCSO bloom cli):

find /usr/bin/ -type f -print0 | xargs -0 sha1sum | awk '{ print $1 }'  | tr a-f A-F | bloom c /home/adulau/hashlookup-full.bloom

The Bloom filter doesn’t contain any metadata. It’s just the SHA-1 hash value stored in CIRCL hashlookup. The Bloom filter file can be check with the bloom cli:

adulau@kolmogorov ~/hashlookup $ bloom s hashlookup-full.bloom
File:			/home/adulau/hashlookup/hashlookup-full.bloom
Capacity:		296893697
Elements present:	296890922
FP probability:		1.00e-04
Bits:			5691486835
Hash functions:		14

The hashlookup forensic analyser supports the Bloom filter and can be also used directly instead of using the online queries.

python3 bin/hashlookup-analyser.py --bloomfilter /home/adulau/hashlookup/hashlookup-full.bloom --include-stats -d /bin

Libraries and Software available to use CIRCL hashlookup

PyHashlookup is a client API in Python to query CIRCL hashlookup.
The Hive Project - Cortex Analyzers pull-request to be integrated in The Hive Cortex Analyzers.
MISP module hashlookup expansion is a MISP module allowing to lookup and expand from hashlookup.
munin - Online Hash Checker for Virustotal and Other Services includes the support for hashlookup.
hashlookup-forensic-analyser Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.
R package to query hashlookup.