Skip to content

03 · Malware Analysis Basics

Once forensics (Module 2) locates a suspicious sample, malware analysis answers: what does it do, how does it persist, what does it communicate with, and how do we detect and remove every instance of it.

Isolated lab only — never on production or your host OS

Always analyze malware in an isolated, snapshot-able VM with networking either disabled or routed through a controlled fake Internet (INetSim/FakeNet-NG). Never run live samples on your host machine or a network-connected device. Use samples from a controlled source such as your own IR case or a training corpus (e.g. theZoo, MalwareBazaar) — never target or weaponize a sample against a real system.

1. Static vs. dynamic analysis

  • Static analysis — examine the file without executing it: strings, hashes, headers, imports, embedded resources.
  • Dynamic analysis — execute it in a sandbox and observe behavior: file/registry changes, network traffic, process activity.

Always do static first — it is safer and often narrows down what to look for in dynamic analysis.

2. Static analysis workflow

# Identify file type and basic metadata
file sample.exe
sha256sum sample.exe        # check against threat intel (Module 4) / VirusTotal

# Extract human-readable strings -- URLs, IPs, file paths, error strings,
# and often crude but revealing indicators
strings -n 8 sample.exe | less

# Inspect PE headers, imports, sections (Windows executables)
pip install pefile
python3 -c "
import pefile
pe = pefile.PE('sample.exe')
for entry in pe.DIRECTORY_ENTRY_IMPORT:
    print(entry.dll)
    for imp in entry.imports:
        print('  ', imp.name)
"

Suspicious imports are a fast signal: VirtualAlloc + WriteProcessMemory + CreateRemoteThread together strongly suggest process injection; InternetOpenUrl/WinHttpOpen suggest network callback capability; CryptEncrypt at scale suggests ransomware.

3. Dynamic analysis in a sandbox

# Reset the analysis VM to a clean snapshot before every run
VBoxManage snapshot analysis-vm restore clean-snapshot

# Inside the isolated VM, monitor behavior with Sysinternals tools
procmon.exe      # file system, registry, process/thread activity
tcpview.exe      # live network connections

Or use an automated sandbox (Cuckoo Sandbox, or a hosted service used only with non-sensitive, already-identified-as-malicious samples) to get a behavioral report: dropped files, registry keys set, mutexes created, and C2 network calls — all inside disposable, network-isolated infrastructure.

4. Identifying persistence mechanisms

Malware wants to survive a reboot. Check the same locations defenders check in IR (Level 1 Module 9), from the attacker's side this time, to understand what to hunt for:

# Registry Run keys
reg query "HKCU\Software\Microsoft\Windows\CurrentVersion\Run"
reg query "HKLM\Software\Microsoft\Windows\CurrentVersion\Run"

# Scheduled tasks
schtasks /query /fo LIST /v

# Services
sc query type= service state= all

5. Network indicators (IOCs)

Dynamic analysis behind a fake Internet (INetSim) reveals what the sample tries to reach even when the real C2 infrastructure is offline:

# INetSim logs every DNS/HTTP/HTTPS request the sample attempts
tail -f /var/log/inetsim/service.log

Extracted indicators — C2 domains/IPs, URI patterns, User-Agent strings, mutex names, file hashes — feed directly into Module 4's threat intelligence pipeline and into detection rules (Snort/Suricata, SIEM).

6. Basic unpacking and de-obfuscation awareness

Many samples are packed (compressed/encrypted) to defeat static analysis. Signs of packing: very high entropy, few recognizable imports, a single suspicious section. Tools like PEiD or entropy analysis in pestudio flag likely packers; x64dbg lets you run to the Original Entry Point (OEP) and dump the unpacked code for a training exercise. This is advanced ground — the goal at this level is recognizing packed samples and knowing the technique exists, not full reverse engineering.

7. Building a detection signature from findings

A YARA rule turns one analyzed sample into scalable detection:

rule Suspicious_Sample_Example
{
    meta:
        description = "Detects strings/behavior seen in analyzed sample X"
        author = "analyst"
        date = "2026-08-31"
    strings:
        $c2_domain = "evil-c2-domain-from-lab.example" nocase
        $mutex = "Global\\SampleMutexNameSeenInSandbox"
        $suspicious_api = "CreateRemoteThread"
    condition:
        2 of them
}

How It Actually Works: how a sandbox actually captures behavior, and how a YARA signature actually matches a family of malware

A dynamic analysis sandbox instruments the guest OS at the hypervisor or kernel driver level, intercepting the same syscall boundary the OS itself uses to mediate every meaningful action a process takes (Level 1 Module 3's kernel permission checks are the same boundary). Every CreateFile, RegSetValue, connect() call the sample makes is logged with its arguments before being allowed to proceed, producing a structured behavioral trace independent of the sample's actual code — which is exactly why sandboxing still works against packed or obfuscated binaries that defeat static signature matching: obfuscation hides what the code looks like, not what the code does once it runs, and the syscall interception layer only cares about the latter. This is also the precise mechanism behind sandbox-evasion checks malware authors add — querying CPUID for hypervisor-only feature bits, checking for a hypervisor-vendor string, or timing a loop and comparing against expected native CPU cycle counts — all of these detect the side effects of running inside instrumented virtualization rather than the instrumentation directly, because the sandbox's monitoring inherently adds overhead or presents artifacts a bare- metal execution would not.

YARA rules generalize a signature beyond one exact file hash by matching on a combination of string and byte patterns and structural conditions:

rule Suspicious_Loader {
  strings:
    $s1 = "VirtualAllocEx" ascii
    $s2 = "WriteProcessMemory" ascii
    $s3 = { 60 89 E5 83 EC ?? }   // a hex byte pattern with a wildcard byte
  condition:
    2 of ($s1, $s2, $s3) and filesize < 500KB
}

The condition clause is what turns a brittle single-string match into a resilient family signature: instead of requiring an exact sequence, requiring "any 2 of these 3 indicators" tolerates the sample being recompiled, repacked, or having irrelevant sections changed, as long as the functionally significant API-usage pattern (allocate remote memory, write to it — the two-call combination that defines process injection generically) survives. This is the direct, mechanism-level reason a well-written YARA rule built from one sample's persistence mechanism and network IOCs (the exact focus of sections 4–7 above) often catches an entire malware family's future variants, while a rule keyed to one exact file hash catches exactly one file and nothing else.

8. Checklist

  • [ ] Analysis performed only in an isolated, snapshot-restorable VM
  • [ ] Static analysis (hash, strings, headers, imports) done first
  • [ ] Network isolated or routed through a fake Internet during execution
  • [ ] Persistence mechanisms documented
  • [ ] IOCs (hashes, domains, mutexes) extracted and exported
  • [ ] Findings turned into a detection rule (YARA / SIEM / IDS)
  • [ ] VM reverted to clean snapshot after analysis

What's next

Module 4 shows how individual IOCs like these become part of an organization-wide threat intelligence program.