Ace CTF Challenge Write-up - Advanced Flutter Windows Reverse Engineering

Titoot

Announcement

I rooted my phone just to use as a bootable device.

Abstract#

This write-up documents the technical analysis and solution for Ace, a hard-category reverse engineering challenge authored for the Egypt National Cybersecurity CTF 2025. The challenge was designed to push the boundaries of Flutter Windows application security research, a relatively unexplored domain compared to its mobile counterparts.

The challenge implemented a multi-layered defensive architecture requiring participants to chain multiple advanced bypass techniques: defeating Windows anti-screen capture mechanisms, circumventing Flutter’s SSL certificate pinning with custom signature extraction, compiling modified Flutter engine binaries for Dart runtime introspection, and performing binary diffing against obfuscated Dart snapshots. This research contributes novel methodologies for Flutter Windows security assessment that can be applied to real-world application penetration testing.

Introduction#

Flutter has emerged as a dominant cross-platform development framework, powering applications across mobile, web, and desktop platforms. While Flutter Android and iOS security research has received considerable attention from the security community, Windows desktop applications remain largely unexplored territory. This gap presents significant implications for organizations deploying Flutter-based desktop applications in security-sensitive environments.

The Ace challenge was specifically designed to address this research gap by implementing multiple defensive mechanisms that participants would need to systematically defeat. Each layer of protection required distinct technical skills and represented realistic security controls that organizations might implement to protect their Flutter Windows applications. The challenge structure served dual purposes: testing participant capabilities and advancing the collective knowledge of Flutter Windows security assessment methodologies.

This writeup serves as both a solution guide and a technical reference for security researchers venturing into Flutter Windows application analysis.

Phase 1: Initial Reconnaissance and Binary Analysis#

Challenge Discovery#

Challenge Interface

upon opening the challenge page we get this description which we might need to decipher but we’ll get to it later, so let’s start the instance and take a look.

challenge instance

File Structure Analysis#

The downloaded challenge archive contained the following file hierarchy:

1
D:.
2
|   ace.exe                 # Flutter application wrapper
3
|   ace.pdb                 # Debug symbols (intentional inclusion)
4
|   dartjni.dll             # Dart JNI bridge library
5
|   flutter_windows.dll     # Flutter engine for Windows
6
|
7
\---data
8
    |   app.so              # Compiled Dart snapshot (AOT)
9
    |   icudtl.dat          # ICU internationalization data
10
    |
11
    \---flutter_assets
12
        |   AssetManifest.bin
13
        |   FontManifest.json
14
        |   NativeAssetsManifest.json
15
        |   NOTICES.Z
16
        |
17
        +---fonts
18
        |       MaterialIcons-Regular.otf
19
        |
20
        \---shaders
21
                ink_sparkle.frag
22
                Stretch_effect.frag

Flutter Windows Architecture Primer#

Understanding Flutter’s Windows deployment model is essential for effective security analysis. Flutter Windows applications follow a specific architecture:

Executable Wrapper (ace.exe): A thin native wrapper responsible for initializing the Flutter engine and loading the application snapshot.
Flutter Engine (flutter_windows.dll): Contains the core Flutter framework code, rendering engine, and Dart VM integration. This is typically consistent across Flutter applications built with the same SDK version.
Dart Snapshot (app.so): Contains the AOT-compiled Dart application code. Unlike JIT-compiled Flutter applications (debug mode), release builds use ahead-of-time compilation to native code, making reverse engineering more challenging.
JNI Bridge (dartjni.dll): Facilitates communication between Dart code and native Windows APIs through the Foreign Function Interface (FFI).

The official Flutter documentation provides reference for this structure at Building Windows apps with Flutter.

Phase 2: Anti-Screen Capture Bypass#

Behavioral Analysis#

Upon initial execution, the application window rendered correctly on the monitor but appeared completely black when attempting to capture screenshots, a clear indication of anti-screen capture protection.

Black Window

Technical Investigation#

The Windows API provides legitimate functionality for protecting window content from screen capture through the SetWindowDisplayAffinity function from user32.dll. This API is commonly used by applications handling sensitive content such as password managers, banking applications, and DRM-protected media.

Win32 API Reference: SetWindowDisplayAffinity#

1
BOOL SetWindowDisplayAffinity(
2
  HWND  hWnd,
3
  DWORD dwAffinity
4
);

Affinity Constants:

WDA_NONE (0x00000000) – No display restrictions
WDA_MONITOR (0x00000001) – Content visible only on physical monitors; excluded from screen capture APIs
WDA_EXCLUDEFROMCAPTURE (0x00000011) – Content completely hidden from all capture mechanisms

String Analysis Confirmation#

Using the strings utility on the app.so file confirmed the presence of screen capture protection:

Strings Output

The presence of the API name in the compiled snapshot indicated direct invocation of this Windows API from the Dart application code.

Bypass Implementation#

Frida, a dynamic instrumentation toolkit, provides an elegant mechanism for API hooking at runtime. The bypass strategy involves intercepting the SetWindowDisplayAffinity call and modifying the affinity parameter to WDA_NONE, effectively disabling the protection while maintaining application stability.

1
const mod = Process.getModuleByName("user32.dll");
2

3
// find the actual export
4
const addr = mod.getExportByName("SetWindowDisplayAffinity");
5

6
// verify pointer
7
console.log("[*] SetWindowDisplayAffinity @ " + addr);
8

9
// hook
10
Interceptor.attach(addr, {
11
    onEnter(args) {
12
        // original affinity passed
13
        const orig = args[1].toInt32();
14

15
        // override to NONE (0)
16
        args[1] = ptr(0);
17

18
        console.log(
19
            "[+] SetWindowDisplayAffinity called with " +
20
            orig +
21
            " → forced 0"
22
        );
23
    },
24

25
    onLeave(retval) {
26
        // force success
27
        retval.replace(1);
28
    },
29
});

Execution and Results#

The Frida script was executed with the following command:

1
frida -f ace.exe -l hook.js

Output:

1
     ____
2
    / _  |   Frida 17.2.17 - A world-class dynamic instrumentation toolkit
3
   | (_| |
4
    > _  |   Commands:
5
   /_/ |_|       help      -> Displays the help system
6
   . . . .       object?   -> Display information about 'object'
7
   . . . .       exit/quit -> Exit
8
   . . . .
9
   . . . .   More info at https://frida.re/docs/home/
10
   . . . .
11
   . . . .   Connected to Local System (id=local)
12
Spawning `ace.exe`...
13
[*] SetWindowDisplayAffinity @ 0x7ffb66e459b0
14
Spawned `ace.exe`. Resuming main thread!
15
[Local::ace.exe ]-> [+] SetWindowDisplayAffinity called with 1 → forced to 0 (WDA_NONE)

With the anti-screen capture protection disabled, the application content became visible for screenshot capture:

Nyan Cat Revealed

Challenge Progression#

Submitting the captured image to the challenge validation endpoint confirmed successful completion of Phase 1:

Phase 1 Complete

Phase 3: Network Communication Analysis#

Initial Network Investigation#

Upon interacting with the application’s “Deploy Paws!” button, an HTTPS handshake error was immediately observed:

HTTPS Error

The challenge description had forewarned participants about expected network errors, suggesting that understanding and resolving these errors was integral to challenge progression.

Flutter Network Traffic Interception#

Flutter applications handle network communication differently from typical Windows applications. The framework implements its own networking stack based on BoringSSL (Google’s fork of OpenSSL), which complicates standard traffic interception techniques.

Initial attempts using HTTP Toolkit with Proxifier (a tool for redirecting application traffic through a proxy) revealed a TLS handshake failure:

HTTP Toolkit Error

This behavior indicates the presence of SSL certificate pinning, a security mechanism that validates server certificates against a predefined set of trusted certificates, preventing man-in-the-middle attacks even when the client trusts a custom CA.

SSL Pinning Bypass Research#

Flutter’s SSL pinning implementation resides in flutter_windows.dll within the ssl_verify_peer_cert function. The standard approach for bypassing Flutter SSL pinning involves locating this function in memory and patching it to always return success (0).

Reference methodology from InfoSec Writeups provided a Frida-based approach, but initial execution failed:

1
[+] Attempting to find and hook ssl_verify_peer_cert (1/5)
2
[!] Flutter library found, but ssl_verify_peer_cert could not be found.
3
[+] Attempting to find and hook ssl_verify_peer_cert (2/5)
4
[!] Flutter library found, but ssl_verify_peer_cert could not be found.
5
...
6
[!] ssl_verify_peer_cert not found. Please open an issue
7
[!] Max attempts reached, stopping

The failure indicated that the byte pattern signature for ssl_verify_peer_cert did not match the actual function implementation in the target Flutter version. This necessitated manual signature extraction.

Manual Function Signature Extraction#

Using IDA Pro for static analysis, the flutter_windows.dll was loaded to identify the correct function signature:

IDA Pro Initial Load

The analysis workflow proceeded as follows:

String Search: Using Shift + F12 to access the strings window and searching for “handshake”:
Cross-Reference Analysis: Locating the string ../../flutter/third_party/boringssl/src/ssl/handshake.cc and tracing its cross-references:
Function Identification: Enabling opcode bytes display in IDA Pro settings and analyzing the disassembly:
Signature Extraction: After examining candidate functions, sub_1804E8BFF exhibited the characteristic pattern of ssl_verify_peer_cert:

The extracted function signature (byte pattern) was:

1
41 57 41 56 41 55 41 54 56 57 55 53 48 83 ec 38 48 89 cf 48 8b 05 ab 45 c2 00

Updated SSL Pinning Bypass Script#

With the correct signature identified, the Frida script was updated:

1
/**
2
 * Flutter Windows SSL Pinning Bypass
3
 *
4
 * Modified version of:
5
 * https://github.com/NVISOsecurity/disable-flutter-tls-verification/blob/main/disable-flutter-tls.js
6
 *
7
 * This version has been updated to support Windows applications with
8
 * Flutter SDK versions where the ssl_verify_peer_cert signature differs
9
 * from the documented patterns.
10
 */
11

12
var config = {
13
   "windows":{
14
       "modulename": "flutter_windows.dll",
15
       "patterns":{
16
           "x64": [
17
               // Updated signature for target Flutter version
18
               "41 57 41 56 41 55 41 54 56 57 55 53 48 83 ec 38 48 89 cf 48 8b 05 ab 45 c2 00"
19
           ],
20
       },
21
   },
22
};
23

24
var TLSValidationDisabled = false;
25
var flutterLibraryFound = false;
26
var tries = 0;
27
var maxTries = 5;
28
var timeout = 1000;
29

30
disableTLSValidation();
31

32
function disableTLSValidation() {
33
   if (TLSValidationDisabled) return;
34

35
   tries++;
36
   if(tries > maxTries){
37
       console.log('[!] Max attempts reached, stopping');
38
       return;
39
   }
40

41
   console.log(`[+] Attempting to find and hook ssl_verify_peer_cert (${tries}/${maxTries})`);
42

43
   var platformConfig = config["windows"];
44
   var m = Process.findModuleByName(platformConfig["modulename"]);
45

46
   if (m === null) {
47
       console.log('[!] Flutter library not found');
48
       setTimeout(disableTLSValidation, timeout);
49
       return;
50
   }
51
   else{
52
       if(flutterLibraryFound == false){
53
           flutterLibraryFound = true;
54
           tries = 1;
55
       }
56
   }
57

58
   if (Process.arch in platformConfig["patterns"]) {
59
       var ranges = m.enumerateRanges('r-x');
60
       findAndPatch(ranges, platformConfig["patterns"][Process.arch], 0);
61
   }
62
   else {
63
       console.log('[!] Processor architecture not supported: ', Process.arch);
64
   }
65

66
   if (!TLSValidationDisabled) {
67
       if(tries < maxTries){
68
           console.log(`[!] Flutter library found, but ssl_verify_peer_cert could not be found.`);
69
       }
70
   }
71
}
72

73
function findAndPatch(ranges, patterns, thumb) {
74
   ranges.forEach(range => {
75
       patterns.forEach(pattern => {
76
           var matches = Memory.scanSync(range.base, range.size, pattern);
77
           matches.forEach(match => {
78
               var info = DebugSymbol.fromAddress(match.address);
79
               console.log(`[+] ssl_verify_peer_cert found at offset: ${info.name}`);
80
               TLSValidationDisabled = true;
81
               hook_ssl_verify_peer_cert(match.address.add(thumb));
82
               console.log('[+] ssl_verify_peer_cert has been patched');
83
           });
84
       });
85
   });
86

87
   setTimeout(disableTLSValidation, timeout);
88
}
89

90
function hook_ssl_verify_peer_cert(address) {
91
   Interceptor.replace(address, new NativeCallback((pathPtr, flags) => {
92
       return 0;  // Always return success (no verification)
93
   }, 'int', ['pointer', 'int']));
94
}

Successful Bypass Execution#

1
     ____
2
    / _  |   Frida 17.2.17 - A world-class dynamic instrumentation toolkit
3
   | (_| |
4
    > _  |   Commands:
5
   /_/ |_|       help      -> Displays the help system
6
   . . . .       object?   -> Display information about 'object'
7
   . . . .       exit/quit -> Exit
8
   . . . .
9
   . . . .   More info at https://frida.re/docs/home/
10
   . . . .
11
   . . . .   Connected to Local System (id=local)
12
Spawning `ace.exe`...
13
[+] Attempting to find and hook ssl_verify_peer_cert (1/5)
14
[+] ssl_verify_peer_cert found at offset: InternalFlutterWindows_WindowManager_GetFullscreen
15
[+] ssl_verify_peer_cert has been patched
16
Spawned `ace.exe`. Resuming main thread!

Network Traffic Analysis#

With SSL pinning disabled, HTTP Toolkit successfully captured the application’s network communication:

HTTP Toolkit Success

Key Findings:

The application sends POST requests to /api/deploy
Request body contains base64-encoded data
A custom header x-ace-signature contains what appears to be a hash of the request body (likely HMAC-based integrity verification)

Phase 4: Dart Snapshot Reverse Engineering#

The Challenge of Dart AOT Snapshots#

Loading app.so into IDA Pro revealed a fundamental challenge: no cross-references to string literals were available. This occurs because Dart AOT snapshots are deserialized at runtime by the Dart VM, with strings and symbols stored in a compressed format that static analysis tools cannot directly interpret.

Initial String Analysis#

The strings utility provided initial insights into the application’s cryptographic implementation:

1
2405    000bf230    000bf230    PT_LOAD(0)    08    A    AESMode.
2
4234    000dbe80    000dbe80    PT_LOAD(0)    0a    A    ::AES_SALT
3
2592    000c21e0    000c21e0    PT_LOAD(0)    0e    A    HMAC is closed
4
5814    000f43c0    000f43c0    PT_LOAD(0)    0b    A    ::HMAC_SALT
5
5854    000f4e90    000f4e90    PT_LOAD(0)    05    A    /HMAC

These strings confirmed:

AES encryption for data encoding
HMAC for request integrity verification
Salt values for cryptographic operations

Modified Flutter Engine Compilation#

To overcome the static analysis limitations, a more sophisticated approach was required: compiling a modified Flutter engine that dumps function offsets during runtime deserialization. This technique, inspired by the reFlutter project, provides accurate function mappings that static analysis cannot achieve.

reFlutter Architecture#

The reFlutter project patches the Flutter engine’s app_snapshot.cc file to intercept snapshot deserialization and log function metadata:

reFlutter

Custom Engine Modification#

Following the compilation guide from Ostorlab’s Medium article, a custom Flutter engine was compiled with the following modification to app_snapshot.cc:

1
// Function offset dumping code inserted during snapshot deserialization
2
ASSERT(code->IsCode());
3
auto& rClass = Class::Handle(func.Owner());
4
auto& rLib = Library::Handle(rClass.library());
5
auto& rlibName = String::Handle(rLib.url());
6

7
char offsetString[70];
8
snprintf(offsetString, sizeof(offsetString),
9
         "0x%016" PRIxPTR,
10
         static_cast<uintptr_t>(code->untag()->monomorphic_unchecked_entry_point_));
11

12
JSONWriter js;
13
js.OpenObject();
14
js.PrintProperty("method_name", func.UserVisibleNameCString());
15
js.PrintProperty("offset", offsetString);
16
js.PrintProperty("library_url", rlibName.ToCString());
17
js.PrintProperty("class_name", rClass.UserVisibleNameCString());
18
js.CloseObject();
19

20
char* buffer = nullptr;
21
intptr_t buffer_length = 0;
22
js.Steal(&buffer, &buffer_length);
23

24
char pat[512] = { 0 };
25
FILE *f = fopen("reflutter_dump.dart", "a+");
26
if (f) {
27
    fprintf(f, "%s\n", buffer);
28
    fclose(f);
29
}

CRITICAL NOTE
The compiled engine’s snapshot hash must match the target application’s hash. The hash can be extracted from app.so using strings. In this case: 1ce86630892e2dca9a8543fdb8ed8e22.

Compilation and Deployment#

Compiling Flutter

After successful compilation, the original flutter_windows.dll was replaced with the modified version, and the application was executed to generate the function offset dump:

1
{"method_name": "qka", "offset": "0x000000000020eaec", "library_url": "dart:core", "class_name": "_RegExp"}
2
{"method_name": "[]", "offset": "0x000000000020eb80", "library_url": "dart:core", "class_name": "_cb"}
3
{"method_name": "<anonymous closure>", "offset": "0x0000000000000080", "library_url": "dart:_http", "class_name": "_kn"}
4
{"method_name": "encode", "offset": "0x000000000000493c", "library_url": "dart:convert", "class_name": "Utf8Codec"}
5
{"method_name": "decode", "offset": "0x000000000000e98c", "library_url": "dart:convert", "class_name": "Utf8Codec"}
6
{"method_name": "normalize", "offset": "0x00000000000127b8", "library_url": "dart:convert", "class_name": "Base64Codec"}
7
...

Addressing Code Obfuscation#

The dumped function names revealed Dart’s code obfuscation, function names like qka and _cb instead of meaningful identifiers. To overcome this, binary diffing was employed using BinDiff for IDA Pro.

The methodology involved:

Creating a new Flutter application with the same Flutter SDK version
Compiling without obfuscation
Performing binary diffing between the obfuscated and unobfuscated app.so files
Mapping function names based on structural similarity

Following the approach documented in Guardsquare’s Flutter reversing guide, the following function mappings were established:

1
{"method_name": "encode", "offset": "0x000000000000493c", "library_url": "dart:convert", "class_name": "Utf8Codec"}
2
{"method_name": "decode", "offset": "0x000000000000e98c", "library_url": "dart:convert", "class_name": "Utf8Codec"}
3
{"method_name": "decode", "offset": "0x000000000000ea48", "library_url": "dart:convert", "class_name": "Utf8Codec"}
4
{"method_name": "normalize", "offset": "0x00000000000127b8", "library_url": "dart:convert", "class_name": "Base64Codec"}
5
{"method_name": "_checkPadding", "offset": "0x0000000000012f94", "library_url": "dart:convert", "class_name": "Base64Codec"}
6
{"method_name": "_inverseAlphabet", "offset": "0x00000000000130f4", "library_url": "dart:convert", "class_name": "_Base64Decoder"}
7
{"method_name": "convert", "offset": "0x00000000001a37a8", "library_url": "dart:convert", "class_name": "JsonDecoder"}
8
{"method_name": "decode", "offset": "0x00000000000296c4", "library_url": "dart:convert", "class_name": "JsonCodec"}
9
{"method_name": "_JsonListener", "offset": "0x000000000000493c", "library_url": "dart:convert", "class_name": "_JsonListener"}

Phase 5: Runtime Instrumentation and Data Extraction#

Isolate Snapshot Calibration#

Dart function offsets in Windows Flutter applications can shift between executions due to Address Space Layout Randomization (ASLR) and other memory mapping factors. To reliably hook functions, a calibration technique was developed using an “anchor function” with a unique byte signature.

Advanced Frida Hook Implementation#

The following script implements isolate snapshot calibration and hooks the base64 encoding function to intercept plaintext data:

1
/**
2
 * Flutter Windows Runtime Instrumentation
3
 *
4
 * This script implements isolate snapshot calibration using an anchor function
5
 * to reliably locate and hook Dart runtime functions despite ASLR.
6
 *
7
 * Technique: Find unique function signature → Calculate isolate base →
8
 *            Apply file offsets to locate target functions
9
 */
10

11
// Configuration: Anchor function for calibration
12
var ANCHOR_FUNC_FILE_OFFSET = 0x1a23e4;     // Base64Encoder.convert offset from FILE START
13
var ISOLATE_SNAPSHOT_BASE_FILE_OFFSET = 0x1D6B80;  // _kDartIsolateSnapshotInstructions from FILE START
14
var ANCHOR_FUNC_SIG = "55 48 89 E5 48 83 EC 18 48 89 F3 48 89 75 F8";
15

16
// Global calibration variables
17
var g_CodeBase = null;        // Base address of ISOLATE SNAPSHOT in memory
18
var g_ElfHeader = null;       // Base address of ELF header in memory
19
var g_GapSize = null;         // Calculated gap between ELF header and isolate snapshot
20

21
console.log("[*] Waiting for application code to load...");
22
console.log("[*] Mode: Isolate Snapshot-Based Calibration");
23

24
var scanInterval = setInterval(function() {
25
    if (g_CodeBase && g_ElfHeader) {
26
        clearInterval(scanInterval);
27
        return;
28
    }
29

30
    // Locate ELF header (magic: 7F 45 4C 46 = \x7FELF)
31
    Process.enumerateRanges('r--').forEach(function (range) {
32
        if (g_ElfHeader) return;
33

34
        try {
35
            var elfMatches = Memory.scanSync(range.base, Math.min(range.size, 0x1000), "7F 45 4C 46");
36
            if (elfMatches.length > 0) {
37
                g_ElfHeader = elfMatches[0].address;
38
                console.log("[+] ELF Header found at: " + g_ElfHeader);
39
            }
40
        } catch (e) {}
41
    });
42

43
    // Locate anchor function in executable memory
44
    Process.enumerateRanges('r-x').forEach(function (range) {
45
        if (g_CodeBase) return;
46

47
        try {
48
            var matches = Memory.scanSync(range.base, range.size, ANCHOR_FUNC_SIG);
49
            if (matches.length > 0) {
50
                var anchorAddr = matches[0].address;
51

52
                if (g_ElfHeader) {
53
                    // Calculate isolate snapshot base
54
                    // anchorAddr = IsolateBase + (ANCHOR_FUNC_FILE_OFFSET - ISOLATE_SNAPSHOT_BASE_FILE_OFFSET)
55
                    var anchorOffsetFromIsolate = ANCHOR_FUNC_FILE_OFFSET - ISOLATE_SNAPSHOT_BASE_FILE_OFFSET;
56
                    g_CodeBase = anchorAddr.sub(anchorOffsetFromIsolate);
57
                    g_GapSize = g_CodeBase.sub(g_ElfHeader).sub(ISOLATE_SNAPSHOT_BASE_FILE_OFFSET);
58

59
                    console.log("\n[+] CALIBRATION SUCCESS!");
60
                    console.log("    ELF Header:           " + g_ElfHeader);
61
                    console.log("    Anchor Found At:      " + anchorAddr);
62
                    console.log("    Anchor File Offset:     0x" + ANCHOR_FUNC_FILE_OFFSET.toString(16));
63
                    console.log("    Isolate Snapshot File Offset: 0x" + ISOLATE_SNAPSHOT_BASE_FILE_OFFSET.toString(16));
64
                    console.log("    Anchor offset from Isolate: 0x" + anchorOffsetFromIsolate.toString(16));
65
                    console.log("    Calculated Gap:       0x" + g_GapSize.toString(16));
66
                    console.log("    -----------------------------------------");
67
                    console.log("    ISOLATE SNAPSHOT BASE: " + g_CodeBase);
68
                    console.log("    -----------------------------------------");
69

70
                    // Verify calibration accuracy
71
                    var expectedAnchor = g_CodeBase.add(anchorOffsetFromIsolate);
72
                    console.log("    Expected Anchor:      " + expectedAnchor);
73
                    console.log("    Actual Anchor:        " + anchorAddr);
74
                    console.log("    Match: " + (expectedAnchor.equals(anchorAddr) ? "✓" : "✗"));
75

76
                    // Proceed to hook target functions
77
                    hookBase64Convert();
78
                }
79
            }
80
        } catch (e) {}
81
    });
82
}, 1000);
83

84
// Convert file offset to memory address
85
function fileOffsetToAddress(fileOffset) {
86
    if (!g_CodeBase) {
87
        console.log("[!] Calibration not complete!");
88
        return null;
89
    }
90
    var offsetFromIsolate = fileOffset - ISOLATE_SNAPSHOT_BASE_FILE_OFFSET;
91
    return g_CodeBase.add(offsetFromIsolate);
92
}
93

94
// Hook Base64Encoder.convert to intercept plaintext data
95
function hookBase64Convert() {
96
    var target = fileOffsetToAddress(ANCHOR_FUNC_FILE_OFFSET);
97

98
    if (!target) {
99
        console.log("[!] Could not calculate target address");
100
        return;
101
    }
102

103
    console.log("[*] Attaching to Base64Encoder.convert at: " + target);
104

105
    Interceptor.attach(target, {
106
        onEnter: function(args) {
107
            console.log("\n[+] Base64Encoder.convert HIT");
108

109
            // Extract input data (Dart List<int> representation)
110
            var inputPtr = args[1];
111

112
            if (!inputPtr.isNull() && (inputPtr.toInt32() & 1) !== 0) {
113
                var realInput = inputPtr.sub(1);
114
                console.log("    [Input] Raw Dump:");
115
                console.log(hexdump(realInput, { length: 128, header: false, ansi: true }));
116
            }
117
        },
118
        onLeave: function(retval) {
119
            console.log("[+] Base64Encoder.convert EXIT");
120

121
            var resPtr = retval;
122
            if (resPtr.isNull() || (resPtr.toInt32() & 1) === 0) return;
123

124
            var realRes = resPtr.sub(1);
125

126
            try {
127
                // Read Dart String length (SMI at offset +8)
128
                var rawLen = realRes.add(8).readU32();
129
                var strLen = rawLen >> 1; // Untag SMI (Small Integer)
130

131
                console.log("    String Length: " + strLen);
132

133
                // Read string data
134
                var dataPtr = realRes.add(16);
135

136
                if (strLen > 0 && strLen < 100000) {
137
                    var txt = Memory.readUtf8String(dataPtr, strLen);
138
                    console.log("    Decoded Content:");
139
                    console.log(txt);
140
                }
141

142
            } catch(e) {
143
                console.log("    [!] Error reading string: " + e.message);
144
                console.log(hexdump(realRes, { length: 128, header: false, ansi: true }));
145
            }
146
        }
147
    });
148
}

Data Extraction Results#

The runtime instrumentation successfully captured the base64 encoding operations:

Base64 Dump

Further hooking of jsonEncode revealed the plaintext request payload:

jsonEncode Dump

The captured JSON payload contained a flag field that could be modified at runtime before encryption and HMAC calculation.

Final Exploitation#

The complete attack chain:

Hook the JSON encoding function to intercept plaintext data
Patch the flag field from false to true in memory
Allow normal encoding and HMAC calculation to proceed
Capture the modified request from HTTP Toolkit
Replay the request via Postman (using HTTP to avoid SSL issues)

Flag Capture:

Flag

Conclusion#

The Ace challenge exemplified the evolving landscape of application security, where cross-platform frameworks introduce both new attack surfaces and new defensive opportunities. The successful solution required chaining multiple advanced techniques spanning Windows API manipulation, cryptographic protocol analysis, custom toolchain compilation, and runtime instrumentation.

As Flutter continues to gain adoption for desktop application development, security researchers and practitioners must develop platform-specific assessment methodologies. This research provides a foundation for Flutter Windows security analysis that can be extended and refined as the framework evolves.

References#

For questions, discussions, or collaboration opportunities, connect with the me on LinkedIn.

Share

If this article helped you, please share it with others!

Ace CTF Challenge Write-up - Advanced Flutter Windows Reverse Engineering

https://titoot.dev/posts/ace-ctf-challenge-writeup/

Author

Titoot

Published at

2026-03-17

License

CC BY-NC-SA 4.0

Some information may be outdated

Custom Communication writeup | ICMTC 2024 CTF

Titoot's Lair

Abstract#

Introduction#

Phase 1: Initial Reconnaissance and Binary Analysis#

Challenge Discovery#

File Structure Analysis#

Flutter Windows Architecture Primer#

Phase 2: Anti-Screen Capture Bypass#

Behavioral Analysis#

Technical Investigation#

Win32 API Reference: SetWindowDisplayAffinity#

String Analysis Confirmation#

Bypass Implementation#

Execution and Results#

Challenge Progression#

Phase 3: Network Communication Analysis#

Initial Network Investigation#

Flutter Network Traffic Interception#

SSL Pinning Bypass Research#

Manual Function Signature Extraction#

Updated SSL Pinning Bypass Script#

Successful Bypass Execution#

Network Traffic Analysis#

Phase 4: Dart Snapshot Reverse Engineering#

The Challenge of Dart AOT Snapshots#

Initial String Analysis#

Modified Flutter Engine Compilation#

reFlutter Architecture#

Custom Engine Modification#

Compilation and Deployment#

Addressing Code Obfuscation#

Phase 5: Runtime Instrumentation and Data Extraction#

Isolate Snapshot Calibration#

Advanced Frida Hook Implementation#

Data Extraction Results#

Final Exploitation#

Conclusion#

References#

Table of Contents