296 words
1 minute
Analyzing and Decrypting Video Encryption Player

A friend of mine found a video player, but a unique one, which only runs custom encrypted videos. I have been interested in video encryption for a while, but this one is different!

It’s a dotNET player that takes an encrypted video file and plays it. It has other mechanisms to prevent screen recording, but from my experience, it’s just a flag and can be bypassed effortlessly, and even hardware acceleration can go through it.

Enough talking. Let’s reverse!

Detect it easy on the main executable.

Detect It Easy showing .NET app

As stated in the image, it’s a dotNET app, which is an interpreted language with a lot of info, so let’s decompile it using dnspy (or any other dotNET decompiler like dotpeek for example)

By looking through it, we find a function that handles video data

It parses a custom header to get the video info (length, name, etc.)

dnSpy showing video header parsing function

Then, it decrypts the video header (100 bytes), but the rest of the video is unencrypted.

It creates a hash of a constant string and XORs the header with it.

dnSpy showing XOR decryption code

But wait, there’s more!#

In an updated version, they edited the algorithm, so lets take a look

dnSpy showing updated version with DLL call

It now passed the file path to an unmanaged DLL

Loading it through IDA and tracing the close, we find the following

IDA Pro disassembly of DLL

For simplicity purposes, the DLL has an MD5 hashing function I named MD5Hash and it gets the packageID as before and hash it and then concat random strings to it and then hash it again

And it does that until it gets to the final hash

That’s so boring, actually, so I’m using another method

Since both versions use XOR cipher that doesn’t stop me from doing XOR brute force of the known MP4 iso base media file format (isobmff) to get the key

XOR brute force result showing MP4 header

Share

If this article helped you, please share it with others!

Analyzing and Decrypting Video Encryption Player
https://titoot.dev/posts/analyzing-video-encryption-algorithm/
Author
Titoot
Published at
2023-12-28
License
CC BY-NC-SA 4.0

Some information may be outdated

Table of Contents