An Analysis of x86 Single-Instruction Compiling as an Obfuscation Technique

Lasse Dessau, Rasmus Bo Kajberg

AbstractIn this thesis we evaluate single instruction compiling as an obfuscation technique. We analyse the inner workings of M/o/Vfuscator2, and propose a tool to aid analysis of programs obfuscated by M/o/Vfuscator2. We find that M/o/Vfuscator2 is similar to Virtual Machine (VM) based obfuscation. However, while state of the art VM based obfuscators can construct a custom instruction set for each protected file, this is not the case for M/o/Vfuscator2. This is a limitation of M/o/Vfuscator2, and not of single instruction compiling in general. Based on this, we conclude that single instruction compiling can be used as an effective alternative architecture in state of the art VM based obfuscators.
TypeMaster's thesis [Academic thesis]
Year2016
PublisherTechnical University of Denmark, Department of Applied Mathematics and Computer Science
AddressRichard Petersens Plads, Building 324, DK-2800 Kgs. Lyngby, Denmark, compute@compute.dtu.dk
SeriesDTU Compute M.Sc.-2016
NoteDTU supervisor: Christian D. Jensen, cdje@dtu.dk, DTU Compute
Electronic version(s)[pdf]
Publication linkhttp://www.compute.dtu.dk/english
BibTeX data [bibtex]
IMM Group(s)Computer Science & Engineering