Where It All Begins – An Introduction Protected Mode [Data Protection Series – Part 1]

In the last decade the “Data protection” has found new expressions in the field of software security. The word ‘data‘ is being replaced with ‘information‘. The line between them is blur however there is a distinction in the context they are being used. I notice that system security experts or system developers generally use the term ‘data protection’. Otherwise, mostly the term ‘data’ and ‘information’ are used indistinguishably. In my opinion, information is what we deduce from the data. It has a meaning associated with it in the forms of an answer to a question. Data can be stored or transferred by a software system. In general, a system should ‘not’ allow the data to be read or written without the user’s consent. To enable this, software systems have many inbuilt mechanisms for data protection.

The objective of this post and upcoming many posts in this series of “Data Protection” is to not only provide conceptual understanding but walk through various mechanisms (hardware and software)  with the help of sample source code and disassembly by reverse engineering tools such as WinDbg. It is expected that the reader has some knowledge of C, CPU Registers, and WinDbg or any other related debugging tool. To begin with, we look into the security or protection for executable code and data at the processor level. Later we explore other levels of system security built on top of it.


Broadly speaking, the software developers and security experts are generally aware of the CPU rings or more specifically privilege levels. But we are not going to start our discussion with CPU rings. Because the discussion will remain a conceptual and a lot of other important aspects related to software security are missed. We are going to start our discussion with a brief introduction to ‘Protected Mode’ of CPU, how to enable the protected mode, and what does it protect? Next, we dig into how we access memory in protected mode.

The P-Mode

Processors before 80286 used to operate popular known ‘Real Mode‘. The real mode had some serious issue and one to call out is that you can write anywhere you want which means you can overwrite code or data. The real mode has no memory-protection. Therefore, there was a need for better memory protection. Protected mode (aka P-mode) is an operation mode available in 80286 and later processors. It was designed to overcome the limitations of real mode and make the system more stable. Many processors still support real mode for backward compatibility. In x86 architecture-based systems, the CPU is initialized in real mode by BIOS. To enter into the protected mode, we need to set the lowest bit of CR0 CPU register. Here is how you check in WinDbg:

1: kd> r cr0

cr0=8001003b  >>>>> (Binary)00000000 00000000 00000000 00001011

Enabling the protected unleashes many features of the 32-bit world. Protected mode offers lots of features such as memory protection, virtual memory, task state switching, interrupts, access to CPU rings, 32-bit registers, access to 4 GB of memory, etc. With the protected mode the usage of the six CPU segment registers is completely changed and they don’t point to the explicit memory address as in the real mode. The figure below gives an overview of address translation in protected mode. The block diagram introduces many new concepts that will be covered one-by-one, some in this post, and others in the upcoming posts.

Switching from real mode to protected mode needs some careful configuration. Before enabling the protected mode system programmer need to make sure:

    1. Interrupts are disabled. (‘cli’ instruction)
    2. Enable A20 Line (No longer supported by modern Intel CPUs).
    3. Load Global Descriptor Table with segment descriptors (covered later).

Switching to protected mode without setting up Global Descriptor Table (GDT) would mean that default values in segment registers would be used to access the descriptors and in the absence of valid descriptors, the system will run into the triple fault.

If an interrupt is triggered and IDT is not set up, it would result in double fault because the memory segment where your interrupt handler resides (pointed by IDTR CPU register) is invalid. Double fault handler is also going to be in IDT ‘nt!KiDoubleFaultAbort’ which is not set up yet so you get into a triple fault and processor resets and system restarts.


Both segmentation and paging mechanisms are independent of each other. We will cover the segmentation part in this post and cover paging in upcoming posts.

Segmentation in protected mode is a little different than the real mode. Segmentation allows dividing the program into the logical block and placing them in different areas of memory. For example, having different memory regions/segments for code and data. Protected mode not only allows creating segment but also controlling the access to the region/segment, its size, granularity, ring (privilege level) of execution, etc. You can say that this is the first level of security offered by this processor mode.

Let’s talk a little about the selectors, descriptor tables and descriptors to understand the use and functionality.

The role of segment registers in protected mode is of ‘selector’. The value in the segment register (CS, DS, SS, etc) is an index in either Global or Local Descriptor tables (GDT or LDT) pointing to a ‘descriptor’.

The ‘descriptor tables’ reside in memory and used by the processor to perform address translation. Each entry in the descriptor table is 8 byte(64 bit) long and describes a segment of memory.

Three types of Descriptor Tables are used in protected mode.

    1. Global Descriptor Table(GDT)
    2. Local Descriptor Table(LDT)
    3. Interrupt Descriptor Table(IDT)

The Location (Base Address and Limit as shown above) of these tables (Address and size) is kept in following CPU registers:

  2. LDTR (stores an index to an entry in GDT and loaded on demand in windows.)

You can dump the content of these registers via WinDgb command as shown below:

1: kd> r gdtr

1: kd> r gdtl

1: kd> r ldtr

1: kd> r idtr

1: kd> r idtl

GDT and IDT are per processor. Address of these tables is stored in the following structure:

1: kd> !pcr
KPCR for Processor 1 at 807c6000:
Major 1 Minor 1
NtTib.ExceptionList: 807e243c
NtTib.StackBase: 00000000
NtTib.StackLimit: 00000000
NtTib.SubSystemTib: 807c9750
NtTib.Version: 0000e5c4
NtTib.UserPointer: 00000002
NtTib.SelfTib: 00000000

SelfPcr: 807c6000
Prcb: 807c6120
Irql: 0000001f
IRR: 00000000
IDR: ffffffff
InterruptMode: 00000000
IDT: 807cf020
GDT: 807cec20
TSS: 807c9750

CurrentThread: 807cb800
NextThread: 8465b608
IdleThread: 807cb800

1: kd> dt nt!_KPRCB 807c6120 processorstate.specialregisters. 
+0x018 ProcessorState : 
+0x2cc SpecialRegisters : 
+0x000 Cr0 : 0x8001003b
+0x004 Cr2 : 0x3aede8
+0x008 Cr3 : 0x185000
+0x00c Cr4 : 0x406f9
+0x010 KernelDr0 : 0x83d33a7c
+0x014 KernelDr1 : 0
+0x018 KernelDr2 : 0
+0x01c KernelDr3 : 0
+0x020 KernelDr6 : 0xffff4ff0
+0x024 KernelDr7 : 0x500
+0x028 Gdtr : _DESCRIPTOR
+0x030 Idtr : _DESCRIPTOR
+0x038 Tr : 0x28
+0x03a Ldtr : 0
+0x03c Reserved : [6] 0

1: kd> dt nt!_KPRCB 807c6120 processorstate.specialregisters.gdtr.
+0x018 ProcessorState : 
+0x2cc SpecialRegisters : 
+0x028 Gdtr : 
+0x000 Pad : 0
+0x002 Limit : 0x3ff
+0x004 Base : 0x807cec20
1: kd> dt nt!_KPRCB 807c6120 processorstate.specialregisters.idtr.
+0x018 ProcessorState : 
+0x2cc SpecialRegisters : 
+0x030 Idtr : 
+0x000 Pad : 0
+0x002 Limit : 0x7ff
+0x004 Base : 0x807cf020

Putting it all together so far:

Let’s talk about the 64-bit descriptor which actually describes the memory segment. It contains the following fields:

    1. Segment base address
    2. Segment limit which tells the size of the segment.
    3. Access rights
    4. Control bits

Fig above gives the storage of various fields in 64-bit size. This messed up structure is due to the backward compatibility. Some of the fields are explained below. Please refer to the AMD manual for the detailed use of bits.

Base Address: The 32-bit starting memory address of the segment

Segment Limit: 20-bit length.

DPL: 2 bits, Privilege level(Ring 0 to Ring 3)

G: Granularity Bit Controls the resolution of the segment limit

0: 1-byte granularity

1: 4KB granularity

P: If the segment is present in the memory or not.

8-12 BITS: These 5 bits tell the type(Code\data\system) and access level.

AVL: Available for use by OS

You can imagine the above complex representation of descriptor with the simple one below:

Putting the segmentation unit together, here is how we access an address:

The linear address is passed to the paging mechanism to translate to a physical address. In the coming post, we discuss this configuration in a windows system and explore more on the paging mechanism. This will set the base for discussing more low-level system security mechanisms.

Leave a Reply

Your email address will not be published. Required fields are marked *