The Rekall Memory Forensic Framework API.

A command is a reusable plugin which implements a user runnable command. For example, when the user issues the pslist command, the WinPSList plugin is run. Therefore in order to be accessible to the user, a command plugin must extend the plugin.Command class.

However it is not sufficient to simply extend the plugin.Command class, the class must also declare what the name of the command it implements is. For example the WinPSList class implements the pslist command.

At any given moment there can be a number of command plugins which define a particular user command. For example, LinPSList, WinPSList and MacPSList all define a command called pslist. The correct class to run is chosen based on the profile.

Here is the base minimum that should be defined for a new command plugin:

args(cls, parser)

This is a classmethod called in order to construct the command line options. See below.

__init__(**args)

This is the constructor which should take any parameters you wish to accept. If you also want the user to be able to provide these parameters through the command line you will need to add them to the parser provided in args() above.

is_active(cls, session)

This method will be called with the session to check if this specific class is active. This mechanism allows multiple implementations to all share the same name, as long as only one is actually active. For example, we can have a linux, windows and mac version of plugins with the same "pslist" name.

render(renderer)

This is the main entry point for the command when called from the Rekall UI. The plugin should render any output using the renderer (see Rendering Output).

In addition to these methods the plugin should define any other methods that can be reusable by other components.

Many of the existing command plugins implement some of these methods via inheritance. For example, plugins which can only operate on a Linux image may extend the rekall.plugins.linux.common.LinuxPlugin class which already implements the correct is_active() method.

Similarly plugins which operate on processes may extend therekall.plugins.linux.common.LinuxProcessFilter plugin. This will give the new plugin the args() method defining all the commandline options which allow the user to select processes (i.e. by pid, by task address etc). In this way the commands which operate on processes can present a consistent user interface easily with the same filtering options.

So what happens when a user invokes rekall from the command line? Assume the following command is issued:

Rekall is strict with the order in which commands are supplied. For example it is not valid to provide the --proc_reg parameter before the pslist keyword (since it is a module level parameter). Similarly it is not legal to provide the --filename parameter after the command name (pslist). This strictness allows different modules to define the same command line option for their own needs and avoids any command line option clashes between different plugins.

This strictness can easily be observed when requesting help. Without a module name, the help output simply lists those options processed by the main program (i.e. global options). It also provides a list of available modules:

Once the module is provided, we see a per-module help output:

When invoked without a command name, Rekall drops into the interactive shell. This mode of operation is more efficient as many commands can be run without needing to reinitialize the framework each time.

This is what happens during initialization:

Most rekall plugins expect to have valid address spaces set in the session object before they run. There are two main session parameters which are commonly required, the session.physical_address_space and session.kernel_address_space. Usually if these parameters are not provided in the session, the plugins will automatically invoke the load_as() plugin.

The load_as plugin is just a regular command plugin, which means that it can be implemented by different plugin.Command() classes (autoselected via the is_active() class method - see Command Plugin). This means we can have one implementation for windows, one for linux etc.

The load_as plugin is responsible for loading two different address spaces. The physical address space refers to loading the image in whatever format it might be into a direct linear address space. The kernel virtual address space is the view of the virtual memory as seen by the kernel.

The physical address space is derived by an automatic voting algorithm to auto-detect the memory image format:

1. Start with the None address space and pass it to all address spaces in their requested order (classes are sorted by their order attribute). Address spaces which are incompatible with the base address space will raise ASAssertionError and will be skipped.

2. The first address space which instantiates successfully, will be accepted as the next base address space.

3. The process is repeated until all address spaces failed to instantiate. We then return the last successfully instantiated address space.

For example, suppose we have a Windows Crash dump image which we compressed using the EWF format. In the first voting round, the EWF address space will detect that this is a valid EWF format, and will be selected. Then all the other image address spaces will be tried on the decompressed EWF image, and the crash dump address space will detect it as a valid crash dump.

In the windows load_as() plugin, the virtual address space is created from the kernel’s Directory Table Base (DTB). If the DTB is not directly provided, theload_as() plugin employs the find_dtb() plugin to detect the dtb. On Windows, the find_dtb() plugin scans the image for the Idle process. In other implementations, the kernel DTB is calculated using some other way (e.g. directly from debug symbols). The correct find_dtb() plugin for the selected profile will be used, allowing a different algorithm to be used for windows or linux.

typedef unsigned char uchar;
enum {
  OPT1,
  OPT2
} options;

struct foobar {
    enum options flags;
    short int bar;
    uchar *foo;
}

class BaseObject(object):
    def __init__(self, theType=None, offset=0, vm=None, profile=None,
                 parent=None, name='', context=None, **kwargs):
        """Constructor for Base object.

        Args:

          theType: The name of the type of this object. This different
             from the class name, since the same class may implement many types
             (e.g. Struct implements every instance in the vtype definition).

          offset: The offset within the address space to this object exists.

          vm: The address space this object uses to read itself from.

          profile: The profile this object may use to dereference other
           types.

          parent: The object which created this object.

          name: The name of this object.

          context: An opaque dict which is passed to all objects created from
            this object. This dict may contain context specific information
            which each derived instance can use.

          kwargs: Arbitrary args this object may accept - these can be passed in
             the vtype language definition.
        """
....

class Struct(BaseAddressComparisonMixIn, BaseObject):
    """ A Struct is an object which represents a c struct

    Structs have members at various fixed relative offsets from our own base
    offset.
    """
    def __init__(self, members = None, struct_size = 0, **kwargs):
       ....

class String(obj.StringProxyMixIn, obj.NativeType):
    """Class for dealing with Null terminated C Strings.
    """
    def __init__(self, length = 1024, term="\x00", **kwargs):
        ....

class Pointer(NativeType):
    """A pointer reads an 'address' object from the address space."""

    def __init__(self, target=None, target_args=None, value=None, **kwargs):
        """Constructor.

        Args:
           target: The name of the target object (A string). We use the profile
             to instantiate it.
           target_args: The target will receive these as kwargs.
        """
        ....

In order to control object creation automatically, we need to describe how they are to be created. This description is termed the vtypes language. It is really a data driven description of how to create instances of the Struct() class.

The precise format of a vtype language struct definition is as follows:

The VTypes language was designed to allow:

• Partial definition of struct members - not all members in the struct must be defined. The offset of the member in the struct is explicitly given. This allows us to create aliases (i.e. many fields which access the same memory location) as well as sparse structs (i.e. structs with only a few fields known).

• Struct members are simply names of object classes (inherited from obj.BaseObject). These classes take care of actually parsing the data. This allows us to interpret the memory offset in arbitrary ways. These classes are instantiated at the required offset.

The following is an example of a vtype definition generated from debugging symbols:

Rekall aims to specify semantic information about each field type. That means that we are really looking for the meaning behind each field, not just the mechanics of how to parse it. For example, the following struct may be defined in C:

The debugging symbols will generate for this field an array of char objects:

However, while technically correct, this is not semantically correct. We know that the array of char objects should really be interpreted as a null terminated unicode string in UTF8. We know that the offset of this field is correct though, just that its meaning according to the debug symbols is inaccurate.

The vtype language allows specification of Overlays to "correct" or adjust the values of lower layers. In this case we load the debug generated vtype first, then we load an overlay like:

Over the top. The overlay may specify a value of None for the offset, or the struct size positions. This will allow these values to "bubble up" from the lower level description. However, specifying a new class name will override the values in the lower vtype description.

In practice this is used to provide higher level semantic information to existing fields in a version independent manner. The exact offsets of fields is obtained from the debugging symbols, but semantic meaning is obtained from the overlay.

The vtype language allows recursive definition of field types. This is encouraged since it leads to semantically readable code which exactly describes the nature of the memory objects. For example:

Specifies the name member to be a unicode string of length 60, while the kp field is a pointer to an array of kernel_param objects. The array size is specified in the module’s num_kp member.

Note that None is specified for some fields in this vtype description. This means that the value in this position will be overlayed (or taken from a previous layer).

In order to simplify the description within the vtypes languages, we can replace many of the fields with python callables (usually lambda ). In the above example, we specified the count parameter of the Array constructor as a callable fetching the value from the module object’s num_kp field:

• Specifying a callable in place of the struct’s size can determine the size from the actual struct itself (e.g. if the size is stored in a member).

• Callables in the field offset position specify the offset of the field. Note that this is evaluated to the absolute offset.

• Callables in the keyword args field are evaluated when the field is accessed.

By convention, Rekall specifies pure data in the lowest vtype description layer (usually extracted from debugging symbols), while callables are only specified in overlays (possibly leaving gaps for the debugging information to bubble through them). This means that the lowest layer vtype descriptions are purely data, and can therefore be encoded in a safe format, such as JSON.

The profile is essentially the factory class for all Rekall objects. A profile is where a number of sources of information are combined in order to produce information consistant with a single uniform compilation unit:

• The vtype descriptions are added to the profile.

• The overlays specific for an operating system are added (these bring semantic information).

• Constants from debugging symbols are introduced.

The profile is built by applying all relevant overlays and classes to parse the compilation unit it cares about. For example the following is a base profile for parsing the Windows kernel:

We can see this profile is applying classes, overlays and constants to the profile. Viewed as a whole, the profile can be said to implement a parsing system for the windows kernel. When a user selects the profile with the --profile command line arg, they are really selecting which profile should be created for parsing the kernel.

In the code, the profile is an instance of the obj.Profile command. Generally however, the profile contains large data structures such as the VType dictionary and constant lists. It is much better to be able to serialize the profile to a standard form (for example for storage in the profile repository as described below).

The Profile File is the serialization of a profile into a single JSON encoded object. The file represents all the data required in order to instantiate the profile instance. Among all the data serialization methods available in python, JSON is perhaps the fastest since it is natively implemented in C, and so makes sense for a permanent storage format.

The JSON file is essentially a dictionary with the following keys:

• $METADATA: This is a dictionary representing the metadata of this profile:

  • Type: Currently can be Profile or Symlink.

  • Version: (Code version) if present (otherwise assumes version 1).

  • ProfileClass: The name of the class to instantiate as the base for this profile.

• $ENUM: These represent dictionary of enum value→name mappings.

• $CONSTANTS: These represent all constant addresses applicable for this profile (i.e. addresses of global symbols).

• $STRUCTS: This is a dict with the descriptions of the structs using the vtypes language.

In order to load the profile, the code parses the json serialized data: . Examine the type of the blob ($METADATA.Type). . If it is a profile, we search for the implementation specified in the ProfileClass and instantiate it. . Call its add_constants() method with the constants found in the $CONSTANTS section. . Call its add_types() method with the $STRUCTS section. . Call its add_enums() method with $ENUMS section.

A special case is when the $METADATA.Type == "Symlink". In that case, the object actually refers to a different named profile (Stored in $METADATA.Target) , and that profile is opened instead. This mechanism allows us to store a specific profiles by build numbers (e.g. for windows 5.1.2600.6165_I386) but still have those accessible via a human readable name like WinXPSP1x86.

Most of the information in a profile is extracted from debugging symbols specific for the executable of interest. In the case of operating systems, debugging information is extracted from the operating system kernels (via DWARF or PDB symbols).

In practice Rekall supports so many different operating systems and versions that it is impractical to ship Rekall with all the profiles it natively supports. For example, each OSX version has a unique set of vtypes extracted for each kernel version (currently over 40 OSX Darwin releases are supported with an average profile size of around 400kb). Additionally each Linux kernel version must use a different profile file for each linux build and kernel version (even the standard distributions like Ubuntu ship many kernels each year). Similarly if Rekall is used as a library in another application, adding these profiles directly into the Rekall source code will needlessly bloat the application.

In order to solve this problem, the Rekall project provides for Profile Repositories. When a profile is specified (using the --profile command line, or when passed to the session.LoadProfile() function), Rekall will search for this profile using the profile path configuration parameter. By adding the public profile repository to the search path, it is possible to automatically use the public repository for profiles that are widely known. It is also possible to add a secondary profile repository for local or less commonly seen profiles.

The following sections give examples of generating new profiles for various operating systems.

4.5.1. Generating Linux Profiles

/tmp$ wget http://downloads.rekall.googlecode.com/git/Linux/linux_pmem_1.0RC1.tgz
--2014-01-17 10:57:19--  http://downloads.rekall.googlecode.com/git/Linux/linux_pmem_1.0RC1.tgz
Resolving downloads.rekall.googlecode.com (downloads.rekall.googlecode.com)... 2a00:1450:4001:c02::52, 173.194.70.82
Connecting to downloads.rekall.googlecode.com (downloads.rekall.googlecode.com)|2a00:1450:4001:c02::52|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10854 (11K) [application/octet-stream]
Saving to: `linux_pmem_1.0RC1.tgz'

100%[=============================================>] 10,854      --.-K/s   in 0.005s

2014-01-17 10:57:20 (2.08 MB/s) - `linux_pmem_1.0RC1.tgz' saved [10854/10854]

/tmp$ tar -xvzf linux_pmem_1.0RC1.tgz
linux/
linux/ko_patcher.py
linux/module.c
linux/pmem.c
linux/README
linux/.gitignore
linux/Makefile
/tmp$ cd linux/

/tmp/linux$ sudo make profile
make -C /usr/src/linux-headers-3.8.0-35-generic CONFIG_DEBUG_INFO=y M=`pwd` modules
make[1]: Entering directory `/usr/src/linux-headers-3.8.0-35-generic'
  CC [M]  /tmp/linux/module.o
  CC [M]  /tmp/linux/pmem.o
  Building modules, stage 2.
  MODPOST 2 modules
  CC      /tmp/linux/module.mod.o
  LD [M]  /tmp/linux/module.ko
  CC      /tmp/linux/pmem.mod.o
  LD [M]  /tmp/linux/pmem.ko
make[1]: Leaving directory `/usr/src/linux-headers-3.8.0-35-generic'
cp module.ko module_dwarf.ko
zip "`uname -r`.zip" module_dwarf.ko /boot/System.map-`uname -r`
  adding: module_dwarf.ko (deflated 66%)
  adding: boot/System.map-3.8.0-35-generic (deflated 79%)

/tmp/linux$ unzip -l 3.8.0-35-generic.zip
Archive:  3.8.0-35-generic.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
   371919  2014-01-17 10:57   module_dwarf.ko
  3192757  2013-12-04 18:49   boot/System.map-3.8.0-35-generic
---------                     -------
  3564676                     2 files

11:01:17> convert_profile "3.8.0-35-generic.zip", "3.8.0-35-generic"

4.5.2. Generating Windows Profiles

$ rekall peinfo --filename ntoskrnl.exe

Attribute            Value
-------------------- -----
Machine              IMAGE_FILE_MACHINE_AMD64
TimeDateStamp        2013-03-19 03:32:06+0000
Characteristics      IMAGE_FILE_EXECUTABLE_IMAGE, IMAGE_FILE_LARGE_ADDRESS_AWARE
GUID                 2c39f687423840e793308f28c4fde0cd

.......
Version Information:
key                  value
-------------------- -----
CompanyName          Microsoft Corporation
FileDescription      NT Kernel & System
FileVersion          6.1.7600.17273 (win7_gdr.130318-1532)
InternalName         ntkrnlmp.exe
LegalCopyright       Microsoft Corporation. All rights reserved.
OriginalFilename     ntkrnlmp.exe
ProductName          Microsoft Windows Operating System
ProductVersion       6.1.7600.17273

$ rekal fetch_pdb --filename ntoskrnl.exe -D .
Trying to fetch http://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/2C39F687423840E793308F28C4FDE0CD2/ntkrnlmp.pd_
Received 2654299 bytes
Extracting cabinet: /tmp/ntkrnlmp.pd_
  extracting ntkrnlmp.pdb

All done, no errors.

$ rekal parse_pdb -f /tmp/ntkrnlmp.pdb --output 2C39F687423840E793308F28C4FDE0CD2 \
  --profile_class Win7x64
$ head 2C39F687423840E793308F28C4FDE0CD2
{
 "$METADATA": {
  "ProfileClass": "Win7x64",
  "Type": "Profile"
  },
 "$STRUCTS": {
  "BATTERY_REPORTING_SCALE": [8, {
   "Capacity": [4, ["unsigned long", {}]],
   "Granularity": [0, ["unsigned long", {}]]
   }],

4.5.3. Symbolic names

$ rekal --profile ntoskrnl.exe/AMD64/6.1.7600.17273/2C39F687423840E793308F28C4FDE0CD2 \
  -f ~/images/win7.elf pslist

{
  "$METADATA": {
    "Type": "Symlink",
    "Target": "ntoskrnl.exe/AMD64/6.1.7601.17514/3844dbb920174967be7aa4a2c20430fa"
  }
}

$ rekal -v --profile Win7SP1x64  -f ~/images/win7.elf pslist
INFO:root:Loaded profile ntoskrnl.exe/AMD64/6.1.7601.17514/3844dbb920174967be7aa4a2c20430fa from URL:http://profiles.rekall.googlecode.com/git/
INFO:root:Loaded profile Win7SP1x64 from URL:http://profiles.rekall.googlecode.com/git/
....

The profile is a self contained system for parsing the kernel data structures. However, some modules would like to alter the profile slightly - for example to add new classes replacing the default classes (with additional methods), or maybe adding new information obtained by reverse engineering certain data structures. In these cases we wish to modify the profile definition by adding an improved class definition system.

It is normally discouraged to directly add new BaseObject class implementations to the rekall framework since the changes will appear in all users of the profile - potentially clashing with others' modifications. In other words we want to modify the profile only for the users of this profile.

This can be done by explicitly calling the ProfileModification class in your plugin. This will install the updated implementation in your profile - without affecting other profiles. This localized change opens the door for multiple implementations of profile parsing systems.

For example consider the standard registry parsing implementation inrekall.plugins.windows.registry.registry. This implementation is a fast, self contained and complete implementation of registry parsing in the windows kernel. For a plugin to use this implementation, they will need to add it to their current profile:

4.6.1. The Registry parsing implementation.

4.6.2. The PE parsing implementation.

[DEFAULT]
--profile = Win7SP1x64
--filename = %(testdir)s/win7.elf

# When any test is looking for a pid, use this one.
pid = 2912

[TestDT]
commandline = dt _EPROCESS

[TestDump]
commandline = dump 0xfa8002193060

[TestVtoP]
commandline = vtop 0xfa8002193060

[TestDisassemble]
func = 0xfa8002193060

$ python tools/testing/build_suite.py template \                
  --file xp-laptop-2005-06-25_trunk/xp-laptop-2005-06-25.img    

$ python tools/testing/build_suite.py baseline \                
  --config xp-laptop-2005-06-25_trunk/tests.config             

{
    "time_used": 4.6139168739318848,
    "output": [
        "Offset(V) ||Name                ||   PID||  PPID||  Thds||    Hnds||  Sess|| Wow64||Start               ||Exit                ",
        "----------||--------------------||------||------||------||--------||------||------||--------------------||--------------------",
        "0x823c87c0||System              ||     4||     0||    61||    1140||------||     0||                    ||                    ",
        "0x81fdf020||smss.exe            ||   448||     4||     3||      21||------||     0||2005-06-25 16:47:28 ||                    ",
        "0x81ed84e8||dd.exe              ||  4012||  2624||     1||      22||     0||     0||2005-06-25 16:58:46 ||                    "
    ],
    "options": {
        "--profile": "WinXPSP2x86",
        "commandline": "pslist",
        "--filename": "/tmp/xp-laptop-2005-06-25_trunk/xp-laptop-2005-06-25.img",
    }
}

$ python tools/testing/build_suite.py test \                    
  --config xp-laptop-2005-06-25_trunk/tests.config