Tag: DEV

A little-known feature in modern Windows is the ability to expose hierarchical data using the file system. This is called Windows Projected File System (ProjFS), available since Windows 10 version 1809. There is even a sample that exposes the Registry hierarchy using this technology. Using the file system as a “projection” mechanism provides a couple of advantages over a custom mechanism:

• Any file viewing tool can present the information such as Explorer, or commands in a terminal.
• “Standard” file APIs are used, which are well-known, and available in any programming language or library.

Let’s see how to build a Projected File System provider from scratch. We’ll expose object manager directories as file system directories, and other types of objects as “files”. Normally, we can see the object manager’s namespace with dedicated tools, such as WinObj from Sysinternals, or my own Object Explorer:

WinObj showing parts of the object manager namespace

Here is an example of what we are aiming for (viewed with Explorer):

Explorer showing the root of the object manager namespace

First, support for ProjFS must be enabled to be usable. You can enable it with the Windows Features dialog or PowerShell:

Enable-WindowsOptionalFeature -Online -FeatureName Client-ProjFS -NoRestart

We’ll start by creating a C++ console application named ObjMgrProjFS; I’ve used the Windows Desktop Wizard project with a precompiled header (pch.h):

#pragma once

#include <Windows.h>
#include <projectedfslib.h>

#include <string>
#include <vector>
#include <memory>
#include <map>
#include <ranges>
#include <algorithm>
#include <format>
#include <optional>
#include <functional>

projectedfslib.h is where the ProjFS declarations reside. projectedfslib.lib is the import library to link against. In this post, I’ll focus on the main coding aspects, rather than going through every little piece of code. The full code can be found at https://github.com/zodiacon/objmgrprojfs. It’s of course possible to use other languages to implement a ProjFS provider. I’m going to attempt one in Rust in a future post 🙂

The projected file system must be rooted in a folder in the file system. It doesn’t have to be empty, but it makes sense to use such a directory for this purpose only. The main function will take the requested root folder as input and pass it to the ObjectManagerProjection class that is used to manage everything:

int wmain(int argc, const wchar_t* argv[]) {
	if (argc < 2) {
		printf("Usage: ObjMgrProjFS <root_dir>\n");
		return 0;
	}

	ObjectManagerProjection omp;
	if (auto hr = omp.Init(argv[1]); hr != S_OK)
		return Error(hr);

	if (auto hr = omp.Start(); hr != S_OK)
		return Error(hr);

	printf("Virtualizing at %ws. Press ENTER to stop virtualizing...\n", argv[1]);
	char buffer[3];
	gets_s(buffer);

	omp.Term();

	return 0;
}

Let start with the initialization. We want to create the requested directory (if it doesn’t already exist). If it does exist, we’ll use it. In fact, it could exist because of a previous run of the provider, so we can keep track of the instance ID (a GUID) so that the file system itself can use its caching capabilities. We’ll “hide” the GUID in a hidden file within the directory. First, create the directory:

HRESULT ObjectManagerProjection::Init(PCWSTR root) {
	GUID instanceId = GUID_NULL;
	std::wstring instanceFile(root);
	instanceFile += L"\\_obgmgrproj.guid";

	if (!::CreateDirectory(root, nullptr)) {
		//
		// failed, does it exist?
		//
		if (::GetLastError() != ERROR_ALREADY_EXISTS)
			return HRESULT_FROM_WIN32(::GetLastError());

If creation fails not because it exists, bail out with an error. Otherwise, get the instance ID that may be there and use that GUID if present:

	auto hFile = ::CreateFile(instanceFile.c_str(), GENERIC_READ, 
		FILE_SHARE_READ, nullptr, OPEN_EXISTING, 0, nullptr);
	if (hFile != INVALID_HANDLE_VALUE && ::GetFileSize(hFile, nullptr) == sizeof(GUID)) {
		DWORD ret;
		::ReadFile(hFile, &instanceId, sizeof(instanceId), &ret, nullptr);
		::CloseHandle(hFile);
	}
}

If we need to generate a new GUID, we’ll do that with CoCreateGuid and write it to the hidden file:

if (instanceId == GUID_NULL) {
	::CoCreateGuid(&instanceId);
	//
	// write instance ID
	//
	auto hFile = ::CreateFile(instanceFile.c_str(), GENERIC_WRITE, 0, nullptr, CREATE_NEW, FILE_ATTRIBUTE_HIDDEN, nullptr);
	if (hFile != INVALID_HANDLE_VALUE) {
		DWORD ret;
		::WriteFile(hFile, &instanceId, sizeof(instanceId), &ret, nullptr);
		::CloseHandle(hFile);
	}
}

Finally, we must register the root with ProjFS:

auto hr = ::PrjMarkDirectoryAsPlaceholder(root, nullptr, nullptr, &instanceId);
if (FAILED(hr))
	return hr;

m_RootDir = root;
return hr;

Once Init succeeds, we need to start the actual virtualization. To that end, a structure of callbacks must be filled so that ProjFS knows what functions to call to get the information requested by the file system. This is the job of the Start method:

HRESULT ObjectManagerProjection::Start() {
	PRJ_CALLBACKS cb{};
	cb.StartDirectoryEnumerationCallback = StartDirectoryEnumerationCallback;
	cb.EndDirectoryEnumerationCallback = EndDirectoryEnumerationCallback;
	cb.GetDirectoryEnumerationCallback = GetDirectoryEnumerationCallback;
	cb.GetPlaceholderInfoCallback = GetPlaceholderInformationCallback;
	cb.GetFileDataCallback = GetFileDataCallback;

	auto hr = ::PrjStartVirtualizing(m_RootDir.c_str(), &cb, this, nullptr, &m_VirtContext);
	return hr;
}

The callbacks specified above are the absolute minimum required for a valid provider. PrjStartVirtualizing returns a virtualization context that identifies our provider, which we need to use (at least) when stopping virtualization. It’s a blocking call, which is convenient in a console app, but for other cases, it’s best put in a separate thread. The this value passed in is a user-defined context. We’ll use that to delegate these static callback functions to member functions. Here is the code for StartDirectoryEnumerationCallback:

HRESULT ObjectManagerProjection::StartDirectoryEnumerationCallback(const PRJ_CALLBACK_DATA* callbackData, const GUID* enumerationId) {
	return ((ObjectManagerProjection*)callbackData->InstanceContext)->DoStartDirectoryEnumerationCallback(callbackData, enumerationId);
}

The same trick is used for the other callbacks, so that we can implement the functionality within our class. The class ObjectManagerProjection itself holds on to the following data members of interest:

struct GUIDComparer {
	bool operator()(const GUID& lhs, const GUID& rhs) const {
		return memcmp(&lhs, &rhs, sizeof(rhs)) < 0;
	}
};

struct EnumInfo {
	std::vector<ObjectNameAndType> Objects;
	int Index{ -1 };
};
std::wstring m_RootDir;
PRJ_NAMESPACE_VIRTUALIZATION_CONTEXT m_VirtContext;
std::map<GUID, EnumInfo, GUIDComparer> m_Enumerations;

EnumInfo is a structure used to keep an object directory’s contents and the current index requested by the file system. A map is used to keep track of all current enumerations. Remember, it’s the file system – multiple directory listings may be happening at the same time. As it happens, each one is identified by a GUID, which is why it’s used as a key to the map. m_VirtContext is the returned value from PrjStartVirtualizing.

ObjectNameAndType is a little structure that stores the details of an object: its name and type:

struct ObjectNameAndType {
	std::wstring Name;
	std::wstring TypeName;
};

The Callbacks

Obviously, the bulk work for the provider is centered in the callbacks. Let’s start with StartDirectoryEnumerationCallback. Its purpose is to let the provider know that a new directory enumeration of some sort is beginning. The provider can make any necessary preparations. In our case, it’s about adding a new enumeration structure to manage based on the provided enumeration GUID:

HRESULT ObjectManagerProjection::DoStartDirectoryEnumerationCallback(const PRJ_CALLBACK_DATA* callbackData, const GUID* enumerationId) {
	EnumInfo info;
	m_Enumerations.insert({ *enumerationId, std::move(info) });
	return S_OK;
}

We just add a new entry to our map, since we must be able to distinguish between multiple enumerations that may be happening concurrently. The complementary callback ends an enumeration which is where we delete the item from the map:

HRESULT ObjectManagerProjection::DoEndDirectoryEnumerationCallback(const PRJ_CALLBACK_DATA* callbackData, const GUID* enumerationId) {
	m_Enumerations.erase(*enumerationId);
	return S_OK;
}

So far, so good. The real work is centered around the GetDirectoryEnumerationCallback callback where actual enumeration must take place. The callback receives the enumeration ID and a search expression – the client may try to search using functions such as FindFirstFile / FindNextFile or similar APIs. The provided PRJ_CALLBACK_DATA contains the basic details of the request such as the relative directory itself (which could be a subdirectory). First, we reject any unknown enumeration IDs:

HRESULT ObjectManagerProjection::DoGetDirectoryEnumerationCallback(
	const PRJ_CALLBACK_DATA* callbackData, const GUID* enumerationId, 
	PCWSTR searchExpression, PRJ_DIR_ENTRY_BUFFER_HANDLE dirEntryBufferHandle) {

	auto it = m_Enumerations.find(*enumerationId); 
	if(it == m_Enumerations.end())
		return E_INVALIDARG;
    auto& info = it->second;

Next, we need to enumerate the objects in the provided directory, taking into consideration the search expression (that may require returning a subset of the items):

	if (info.Index < 0 || (callbackData->Flags & PRJ_CB_DATA_FLAG_ENUM_RESTART_SCAN)) {
		auto compare = [&](auto name) {
			return ::PrjFileNameMatch(name, searchExpression);
			};
		info.Objects = ObjectManager::EnumDirectoryObjects(callbackData->FilePathName, nullptr, compare);
		std::ranges::sort(info.Objects, [](auto const& item1, auto const& item2) { 
			return ::PrjFileNameCompare(item1.Name.c_str(), item2.Name.c_str()) < 0; 
			});
		info.Index = 0;
	}

There are quite a few things happening here. ObjectManager::EnumDirectoryObjects is a helper function that does the actual enumeration of objects in the object manager’s namespace given the root directory (callbackData->FilePathName), which is always relative to the virtualization root, which is convenient – we don’t need to care where the actual root is. The compare lambda is passed to EnumDirectoryObjects to provide a filter based on the search expression. ProjFS provides the PrjFileNameMatch function we can use to test if a specific name should be returned or not. It has the logic that caters for wildcards like * and ?.

Once the results return in a vector (info.Objects), we must sort it. The file system expects returned files/directories to be sorted in a case insensitive way, but we don’t actually need to know that. PrjFileNameCompare is provided as a function to use for sorting purposes. We call sort on the returned vector passing this function PrjFileNameCompare as the compare function.

The enumeration must happen if the PRJ_CB_DATA_FLAG_ENUM_RESTART_SCAN is specified. I also enumerate if it’s the first call for this enumeration ID.

Now that we have results (or an empty vector), we can proceed by telling ProjFS about the results. If we have no results, just return success (an empty directory):

if (info.Objects.empty())
	return S_OK;

Otherwise, we must call PrjFillDirEntryBuffer for each entry in the results. However, ProjFS provides a limited buffer to accept data, which means we need to keep track of where we left off because we may be called again (without the PRJ_CB_DATA_FLAG_ENUM_RESTART_SCAN flag) to continue filling in data. This is why we keep track of the index we need to use.

The first step in the loop is to fill in details of the item: is it a subdirectory or a “file”? We can also specify the size of its data and common times like creation time, modify time, etc.:

while (info.Index < info.Objects.size()) {
	PRJ_FILE_BASIC_INFO itemInfo{};
	auto& item = info.Objects[info.Index];
	itemInfo.IsDirectory = item.TypeName == L"Directory";
	itemInfo.FileSize = itemInfo.IsDirectory ? 0 : 
		GetObjectSize((callbackData->FilePathName + std::wstring(L"\\") + item.Name).c_str(), item);

We fill in two details: a directory or not, based on the kernel object type being “Directory”, and a file size (in case of another type object). What is the meaning of a “file size”? It can mean whatever we want it to mean, including just specifying a size of zero. However, I decided that the “data” being held in an object would be text that provides the object’s name, type, and target (if it’s a symbolic link). Here are a few example when running the provider and using a command window:

C:\objectmanager>dir p*
 Volume in drive C is OS
 Volume Serial Number is 18CF-552E

 Directory of C:\objectmanager

02/20/2024  11:09 AM                60 PdcPort.ALPC Port
02/20/2024  11:09 AM                76 PendingRenameMutex.Mutant
02/20/2024  11:09 AM                78 PowerMonitorPort.ALPC Port
02/20/2024  11:09 AM                64 PowerPort.ALPC Port
02/20/2024  11:09 AM                88 PrjFltPort.FilterConnectionPort
               5 File(s)            366 bytes
               0 Dir(s)  518,890,110,976 bytes free

C:\objectmanager>type PendingRenameMutex.Mutant
Name: PendingRenameMutex
Type: Mutant

C:\objectmanager>type powerport
Name: PowerPort
Type: ALPC Port

Here is PRJ_FILE_BASIC_INFO:

typedef struct PRJ_FILE_BASIC_INFO {
    BOOLEAN IsDirectory;
    INT64 FileSize;
    LARGE_INTEGER CreationTime;
    LARGE_INTEGER LastAccessTime;
    LARGE_INTEGER LastWriteTime;
    LARGE_INTEGER ChangeTime;
    UINT32 FileAttributes;
} PRJ_FILE_BASIC_INFO;

What is the meaning of the various times and file attributes? It can mean whatever you want – it might make sense for some types of data. If left at zero, the current time is used.

GetObjectSize is a helper function that calculates the number of bytes needed to keep the object’s text, which is what is reported to the file system.

Now we can pass the information for the item to ProjFS by calling PrjFillDirEntryBuffer:

	if (FAILED(::PrjFillDirEntryBuffer(
		(itemInfo.IsDirectory ? item.Name : (item.Name + L"." + item.TypeName)).c_str(), 
		&itemInfo, dirEntryBufferHandle)))
		break;
	info.Index++;
}

The “name” of the item is comprised of the kernel object’s name, and the “file extension” is the object’s type name. This is just a matter of choice – I could have passed the object’s name only so that it would appear as a file with no extension. If the call to PrjFillDirEntryBuffer fails, it means the buffer is full, so we break out, but the index is not incremented, so we can provide the next object in the next callback that does not requires a rescan.

We have two callbacks remaining. One is GetPlaceholderInformationCallback, whose purpose is to provide “placeholder” information about an item, without providing its data. This is used by the file system for caching purposes. The implementation is like so:

HRESULT ObjectManagerProjection::DoGetPlaceholderInformationCallback(const PRJ_CALLBACK_DATA* callbackData) {
	auto path = callbackData->FilePathName;
	auto dir = ObjectManager::DirectoryExists(path);
	std::optional<ObjectNameAndType> object;
	if (!dir)
		object = ObjectManager::ObjectExists(path);
	if(!dir && !object)
		return HRESULT_FROM_WIN32(ERROR_FILE_NOT_FOUND);

	PRJ_PLACEHOLDER_INFO info{};
	info.FileBasicInfo.IsDirectory = dir;
	info.FileBasicInfo.FileSize = dir ? 0 : GetObjectSize(path, object.value());
	return PrjWritePlaceholderInfo(m_VirtContext, callbackData->FilePathName, &info, sizeof(info));
}

The item could be a file or a directory. We use the file path name provided to figure out if it’s a directory kernel object or something else by utilizing some helpers in the ObjectManager class (we’ll examine those later). Then the structure PRJ_PLACEHOLDER_INFO is filled with the details and provided to PrjWritePlaceholderInfo.

The final required callback is the one that provides the data for files – objects in our case:

HRESULT ObjectManagerProjection::DoGetFileDataCallback(const PRJ_CALLBACK_DATA* callbackData, UINT64 byteOffset, UINT32 length) {
	auto object = ObjectManager::ObjectExists(callbackData->FilePathName);
	if (!object)
		return HRESULT_FROM_WIN32(ERROR_FILE_NOT_FOUND);

	auto buffer = ::PrjAllocateAlignedBuffer(m_VirtContext, length);
	if (!buffer)
		return E_OUTOFMEMORY;

	auto data = GetObjectData(callbackData->FilePathName, object.value());
	memcpy(buffer, (PBYTE)data.c_str() + byteOffset, length);
	auto hr = ::PrjWriteFileData(m_VirtContext, &callbackData->DataStreamId, buffer, byteOffset, length);
	::PrjFreeAlignedBuffer(buffer);

	return hr;
}

First we check if the object’s path is valid. Next, we need to allocate buffer for the data. There are some ProjFS alignment requirements, so we call PrjAllocateAlignedBuffer to allocate a properly-aligned buffer. Then we get the object data (a string, by calling our helper GetObjectData), and copy it into the allocated buffer. Finally, we pass the buffer to PrjWriteFileData and free the buffer. The byte offset provided is usually zero, but could theoretically be larger if the client reads from a non-zero position, so we must be prepared for it. In our case, the data is small, but in general it could be arbitrarily large.

GetObjectData itself looks like this:

std::wstring ObjectManagerProjection::GetObjectData(PCWSTR fullname, ObjectNameAndType const& info) {
	std::wstring target;
	if (info.TypeName == L"SymbolicLink") {
		target = ObjectManager::GetSymbolicLinkTarget(fullname);
	}
	auto result = std::format(L"Name: {}\nType: {}\n", info.Name, info.TypeName);
	if (!target.empty())
		result = std::format(L"{}Target: {}\n", result, target);
	return result;
}

It calls a helper function, ObjectManager::GetSymbolicLinkTarget in case of a symbolic link, and builds the final string by using format (C++ 20) before returning it to the caller.

That’s all for the provider, except when terminating:

void ObjectManagerProjection::Term() {
	::PrjStopVirtualizing(m_VirtContext);
}

The Object Manager

Looking into the ObjectManager helper class is somewhat out of the focus of this post, since it has nothing to do with ProjFS. It uses native APIs to enumerate objects in the object manager’s namespace and get details of a symbolic link’s target. For more information about the native APIs, check out my book “Windows Native API Programming” or search online. First, it includes <Winternl.h> to get some basic native functions like RtlInitUnicodeString, and also adds the APIs for directory objects:

typedef struct _OBJECT_DIRECTORY_INFORMATION {
	UNICODE_STRING Name;
	UNICODE_STRING TypeName;
} OBJECT_DIRECTORY_INFORMATION, * POBJECT_DIRECTORY_INFORMATION;

#define DIRECTORY_QUERY  0x0001

extern "C" {
	NTSTATUS NTAPI NtOpenDirectoryObject(
		_Out_ PHANDLE hDirectory,
		_In_ ACCESS_MASK AccessMask,
		_In_ POBJECT_ATTRIBUTES ObjectAttributes);

	NTSTATUS NTAPI NtQuerySymbolicLinkObject(
		_In_ HANDLE LinkHandle,
		_Inout_ PUNICODE_STRING LinkTarget,
		_Out_opt_ PULONG ReturnedLength);

	NTSTATUS NTAPI NtQueryDirectoryObject(
		_In_  HANDLE hDirectory,
		_Out_ POBJECT_DIRECTORY_INFORMATION DirectoryEntryBuffer,
		_In_  ULONG DirectoryEntryBufferSize,
		_In_  BOOLEAN  bOnlyFirstEntry,
		_In_  BOOLEAN bFirstEntry,
		_In_  PULONG  EntryIndex,
		_Out_ PULONG  BytesReturned);
	NTSTATUS NTAPI NtOpenSymbolicLinkObject(
		_Out_  PHANDLE LinkHandle,
		_In_   ACCESS_MASK DesiredAccess,
		_In_   POBJECT_ATTRIBUTES ObjectAttributes);
}

Here is the main code that enumerates directory objects (some details omitted for clarity, see the full source code in the Github repo):

std::vector<ObjectNameAndType> ObjectManager::EnumDirectoryObjects(PCWSTR path, 
	PCWSTR objectName, std::function<bool(PCWSTR)> compare) {
	std::vector<ObjectNameAndType> objects;
	HANDLE hDirectory;
	OBJECT_ATTRIBUTES attr;
	UNICODE_STRING name;
	std::wstring spath(path);
	if (spath[0] != L'\\')
		spath = L'\\' + spath;

	std::wstring object(objectName ? objectName : L"");

	RtlInitUnicodeString(&name, spath.c_str());
	InitializeObjectAttributes(&attr, &name, 0, nullptr, nullptr);
	if (!NT_SUCCESS(NtOpenDirectoryObject(&hDirectory, DIRECTORY_QUERY, &attr)))
		return objects;

	objects.reserve(128);
	BYTE buffer[1 << 12];
	auto info = reinterpret_cast<OBJECT_DIRECTORY_INFORMATION*>(buffer);
	bool first = true;
	ULONG size, index = 0;
	for (;;) {
		auto start = index;
		if (!NT_SUCCESS(NtQueryDirectoryObject(hDirectory, info, sizeof(buffer), FALSE, first, &index, &size)))
			break;
		first = false;
		for (ULONG i = 0; i < index - start; i++) {
			ObjectNameAndType data;
			auto& p = info[i];
			data.Name = std::wstring(p.Name.Buffer, p.Name.Length / sizeof(WCHAR));
			if(compare && !compare(data.Name.c_str()))
				continue;
			data.TypeName = std::wstring(p.TypeName.Buffer, p.TypeName.Length / sizeof(WCHAR));
			if(!objectName)
				objects.push_back(std::move(data));
			if (objectName && _wcsicmp(object.c_str(), data.Name.c_str()) == 0 || 
				_wcsicmp(object.c_str(), (data.Name + L"." + data.TypeName).c_str()) == 0) {
				objects.push_back(std::move(data));
				break;
			}
		}
	}
	::CloseHandle(hDirectory);
	return objects;
}

NtQueryDirectoryObject is called in a loop with increasing indices until it fails. The returned details for each entry is the object’s name and type name.

Here is how to get a symbolic link’s target:

std::wstring ObjectManager::GetSymbolicLinkTarget(PCWSTR path) {
	std::wstring spath(path);
	if (spath[0] != L'\\')
		spath = L"\\" + spath;

	HANDLE hLink;
	OBJECT_ATTRIBUTES attr;
	std::wstring target;
	UNICODE_STRING name;
	RtlInitUnicodeString(&name, spath.c_str());
	InitializeObjectAttributes(&attr, &name, 0, nullptr, nullptr);
	if (NT_SUCCESS(NtOpenSymbolicLinkObject(&hLink, GENERIC_READ, &attr))) {
		WCHAR buffer[1 << 10];
		UNICODE_STRING result;
		result.Buffer = buffer;
		result.MaximumLength = sizeof(buffer);
		if (NT_SUCCESS(NtQuerySymbolicLinkObject(hLink, &result, nullptr)))
			target.assign(result.Buffer, result.Length / sizeof(WCHAR));
		::CloseHandle(hLink);
	}
	return target;
}

See the full source code at https://github.com/zodiacon/ObjMgrProjFS.

Conclusion

The example provided is the bare minimum needed to write a ProjFS provider. This could be interesting for various types of data that is convenient to access with I/O APIs. Feel free to extend the example and resolve any bugs.