Virtual machines. These days it seems like you can’t even turn a corner without running into one of these faux operating systems. As a matter of fact, you probably already have a virtual machine of choice. My coworkers and I tend to go with VirtualBox. The following is how we usually roll. 


  • Host OS: Mac/Windows

  • Guest OS: Linux

  • File sharing: Shared folders connecting host and guest OSes


Although it doesn’t exactly stand out from the crowd, the most important part of this setup may very well be shared folders.


The shared folder system provided by VirtualBox allows the user to mount the host OS’s file system from inside the guest OS. While this is extremely useful, the time it takes to access files across OSes feels excruciatingly slow. find never ends, and git status seems to drag on forever.


I tried to find a way around these problems, and here’s what I found.



However, neither of these “solutions” offered any real answers to the problem at hand. That’s when it occurred to me to look into how vboxsf (VBox’s shared folder system) works compared to other file systems and see if I could find a way to make vboxsf run faster. What I found may surprise you.


The first discovery I’d like to highlight is the fact that find is surprisingly slow.


Test Environment

The test environments I used for these trials are laid out in the table below. The directory I used contained around 30,000 files, weighing in at a 4GB.


Environment

Tool Name

Version

VirtualBox


4.3.28

VMWare Fusion


7.1.2

Host OS


OS X Yosemite 10.10.2

Guest OS


Debian 8.1 (Jessie)


Findutils

GNU findutils 4.4.2


Coreutils

GNU coreutils 8.23


Glibc

2.19


Linux Kernel

3.16.0-4-amd64


VBoxGuestAdditions

4.3.18


VMWareTools

9.9.3


Detective Work

First things first. I needed to determine if the find command itself was running slowly, so I decided to run a test with a few other file systems. For this comparison, I selected vmhgfs, the file system VMWare uses for sharing files between the host and guest OS.


I chose this particular file system because vmhgfs is very similar to vboxsf. The chart below shows the time/strace results that were produced when I ran the find command in both file systems. One look at the results reveals that vboxsf is significantly slower than vmhgfs. Additionally, when we use strace to look at the system calls being run, we see that the call taking up most of vboxsf’s time is newfstatat. It also becomes painfully obvious that the number of times it’s called far surpasses vmhgfs.


I also found that the processes being called inside the find command are different for vboxsf and vmhgfs, for reasons we’ll look into next.


Parameter

vmhgfs

vboxsf

time(real)

0m7.205s

0m19.774s

time(sys)

0m5.740s

0m8.088s


// vboxsf
$ strace -c find .
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
76.59    0.509083          15     34145           newfstatat   <-Here
10.86    0.072196          73       993           openat
10.24    0.068043          68      1005         6 open
 2.14    0.014249           0     34145           write
 0.14    0.000952           1       998           fstat
 0.03    0.000173           0      2013           getdents
(omitted)


// vmhgfs
$ strace -c find .
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
94.99    3.207002        1593      2013           getdents
 2.36    0.079565          79      1005         6 open
 1.96    0.066032          66       993           openat
 0.38    0.012751          13       993           newfstatat   <-Here
 0.19    0.006551           0     34145           write
 0.12    0.004198           2      1993           close
 0.00    0.000139           0       998           fstat
(omitted)


Follow That Code!

Next, I decided to take a closer look into why the number of times newfstatat is called differs for vboxsf and vmhgfs.


The Code for find

I found that when the code below is run, the result determines whether or not newfstatat is called.


// findutils-4.4.2/gnulib/lib/fts.c 1135-1140
           bool skip_stat = (ISSET(FTS_PHYSICAL)
                     && ISSET(FTS_NOSTAT)
                     && DT_IS_KNOWN(dp)
                     && ! DT_MUST_BE(dp, DT_DIR));
           p->fts_info = FTS_NSOK;
           fts_set_stat_required(p, !skip_stat);


dp is a dirent structure retrieved using glibc’s readdir() function. dirent structures are made up of cached file paths and inode numbers. They are mainly used for converting paths to inode numbers.


In order to make a decision, the program needs to know the value for d_type. If it’s DT_UNKNOWN or DT_DIR, it runs stat. When we examine the values at this point in the process for vboxsf and vmhgfs via gdb, we see that vboxsf is dp->d_type=0 (DT_UNKNOWN), while vmhgfs returns dp->d_type=4 (DT_DIR) and dp->d_type=8 (DT_REG). The following is also included in the explanation of readdir().


Linux aside, d_type fields generally only exist in BSD systems.

d_type This field contains a value indicating the file type, making
             it possible to avoid the expense of calling lstat(2) if
             further actions depend on the type of the file.
             When a suitable feature test macro is defined

             (_DEFAULT_SOURCE on glibc versions since 2.19, or

              _BSD_SOURCE on glibc versions 2.19 and earlier), glibc

             defines the following macro constants for the value

             returned in d_type.


This problem can be explained as follows. Since d_type is always returned as DT_UNKNOWN in vboxsf, the program is forced to call stat more times than it needs to, making vboxsf run that much slower than vmhgfs.


Code for Glibc and Linux Kernel

Having gleaned this valuable information from examining the source code, I decided to press deeper into the heart of find. I now needed to find out why vboxsf caused all d_types to be returned as DT_UKNOWN. To that end, I wanted to track down how readdir() was acquiring information from the file system. readdir() is defined inside glibc. We can see that it calls the syscall getdents() in the following code excerpt.


# glibc-2.19/sysdeps/posix/readdir.c
     bytes = __GETDENTS (dirp->fd, dirp->data, maxread);


getdents() is defined inside the Linux kernel, which is where file->f_op->iterate is called.


// linux/fs/readdir.c
   if (!IS_DEADDIR(inode)) {
       ctx->pos = file->f_pos;
       res = file->f_op->iterate(file, ctx);
       file->f_pos = ctx->pos;
       fsnotify_access(file);
       file_accessed(file);
   }


file->f_op, defined per file system, is a collection of methods used for handling files.


Code for vboxsf

Since file->f_op is defined within the file system, we now need to take a look at the implemention for vboxsf. file->f_op->iterate is defined in the following way for vboxsf. As with other file systems, a little fishing around with grep turns up the results we’re looking for fairly easily.


// VirtualBox-4.3.28/src/VBox/Additions/linux/sharedfolders/dirops.c
struct file_operations sf_dir_fops =
{
   .open    = sf_dir_open,
   .iterate = sf_dir_iterate,
   .release = sf_dir_release,
   .read    = generic_read_dir,
   .llseek  = generic_file_llseek
};


From there, we found the following lines of code when we followed up on vboxsf’s sf_dir_iterate.


// VirtualBox-4.3.28/src/VBox/Additions/linux/sharedfolders/dirops.c
       if (!dir_emit(ctx, d_name, strlen(d_name), fake_ino, DT_UNKNOWN))
       {
           LogFunc(("dir_emit failed\n"));
           return 0;
       }


dir_emit() is a function used to register acquired file names and inode numbers as directory entries. In the same way, an inode number and a file name are returned when getdents() is called in vboxsf. We also find the root of the speed issue―d_type is returned as DT_UNKNOWN.


The solution I came up with to fix the problem at hand was fairly simple. If I could retrieve the dentry type from the host side via vboxsf and use that to return d_type instead, find should become much faster, even for vboxsf.


Hot-Rodding vboxsf

In order to register the correct d_type, we need to retrieve the d_type on the host side. A quick glance at how dentry is retrieved reveals that vboxsf, running on the guest side, is asking for the result from the service running on the host side. The code on the host side for retrieving dentry for Mac and Windows machines is as follows.


// VirtualBox-4.3.28/src/VBox/Runtime/r3/posix/dir-posix.cpp
RTDECL(int) RTDirRead(PRTDIR pDir, PRTDIRENTRY pDirEntry, size_t *pcbDirEntry)
...
           pDirEntry->INodeId = pDir->Data.d_ino; /* may need #ifdefing later */
           pDirEntry->enmType = rtDirType(pDir->Data.d_type);
           pDirEntry->cbName  = (uint16_t)cchName;

// VirtualBox-4.3.28/src/VBox/Runtime/r3/win/direnum-win.cpp
RTDECL(int) RTDirRead(PRTDIR pDir, PRTDIRENTRY pDirEntry, size_t *pcbDirEntry)
...
   pDir->fDataUnread  = false;
   pDirEntry->INodeId = 0; /** @todo we can use the fileid here if we must (see GetFileInformationByHandle). */
   pDirEntry->enmType = pDir->Data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY
                      ? RTDIRENTRYTYPE_DIRECTORY : RTDIRENTRYTYPE_FILE;
   pDirEntry->cbName  = (uint16_t)cchName;


This code shows that when the info for dentry is retrieved, d_type(pDirEntry->enmType) is retrieved along with it.


Herein lies a problem. The value is being returned from the guest side, but it seems like it’s not being set. That’s all, open and shut. I went ahead and modified the code so that the d_type retrieved from the host side is returned like this.


Results

After making these modifications to the code, I tried running the find command again. The time/strace results are organized in the chart below. As you can see, the number of times newfstatat is called has dropped significantly, and the amount of time used to process these calls has become much shorter, transforming vboxsf into a real speed machine that’s almost as fast as vmhgfs.



Parameter

vmhgfs

vboxsf (Before)

vboxsf (After)

time(real)

0m7.205s

0m19.774s

0m3.385s

time(sys)

0m5.740s

0m8.088s

0m0.860s


% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
55.75    0.090346          90      1005         6 open
37.71    0.061110          62       993           openat
 4.58    0.007425           0     34145           write
 0.97    0.001571           2       993           newfstatat   <- Here
 0.66    0.001071           1       998           fstat
 0.20    0.000326           0      1989           fchdir
 0.12    0.000194           0      2013           getdents
 0.00    0.000000           0         4           read
(omitted)


Bringing It Home

By returning the correct d_type value to vboxsf, we can make find and other commands that rely on d_type much faster by cutting out some of the inefficient returns. It’s also interesting to note that my revision was included in the fixes for VirtualBox 5.0.2.


Thanks for reading!


by kokukuma2