|
|
abukta
super admin
profile |
|
Join Date: 29 Mar 2005 21:03
Posts: |
|
|
Performance Costs of using LVM |
The LVM layer in the kernel keeps a mapping of logical storage locations to real (physical) storage locations. That means that for each read or write access to storage LVM has to translate the location of the asked-for piece of information on the disks by looking it up in a table. While this "costs" in terms of CPU usage the operations done are "cheap" (i.e. done very fast), since doing a lookup in a mapping table is one of the fastest and most basic operations computers do.
Each LVM physical volume contains a small area with information about the PV itself (for example the ID of this volume, because LVM identifies it via this ID and not via its name on the system, because the name can change, e.g. if the SCSI id changes!), about the volume group the PV belongs to and about the LVs stored on it. Since this area is pretty small (about 128k is used on every PV on a VG of 8.5 G with 5 PVs) and often used - for every single read/write operation - it is stored in the buffer cache of the Linux kernel, i.e. in RAM. Even more, those parts of it actually used by the piece of code the processor executes are most likely stored in the 1st-level cache, i.e. on the processor itself. Doing the mapping lookup therefore takes no time at all on today's ultra-high frequency CPUs compared to the very long time it takes to access the mechanical storage on the disks.
Also keep in mind that LVM data is only written when the LVM configuration is changed. During runtime, all operations involving the LVM mapping table are read-only. That is very good for caching, especially on multi-processor systems where the 1st-level caches on all the CPUs contain parts of the mapping table.
Real Test - A Benchmark
Theory is nice, but how about the real world?
Here are some numbers from a (non scientific) benchmark that compares I/O on raw disks with I/O over LVM and just one disk, to see the LVM overhead directly, and LVM and 4 disks in a striped configuration.
We used the Bonnie benchmark, which is also included in SUSE Linux under that same name. See the documentation that comes with that package for details about this particular benchmark. The machine used was a Compaq ProLiant 1850 with 6 disks. The Smart-Array RAID controller in this server has been configured not to do anything, that means we have direct access to the disks. The machine has 512 MB of RAM. Since the filesize used in the benchmark is 1024MB we manage to defeat caching to get to the "real" I/O performance. The filesystem is ext2 with 4k blocksize in table 1 and reiserfs in table 2. This adds the opportunity to compare ext2 and reiserfs, but please do not let that aspect confuse you: Look at only one of the tables at a time when you check for the results for LVM.
At first the four disks are tested with no LVM involved. These numbers can later be used for comparison.
The next test includes LVM. A VG is created that contains just one PV, and that is disk HD 1. We can compare the numbers achieved directly with those we got from testing HD 1 directly, without LVM, and get to the LVM overhead this way. As promised, there is no visible overhead. The data we get there does not look any different from what we got when testing the disks without LVM.
The last test creates a VG that contains all four disks involved in this test. On this VG an LV is created that is striped over all four PVs (disks) to give us the opportunity to assess any possible performance gains or losses when using LVM striping (software RAID0).
Table 1: Filesystem is ext2
---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-
K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
-------------------------------------------------------------------------
HD 1 (no LVM)
8500 98.9 20073 17.2 9070 15.7 7970 88.4 22082 16.2 251.2 2.6
HD 2 (no LVM)
8358 97.1 20927 18.0 9129 16.2 7968 88.6 22096 15.7 258.9 3.6
HD 3 (no LVM)
8309 96.5 20525 17.8 9079 15.8 7729 85.9 22091 15.7 249.9 3.4
HD 4 (no LVM)
8343 97.0 21295 18.4 9065 16.3 7938 88.2 22017 14.9 258.7 3.2
LVM - HD 1 only, shows pure LVM overhead, compare with result 1st HD above
8395 97.8 21574 18.8 9106 16.0 7957 88.5 22097 16.6 256.0 2.6
LVM - HDs 1,2,3,4 in one VG, one striped LV (striped over all 4 HDs)
8474 98.9 32048 27.8 6138 11.1 7142 79.8 23983 17.8 385.6 6.4
8428 98.3 31942 28.4 6064 10.7 6875 76.5 24198 17.6 372.1 4.4
Table 2: Filesystem is reiserfs
---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-
K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
-------------------------------------------------------------------------
HD 1 (no LVM)
7775 98.9 22793 76.2 8686 16.9 7265 81.4 21856 17.1 255.9 3.1
HD 2 (no LVM)
8399 97.8 22599 19.8 9233 16.2 7958 88.4 22077 16.6 250.4 2.6
HD 3 (no LVM)
8481 98.8 21080 18.5 9080 15.9 7969 88.5 22069 16.3 249.5 3.1
HD 4 (no LVM)
8487 98.8 21149 18.4 9052 16.0 7956 88.5 22010 15.0 260.9 3.8
LVM - HD 1 only, shows pure LVM overhead, compare with result 1st HD above
7827 98.7 22609 81.8 8682 16.9 7223 80.9 21787 18.0 263.7 2.8
LVM - HDs 1,2,3,4 in one VG, one striped LV (striped over all 4 HDs)
7837 99.1 26304 85.9 5718 11.1 6744 75.6 23229 18.3 372.2 5.7
7824 98.9 29375 93.4 5787 11.1 6822 76.6 24296 18.9 392.7 4.7
message edited by abukta (27 Jul 2006 00:20)
|
Home 
Forum
