✅Integrating LVM with Hadoop and
providing Elasticity to DataNode Storage
✅Increase or Decrease the Size of Static Partition in Linux.
✅Automating LVM Partition using Python-Script.

Logical Volume Management:

Consider we are having two harddisk with size 10 GB and 20 GB, suppose we are having an scenario where we need to store a single file of 25 GB, can we store it? Answer yes, the solution is we can combine these harddisk. Then they wil be considered as single device. This concept is known as LVM (Logical Volume Management).

  • Create a group of these physical volumes which is called as Volume Group (VG).
  • Now we have to make partition using LV (Logical Volume).
  • Then format it
  • Finally mount.

Physical Volume :

I have two harddisks sdc, sdd of size 15 GiB, 10 GiB respectively.

Volume Group :

Now we have to create a VG (Volume Group) with those PVs created about. Command: vgcreate <vg-name> <PV> and use vgdisplay <vg-name> to show the details.

Logical Volume :

Now we have to create LV(Logical Volume) of our desired size. Lets create a LV of 20 GiB size. Command : lvcreate — size <value> — name <lv-name> <vg-name>.

Format :

We have successfully created partition, the next step we have to do is formatting the partition. I have used ext4 file system for formatting.

Mount:

The last step we have to do is mounting the lv. Mount it on existing directory or create a new one and then mount it. command : mount /lv /directory.

Extending the size of LV:

We have created a LV os size 20 GiB, can we extend the size to 23 GiB? answer is yes because our VG has 5 GiB more space. For extending the LV we need to follow 2 steps

  1. Format the extended part.

1. Lvextend :

command to extend the partition is lvextend — size +<value>G /lv

2. Format :

Now we have to use resize2fs to format the remaining part. Note : It will only format the part that is not formatted.

Reducing the LV size:

We have successfully extended the LV size. Now lets try to reduce the size. Foe extending we need to follow 2 steps and for reducing we need to follow 5 steps.

  • Scan and clean
  • Change the format size
  • Reduce the partition
  • Mount

1. Unmount and clean :

First step to reduce the LV size we need to unmont it and make it offline. Then it is good to clean the wanted data in Inode table. For this we are going to use e2fsck (examine errors and clean).

2. Change the format size :

We have already formatted 23 GiB, now we only to need Inode table for say 16 Gib and not 23 GiB. So use resize2fs to change the format size to 16 GiB. Now use lvdisplay command to see the details of lv. Technically we only changed the format size, but not removed the partition. Therefore 16 GiB it is not reflected. Still it shows 23 GiB.

3. Lvreduce :

Now we are going to reduce the partition using lvreduce. Command : lvreduce — size <value>G /lv

4. Mount

Finally mount it and use df -h to see the details. Successfully reduced the size of LV.

AUTOMATING LVM USING PYTHON SCRIPT:

Providing Elasticity to data node storage by LVM :

Before Creating LV and providing it to datanode directory. It takes whole space from / drive. Hadoop Master report :

Increasing the size using Python Script :

Decrease the size by python script: