Gentoo Wiki ArchivesGentoo Wiki

TIP_Speeding_up_portage

This article is part of the Tips & Tricks series.
Terminals / Shells Network X Window System Portage System Filesystems Kernel Other

Contents

Introduction

This "speed up" is intended for those that have an "older" gentoo system, or those that "emerge sync" far too often. "Far too often" being more than 2 times a week.

What To Do

To speed it up slightly I suggest you put /usr/portage on its own partition. Doing so will allow for faster read times when accessing emerge.

Note that reiserfs is a lot faster than ext3 for this but slows down with use due to fragmentation. ext3 has lots of options with which you can make it much faster than it is by default. ext2 is very fast if you are happy with a simple non journalling fs. Since you wont lose difficult to recover data and only need to run emerge --sync if it gets a glitch this may be a good option.

Reiser4 is very efficient for portage if you don't mind a "still not quite in the mainstream kernel" fs. See "unsupported software" forum for more info.

Also *note* this partition should be at least 5gb in size to allow for /usr/portage/distfiles. Or you can change your DISTDIR in /etc/make.conf and place them wherever you want. If you change your DISTDIR to use this partition only for portage, you can use a smaller block-size, because the files in the portage tree normally are much smaller than 4k. I suggest a block-size of 2k. This decreases the size of the portage-tree to about two-thirds of the original size for me.

A note...I tried doing this while doing the initial install of Gentoo. Unfortunately, it resulted in the inability to "emerge sync", so I ended up having to do the install (up to the last parts, 9. Installing Necessary System Tools) just mounting a partition to /usr and not mounting /usr/portage separately. However, I created a separate partition to eventually mount to /usr/portage. I mounted /usr/portage at step 9. For some reason, it worked for me then. However, after you mount it at this stage, you'll have to redo "emerge sync" to avoid getting errors while emerging -- dh003i2 from Gentoo Forums.

Try extracting a portage snapshot like this

tar xvjf portage-20051029.tar.bz2 -C /usr

while having /usr/portage mounted correctly, then doing emerge sync to have the most up-to-date tree and the metadata taken care of. -- Tuxp3

Trouble with full disk that isn't actually full

If you are making a small partition for portage, like 512MB or even a gig, and you have trouble copying the files or extracting the bz2 remake your filesystem making sure there are enough inodes. I made the partition and used 2k block sizes so I set the bytes per inode with -i 2048. This was for an ext3 filesystem I don't know how any other mkfs handle it.

If you are using ReiserFS, don't use the "notail" option.

What It Helps With

This will help when running emerge search or when calculating dependencies for emerge -p something.

NOTE: Also consider using --ask (-a) instead of --pretend (-p) which does the same thing, but gives the opportunity to emerge right away instead of having to run 'emerge' again. One useful combination is emerge -tav <something> which shows the dependency tree, asking for permission and showing verbose list of USE flags.

Also: it has been shown that after a long time and many syncs this will actually make for a smaller space on disk. ~200MB vs. ~600mb for the ebuilds due to less fragmentation of the file system

Why It Works

When you emerge sync you delete and add many new files, over time these files get scattered about your hard drive. when emerge does anything it reads ALL of these files, so if they are closer together you get faster read times from your hard drive.

Another hint on speeding up searches

Look here: TIP Speed up searches in portage


MultiPurpose Trick

This trick does several things

Relocate non-syncable data

In my opinion, its bad practise to keep distfiles and packages directly in the portage directory. For this, I suggest the relocation of packages and distfiles to their own directorys, ie:

/mnt/seagate/packages

and

/mnt/seagate/distfiles

and then symlink them from portage

mv /usr/portage/distfiles /mnt/seagate/distfiles
ln -s /mnt/seagate/distfiles /usr/portage/distfiles
mv /usr/portage/packages /mnt/seagate/packages
ln -s /mnt/seagate/packages /usr/portage/packages

this will have the added benefit that any indexing app ( ie: locate/find/du/filelight ) won't try following the symlinks and waste time trawling useless data.

The alternate method of separating those directiories is setting proper variables in make.conf. In our example we could do this by:

echo 'PKGDIR="/mnt/seagate/packages"' >> /etc/make.conf
echo 'DISTDIR="/mnt/seagate/distfiles"' >> /etc/make.conf

(more of those variables are described in man make.conf)

Make A Sparse File to create portage in

A sparse file is a handy trick which allows creation of a file which has a maximum potential capacity, but an initial small file size, which will only grow as the file is edited. So feel free to create it as big as you want.

dd if=/dev/zero of=/opt/portagefile bs=1k seek=1024k count=1

this will create a 1GB file which is how much space we want to permit portage to consume. You can read more about portage size later in section #Mount our File. And you can grow it with the instructions in section #Howto Resize your portagefile.

Format the Sparse File

Heres where most of you are going to go "eh what?", this is a trick i've been using for ages as for me its much more convenient than TAR files, albeit a little more space inefficient than tar, you can pull one file out of a gig very quickly. (ie: as fast as if it wasnt TAR'ed in the first place )

For my example I use reiserfs

mkfs.reiserfs -f /mnt/seagate/portageFile 

and say yes to the prompt.

As an alternative, you can use an ext2 fs with 1Kb blocks, 200000 inodes, no space reserved for the superuser and hashed b-trees:

(Note: it would be better to use -i 2048 instead of -N 200000; this would allow mke2fs to calculate the correct number of inodes while taking into account the size of the loop disk and the amount of space reserved for the directory structure. see man mke2fs).

mke2fs -b 1024 -N 200000 -m 0 -O "dir_index" -F /mnt/seagate/portageFile

#this is to stop it from complaining about mounting an unchecked fs
tune2fs -c 0 -i 0 /mnt/seagate/portageFile  
 

You now have a formatted file system in a file!...

Now a Word of warning

Be careful as to what filesystems choose, for I know one thing, from experience, that if you store a ReiserFS formatted file on reiserfs formatted system, you have a potential death trap awaiting you. If for some reason, the parent reiserfs partition becomes corrupt, and you need to do a rebuild tree, the recovery tool will see data in both the reiserfs partition AND in the formatted file, and then procede to unpack the file onto the partition ( reiserfsck is very smart it seems ). This will leave you with a rather trashed filesystem, which you will have to weed apart by hand.

Mount our File

Ok, we have to mount our new file somewhere to copy our existing portage into.

mkdir /mnt/portage
mount /mnt/seagate/portageFile /mnt/portage -o loop,noatime,notail,nodev

If you use reiser4, then don't use notail:

mount /mnt/seagate/portageFile /mnt/portage -o loop,noatime,nodev

If you use ext2/ext3, then don't use notail:

mount /mnt/seagate/portageFile /mnt/portage -o loop,noatime,nodev

Importantly noting the loop option, which is the little bit of magic which makes it work. You also want to make sure you use noatime and notail to get the best speed out of it.

Here you can see a recent copied portage tree over reiserfs partitions with and without notail:

/usr/portagefs.notail   600M  565M    36M  95% /usr/portage.notail 
/usr/portagefs          600M  224M   377M  38% /usr/portage 

Using notail each block will contain fragments from only one file. With tail, reiserfs will pack file fragments into the unused space of the blocks, reducing the amount of real disk space used. However, you leave no or little room for files to grow causing your file system to more easily fragment. Reiserfs with tail is good if you have many small files that don't change in size. Only the first holds for the portage tree. So you should only skip the notail option if you really need free space on /.

If you don't like to have your portage tree in /mnt, you could also mount it in some other empty directory (i.e. /usr/portage/tree)

Here another sparse file with an ext2 filesystem and a fresh portage tree copied into it:

365M -rw-r--r--  1 root root    1.1G 2008-08-05 20:21 portagefile

It's easy to see that the 1024MB sparse file in fact occupies 365MB on disk.

Copy Portage into our File/Filesystem

You're going to want to copy the entierty of your portage tree into this file, as this is going to be our new portage, but you must make sure to copy everything -as-is- or the timestamps will go wonky and on your next sync you'll be downloading the entirety of portage all over again.

cp -a /usr/portage/* /mnt/portage/

this should result in a perfect clone of portage in your new home, portageFile

If you intend to move only the portage tree, you will end up with the following two directories in /usr/portage: 'distfiles' (which could be moved with DISTDIR variable in make.conf) and, if you have one, 'packages' (PKGDIR variable).

Rename old portage just in case

just in case our new portage goes hellishly bad, we should leave normal portage in a place where we can bring it back.

mv /usr/portage /usr/portage.old

Set up automatic mounting of portage

we should first unmount our custom portage

umount /mnt/portage

and then remake normal portage seeing we moved it

mkdir /usr/portage

then add the following line to the very end of /etc/fstab

/mnt/seagate/portageFile /usr/portage auto loop,notail,noatime,nodev   0  0

and then test it by mounting it

mount /usr/portage

if all went well, there should now be your entire portage tree stored in /usr/portage, just like new

Update profile symlink

/etc/make.profile is a symlink to a dir in $PORTDIR/profiles, so if you change $PORTDIR you have to update that symlink to point to the new location. Should be a simple [1]

ln -nsf /mnt/portage/profiles/<your-profile> /etc/make.profile
  

You can determine your profile with

ls -ld /etc/make.profile

(make sure there is no trailing slash)

Speed test it

tests to run in /usr/portage and /usr/portage.old

time du -sh .
time find . > /dev/null

you will quickly note, especially if you've had gentoo installed for a while ( a year or so ) that the new portage in /usr/portage is _SO_ much faster than /usr/portage.old its not funny.

I have performed these easy speed tests using various FS with various block sizes. Results are in table:

FS block size time du -sh time find . size
ext2 4096 0.333s 0.288s 542MB
ext2 2048 0.328s 0.292s 322MB
ext2 1024 0.329s 0.282s 227MB
ext3 4096 0.774s 0.372s 543MB
ext3 2048 0.430s 0.359s 323MB
ext3 1024 0.422s 0.363s 228MB
reiserfs 4096 0.379s 0.452s 469MB
reiserfs 2048 0.383s 0.454s 286MB
reiserfs 1024 0.390s 0.461s 208MB
reiserfs 512 0.411s 0.485s 179MB
reiser4 4096 0.718s 0.721s 191MB
jfs 4096 0.323s 0.394s 496MB
xfs 4096 0.382s 0.455s 480MB
xfs 2048 0.410s 0.495s 297MB
xfs 1024 0.412s 0.496s 219MB

Note: shorter time is better.

New Tricks we can do

You can now, at any time see, without much delay, exactly how much portage takes up by

df -h

and you can see how efficient it is by comparing that with

du -sh /mnt/seagate/portageFile

on my machine, the actual portageFile is 199M, and the files contained take up 176M which is approximately 88% space efficiency.

Acid Test

emerge --sync and if that works, you're away laughing.

Also note, if you have any nodes downstream of you via GIGABIT lan and they use the same trick, I would expect to see much faster syncs.

Howto Resize your portageFile

So, it may come so far one day, that you start getting "disk full" messages... Then it's time to resize your portageFile! I ran these commands on a mounted portageFile... So you should also be able to "hot"-resize :) You should check to which /dev/loop# device your portageFile is linked. Normally this should be /dev/loop0. You can check it with

# losetup /dev/loop0
/dev/loop0: [0901]:1082010 (portageFile)

Following lines will hot resize to 800mb. (note: of=/location/to/portageFile)

# dd if=/dev/null of=/mnt/seagate/portageFile bs=1M count=0 seek=800
0+0 records in
0+0 records out
# resize_reiserfs -f /dev/loop0
resize_reiserfs 3.6.19 (2003 www.namesys.com)


resize_reiserfs: On-line resizing finished successfully.

For the Really Adventurous

Possibly placing the portageFile in a RAMFS ( ie: /dev/shm ) would make it even faster still, and if you want some real speed, you could do the same with /var/db/pkg and /var/cache/edb ( faster dependency checks etc ) , but doing that is bound to end in tears ;)

Apparently placing the /var/ files in a sparse file will actually slow things down:

15:53:29 [/var/cache/edb.old] # time find > /dev/null

real    0m0.106s
user    0m0.033s
sys     0m0.062s

15:53:13 [/var/cache/edb] # time find > /dev/null

real    0m0.152s
user    0m0.040s
sys     0m0.067s

Same goes for /var/db/pkg. Seems like the structure of these dirs is different from /usr/portage making them react faster as common ext3 directories.

Retrieved from "http://www.gentoo-wiki.info/TIP_Speeding_up_portage"

Last modified: Sat, 09 Aug 2008 23:19:00 +1000 Hits: 50,897

Created by NickStallman.net, Luxury Homes Australia
Real estate agents should list their apartments, townhouses and units in Australia.