|Terminals / Shells • Network • X Window System • Portage • System • Filesystems • Kernel • Other|
This "speed up" is intended for those that have an "older" gentoo system, or those that "emerge sync" far too often. "Far too often" being more than 2 times a week.
What To Do
To speed it up slightly I suggest you put /usr/portage on its own partition. Doing so will allow for faster read times when accessing emerge.
Note that reiserfs is a lot faster than ext3 for this but slows down with use due to fragmentation. ext3 has lots of options with which you can make it much faster than it is by default. ext2 is very fast if you are happy with a simple non journalling fs. Since you wont lose difficult to recover data and only need to run emerge --sync if it gets a glitch this may be a good option.
Reiser4 is very efficient for portage if you don't mind a "still not quite in the mainstream kernel" fs. See "unsupported software" forum for more info.
Also *note* this partition should be at least 5gb in size to allow for /usr/portage/distfiles. Or you can change your DISTDIR in /etc/make.conf and place them wherever you want. If you change your DISTDIR to use this partition only for portage, you can use a smaller block-size, because the files in the portage tree normally are much smaller than 4k. I suggest a block-size of 2k. This decreases the size of the portage-tree to about two-thirds of the original size for me.
A note...I tried doing this while doing the initial install of Gentoo. Unfortunately, it resulted in the inability to "emerge sync", so I ended up having to do the install (up to the last parts, 9. Installing Necessary System Tools) just mounting a partition to /usr and not mounting /usr/portage separately. However, I created a separate partition to eventually mount to /usr/portage. I mounted /usr/portage at step 9. For some reason, it worked for me then. However, after you mount it at this stage, you'll have to redo "emerge sync" to avoid getting errors while emerging -- dh003i2 from Gentoo Forums.
Try extracting a portage snapshot like this
tar xvjf portage-20051029.tar.bz2 -C /usr
while having /usr/portage mounted correctly, then doing emerge sync to have the most up-to-date tree and the metadata taken care of. -- Tuxp3
Trouble with full disk that isn't actually full
If you are making a small partition for portage, like 512MB or even a gig, and you have trouble copying the files or extracting the bz2 remake your filesystem making sure there are enough inodes. I made the partition and used 2k block sizes so I set the bytes per inode with -i 2048. This was for an ext3 filesystem I don't know how any other mkfs handle it.
If you are using ReiserFS, don't use the "notail" option.
What It Helps With
This will help when running emerge search or when calculating dependencies for emerge -p something.
NOTE: Also consider using --ask (-a) instead of --pretend (-p) which does the same thing, but gives the opportunity to emerge right away instead of having to run 'emerge' again. One useful combination is emerge -tav <something> which shows the dependency tree, asking for permission and showing verbose list of USE flags.
Also: it has been shown that after a long time and many syncs this will actually make for a smaller space on disk. ~200MB vs. ~600mb for the ebuilds due to less fragmentation of the file system
Why It Works
When you emerge sync you delete and add many new files, over time these files get scattered about your hard drive. when emerge does anything it reads ALL of these files, so if they are closer together you get faster read times from your hard drive.
Another hint on speeding up searches
Look here: TIP Speed up searches in portage
This trick does several things
- Makes portage simple to back up
- Makes portage do less disk thrashing
- Makes portage in such a way it can be copied and mounted in a RAMfs ( ie: /dev/shm ) for those high-load sync servers
- Makes directory searches in portage faster by a factor of 10.
Relocate non-syncable data
In my opinion, its bad practise to keep distfiles and packages directly in the portage directory. For this, I suggest the relocation of packages and distfiles to their own directorys, ie:
and then symlink them from portage
mv /usr/portage/distfiles /mnt/seagate/distfiles ln -s /mnt/seagate/distfiles /usr/portage/distfiles mv /usr/portage/packages /mnt/seagate/packages ln -s /mnt/seagate/packages /usr/portage/packages
this will have the added benefit that any indexing app ( ie: locate/find/du/filelight ) won't try following the symlinks and waste time trawling useless data.
The alternate method of separating those directiories is setting proper variables in make.conf. In our example we could do this by:
echo 'PKGDIR="/mnt/seagate/packages"' >> /etc/make.conf echo 'DISTDIR="/mnt/seagate/distfiles"' >> /etc/make.conf
(more of those variables are described in man make.conf)
Make A Sparse File to create portage in
A sparse file is a handy trick which allows creation of a file which has a maximum potential capacity, but an initial small file size, which will only grow as the file is edited. So feel free to create it as big as you want.
dd if=/dev/zero of=/opt/portagefile bs=1k seek=1024k count=1
this will create a 1GB file which is how much space we want to permit portage to consume. You can read more about portage size later in section #Mount our File. And you can grow it with the instructions in section #Howto Resize your portagefile.
Format the Sparse File
Heres where most of you are going to go "eh what?", this is a trick i've been using for ages as for me its much more convenient than TAR files, albeit a little more space inefficient than tar, you can pull one file out of a gig very quickly. (ie: as fast as if it wasnt TAR'ed in the first place )
For my example I use reiserfs
mkfs.reiserfs -f /mnt/seagate/portageFile
and say yes to the prompt.
As an alternative, you can use an ext2 fs with 1Kb blocks, 200000 inodes, no space reserved for the superuser and hashed b-trees:
(Note: it would be better to use -i 2048 instead of -N 200000; this would allow mke2fs to calculate the correct number of inodes while taking into account the size of the loop disk and the amount of space reserved for the directory structure. see man mke2fs).
mke2fs -b 1024 -N 200000 -m 0 -O "dir_index" -F /mnt/seagate/portageFile #this is to stop it from complaining about mounting an unchecked fs tune2fs -c 0 -i 0 /mnt/seagate/portageFile
You now have a formatted file system in a file!...
Now a Word of warning
Be careful as to what filesystems choose, for I know one thing, from experience, that if you store a ReiserFS formatted file on reiserfs formatted system, you have a potential death trap awaiting you. If for some reason, the parent reiserfs partition becomes corrupt, and you need to do a rebuild tree, the recovery tool will see data in both the reiserfs partition AND in the formatted file, and then procede to unpack the file onto the partition ( reiserfsck is very smart it seems ). This will leave you with a rather trashed filesystem, which you will have to weed apart by hand.
Mount our File
Ok, we have to mount our new file somewhere to copy our existing portage into.
mkdir /mnt/portage mount /mnt/seagate/portageFile /mnt/portage -o loop,noatime,notail,nodev
If you use reiser4, then don't use notail:
mount /mnt/seagate/portageFile /mnt/portage -o loop,noatime,nodev
If you use ext2/ext3, then don't use notail:
mount /mnt/seagate/portageFile /mnt/portage -o loop,noatime,nodev
Importantly noting the loop option, which is the little bit of magic which makes it work. You also want to make sure you use noatime and notail to get the best speed out of it.
Here you can see a recent copied portage tree over reiserfs partitions with and without notail:
/usr/portagefs.notail 600M 565M 36M 95% /usr/portage.notail /usr/portagefs 600M 224M 377M 38% /usr/portage
Using notail each block will contain fragments from only one file. With tail, reiserfs will pack file fragments into the unused space of the blocks, reducing the amount of real disk space used. However, you leave no or little room for files to grow causing your file system to more easily fragment. Reiserfs with tail is good if you have many small files that don't change in size. Only the first holds for the portage tree. So you should only skip the notail option if you really need free space on /.
If you don't like to have your portage tree in /mnt, you could also mount it in some other empty directory (i.e. /usr/portage/tree)
Here another sparse file with an ext2 filesystem and a fresh portage tree copied into it:
365M -rw-r--r-- 1 root root 1.1G 2008-08-05 20:21 portagefile
It's easy to see that the 1024MB sparse file in fact occupies 365MB on disk.
Copy Portage into our File/Filesystem
You're going to want to copy the entierty of your portage tree into this file, as this is going to be our new portage, but you must make sure to copy everything -as-is- or the timestamps will go wonky and on your next sync you'll be downloading the entirety of portage all over again.
cp -a /usr/portage/* /mnt/portage/
this should result in a perfect clone of portage in your new home, portageFile
If you intend to move only the portage tree, you will end up with the following two directories in /usr/portage: 'distfiles' (which could be moved with DISTDIR variable in make.conf) and, if you have one, 'packages' (PKGDIR variable).
Rename old portage just in case
just in case our new portage goes hellishly bad, we should leave normal portage in a place where we can bring it back.
mv /usr/portage /usr/portage.old
Set up automatic mounting of portage
we should first unmount our custom portage
and then remake normal portage seeing we moved it
then add the following line to the very end of /etc/fstab
/mnt/seagate/portageFile /usr/portage auto loop,notail,noatime,nodev 0 0
and then test it by mounting it
if all went well, there should now be your entire portage tree stored in /usr/portage, just like new
Update profile symlink
/etc/make.profile is a symlink to a dir in $PORTDIR/profiles, so if you change $PORTDIR you have to update that symlink to point to the new location. Should be a simple 
ln -nsf /mnt/portage/profiles/<your-profile> /etc/make.profile
You can determine your profile with
ls -ld /etc/make.profile
(make sure there is no trailing slash)
Speed test it
tests to run in /usr/portage and /usr/portage.old
time du -sh . time find . > /dev/null
you will quickly note, especially if you've had gentoo installed for a while ( a year or so ) that the new portage in /usr/portage is _SO_ much faster than /usr/portage.old its not funny.
I have performed these easy speed tests using various FS with various block sizes. Results are in table:
|FS||block size||time du -sh||time find .||size|
Note: shorter time is better.
New Tricks we can do
You can now, at any time see, without much delay, exactly how much portage takes up by
and you can see how efficient it is by comparing that with
du -sh /mnt/seagate/portageFile
on my machine, the actual portageFile is 199M, and the files contained take up 176M which is approximately 88% space efficiency.
emerge --sync and if that works, you're away laughing.
Also note, if you have any nodes downstream of you via GIGABIT lan and they use the same trick, I would expect to see much faster syncs.
Howto Resize your portageFile
So, it may come so far one day, that you start getting "disk full" messages... Then it's time to resize your portageFile! I ran these commands on a mounted portageFile... So you should also be able to "hot"-resize :) You should check to which /dev/loop# device your portageFile is linked. Normally this should be /dev/loop0. You can check it with
# losetup /dev/loop0 /dev/loop0: :1082010 (portageFile)
Following lines will hot resize to 800mb. (note: of=/location/to/portageFile)
# dd if=/dev/null of=/mnt/seagate/portageFile bs=1M count=0 seek=800 0+0 records in 0+0 records out # resize_reiserfs -f /dev/loop0 resize_reiserfs 3.6.19 (2003 www.namesys.com) resize_reiserfs: On-line resizing finished successfully.
For the Really Adventurous
Possibly placing the portageFile in a RAMFS ( ie: /dev/shm ) would make it even faster still, and if you want some real speed, you could do the same with /var/db/pkg and /var/cache/edb ( faster dependency checks etc ) , but doing that is bound to end in tears ;)
Apparently placing the /var/ files in a sparse file will actually slow things down:
15:53:29 [/var/cache/edb.old] # time find > /dev/null real 0m0.106s user 0m0.033s sys 0m0.062s 15:53:13 [/var/cache/edb] # time find > /dev/null real 0m0.152s user 0m0.040s sys 0m0.067s
Same goes for /var/db/pkg. Seems like the structure of these dirs is different from /usr/portage making them react faster as common ext3 directories.
Created by NickStallman.net, Luxury Homes Australia
Real estate agents should list their apartments, townhouses and units in Australia.