TIP_speed_up_portage_with_sqlite

Introduction

this is simliar to TIP speed up portage with cdb ... and assumes you have already updated portage to stable version ~2.1

The advantages of using the sqlite module are:

1. the code of the module is maintained by gentoo developers
2. it is fast (metadata regen with cache.metadata_overlay.database is much faster)
3. it is not that hackish like the cdb module

The disadvantages:

1. it might not be as fast (according to the module's author: "it's not any faster than the default cache. a little slower actually. maybe depends on your use case"
   1. with fast storage backend (e.g. raid) it doesn't matter whether you have a lot of small cache files or one big (sqlite database) file
   2. backend metadata_overlay is about three times faster (on fast storage) both in syncing and depscanning
2. you can't disable metadata-transfer in FEATURES which is possible with metadata_overlay module, saving quite lot of time when doing emerge --sync

Installation

# emerge -av pysqlite
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild  N    ] dev-db/sqlite-3.2.1-r3  USE="-doc -nothreadsafe" 1,320 kB
[ebuild  N    ] dev-python/pysqlite-2.0.5  USE="-doc" 56 kB

after that modify the /etc/portage/modules file in the way shown below:

portdbapi.auxdbmodule = cache.sqlite.database

also, with >=sys-apps/portage-2.1.5_rc6 it is necessary to add "metadata-transfer" to FEATURES:

FEATURES="metadata-transfer"

to save space (around 100 megs) you can safely delete the old dependency cache

rm -rf /var/cache/edb/dep/*

finally regenerate your cache by invoking

emerge --metadata

Extra Tricks

Due to the inherent benefits of SQL backing, a rather quick reverse-dependency lookup can be performed by

sqlite3 /var/cache/edb/dep/usr/portage.sqlite 
SELECT portage_package_key FROM portage_packages WHERE DEPEND GLOB "*PKGNAMEPARTHERE*" ;

I have written a ruby script which utilizes this ability of SQlite to find RDEPENDS and DEPENDS very quickly to rapidly compute the reverse-dependencies of a named package, much faster than "qdepends -Q" .

#!/usr/bin/env ruby
#
#  Fast RevDep Lookup for portage/sqlite
#   V1.0
#
# -- Kent Fredric

require 'sqlite3'

# DEPEND tokenizer
# Breaks each seperate atom down 
def splitX( i )
	arry=[]
	items=0
	tokes=i.split(" ")
	tokes.length.times{ |n| 
		thistoke=tokes[n]
		if thistoke.match( /^\|\|/) or thistoke.match( /\?$/ )
			# Glob to the closing bracket 
			i=n+1 #jump to the ( following blah? 
			depth=1
			while( depth > 0 )
				i=i+1
				if tokes[i].match(/\(/)
					depth=depth+1
				end
				if tokes[i].match(/\)/)
					depth=depth-1
				end
			end
			output=tokes[n..i].join(" ")
			tokes[n]=output
			((n+1)..i).each{ |x|
				tokes[x]=""
			}
		end
	}
  	tokes.delete_if{ |x| x==""}
	return tokes
end
pkg=ARGV[0]

db=SQLite3::Database.new( "/var/cache/edb/dep/usr/portage.sqlite" )
db.execute( "select portage_package_key,DEPEND,RDEPEND 
	     FROM portage_packages 
	     WHERE DEPEND GLOB '*#{pkg}*' 
		OR RDEPEND GLOB '*#{pkg}*'") do |row|
	pname=row[0]
	print pname
	print "\n"
	deps={}
	deps[:RDEPEND]=[]
	deps[:DEPEND]=[]
	splitX(row[1]).each{ |x|
		if x.match(pkg)
			deps[:DEPEND] << x
		end
	}
	splitX(row[2]).each{ |x|
		if x.match(pkg)
			deps[:RDEPEND] << x
		end
	}
	if deps[:DEPEND].length > 0 
		print "DEPEND = "
		print deps[:DEPEND].join(" , ")
		print "\n"
	end
	if deps[:RDEPEND].length > 0 
		print "RDEPEND = "
		print deps[:RDEPEND].join(" , ")
		print "\n"
	end
	puts "\n"
end

db.close

this can then be executed like so

ruby /path/to/this/ruby/script.rb package-part

ie:

ruby /tmp/rdep.rb xmms

Note: it uses "globbing" instead of regexp, so you type any part of a package and it will spit out everything that matches. This is why I've added the functionality to show which atom/rule references the key you looked for.

Sample output with "xmms"

kde-base/kopete-3.5.4
DEPEND = xmms? ( media-sound/xmms )
RDEPEND = xmms? ( media-sound/xmms )

media-libs/faad2-2.0-r3
DEPEND = xmms? ( >=media-sound/xmms-1.2.7 media-libs/id3lib )
RDEPEND = xmms? ( >=media-sound/xmms-1.2.7 media-libs/id3lib )

media-libs/faad2-2.0-r13
DEPEND = xmms? ( >=media-sound/xmms-1.2.7 media-libs/id3lib )
RDEPEND = xmms? ( >=media-sound/xmms-1.2.7 media-libs/id3lib )

media-libs/faad2-2.0-r7
DEPEND = xmms? ( >=media-sound/xmms-1.2.7 media-libs/id3lib )
RDEPEND = xmms? ( >=media-sound/xmms-1.2.7 media-libs/id3lib )

media-libs/faad2-2.0-r12
DEPEND = xmms? ( >=media-sound/xmms-1.2.7 media-libs/id3lib )
RDEPEND = xmms? ( >=media-sound/xmms-1.2.7 media-libs/id3lib )

media-libs/faad2-2.0-r11
DEPEND = xmms? ( >=media-sound/xmms-1.2.7 media-libs/id3lib )
RDEPEND = xmms? ( >=media-sound/xmms-1.2.7 media-libs/id3lib )

net-irc/bitchx-1.1-r2
DEPEND = !arm? ( xmms? ( media-sound/xmms ) esd? ( >=media-sound/esound-0.2.5 >=media-libs/audiofile-0.1.5 ) !amd64? ( gtk? ( =x11-libs/gtk+-1.2* >=media-libs/imlib-1.9.10-r1 ) ) gnome? ( >=gnome-base/gnome-libs-1.4.1.2-r1 ) )
RDEPEND = !arm? ( xmms? ( media-sound/xmms ) esd? ( >=media-sound/esound-0.2.5 >=media-libs/audiofile-0.1.5 ) !amd64? ( gtk? ( =x11-libs/gtk+-1.2* >=media-libs/imlib-1.9.10-r1 ) ) gnome? ( >=gnome-base/gnome-libs-1.4.1.2-r1 ) )

net-irc/bitchx-1.1-r3
DEPEND = xmms? ( media-sound/xmms )
RDEPEND = xmms? ( media-sound/xmms )

net-irc/bitchx-1.1-r1
DEPEND = !arm? ( xmms? ( media-sound/xmms ) esd? ( >=media-sound/esound-0.2.5 >=media-libs/audiofile-0.1.5 ) gtk? ( =x11-libs/gtk+-1.2* >=media-libs/imlib-1.9.10-r1 ) gnome? ( >=gnome-base/gnome-libs-1.4.1.2-r1 ) )
RDEPEND = !arm? ( xmms? ( media-sound/xmms ) esd? ( >=media-sound/esound-0.2.5 >=media-libs/audiofile-0.1.5 ) gtk? ( =x11-libs/gtk+-1.2* >=media-libs/imlib-1.9.10-r1 ) gnome? ( >=gnome-base/gnome-libs-1.4.1.2-r1 ) )

( items selected to give a view of the point :) )

Here's a simple reverse dependency lookup script that returns a sorted version-striped list of dependencies.

#!/bin/env python2.5
# -*- coding: ascii -*-
#
#    Copyright 2007 Kevin Croft
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
"""
rdep.py 1.0

A simple reverse dependency lookup for portage.
Prints a sorted list of dependencies without versions.

Usage:
  %(program)s package

"""
import os
import re
import sys
import sqlite3

def main(args):
    rcode = 1
    if len(args) != 2:
        program = args[0]
        print __doc__ % locals()
        return rcode

    package = args[1]
    portage_db = '/var/cache/edb/dep/usr/portage.sqlite'
    if not os.path.isfile(portage_db):
        raise IOError(portage_db + ' does not exist. '
        + 'You may not be using sqlite to manage portage '
        + 'or perhaps the file has moved. '
        + 'See http://www.gentoo-wiki.info/TIP_speed_up_portage_with_sqlite')

    con = sqlite3.connect(portage_db)
    rows = con.execute("\
           SELECT portage_package_key        \
           FROM portage_packages             \
           WHERE depend GLOB '*%(package)s*' \
           OR rdepend glob '*%(package)s*'"
                       % {'package':package} )

    sub = re.compile('-\d.*').sub # regex to find the trailing version pattern
    deps = set([sub('',row[0]) for row in rows]) # remove dupes using a set
    for dep in sorted(deps):
        print dep

    rcode = 0
    return rcode

if __name__ == '__main__':
    sys.exit(main(sys.argv))

Sample output with "rrdtool"

/usr/bin/time -p rdep.py rrdtool
app-admin/recursos
dev-java/jrrd
dev-python/py-rrdtool
dev-ruby/ruby-rrd
games-util/uglygs
mail-filter/dcc
net-analyzer/FlowScan
net-analyzer/argus-clients
net-analyzer/bmon
net-analyzer/cacti
net-analyzer/jffnms
net-analyzer/munin
net-analyzer/ntop
net-analyzer/rrdcollect
net-analyzer/smokeping
net-mail/mailgraph
sys-apps/lm_sensors
sys-cluster/ganglia
www-apps/drraw
www-servers/lighttpd
real 0.11
user 0.06
sys 0.05