Graph log: mercurial and git

Lately, the version control system of my choice for personal small projects is mercurial. I find it easy to use and to install (I’ve never encountered a system without python) so it is always my first choice. However, a friend of mine asked me to go through his project’s code that was hosted on a git repo. I must say that before this I had never used git, I was reluctant and git’s weird options and arguments kept me away initially.

Enough with the introduction. One command used extensively by me (while using mercurial) was:
hg glog

which has the following output (copied shamelessly from http://mercurial.selenic.com/wiki/TutorialMerge)):

$ hg glog
@ changeset: 4:d2ecac0134d8
|\ tag: tip
| | parent: 2:c3844fde99f0
| | parent: 3:86794f718fb1
| | user: mpm@selenic.com
| | date: Tue May 06 23:44:19 2008 +0200
| | summary: Merged changes from my-hello-new-output
| |
| o changeset: 3:86794f718fb1
| | parent: 1:82e55d328c8c
| | user: mpm@selenic.com
| | date: Mon May 05 01:20:46 2008 +0200
| | summary: Express great joy at existence of Mercurial
| |
o | changeset: 2:c3844fde99f0
|/ user: mpm@selenic.com
| date: Tue May 06 20:10:35 2008 +0200
| summary: Add description of hello.c
|
o changeset: 1:82e55d328c8c
| user: mpm@selenic.com
| date: Fri Aug 26 01:21:28 2005 -0700
| summary: Create a makefile
|
o changeset: 0:0a04b987be5a
user: mpm@selenic.com
date: Fri Aug 26 01:20:50 2005 -0700
summary: Create a standard "hello, world" program


The above ascii graphlog is way too useful. Especially if you are a DVCS newbie! As you understand I needed such a command for git. Luckily enough, a quick glance at git-log manual’s page revealed the following:

git log --graph

which renders the desired output (django-cms2’s repository):


* commit 0e2d41477ceeb4482b9743b61792d523034db586
| Author: Peter-Paul van Gemerden
| Date: Fri Nov 20 19:27:04 2009 +0000
|
| l10n: First version of the Dutch translation.
|
| This is a nearly complete and fairly natural Dutch translation. It's pretty consistent and there's very little 'forced' translations. I have tested it, but not thoroughly.
|
| Transmitted-via: Transifex (www.transifex.net)
|
* commit f91fc2cbbd2001d9ebbe5ac1e4df70567295294a
|\ Merge: c0f5c6e 51b8d27
| | Author: Patrick Lauber
| | Date: Fri Nov 20 09:17:34 2009 +0100
| |
| | Merge branch '2.0.X'
| |
| * commit 51b8d272f6202574062739d63d8626d184a9a4d3
| | Author: dstufft
| | Date: Fri Nov 20 07:41:48 2009 +0800
| |
| | fixed issue where pages 3+ levels deep wasn't stripping the home page slug
| |
| * commit 2a58eae233d82358b847ff99a1daf6bc4a5b7e8c
| | Author: stefanfoulis
| | Date: Tue Nov 3 22:44:18 2009 +0800
| |
| | added buildout stuff to gitignore
output snipped here


It would be intresting to find/match the most used commands/workflows while using mercurial and git and compare them. Maybe a feature post although I’m sure that if I google it a bit around, surely something like this will come up.

 

Emacs uniquify package – Making buffer names unique

Tired of opening files with the same filename in Emacs and having that horrible <2>, <3>, <n> appended to the end of the buffer name ? I was, but luckily a friend of mine pointed me to the uniquify library that ships with Emacs.

Just put these two lines in your .emacs file and enjoy:

(require 'uniquify)
(setq uniquify-buffer-name-style 'forward)


This is extremely useful when editing django projects where each app usually has a models.py and a views.py. With this, you avoid confusion! (and save some time).

ENJOY!

 

Eve Online’s database schema graph

Too much free time on my hands and well, some good guy at the CCP forums had converted CCP’s latest Apocrypha (Eve version tag) MSSQL dump in PostgreSQL’s native format. What the heck! Of course, I imported the monster instantly:

[abrown@bifteki]~ $ createdb -U pgsql -E UTF-8 eve
[abrown@bifteki]~ $ time psql -q -U pgsql eve < ~/Downloads/apo15-pgsql-v1.sql
real	1m19.870s
user	0m1.078s
sys	0m0.673s

Running a ‘python manage.py inspectdb’ didn’t help at all. The resulting code was huge (over 800 lines of machine generated python. hell!). So, even If I was acquainted with the various names of the tables/columns due to my exposure to Eve Online (will blog about that in the future), I thought that it would be a nice idea to learn to produce database schema graphs. Googling around, I found out the postgresql_autodoc tool. Hapilly, this was already present in the FreeBSD ports tree:

[abrown@bifteki]ports $ make search name=autodoc
Port:	postgresql_autodoc-1.40
Path:	/usr/ports/databases/postgresql_autodoc
Info:	Automatic documentation generator for postgresql databases
Maint:	olgeni@FreeBSD.org
B-deps:	gettext-0.17_1 gmake-3.81_3 libiconv-1.13.1 p5-DBD-Pg-2.15.1 p5-DBI-1.60.9 p5-HTML-Template-2.9_1 p5-Storable-2.21 p5-Term-ReadKey-2.30 p5-version-0.78 perl-5.8.9_3 postgresql-client-8.2.13
R-deps:	perl-5.8.9_3
WWW:	http://www.rbt.ca/autodoc/index.html

Using this tool was pretty straightforward as well:

[abrown@bifteki]~ $  postgresql_autodoc -d eve -f eve.out -u pgsql
Producing eve.out.dia from /usr/local/share/postgresql_autodoc/dia.tmpl
Producing eve.out.dot from /usr/local/share/postgresql_autodoc/dot.tmpl
Producing eve.out.html from /usr/local/share/postgresql_autodoc/html.tmpl
Producing eve.out.neato from /usr/local/share/postgresql_autodoc/neato.tmpl
Producing eve.out.xml from /usr/local/share/postgresql_autodoc/xml.tmpl
Producing eve.out.zigzag.dia from /usr/local/share/postgresql_autodoc/zigzag.dia.tmpl
[ltsampros@bifteki]~ $ time dot -Tpng -o eve.out.png eve.out.dot
real	0m13.943s
user	0m11.637s
sys	0m0.203s

And that was it. Running graphviz’s dot using the dot file produced by postgresql_autodoc did the trick instantly. The resulting png file is a little bit huge (>3MB) but well it’s a good reference. You can find the image here.

 

Django ORM: effective query use

Hello everyone,

Eventually I managed to take off my cowboy hat and organize my thoughts/findings while using the Django ORM library. My needs, db-wise, were pretty simple:

a) update several rows in a one shot. (i.e. one simple UPDATE query)

b) access columns from referenced tables (using foreign keys) in an efficient manner (use of INNER JOIN)

To demonstrate how to do things, I’ll use the following sample (and simple) models:

class Staff(models.Model):
    id = models.IntegerField(primary_key=True)
    name = models.CharField(max_length=-1)
    department = models.ForeignKey(Department,
                                   db_column='department')

class Department(models.Model):
    id = models.IntegerField(primary_key=True)
    name = models.CharField(max_length=-1)
    money = models.IntegerField()

For example, we have 4 departments in our Department table and each department has a money attribute that holds the amount of money spent by each one. At the start of the fiscal year however, all these values need to be zeroed. Wanting to update the money field with the value 0 for ALL is pretty straightfoward as you can see below. The second example shows such an update with a filter i.e. a WHERE clause (yes, these departments owes money from last year):

In [7]: s = Department.objects.all()

In [8]: s.update(money=0)
Out[8]: 4

In [9]: Department.objects.filter(name__contains="3").update(money=1000)
Out[9]: 2

The above pieces produce and execute these (as observed in the PostgreSQL query logging) queries as expected:

LOG:  BEGIN; SET TRANSACTION ISOLATION LEVEL READ COMMITTED
LOG:  UPDATE "department" SET "money" = 0
LOG:  COMMIT

LOG:  BEGIN; SET TRANSACTION ISOLATION LEVEL READ COMMITTED
LOG:  SELECT "department"."id", "department"."name", "department"."money" FROM "department" WHERE "department"."name"::text LIKE E'%3%'  LIMIT 21
LOG:  UPDATE "department" SET "money" = 1000 WHERE "department"."name"::text LIKE E'%3%'
LOG:  COMMIT

This was the first and simple case. The second case proved a little bit more tricky. At first, I want to print for each Staff row it’s department name. This is pretty simple to achieve:

In [2]: s = Staff.objects.all()

In [3]: for i in s:
   ...:     print i.department
   ...:
   ...:
1 test 1 1
2 test 2 2
3 test 3 3

The QuerySet command 3 (QuerySets are lazy) in the above output produced the following 4 queries (for simplicity, I omit various statements that the ORM executes like setting the timezone when the connection is opened etc etc):

LOG:  SELECT "staff"."id", "staff"."name", "staff"."department" FROM "staff"
LOG:  SELECT "department"."id", "department"."name", "department"."money" FROM "department" WHERE "department"."id" = 1
LOG:  SELECT "department"."id", "department"."name", "department"."money" FROM "department" WHERE "department"."id" = 2
LOG:  SELECT "department"."id", "department"."name", "department"."money" FROM "department" WHERE "department"."id" = 3

As you see, a perfect normal SELECT query is produced that fetches from the database the desired row. However, accessing the name of the department, did fire a new queryin in order to accesse the department table and get the relevant data.

As expected, the departments to which our Staff belong is printed. However that’s 4 different queries. This is pretty ineffective, if we have tens of objects to process (which I think is a usual case). Worry not. QuerySet objects are a good friend of yours, as they have available the select_related() method.

Using the select_related() method foreign key relationships are followed automatically and as a result only one (bigger) query is executed:

In [3]: s = Staff.objects.select_related().all()
In [4]: for i in s:
   ...:     print i.department
   ...:
1 test 1 1
2 test 2 2
3 test 3 3

LOG:  statement: SELECT "staff"."id", "staff"."name", "staff"."department", "department"."id", "department"."name", "department"."money" FROM "staff" INNER JOIN "department" ON ("staff"."department" = "department"."id") LIMIT 21

As you see, the use of select_related() brought all available data following the foreign key using a perfectly valid INNER JOIN query. I think that this is much more effective than the previous example. Simple, huh?

A small note: in case you have a model with more than one Foreign Keys, it might be a good idea to set explicitly which keys you want to be followed, like this:

In [4]: s = Staff.objects.select_related("department").all()

If no arguments are given to select_related() then django follows all the available foreign keys defined in your model. This can lead to pretty complex queries which are not worth the cost if you are not intrested in all the returned values.

I hope I didn’t get you bored
Next week, I’ll try to set examples of using many to many relationships.

Have fun!

 

Most useful Emacs Command #1

Have you ever received a file from someone that always leaves trailing spaces on random lines ?

Worry not. delete-trailing-whitespace is your friend. Just open up your violated file and press :
M-x delete-trailing-whitespace RET

The numbering (the one in the heading) in this series of articles does not represent the order of importance. Just a numbering scheme.
I’m sure I can recollect tens of useful commands, but this one is special, especially in cases when more than one person touches the same file

 

Mozilla’s Weave and cryptography.

Recently I had delved into the source code of Mozilla’s Weave (the firefox extension part) to check how things are done. I did that because in the recent years, I never found a way to sync different parts of my digital life in a painless way and as a result I’m always in the look for new tools/products that might ease my pain (explaining why nothing works for me, will be analyzed in a different post).

So, during my adventures with the Weave extension, I stumbled upon the “crypto” directory:

[abrown@bifteki]weave $ pwd
/usr/home/abrown/clones/weave
[abrown@bifteki]weave $ ls -1
Makefile
README
crypto/
source/
tests/
tools/


Unfortunately, I didn’t even try building the component due to lack of time. But before bringing my little exploration to a halt, I tried to find some more information about how their cryptography module is utilized by the Weave extension but I didn’t had any luck. However, after a month or so, a friend of mine passed me this wonderful link:
How does Weave use Cryptography?

Really a very nice article explaining Weave’s crypto scheme. Thanks!

 

First post

Hello world!

After many many years I finally decided to start my own blog! This blog will serve my need for communication with the world aka my need to complain and whine about all kind of things that you probably don’t care about!