Great land of Africa here I come!

Going on vacation tomorrow, finally got a chance to do a 9 day safari trip in Kenya with some friends and then stop over at Switzerland for another 4 days on the way back to Bay area. Still packing at last minute but I’m so excited!

EmailDiggFacebookDeliciousStumbleUponTwitterTumblrGoogle GmailBlogger PostSina WeiboBox.netEvernoteFriendFeedGoogle BookmarksLiveJournalLinkedInPrintPrintFriendlyRedditSlashdotWordPressShare

Tags: ,

Saturday, August 13th, 2011 Uncategorized No Comments

Cars 2 is out this weekend

Be sure to check it out in theater, my friend, although it is not so much favored by movie critics, mostly due to not having as deep of emotional storyline as last year’s toy story 3,
I think it is still a solid, entertaining family film. And kids would like it for sure.

Also be sure to check out the teaser of our next year’s ‘Brave’.  My colleagues and I have spent years on the new software platform
and ‘Brave’ is the first one that uses it.

EmailDiggFacebookDeliciousStumbleUponTwitterTumblrGoogle GmailBlogger PostSina WeiboBox.netEvernoteFriendFeedGoogle BookmarksLiveJournalLinkedInPrintPrintFriendlyRedditSlashdotWordPressShare
Thursday, June 23rd, 2011 Uncategorized No Comments

Install Ruby 1.9.2 and Rails 3 on Ubuntu 11.04

It seems apt-get hasn’t updated with ruby 1.9.2 and rails 3, so apt-get install ruby/rails won’t get ruby 1.9.2 and rails 3 (at least not for me at the moment).
But I really want to explore the world of rails 3, especially I’ve heard a lot has been changed since rails 2, which was the version last time I touched it while I was at CMU.
I was going to build from source but luckily I found this blog post which saved me quite a bit of time and effort.
However, I did encounter some problems by following the steps from that post, so I’m posting my steps here hopefully this could be useful to some else.

first you need to install bunch of packages via apt-get if you haven’t already:

1
apt-get install curl git-core build-essential bison openssl libreadline5 libreadline5-dev zlib1g zlib1g-dev libssl-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libmysqlclient-dev mysql-client mysql-server autoconf

then you need install rvm, which is a great tool to manage your ruby version (I guess it’s like python_select in ruby world)

1
curl -s https://rvm.beginrescueend.com/install/rvm

add the following line to your .bashrc

1
[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"  # This loads RVM into a shell session.

source it so rvm is available in your shell session, I think the original post didn’t mention this step so if you follow it strictly like I did you will get an error like svmsudo command not found in further steps.

1
source ~/.rvm/scripts/rvm

then install ruby and rails

1
2
3
4
5
6
7
8
rvm install 1.9.2
rvm use 1.9.2 --default
rvm 1.9.2 --passenger
rvm 1.9.2
gem install passenger
apt-get install libcurl4-openssl-dev
rvmsudo passenger-install-nginx-module
gem install rails

you should be able to use rails now:

shiqi@ubuntu:~$ which ruby
/home/shiqi/.rvm/rubies/ruby-1.9.2-p180/bin/ruby
shiqi@ubuntu:~$ ruby -v
ruby 1.9.2p180 (2011-02-18 revision 30909) [x86_64-linux]
shiqi@ubuntu:~$ which rails
/home/shiqi/.rvm/gems/ruby-1.9.2-p180/bin/rails
shiqi@ubuntu:~$ rails -v
Rails 3.0.8

easy, huh?

kudos to the author of the following references:
1. http://stuffingabout.blogspot.com/2011/04/installing-rails-3-on-ubuntu-1104.html
2. http://thekindofme.wordpress.com/2010/10/24/rails-3-on-ubuntu-10-10-with-rvm-passenger-and-nginx/
3. https://rvm.beginrescueend.com/rvm/install/

EmailDiggFacebookDeliciousStumbleUponTwitterTumblrGoogle GmailBlogger PostSina WeiboBox.netEvernoteFriendFeedGoogle BookmarksLiveJournalLinkedInPrintPrintFriendlyRedditSlashdotWordPressShare

Tags: , , , ,

Friday, June 10th, 2011 linux, rails, ruby, ubuntu No Comments

My new toy arrived

Dell XPS 8300 desktop to replace my old (yet still robust) Lenovo T60p laptop (for the past 2 years I’ve been using it as a desktop).
the newbie comes with i5 processor and 6G RAM, and I just put ubuntu 11.04 and fedora 15 on it, plus the existing Windows 7, it is really hard to decide which linux I should boot with ;)
It will take me some time to install all the things I need, but nagios should be relatively easy, was trying to install that on my Macbook Pro with Macport, such a nightmare.
I spent 2 days trying to figure out the usage of gcc –arch flags and still have no clue how to fix that. decided to give up on that.
So far everything went pretty smooth, found this blog post to help with install rails 3 on ubuntu 11.04 but still having some minor troubles with rvm. Also the dual screen setting doesn’t look very easy to deal with in Ubuntu (my machine comes with only one HDMI plug and I had to use a Diamond BUV195 USB to DVI adapter to extend the screen). Fedora 15 meanwhile is giving me a hard time on the wireless adapter. I have to say under windows 7 those were handled much better (mostly because some of the drivers are only available for windows). All my wacky devices are recognized and installed, no question asked.
Looks I just found some gigs for the weekend to figure out, can’t wait! :P

EmailDiggFacebookDeliciousStumbleUponTwitterTumblrGoogle GmailBlogger PostSina WeiboBox.netEvernoteFriendFeedGoogle BookmarksLiveJournalLinkedInPrintPrintFriendlyRedditSlashdotWordPressShare

Tags: , , ,

Thursday, June 9th, 2011 life No Comments

Sqlalchemy: merge on duplicates, primary key, unique and python class

For a pythonista an amateur python code writer like me to work with sql, especially mysql there used to be very limited choices and I remember the days I used MySQLdb module with raw SQL statement string all over the place in my python code. My code looked so tedious that I had to quit my job at yahoo to stay away from it. OK that wasn’t the reason why I quit, but the code I wrote for that tool to query bugzilla database was pretty lame indeed. I even had a class dedicated to generate sql statements based on different parameters passed in, and as expected I spent shit load of time maintaining and refactoring that class. Alright, in all fairness I have to admit that I shouldn’t attribute my horrible experiences solely to MySQLdb, it is a fantastic base data access layer implementation between python and mysql, lots of amazing tools/modules are based on it, my unpleasantness was actually almost entirely due to my bad sense of design pattern and refactoring.

So anyways, as history moves on and human nature of life keeps getting better and better, people, great and smart people created SqlAlchemy. The great thing (maybe just for me) is that it comes with data abstraction layer ORM. and suddenly I can access my data in an elegant, object-oriented way. I like it so much that I can dedicate a whole blog just talk about how it helped me become a better person and thus changed my life completely, Not.

But like all other powerful things SqlAlchemy has its Achilles’ Heel…… well, I’m not going to talk about that here, LOL. What I want to talk about is just something I always feel inconvenient and really wish they could make it part of the SqlAlchmey library. So the here is the task I’m facing more often than not, in a nutshell: So I have two tables users and addresses, and users have a foreign key constrain to addresses, which means multiple users can share one address, but not vise versa. So the my db will look like this:

1
2
3
4
5
6
7
8
9
10
        user = Table('user', metadata,
            Column('id', Integer, primary_key = True),
            Column('name', String(9)),
            Column('addr_id', Integer, ForeignKey('addr.id'))
        )
 
        addr = Table('addr', metadata,
            Column('id', Integer, primary_key = True),
            Column('addr_info', String(10), unique=True)
        )

and of course my table classes will look like this:

1
2
3
4
5
6
7
8
9
10
11
12
class User(object):
    def __init__(self, name=None):
        self.name = name
 
    def __repr__(self):
        return ','.join([self.name,
                         self.addr_id])
class Addr(object):
    def __init__(self, addr_info=None):
        self.addr_info = addr_info
    def __repr__(self):
        return self.addr_info

and of course the mappers:

1
2
mapper(User, user, properties={'addr': relation(Addr)})
mapper(Addr, addr)

now I’m reading my data from a file, with each line describing a resident and an address, my code is supposed to read this file and put them into the database. The logic is fairly simple here, just parse the file, initialize my data class instance with data, then put them into the database.

However, the insertion process needs special attention here: if the address you parsed already exist in the db, you don’t want to insert another row in the address table to create a duplicate record, you really just want to update your foreign key and only insert your new resident here to avoid redundant insertion in the address table. The operation sounds very generic and I was so certain that SqlAlchemy would have something magical I can use to make this “update if exist” happen automatically. And Session.merge() at some point was so close to the pony in my dream until I found it only checks on primary key (see discussion here). So my first and brutal force solution was always do a query first to query out the address id of existing record (None if not) and then assign the id explicitly to the foreign key for resident. Then my first cut looks like this:

1
2
3
4
5
6
7
        user, addr = self.parser.parse_data(line)
        known_addr = self.session.query(Addr).filter(Addr.addr_info==addr.addr_info).all()
        if known_addr:
            addr = self.session.merge(known_addr[0])
        else:
            self.session.add(addr)
        user.addr = addr

While this is essentially what needs to be done on logical level (unfortunately there are no magical APIs to handle this situation from SqlAlchemy, due to performance reasons, more details here), it really makes the code long and hard to maintain. without putting this logic into some code block, almost every data insertion would require this kind of query/validation first. So what would be the best way to refactor this?

the first thing came up in my mind was to put this in my utility/helper class for database, the function could be shortened like this:

1
2
3
        user, addr = self.parser.parse_data(line)
        addr = db_helper.sync_with_db(addr)    #all query_check_merge biz goes here
        user.addr = addr

I’m not really in favor of this solution because I still need to remember to call the helper function from my utility/helper class after I parse my data. plus if I have more tables like address with slightly different logic I would ended up writing functions for each of them in my db_helper, and I have to remember to call all of them as well, it could get very ugly with a super long list of function calling.
Generally speaking, the duplication check is really the business of a table itself since different table has different column names, types and requirements built around them, so it feels natural to build this logic into the table class. Is there a way to embed this into my table class so the duplication check is performed when my instance is formed? There are some smart solutions I found online and my favorite one is to utilize the __new__ class method to check the duplication during class instantiation. Follow this recipe I could make my case work as simple as the following:

1
2
3
4
5
6
7
8
9
class Addr(object):
    def __init__(self, addr_info):
        self.addr_info =addr_info
 
mapper(Addr, addr)
Addr = unique_constructor(Session, 
            lambda addr_info:addr_info, 
            lambda query, name:query.filter(Addr.addr_info==addr_info)
)(Addr)

with declarative this could be simpler even more, as indicated on the recipe page above.

after that the call of

1
Addr('blah')

will grant me a instance representing either an existing row or a new record. No more clumsy query_first_and_then_add business.

I was trying to customized it to make it look simpler and only work the class I created, until I realized how silly that is, it’s like changing the super key to 1000 doors to make it a normal key to only open 1. I’m glad I stopped myself before I face another door and start to miss the super key.

but to keep the record of my stupidity and help me learn, here is my super hacky class, it passed my test of 3 line of data so I’m sure it is useful to someone out there, LOL:

1
2
3
4
5
6
7
8
9
10
11
class Addr(object):
    def __init__(self, addr_info):
        self.addr_info = addr_info
    def __repr__(self):
        return self.addr_info or " "
    def __new__(cls, addr=None):
        known_addr = sessionmaker()().query(cls).filter(Addr.addr_info==addr).all()
        if known_addr:
            return known_addr[0]
        else:
            return object.__new__(cls)
EmailDiggFacebookDeliciousStumbleUponTwitterTumblrGoogle GmailBlogger PostSina WeiboBox.netEvernoteFriendFeedGoogle BookmarksLiveJournalLinkedInPrintPrintFriendlyRedditSlashdotWordPressShare

Tags: , , ,

Saturday, June 4th, 2011 python, sql, tech No Comments

making hooq work with pyQt and Qt4.7

GUI automation test for desktop apps with Qt could be a PIA, not only because GUI automation is simply a PIA in general,
but there are just not that much free open source tools you can make use of for Qt. Or I should say at least for me it is not easy to find any.
Almost all tools I found online are heavy weight commercial software
that have this long and fancy feature set that I have to buy as a whole, out of which most I may never touch.

Luckily I found hooq, a free, open source tool to help recording qt events
and replay them (which is exactly what I need, for now). Like all other free lunches it does take me some effort to make it useful for me. And kudos to my genius coworker Leaf
who figured out how to build all the packages hooq requires to make this happen. It seems most people got stuck on this very first step.

But then I got stuck on another problem:
although hooq can be executed fine and it can launch my app correctly, it won’t record any of my actions.
All I got was requireHooqScriptVersion(2);
I spent a whole day digging into this and it seems to be a timing issue.
fundamentally hooq uses gdb to track all the qt events by inserting breaks on all QCoreApplication::exec
and sometimes (I’m not sure if this is a system thing) the timing of loading the shared libraries could be different
and that will cause the breaks failed to be set, thus the recording won’t take effect

after trying different combinations, I modify void GdbInjector::startProcess() in GdbInjecter.cpp into the following
and this seem to work for me:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
void GdbInjector::startProcess()
{
        QString argumentsString;
        Q_FOREACH(QString argument, m_applicationArguments)
        {
         	argument.replace('"', "\\\"");
                argumentsString.append(QString(" \"%1\"").arg(argument));
	}
 
        Q_ASSERT(m_gdb->state() == QProcess::Running);
	Q_ASSERT(m_gdb->isWritable());
        m_gdbStream.setDevice(m_gdb);
        m_gdbStream << "set breakpoint pending on" << endl;
        m_gdbStream << "set auto-solib-add off" << endl; // avoid loading symbols for shared libraries                                                                                                         
                                                         // that we do not need to interact with from GDB                                                                                                      
        m_gdbStream << "break _start" << endl; // C entry point - after main libraries have been loaded                                                                                                        
	m_gdbStream << "set args" << argumentsString << endl;
        m_gdbStream << "run" << endl; // run until we hit it, and therefore Qt shared libraries are loaded                                                                                                     
 
        m_gdbStream << "sharedlibrary libdl" << endl; // load the libdl library so that we can call __dlopen()                                                                                                 
	m_gdbStream << "sharedlibrary libc" << endl;  // load the libc library so that we can call __dlopen()                                                                                                  
        //m_gdbStream << QString("call dlopen(\"/usr/lib64/libQtCore.so.4\",%1)").arg(QString::number(RTLD_NOW)) << endl;                                                                                      
        // Newer systems                                                                                                                                                                                       
        m_gdbStream << QString("call dlopen(\"%1\", %2)").arg(libraryPath()).arg(QString::number(RTLD_NOW)) << endl; // load our library                                                                       
	// Older systems                                                                                                                                                                                       
        m_gdbStream << QString("call __dlopen(\"%1\", %2)").arg(libraryPath()).arg(QString::number(RTLD_NOW)) << endl; // load our library                                                                     
        m_gdbStream << "sharedlibrary libQtCore" << endl; // load QtCore for breaking on QCoreApplication::exec()                                                                                              
        m_gdbStream << "sharedlibrary libQtGui" << endl; // load QtCore for breaking on QCoreApplication::exec()                                                                                               
        m_gdbStream << "break QCoreApplication::exec" << endl; // now, we can set this breakpoint...                                                                                                           
        m_gdbStream << "continue" << endl;
 
        m_gdbStream << "sharedlibrary injectedHooq" << endl; // load the hooq injector library so that we can call startHooq()                                                                                 
 
        m_gdbStream << "call startHooq()" << endl; // install our plugin (which required QCoreApplication setup)                                                                                               
	m_gdbStream << "continue" << endl; // run the app                                                                                                                                                      
        m_gdbStream << "backtrace" << endl; // just in case (use with --spam)                                                                                                                                  
        m_gdbStream << "quit" << endl; // after the application has exited, quit gdb                                                                                                                           
}
EmailDiggFacebookDeliciousStumbleUponTwitterTumblrGoogle GmailBlogger PostSina WeiboBox.netEvernoteFriendFeedGoogle BookmarksLiveJournalLinkedInPrintPrintFriendlyRedditSlashdotWordPressShare

Tags: , , , ,

Thursday, June 2nd, 2011 python, qt, tech 3 Comments

Playing with wordpress plugins

I spent some time tonight playing with some of the wordpress plugins, one of them is ‘Facebook comments for wordpress’. Works great except I had a hard time to adjust the width of the comment box. I spent quite some time and still couldn’t figured out why the width is always set to 300px, firebug told me it is not set explicitly in any css file and there seems no good way to find how this number is calculated. well, my hacky solution for now is to add the following css definition to my  facebook-comments-widgets.css

1
2
.fb_ltr {width:100%}
.fb_iframe_widget {width:610px} /*this is according to my theme layout setup*/

would be nice to get to the bottom of it but it works fine for now.

Other plugins works great after installation.
I really like the charming new look WPtouch gave under iphone/itouch
and the many share buttons by AddtoAny

EmailDiggFacebookDeliciousStumbleUponTwitterTumblrGoogle GmailBlogger PostSina WeiboBox.netEvernoteFriendFeedGoogle BookmarksLiveJournalLinkedInPrintPrintFriendlyRedditSlashdotWordPressShare

Tags: , , , , , ,

Sunday, May 29th, 2011 css, life, tech No Comments

Finding the continuous sequence that has largest sum from a list of integers(or floats)

Long weekend could be boring, especially when you are just 3 weeks out of a major (relatively) surgery and can’t do serious outdoor activities. So I decided to dive into some of the basic algorithms and implement them in python. I used to just look at the pseudo code and thought “this is easy”. but for the past week I’ve been struggling to implement another problem (post in draft now) which really made me realize it could be much harder said than done. Admittedly some of those problems are really fundamental and most of the times people won’t bother to implement them because there are various libs to take advantage of (plus implementing your own version of same functionality from standard lib is really not a very good practice anyways). However, it is fun and really kills time. Besides, my biggest fear never is to be dumb but to be dumb without realizing it and the will the change it, struggling with those problems really help me to remind myself how bad of a coder I am and inspire me to improve it everyday.

So here is today’s problem:

Finding the continuous sequence that has largest sum from a list of integers(or floats)

The algorithm itself is pretty obvious: keep track of your sum, if it is below 0 then it won’t contribute to the max,

at that point keep what you have and then start a new iteration on the rest of the list.

My first implementation was to return the whole list, then I realized that the result will always be a subset of the original list

so just return the start and end index should be enough, it would save some space in this way.

Now it is arguable how to handle the situation when the max you can find is negative, which means you have nothing but negative numbers

in your list. I took a short cut here by returning an empty list in this case, my idea is that by returning an empty list I’m sending the message saying that you best choice in this situation is to not have anything from this list. An alternative could be scan the whole list once and initialize

the max_sum with the biggest element in the list. This won’t do harm to the O(n) time complexity.

Update: the wordpress coding syntax plugin reminds me that sum is a preserved keyword in python for sum function (shame on me that I even called that function in my code below for printing out without realizing it). Ahh, isn’t the highlighting nice? LOL

def largest_sum(self, list_of_int):
	max_sum = 0
	sum = 0
	i_max_start = 0
	i_max_end = -1
	current_start = 0		
	if list_of_int and type(list_of_int)==type([]):
		list_size = len(list_of_int)
		for index, item in enumerate(list_of_int):
			sum += item
			if sum > 0:
				if sum > max_sum:
					max_sum = sum
					i_max_end = index
					i_max_start = current_start				
			else:
				sum = 0
				if index < list_size-1:
					current_start = index +1
	return i_max_start, i_max_end				
 
def test_largest_sum(self):
	data = [[3,-7,5,-7,20,6,2,-6,4],
			[],
			[1,-2],
			[3,-7,1,-7,20,6,2,-1,4],
			[3,7,-1,-7,-20,6,2,-1,4],
			[3],
			[-7],
			[-5,10],
			[3,-2,4],
			'test',
			None]
	result = [(start, end) for (start, end) in map(self.largest_sum, data)]
	for index, d in enumerate(data):
		start, end = result[index]
		if end>=0:
			print sum(d[start:end+1]), d[start:end+1]
		else:
			print None, []

and here are the output:

28 [20, 6, 2]
None []
1 [1]
31 [20, 6, 2, -1, 4]
11 [6, 2, -1, 4]
3 [3]
None []
10 [10]
5 [3, -2, 4]
None []
None []

Something I thought about that could be interesting extension on this but yet have time to implement it or think it though:

  • 1. find all sequences with largest sum if there are more than one sequence with that sum
  • 2. find the longest/shortest one among the result from the result of 1 above (I feel that doesn’t mean don’t know if this necessarily means that I have to get the result from 1. Update: I probably still do. won’t know the largest sum if I don’t go thru them all)
  • 3. turn this question around: given the largest and smallest sum and the elements (unordered), construct the list (the answer of course won’t may not be unique)
  • 4. same as above, but construct the list so that the list has longest/shortest sequence with largest sum (first impression: separate the positive and negative value into two lists, sort them and then intersect them might be enough to achieve this… but again, ‘easier said than done’ LOL)
  • EmailDiggFacebookDeliciousStumbleUponTwitterTumblrGoogle GmailBlogger PostSina WeiboBox.netEvernoteFriendFeedGoogle BookmarksLiveJournalLinkedInPrintPrintFriendlyRedditSlashdotWordPressShare

    Tags: , ,

    Saturday, May 28th, 2011 algorithm, python, tech No Comments

    Find the largest possible number of people in circus tower

    A circus is designing a tower routine consisting of people standing atop one another’s shoulders. For practical and aesthetic reasons, each person must be both shorter and lighter than the person below him or her. Given the heights and weights of each person in the circus, write a method to compute the largest possible number of people in such a tower.

    EXAMPLE:
    Input:
            (ht, wt): (65, 100) (70, 150) (56, 90) (75, 190) (60, 95) (68, 110)
    Output: The longest tower is length 6 and includes from top to bottom: 
            (56, 90) (60,95) (65,100) (68,110) (70,150) (75,190)
    

    I really love one of the solutions I found online.
    It took several advantages of python language:

    1.lambda function
    2.bisect
    3.how python compare iterables like list and tuple by default
    (compare the first element and it is recursive!)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    
        import bisect
        def circus(input_set):
            input_set_by_y_order = sorted(input_set, key=lambda item:item[1])
            result_set = []
            for input_item in input_set_by_y_order:
                    new_guy = [input_item]
                    i = bisect.bisect_left(result_set, new_guy)
                    if i != len(result_set):
                        result_set[i].insert(0,new_guy)
                    else:
                        result_set.append(new_guy)
            return len(result_set)
     
        def test_circus():
            data_set = [(65,100), (70,50), (56,90), (75,190), (60,95),
                        (68,110),(62,144)]
            print circus(data_set)
    EmailDiggFacebookDeliciousStumbleUponTwitterTumblrGoogle GmailBlogger PostSina WeiboBox.netEvernoteFriendFeedGoogle BookmarksLiveJournalLinkedInPrintPrintFriendlyRedditSlashdotWordPressShare

    Tags: , , ,

    Monday, May 23rd, 2011 python, tech No Comments

    My weight loss trend

    There is really no secret on this: exercise and diet are your best friend.

    I’m heading toward my 100 lb weight loss goal now and I’m very close.

    dyerware.com


    EmailDiggFacebookDeliciousStumbleUponTwitterTumblrGoogle GmailBlogger PostSina WeiboBox.netEvernoteFriendFeedGoogle BookmarksLiveJournalLinkedInPrintPrintFriendlyRedditSlashdotWordPressShare

    Tags: , ,

    Saturday, May 21st, 2011 life No Comments