August 26, 2007

ladakh, land of peace and quiet - part I

Have you ever felt the power of space? When you feel that the space, just space around you, affects you strongly. Almost all of us have experienced it for short duration in some way or other, for example when we go to a temple. I felt it for a much longer duration. It happened to us when we visited Ladakh last month. By we, I mean Pankaj and I. For those who don't know, Pankaj and I are best buddies.

So, we went to this land of peace and quiet. There are some obvious things that make Ladakh different from all other hill stations. Altitude so high that AMS (Acute Mountain Sickness) comes to you easily, different kind of people, and proximity to both Pakistan and China borders. But, there are some things which are not easy to imagine. Things like how can it calm you beyond your imagination.

We were very excited about this trip. We decided to go by Manali-Leh road and come back by air. Our route was something like: Hyderabad -> Delhi -> Chandigarh -> Manali -> Leh -> Delhi -> Hyderabad. We had started preparing for it well in advance. Booking flight tickets from Leh to Delhi, hotels in Manali and Keylong, train tickets for Delhi to Chandigarh, guest house in Leh etc. And what date did we chose to travel - July 13th, Friday. Yes, Friday the 13th. It was accidental. We didn't notice it until the last day :)


My flight from Hyderabad to Delhi was little delayed, but I managed to reach Delhi just in time. Pankaj had come to airport to pick me up by a cab. From there we went together to Delhi Railway station by the same cab. We reached on time but no time was left for dinner. So, that's how we started our journey. Everything went as planned. We reached Manali by next day evening. After cleaning ourselves up and having snacks, we visited Hidimba temple. It's really a nice temple and in the late evening, it was a quiet place and had a calming effect in it. Next day morning we started for Leh. Our next destination was Keylong which is about 8 hours away from Manali. We passed Rohtang La on our way. We reached Keylong by evening 4 pm. This whole region is very beautiful. Air is fresh, land is green all around, mountains are capped with snow, space is free of any kind of pollution. We absolutely loved that place. Walking around in the town, watching sky full of stars in the night. Next day we started early, at about 6 am. And then we realized, Mountains look even more beautiful in morning :)


Next we encounterd a broken bridge which was being repaired. We took this as an opportunity to feel our surroundings. We hiked down to the shore of the river and sat there to appreciate the lovely morning. It was about 9:30 am in the morning. It wasn't windy, sun was shining and temperature was little bit on the colder side, but not too cold. I touched the water, it was chilling. In the mean time, our driver decided to cross the river through water and called us back. It was a bad move. We got stuck in the middle of the river. And then, we got a little adventurous. We removed our shoes and got into the water to push the car. But water was unthinkably cold. Our feet became red and nails started becoming yellow. Even after all the effort, we managed to move the car only a bit. After trying all the things, we finally gave up. Driver went out to fetch some help to tow the car. Luckily he found a tractor that was working at a road repair site not too far off. That's how we finally came out of river :)

Out next stop was at Sarchu. We had our lunch there in a mini-restaurant. It was in a tent and was run by 2 people exactly - a man and a woman, a couple probably. We had dal-roti, paratha and tea there. After that we started for Leh again. It was still a long drive from here. We passed mighty Tanglang La on our way. We reached Leh in the night at about 9:30 PM. Market was almost closed and it was completely dark there. After asking a couple of people there, we found our guest house and got in :)



(..to be continued. Photographs of the trip so far - http://picasaweb.google.com/manugarg/ReachingLeh)

Hacking squid

In this post, I would like to talk about the recent fun I had with squid. It involved some troubleshooting and some hacking.
Problem: Squid will stop responding after running for some random period of time, say 10 to 40 min and cpu usage will shoot up to 95-100%.

I started with strace, but everything looked fine there. Then I tried ltrace and there I got the first clue. squid was comparing 2 strings in an infinite loop:
strcmp("thumbnail.videoegg.com", "i12.ebaystatic.com") = -1
strcmp("thumbnail.videoegg.com", "i12.ebaystatic.com") = -1
strcmp("thumbnail.videoegg.com", "i12.ebaystatic.com") = -1

Looks like some bad 'for' loop. But, what part of code and why? It needed little more debugging to answer these questions. The squid binary that I was running was installed from a debian package and thus was stripped off debugging symbols. To fix that problem, I rebuilt the squid package with debugging information. On debian, you do that by supplying "DEB_BUILD_OPTIONS = nostrip" as an environment variable or in debian/rules file, while building the package. I ran this newly compiled squid within gdb. And, when it was hung up again, I took the backtrace:
gdb) bt
#0 0x00002ae1d60c6914 in strcmp () from /lib/libc.so.6
#1 0x000000000049f269 in hash_lookup (hid=0x79aae0, k=0x299f3e0) at hash.c:192
#2 0x000000000042c9bd in idnsCachedLookup (key=0xa733a8 "i12.ebaystatic.com",
callback=0x44ffe0 , data=0x7ad400) at dns_internal.c:1016
#3 0x000000000042d380 in idnsALookup (name=0x299f3e0 "thumbnail.videoegg.com",
callback=0xa733a8, data=0x7ad400) at dns_internal.c:1042
#4 0x00000000004506be in ipcache_gethostbyname (name=0x32bc628 "thumbnail.videoegg.com",
flags=1) at ipcache.c:521
#5 0x000000000040a3b4 in aclMatchAclList (list=0x78fe90, checklist=0xa730a8) at acl.c:1800
#6 0x000000000040ac50 in aclCheck (checklist=0xa730a8) at acl.c:2160
#7 0x00000000004200ae in clientReadRequest (fd=542, data=) at
client_side.c:4058
#8 0x00000000004288f8 in comm_select (msec=) at comm_generic.c:264
#9 0x0000000000452baa in main (argc=3, argv=) at main.c:858

On the first look it seems to be the problem with hash_lookup function as it's calling strcmp repetitively. hash_lookup seems to be pretty generic function however and it's unlikely that it will do something that stupid. Looking at the rest of the stack, it seems pretty clear that the problem is either in the internal dns part of the code (dns_internal.c) or hash routines. After this it was all dirty work. Going through various parts of the squid code, mainly internal dns and hash routines, and trying different things. Interestingly, I nailed down the cause of the problem while sitting at a bus stop, waiting for the bus to Pune :) Luckily I had my laptop with me at that time and I could verify it immediately. For the results of my findings, I'll quote myself from the bug report:
The problem seems to be in the way squid's internal DNS system (dns_internal.c) keeps record of looked up but not yet answered DNS queries. This bug is hit specifically when multiple search paths are used in /etc/resolv.conf.

Squid caches all dns queries before sending them to avoid duplicate queries for the same name. (look at: idnsCacheQuery(q) and hash_table *idns_lookup_hash, in dns_internal.c). This mechanism works well unless multiple search paths are defined in /etc/resolv.conf. When multiple dns search paths are defined, same query object is modified and next search path is concatenated to it's name. This query is cached again and resent.

Problem is that the query is not unlinked before being cached and thus linked again. Only the key of hash object (that's actually name) changes this time; object itself remains same. This corrupts the hash table of looked up queries.

Once you know the problem and what's causing it, it becomes pretty easy to fix that. That's true at least in the world of computers, especially open source world, if not generally in life :) So, I wrote a patch to fix it and it worked. This patch made into both the branches of squid - 2.6 and 3.0. Also, it got me my name listed on the changesets page as a contributor :)

It was really exciting as it was my first contribution (in the form of code) to an existing open source project. I have started a couple of small time open source projects, but contributing to an existing and mature project was another type of fun :-)