Inspirated

 
 

May 28, 2010

Summer of Code Progress: REST collections

Filed under: Blog — krkhan @ 6:44 pm

Starting from this post, I will be documenting my SoC progress according to the Ubuntu guidelines. This will enable me to log my activities in an organized manner. The inaugural report:

Related Links

Summer of Code Archive Inspirated Code
Report Guidelines Ubuntu Wiki
Original Proposal Ubuntu Wiki

Time Spent

8 hours.

Highlights

This is my first experience of developing on a Zope project. The fact that the project is already deployed in a large-scale production environment is daunting as well as exciting at the same time. For Launchpad work, I learned only the bare essentials of Zope, i.e., things that are directly related with my project such as interfaces and adapters. After SoC however, I plan on exploring it in more detail.

Concerns

Testing Launchpad modifications is a beast of a task. After making each set of changes to the code, I have to stop the server and relaunch it again which takes at least 30 seconds. It becomes especially frustrating when I’m in the process of making tiny changes in order to explore the API.

I’ve tried to work around this issue by using pdb.set_trace(). Still, it’s no substitute for being able to edit whole files in one go.

Waiting Items

None.

Stalled Tasks

None.

Accomplishments

  • Test locally running Launchpad with a RESTClient.
  • Go through the code-base and figure out the portions which deal with API services.
  • Export a bug collection for every project:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    
    <rest-client version="2.3"><request>
    <http-version>1.1</http-version>
    <URL>https://api.launchpad.dev/beta/alsa-utils/bugs</URL>
    <method>GET</method>
    <ssl-truststore>/home/inspirated/.keystore</ssl-truststore>
    <ssl-truststore-password>rO0ABXQAB2tyYWNrM2Q=</ssl-truststore-password>
    <ssl-hostname-verifier>ALLOW_ALL</ssl-hostname-verifier>
    <headers>
    <header key="Authorization" value="OAuth realm="OAuth", oauth_nonce="FFFF", oauth_timestamp="1275040360", oauth_consumer_key="just%20testing", oauth_signature_method="PLAINTEXT", oauth_version="1.0", oauth_token="0NWl33zGj3QDtxFWdHCt", oauth_signature="%261XP9XCptHQ3S8g49RjFTr3xFlrK0H73Z5QjbGbxs1pMTc5mzptq0Z5Qs93gjcXDQtqlgFSVVQBPLdK47""/>
    <header key="Accept" value="application/json"/>
    <header key="Host" value="api.launchpad.dev"/>
    </headers>
    </request>
    </rest-client>
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    
    <rest-client version="2.3">
    <response>
    <execution-time>1833</execution-time>
    <status code="200">HTTP/1.1 200 Ok</status>
    <headers>
    <header key="Date" value="Fri, 28 May 2010 12:50:18 GMT"/>
    <header key="Server" value="zope.server.http (DebugLayerHTTP)"/>
    <header key="X-Powered-By" value="Zope (www.zope.org), Python (www.python.org)"/>
    <header key="Content-Type" value="application/json"/>
    <header key="Content-Length" value="10736"/>
    <header key="Vary" value="Accept,Accept-Encoding"/>
    <header key="Via" value="1.1 launchpad.dev"/>
    <header key="Keep-Alive" value="timeout=15, max=100"/>
    <header key="Connection" value="Keep-Alive"/>
    </headers>
    <body>
    {
       total_size           : 15,
       start                : 0,
       resource_type_link   : 'https://api.launchpad.dev/beta/#bug-page-resource',
       next_collection_link : 'https://api.launchpad.dev/beta/alsa-utils/bugs?ws.start=5&amp;ws.size=5',
       entries              : [
          {
             users_unaffected_collection_link          : 'https://api.launchpad.dev/beta/bugs/15/users_unaffected',
             latest_patch_uploaded                     : null,
             users_affected_count_with_dupes           : 0,
             security_related                          : false,
             private                                   : false,
             bug_watches_collection_link               : 'https://api.launchpad.dev/beta/bugs/15/bug_watches',
             date_made_private                         : null,
             linked_branches_collection_link           : 'https://api.launchpad.dev/beta/bugs/15/linked_branches',
             subscriptions_collection_link             : 'https://api.launchpad.dev/beta/bugs/15/subscriptions',
             number_of_duplicates                      : 0,
             id                                        : 15,
             users_unaffected_count                    : 0,
             title                                     : 'Nonsensical bugs are useless',
             name                                      : null,
             http_etag                                 : '"23e175849b60b27f5553ddb56995e044c71d745a-ce4f09c837a78ed5fdf23f759020058f3423a804"',
             messages_collection_link                  : 'https://api.launchpad.dev/beta/bugs/15/messages',
             self_link                                 : 'https://api.launchpad.dev/beta/bugs/15',
             who_made_private_link                     : null,
             attachments_collection_link               : 'https://api.launchpad.dev/beta/bugs/15/attachments',
             resource_type_link                        : 'https://api.launchpad.dev/beta/#bug',
             date_last_updated                         : '2007-12-18T16:31:34.972893+00:00',
             description                               : 'Like this one, natch.',
             duplicates_collection_link                : 'https://api.launchpad.dev/beta/bugs/15/duplicates',
             tags                                      : [],
             message_count                             : 7,
             heat                                      : 0,
             bug_tasks_collection_link                 : 'https://api.launchpad.dev/beta/bugs/15/bug_tasks',
             cves_collection_link                      : 'https://api.launchpad.dev/beta/bugs/15/cves',
             users_affected_with_dupes_collection_link : 'https://api.launchpad.dev/beta/bugs/15/users_affected_with_dupes',
             duplicate_of_link                         : null,
             users_affected_count                      : 0,
             owner_link                                : 'https://api.launchpad.dev/beta/~name16',
             date_created                              : '2007-12-18T16:30:19.103679+00:00',
             can_expire                                : false,
             date_last_message                         : '2007-12-18T16:31:34.790641+00:00',
             users_affected_collection_link            : 'https://api.launchpad.dev/beta/bugs/15/users_affected'
          },
          <!-- entries omitted -->
          {
             users_unaffected_collection_link          : 'https://api.launchpad.dev/beta/bugs/11/users_unaffected',
             latest_patch_uploaded                     : null,
             users_affected_count_with_dupes           : 0,
             security_related                          : false,
             private                                   : false,
             bug_watches_collection_link               : 'https://api.launchpad.dev/beta/bugs/11/bug_watches',
             date_made_private                         : null,
             linked_branches_collection_link           : 'https://api.launchpad.dev/beta/bugs/11/linked_branches',
             subscriptions_collection_link             : 'https://api.launchpad.dev/beta/bugs/11/subscriptions',
             number_of_duplicates                      : 0,
             id                                        : 11,
             users_unaffected_count                    : 0,
             title                                     : 'Make Jokosher use autoaudiosink',
             name                                      : null,
             http_etag                                 : '"8675b8d2649cf0c08d58681036ef1e8bb6a20e78-d7d969f7ed8c2f2def4d9e429f23ac41f7fec4dc"',
             messages_collection_link                  : 'https://api.launchpad.dev/beta/bugs/11/messages',
             self_link                                 : 'https://api.launchpad.dev/beta/bugs/11',
             who_made_private_link                     : null,
             attachments_collection_link               : 'https://api.launchpad.dev/beta/bugs/11/attachments',
             resource_type_link                        : 'https://api.launchpad.dev/beta/#bug',
             date_last_updated                         : '2007-03-15T20:37:51.603369+00:00',
             description                               : 'I\'ve had problems when switching from Jokosher to Totem to play an Ogg.\n\nTotem appears to be playing normally but does not produce any sound.\nIf I close Jokosher then you can hear totem.\n\nI\'ve also had a problem when trying to playback sound within jokosher\nafter switch from another app (I think it was xmms). I get a dialog\nwith the following text:\n\n     Argh! Something went wrong and a serious error occurred:\n\n     Resource busy or not available.\n\n     gstalsasink.c(636): gst_alsasink_open (): /timeline/playbackbin/\n     alsasink0:\n     Device \'hw:0\' is busy\n\nAfter closing and reopening Jokosher, i got the same error again.\nAfter logging out of gnome and logging in again - I still got the same\nerror message.\n\nI had to restart!\n\nIs there some way to reset alsa or the device driver - with out having\nto restart?\n\nRunning on Ubuntu - with Jokosher 0.2 runscript.',
             duplicates_collection_link                : 'https://api.launchpad.dev/beta/bugs/11/duplicates',
             tags                                      : [],
             message_count                             : 7,
             heat                                      : 0,
             bug_tasks_collection_link                 : 'https://api.launchpad.dev/beta/bugs/11/bug_tasks',
             cves_collection_link                      : 'https://api.launchpad.dev/beta/bugs/11/cves',
             users_affected_with_dupes_collection_link : 'https://api.launchpad.dev/beta/bugs/11/users_affected_with_dupes',
             duplicate_of_link                         : null,
             users_affected_count                      : 0,
             owner_link                                : 'https://api.launchpad.dev/beta/~name16',
             date_created                              : '2007-03-15T20:33:56.678930+00:00',
             can_expire                                : false,
             date_last_message                         : null,
             users_affected_collection_link            : 'https://api.launchpad.dev/beta/bugs/11/users_affected'
          }
       ]
    }
    </body>
    </response>
    </rest-client>

Minor Tasks

Having gotten stuck on the task of running RESTClient against my Launchpad instance, I ran into a few issues such as RESTClient’s refusal to accept the local SSL certificate and confusion regarding domain names. The former issue was solved by using keytool to create a trusted keystore while the latter was addressed by using correct domain names as pointed out by William Grant.

Actions for the Following Report

While I have managed to export a web service property for Launchpad projects, I haven’t yet figured out a way to group bugs by projects. This will require a clear understanding of the schemas involved as well as some skills with SQL. My next task therefore will be to export only the bugs that are related to a particular project.

Tags: , , , , , , , ,

May 22, 2010

Summer of Code Progress: Debugging Launchpad’s RESTful API

Filed under: Blog — krkhan @ 8:03 pm

Related Links

Summer of Code Archive Inspirated Code
Original Proposal Ubuntu Wiki

Report

RESTful web services are very fun once they get properly integrated in end-user applications. Testing those services is a whole different thing and I wanted to dive deep into the bare essentials of REST communication performed by Launchpad for its API. The most straightforward way was to write a Python script using urllib2 and/or httplib for the HTTP request/response mantra. Obviously I found it to be a cumbersome solution and then found this little beauty:

RESTClient Screenshot
(Click on the thumbnail for larger version.)

RESTClient does one thing and does it sweetly: it gives you a nice GUI for toying with REST requests. Getting it to work with Launchpad was not as straightforward though as all requests have to be signed to be of any real value. I also had to spent a few hours trying to understand why api.edge.launchpad.net was not recognizing my access tokens until by sheer stroke of luck I used api.staging.launchpad.net and it started working perfectly.

1
2
3
4
5
6
7
8
9
10
11
12
<rest-client version="2.3">
<request>
<http-version>1.1</http-version>
<URL>https://api.staging.launchpad.net/beta/bugs/1</URL>
<method>GET</method>
<headers>
<header key="Authorization" value="OAuth realm="OAuth", oauth_nonce="77848601", oauth_timestamp="1274537900", oauth_consumer_key="just%20testing", oauth_signature_method="PLAINTEXT", oauth_version="1.0", oauth_token="6dwHFn3CPzxjNrrdp3r9", oauth_signature="%26xxxxxx""/>
<header key="Accept" value="application/json"/>
<header key="Host" value="api.staging.launchpad.net"/>
</headers>
</request>
</rest-client>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
<rest-client version="2.3">
<response>
<execution-time>897</execution-time>
<status code="200">HTTP/1.1 200 Ok</status>
<headers>
<header key="Date" value="Sat, 22 May 2010 14:26:32 GMT"/>
<header key="Server" value="zope.server.http (HTTP)"/>
<header key="X-Powered-By" value="Zope (www.zope.org), Python (www.python.org)"/>
<header key="Content-Type" value="application/json"/>
<header key="Content-Length" value="2888"/>
<header key="Etag" value=""d9b357c932ada0a71e96401a8c87368d4704a500-dec69b947b887c4addba8eb4aae5ca64c94f616c""/>
<header key="Vary" value="Accept,Accept-Encoding"/>
<header key="Via" value="1.1 wildcard.staging.launchpad.net"/>
<header key="Keep-Alive" value="timeout=15, max=100"/>
<header key="Connection" value="Keep-Alive"/>
</headers>
<body>
{
   users_unaffected_collection_link          : 'https://api.staging.launchpad.net/beta/bugs/1/users_unaffected',
   latest_patch_uploaded                     : null,
   users_affected_count_with_dupes           : 378,
   security_related                          : false,
   private                                   : false,
   bug_watches_collection_link               : 'https://api.staging.launchpad.net/beta/bugs/1/bug_watches',
   date_made_private                         : null,
   linked_branches_collection_link           : 'https://api.staging.launchpad.net/beta/bugs/1/linked_branches',
   subscriptions_collection_link             : 'https://api.staging.launchpad.net/beta/bugs/1/subscriptions',
   number_of_duplicates                      : 0,
   id                                        : 1,
   users_unaffected_count                    : 6,
   title                                     : 'Microsoft has a majority market share',
   name                                      : 'liberation',
   http_etag                                 : '"d9b357c932ada0a71e96401a8c87368d4704a500-dec69b947b887c4addba8eb4aae5ca64c94f616c"',
   messages_collection_link                  : 'https://api.staging.launchpad.net/beta/bugs/1/messages',
   self_link                                 : 'https://api.staging.launchpad.net/beta/bugs/1',
   who_made_private_link                     : null,
   attachments_collection_link               : 'https://api.staging.launchpad.net/beta/bugs/1/attachments',
   resource_type_link                        : 'https://api.staging.launchpad.net/beta/#bug',
   date_last_updated                         : '2010-05-13T23:24:26.362094+00:00',
   description                               : 'Microsoft has a majority market share in the new desktop PC marketplace.\nThis is a bug, which Ubuntu is designed to fix.\n\nNon-free software is holding back innovation in the IT industry, restricting access to IT to a small part of the world\'s population and limiting the ability of software developers to reach their full potential, globally. This bug is widely evident in the PC industry.\n\nSteps to repeat:\n\n1. Visit a local PC store.\n\nWhat happens:\n2. Observe that a majority of PCs for sale have non-free software pre-installed.\n3. Observe very few PCs with Ubuntu and free software pre-installed.\n\nWhat should happen:\n1. A majority of the PCs for sale should include only free software like Ubuntu.\n2. Ubuntu should be marketed in a way such that its amazing features and benefits would be apparent and known by all.\n3. The system shall become more and more user friendly as time passes.\n\n',
   duplicates_collection_link                : 'https://api.staging.launchpad.net/beta/bugs/1/duplicates',
   tags                                      : [
      'iso-testing',
      'ubuntu'
   ],
   message_count                             : 1198,
   heat                                      : 2138,
   bug_tasks_collection_link                 : 'https://api.staging.launchpad.net/beta/bugs/1/bug_tasks',
   cves_collection_link                      : 'https://api.staging.launchpad.net/beta/bugs/1/cves',
   users_affected_with_dupes_collection_link : 'https://api.staging.launchpad.net/beta/bugs/1/users_affected_with_dupes',
   duplicate_of_link                         : null,
   users_affected_count                      : 378,
   owner_link                                : 'https://api.staging.launchpad.net/beta/~sabdfl',
   date_created                              : '2004-08-20T00:00:00+00:00',
   can_expire                                : false,
   date_last_message                         : '2010-05-13T23:24:21.253673+00:00',
   users_affected_collection_link            : 'https://api.staging.launchpad.net/beta/bugs/1/users_affected'
}
</body>
</response>
</rest-client>

With a local Launchpad ready to do my bidding and a debugging process in place for elaborating its API, I’m looking forward to the coding phase. Which, coincidentally, begins on the same day I give my final exam for the current semester.

Rest REST: the sweet sauce of labor.” — Plutarch

Tags: , , , , , , , , ,

May 14, 2010

Summer of Code Progress: Ready to Launch(pad)

Filed under: Blog — krkhan @ 11:51 pm

As the initial stage of my GSoC project is heavily involved with the Launchpad core, the first step for being ready was to get it running in a clean lean virtual machine. Bad rhyming I know. But trust me, trying to setup Launchpad locally with a borked internet connection is worse.

So, after some ∞ attempts at using rocketfuel-setup, I got stuck because my internet gave up on each run before Bazaar repositories could download. To top it off, once the repositories did download the VM decided to become unresponsive as it had only 384 MB of RAM and unpacking a huge repository like Launchpad’s core required a lot more resources than that.

Love for open-source followed as I downloaded the repositories on a separate machine, transferred them to the VM using NFS and then was able to modify the rocketfuel-setup so that it would do only the remaining work. The result:

Virtual Launchpad Snapshot #1
(Click on the thumbnail for larger version.)

The added bonus of doing the whole process on VM was the ability to go back to a clean state in case something went haywire. A few months ago, I had moved over to Sun’s VirtualBox from Redhat’s Virtual Machine Manager and this proved to be helpful on many accounts. Apart from being sleeker and more friendlier, VirtualBox offers advanced features such as PulseAudio integration on host and hierarchical snapshots. Using the shots, now I can just select a state where Launchpad was launched clean … :

Virtual Launchpad Snapshot #2
(Click on the thumbnail for larger version.)

… click on restore and viola, I’ll have the same instance up and running again within a few seconds.

Tags: , , , , , ,

May 11, 2010

Summer of Code Progress: Ubuntu Community Bonding

Filed under: Blog — krkhan @ 8:53 pm

Over the past few days, I have been preparing myself for GSoC work in various ways. Officially, these few weeks are defined as “Community Bonding Period” for Summer of Code participants. Meaning, selected students are expected to spend time for getting everything set for the real development which starts on May 26th.

The first thing on my priority list was to find a way for fail-safe IRC communication. The end solution was documented in my previous post. In a nutshell, now I have this IRC client running all the time to which I can connect occasionally and go-through the activity since my last visit.

Next up was discussing with Bryce the list of things that should be ready before the next stage. Setting up Launchpad locally was pinpointed as the real PITA for beginning the development. He advised me on a few other aspects as well, including why I should aim to complete the major coding portions of my work before the mid-term evaluation. Bryce has been the X.org maintainer for Ubuntu so he’s also new to Launchpad development like me. However, unlike me he has been able to set up a Launchpad instance on Lucid Lynx. I have been trying a bit unsuccessfully to do the same with Karmic Koala for a few days now. The rocketfuel-setup quits on me after complaining about download-cache directory not being available in the devel tree. I’m hoping to look more into that tonight.

As for the miscellaneous things, Google gifted all the participants an year of ACM Student Membership. Without sarcasm, ACM has one of the most confusing user-management system I have ever used. I already had a web-account on ACM and hence my username was not available for using this membership. Doesn’t make sense? Exactly. With that said, on the brighter side of things was the fact that ACM’s help-desk was prompt and solved my registration problems within a few emails of correspondence. The ACM email address still isn’t forwarding emails properly but I guess that’ll take a few days to resolve.

On the community side of things, I thought it would be convenient to have important information about my organization’s projects in one place. On that account, here’s a list of my fellow Ubuntu participants for this year’s Summer of Code:

Student Project Mentor
Sarah Strong The Great Clipboard Fixing Galore Project Ted Gould
Michal Karnicki Android U1: Ubuntu One client for Android Stuart Langridge
Jacob Peddicord services-admin configuration and Upstart-ification David Bensimon
Dylan McCall Harvest user interface improvements Daniel Holbach
Urban Skudnik Home user backup solution/Deja Dup improvements Michael Terry
Dmitrijs Ledkovs USB-creator Improvements Evan Dandrea
Kamran Khan Bug Triaging Improvements for Launchpad/Arsenal Bryce Harrington
Harald Sitter Ubuntu One for the KDE workspace Jonathan Riddell
Peter Gardenier Software Center Improvements Matthew Thomas
Andres Rodriguez Lazo Testdrive Front End Dustin Kirkland

Good luck to everyone on board!

Tags: , , , , , , ,

April 27, 2010

Finally there — Google Summer of Code 2010

Filed under: Blog — krkhan @ 1:02 am

GSoC 2010 Logo

Fate has a curious sense of humor. After having 8 of my proposals rejected in past 3 years for Google Summer of Code, I had 2 of them selected in the same year. In GSoC terminology, I became a “duplicate” student. Now, the standard practice for such cases is that the administrators of both organizations interested in the student get together in a “deduplication” meeting and resolve the conflict. However, Maria Randazzo (the program administrator from the Ubuntu organization) was kind enough to ask my preference regarding the project I’d like to work upon.

This put me in a rather uneasy position as mentors from both organizations (Bryce Harrington from Ubuntu & Alexey Khoroshilov from The Linux Foundation) had been really helpful during the application process. In the end however, I chose Ubuntu since its proposal focused on Arsenal and Python which I found relatively more familiar than Alien and Perl. I had also already collaborated with Bryce for some Arsenal patches before so it was easier for me to get up-to-speed with the Ubuntu proposal.

The proposal itself is in fact publicly viewable at this link. Summarizing my feelings, I’ve never looked forward to a summer as much as this one.

“If a June night could talk, it would probably boast it invented romance code.” —Bern Williams

Tags: , , , , , , , , ,
« Previous Page