Inspirated

 
 

July 12, 2010

Summer of Code Progress: Curl It Unlike Beckham

Filed under: Blog — krkhan @ 11:10 pm

Related Links

Summer of Code Archive Inspirated Code
Report Guidelines Ubuntu Wiki
Original Proposal Ubuntu Wiki

Time Spent

50 hours.

Highlights

The send-attachments-upstream.py script was migrated from XML-RPC to Curl for communicating with Bugzilla. The script was the refactored in order to provide capabilities such as attachment filtering. Various bugfixes and improvements were catered to along the way.

Concerns

None.

Waiting Items

None.

Stalled Items

None.

Accomplishments

  • Branch, Merge Revision: Reimplemented attachment sending using pycurl.
  • Branch, Merge Revision: Refactored the script in order to provide options such as -o (copy only attachments uploaded by bug owner).

Minor Tasks

  • Branch, Merge Revision: Fixed regular expressions for parsing results and handling of Unicode attachment titles.

Actions for the Following Report

Implement the following improvements in send-attachments.py:

  • Caching Bugzilla credentials.
  • Filename exclusion for attachments.
  • Archive extraction.
  • File size limits.
  • A command-line switch to enforce MIME content-types based on file extensions.
Tags: , , , , , , , , , , ,

July 3, 2010

Summer of Code Progress: Attachment Upstreamer

Filed under: Blog — krkhan @ 5:04 pm

Related Links

Summer of Code Archive Inspirated Code
Report Guidelines Ubuntu Wiki
Original Proposal Ubuntu Wiki

Time Spent

10 hours.

Highlights

Communicating with Bugzilla is done through the python-bugzilla wrapper library. This could have been achieved by using xmlrpclib directly but doing that would require reinventing a whole lot of wheels by handling Bugzilla specific XML-RPC eccentricities.

Concerns

None.

Waiting Items

None.

Stalled Items

None.

Accomplishments

  • Branch: Added support for copying attachments to a remote bugzilla:
    $ ./send-attachments-upstream.py --user=krkhan@inspirated.com --pass=xxx https://bugs.launchpad.net/ubuntu/+bug/223435  https://partner-bugzilla.redhat.com/show_bug.cgi?id=593603
    Logging in Launchpad [Success <Logged in as Kamran Riaz Khan>]
    Logging in Bugzilla [Success <Logged in as krkhan@inspirated.com>]
    Uploading: Dependencies.txt [Success]
    Uploading: Disassembly.txt [Success]
    Uploading: ProcMaps.txt [Success]
    Uploading: ProcStatus.txt [Success]
    Uploading: Registers.txt [Success]
    Uploading: Stacktrace.txt [Success]
    Uploading: ThreadStacktrace.txt [Success]
    Uploading: Stacktrace.txt (retraced) [Success]
    Uploading: ThreadStacktrace.txt (retraced) [Success]

Minor Tasks

  • Revision: Added python-bugzilla in lib and modified setup.py accordingly.
  • Revision: Initial commit for sending attachments to a remote Bugzilla.
  • Revision: Added error handling for API calls.

Actions for the Following Report

Add support for creating new bugs in a remote Bugzilla based on data from a Launchpad bug.

Tags: , , , , , , , , , ,

June 27, 2010

Find your most used words in Pidgin logs with Python

Filed under: Blog — krkhan @ 12:08 am

Here’s a quick little script which I wrote to tabulate the word frequencies in Pidgin logs. Simple, you point it towards a contact’s log directory:

$ ./purple-stats.py /home/krkhan/.purple/logs/msn/krkhan\@inspirated.com/some.friend\@some.gmail.com

And it gives you the words in their descending order of usage:

         0: you        (38)
         1: it         (30)
         2: to         (24)
         3: the        (22)
         4: in         (22)
         5: lol        (22)
         6: of         (22)
         7: so         (18)
         8: is         (18)
         9: what       (16)
         ...

As usual, Python was used for the dirty work:

purple-stats.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#!/usr/bin/env python
 
from operator import itemgetter
from string import punctuation
 
import locale
import os
import sys
 
from BeautifulSoup import BeautifulSoup
 
if __name__ == "__main__":
    if len(sys.argv) < 2:
        print "usage:", sys.argv[0], "<logs directory>"
    dir = sys.argv[1]
 
    contents = filter(lambda x: x[-5:] == '.html', os.listdir(dir))
    stats = {}
    for entry in contents:
        path = os.path.join(dir, entry)
        with open(path, 'r') as fd:
            data = fd.read()
        soup = BeautifulSoup(data,
            convertEntities=BeautifulSoup.ALL_ENTITIES)
        spans = soup.findAll('span')
        for span in spans:
            for word in span.text.split():
                word = word.strip(punctuation).lower()
                if len(word) < 2:
                    continue
                stats[word] = stats.get(word, 0) + 1
 
    sorted_stats = sorted(stats.iteritems(), key=itemgetter(1))
    sorted_stats.reverse()
    for num, (word, count) in enumerate(sorted_stats):
        line = "%10d: %-10s (%d)" % (num, word, count)
        line = line.encode(locale.getpreferredencoding())
        print line
Tags: , , , , , , , ,

June 22, 2010

Summer of Code Progress: Merging Launchpad branches

Filed under: Blog — krkhan @ 5:21 pm

Related Links

Summer of Code Archive Inspirated Code
Report Guidelines Ubuntu Wiki
Original Proposal Ubuntu Wiki

Time Spent

40 hours.

Highlights

This week was spent on trying to get my Launchpad branches merged upstream. During the process many concerns were raised which resulted in a number of patches and discussions.

Concerns

Quoting Stuart Bishop’s response from Launchpad-dev:

I’m really not sure of the best way to tackle this problem. The
Librarian data is not stored in the database because there are
multiple TB of files. The team membership information is in the
relational database. There are no indexes anywhere to the contents of
the Librarian files. I think we need some sort of external search
engine (I don’t think we don’t want to integrate this into the
Librarian core). Ideally we could feed it subscriber information
allowing it to determine the set of 32000 attachments that ubuntu-bugs
has access to rather than having to calculate this information from
the relational db and then feed the ids to the search engine.

Whatever approach certainly needs signoff from the LP team leads, as
the resource requirements are non trivial and someone needs to pay for
the hardware.

Waiting Items

None.

Stalled Items

  • Implementation of Bug.findAttachments().

Accomplishments

  • Merge Proposal: Got the export-Person-getBugSubscriberPackages branch approved after fixing various tests and bugs.
  • Merge Proposal: Implemented Horspool’s algorithm and fixed various bugs in the implement-Bug-findAttachments branch. The branch itself didn’t get approved because of its design approach for searching the attachments:

    I’m going to mark this review as ‘disapproved,’ not because the code is
    bad (it isn’t) but because I don’t think this is the right solution to
    the problem. I’m sorry to say that I don’t know what the right solution
    to the problem actually is at this point, but I’d guess that something
    involving FTIs would be a start, or some kind of asynchronous processing
    of searches (though then you get into all kinds of knotty stuff with
    callbacks).

Minor Tasks

For reading the file in chunks, I took the Wiki code for Horspool algorithm, converted it to Python and modified a little so that it would work with stream files.

Actions for the Following Report

There doesn’t appear to be a straightforward efficient way for searching bug attachments. I’ll discuss the course of my future development with Bryce tonight and decide whether I should head over to Arsenal development or should I focus on the proposed (albeit germinal) solutions from IRC and the mailing list.

Tags: , , , , , , , ,

June 19, 2010

Using Boyer-Moore-Horspool algorithm on file streams in Python

Filed under: Blog — krkhan @ 4:53 am

Horspool’s algorithm is a simple and efficient string-searching algorithm which trades space for time and performs better as length of search string is increased. Another (perhaps overlooked) advantage of this algorithm is its ability to search through stream files without requiring random access. As I was working on Launchpad for my SoC project I required this particular stream-handling attribute as the file descriptors opened by urllib2 didn’t support seek()ing. Modifying the example code from Wiki page a little, I was able to read() only the required bytes sequentially:

horspool.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
#!/usr/bin/env python
 
import locale
import os
import sys
import urllib2
 
def boyermoore_horspool(fd, needle):
    nlen = len(needle)
    nlast = nlen - 1
 
    skip = []
    for k in range(256):
        skip.append(nlen)
    for k in range(nlast):
        skip[ord(needle[k])] = nlast - k
    skip = tuple(skip)
 
    pos = 0
    consumed = 0
    haystack = bytes()
    while True:
        more = nlen - (consumed - pos)
        morebytes = fd.read(more)
        haystack = haystack[more:] + morebytes
 
        if len(morebytes) < more:
            return -1
        consumed = consumed + more
 
        i = nlast
        while i >= 0 and haystack[i] == needle[i]:
            i = i - 1
        if i == -1:
            return pos
 
        pos = pos + skip[ord(haystack[nlast])]
 
    return -1
 
if __name__ == "__main__":
    if len(sys.argv) < 3:
        print "Usage: horspool.py <url> <search text>"
        sys.exit(-1)
 
    url = sys.argv[1]
    needle = sys.argv[2]
    needle = needle.decode('string_escape')
 
    fd = urllib2.urlopen(url)
    offset = boyermoore_horspool(fd, needle)
    print hex(offset), '::', offset
    fd.close()

Now comes the fun part:

  • The code can search through any URL without downloading it completely, stopping at the first match. For example, the following command will download only the first few bytes of the provided URL:
    $ ./horspool.py http://www.gutenberg.org/files/132/132.txt "The Art of War"

    0x1d :: 29

  • Unicode searches work perfectly as well. Although the matching takes place according to the character encoding of the terminal used. That’s to say, since I’m using a UTF-8 terminal the “bytes” searched were assumed to be UTF-8 encoded as well:
    $ ./horspool.py http://www.gutenberg.org/files/29011/29011-0.txt "Σημείωση: Ο Πίνακας περιεχομένων"

    0x44f :: 1103

  • Same goes for multi-line searches:
    $ ./horspool.py http://www.gutenberg.org/files/29011/29011-0.txt "διευκόλυνση\r\nτου αναγνώστη"

    0x4b5 :: 1205

Tags: , , , , , , , , ,

June 14, 2010

Summer of Code Progress: Searching bug attachments for team subscriptions

Filed under: Blog — krkhan @ 5:39 pm

Related Links

Summer of Code Archive Inspirated Code
Report Guidelines Ubuntu Wiki
Original Proposal Ubuntu Wiki

Time Spent

15 hours.

Highlights

During the course of my SoC work on Launchpad I have been following a simple procedure for developing API calls:

  • Find a similar functionality which is already implemented upstream.
  • Log PostgreSQL queries while using that functionality and then analyze the queries made along with database schemas to devise a plan for implementing the new feature.

As documented in my previous report, I had to implement a property/method for exporting “Source Packages” a “Team” was subscribed to. Going through the chores listed above, I ended up with a query for structural subscriptions table which joined teams and source packages. The method worked perfectly. That is, until I reopened the interface definition today and searched for my changes. I landed on something that looked similar and then realized that the method was already there in Launchpad, just not exported yet.

That condensed my work to just a few lines as I discarded my own changes and their unit tests and exported the method that was already present upstream. Lesson learned: Search your project thoroughly before starting the coding phase in case you’re duplicating someone else’s efforts.

Concerns

The method for getting bug subscriptions for a team had this comment on top:

    # XXX: Tom Berger 2008-04-14 bug=191799:
    # The implementation of these functions
    # is no longer appropriate, since it now relies on subscriptions,
    # rather than package bug supervisors.
    def getBugSubscriberPackages(self):

I’m not sure if a merge proposal for exporting this operation would be accepted since the implementation is “no longer appropriate”. Will discuss this with Bryce in detail tonight.

Waiting Items

None.

Stalled Items

None.

Accomplishments

  • Patch: Exported Person.getBugSubscriberPackages() as read only operation.

Minor Tasks

The process of returning a collection of source packages created a circular import dependency which broke down the process of WADL resource generation. Again, going around Launchpad code revealed that this could be fixed by just returning a generic Interface and then patching it later on in lib/canonical/launchpad/interfaces/_schema_circular_imports.py.

Actions for the Following Report

Searching for bug attachments in source packages is already possible using my bugs-findAttachment branch. In case Person.getBugSubscriberPackages() is considered okay, I have all the Launchpad pieces in place now for doing this in Arsenal: “Search for text in all bug attachments for packages this team is subscribed to”. Really looking forward to having a bit more “freedom” with the codebase than I had with Launchpad.

Tags: , , , , , , , ,

June 9, 2010

Summer of Code Progress: Unit testing and merge proposals

Filed under: Blog — krkhan @ 4:30 pm

Related Links

Summer of Code Archive Inspirated Code
Report Guidelines Ubuntu Wiki
Original Proposal Ubuntu Wiki

Time Spent

20 hours.

Highlights

The process of “presenting” my code to a giant project like Launchpad made me learn a lot about how to plan, execute and test modifications for large-scale open source projects. It’ll be fun to see the final outcome merged upstream.

I am also feeling more and more confident dealing with Launchpad development. It’s gratifying to know that if I need some API calls during the course of my SoC work which aren’t already there, I’ll be able to implement them on short notices.

Concerns

As per the discussion with Bryce, the approach for attachment search needs to be changed a bit. Instead of going through the “Product -> Bugs -> Attachments” route, we need to instead focus on the “Team -> Source Packages -> Bugs -> Attachments” alternate.

Waiting Items

None.

Stalled Items

None.

Accomplishments

  • Patch: Added unittest for bugs collection.
  • Patch: Added unittests for bugs.findAttachments.
  • Written merge proposal for project-bug-collection branch:

    Merge proposal for lp:~inspirated/launchpad/project-bug-collection

    Summary

    As part of my Google Summer of Code project, I had to implement attachment searching functionality in Arsenal. The end-product would allow user to specify a search string which would be searched in all the bug attachments for a project.

    Doing this efficiently required two modifications in Launchpad:

    • Exposing a bug collection for a particular product (this branch)
    • Implement an attachment search method for a particular bug
    Proposed fix

    Export a bugs CollectionField for IProductPublic.

    Pre-implementation Notes
    • The exported field should be read-only
    Implementation Details
    • Bug schema does not refer to Products directly. Therefore, a join was required to query BugTasks which point towards the Product in question and then query the Bugs related to that BugTask.
    Tests
    $ bin/test -vv -m lp.registry.tests.test_product -t test_bug_collection
    Demo and Q/A

    Create a Launchpad instance:

    	>>> from launchpadlib.launchpad import Launchpad
    	>>> launchpad = Launchpad.login_with('just testing', 'https://api.launchpad.dev/beta/')
    	The authorization page:
    	   (https://launchpad.dev/+authorize-token?oauth_token=ppd2bRZDnN6VX94lRqrq)
    	should be opening in your browser. After you have authorized
    	this program to access Launchpad on your behalf you should come
    	back here and press <Enter> to finish the authentication process.

    Load a project:

    	>>> project = launchpad.projects['firefox']
    	>>> project
    	<project at https://api.launchpad.dev/beta/firefox>
    	>>> project.bugs
    	<lazr.restfulclient.resource.Collection object at 0x925c04c>

    Use the bugs collection:

    	>>> for bug in project.bugs:
    	...     print bug.title
    	... 
    	Firefox does not support SVG
    	Reflow problems with complex page layouts
    	Firefox install instructions should be complete
    	Firefox crashes when Save As dialog for a nonexistent window is closed
    lint

    The changes are lint clean.

  • Written merge proposal for implement-Bug-findAttachments branch:

    Merge proposal for lp:~inspirated/launchpad/implement-Bug-findAttachments

    Summary

    As part of my Google Summer of Code project, I had to implement attachment searching functionality in Arsenal. The end-product would allow user to specify a search string which would be searched in all the bug attachments for a project.

    Doing this efficiently required two modifications in Launchpad:

    • Exposing a bug collection for a particular product
    • Implement a attachment search method for a particular bug (this branch)
    Proposed fix

    Export a read operation findAttachment in IBug which returns a collection of IBugAttachment.

    Pre-implementation Notes
    • The search operation is essentially a text-only grep.
    Implementation Details
    • Zope configuration had to be updated to export the method
    • Multi-line searches break the GET request with a “HTTP/1.1 400 Bad Request” response
    • The attachments are all opened as regular file objects and then searched for the target pattern.
    Tests
    bin/test -vv -m lp.bugs.tests.test_bug_find_attachment
    Demo and Q/A

    Open any Launchpad bug in a browser:

    	https://launchpad.dev/bugs/15

    Create an attachment and upload any text file containing the string ‘char buf’.

    Create a Launchpad instance:

    	>>> from launchpadlib.launchpad import Launchpad
    	>>> launchpad = Launchpad.login_with('just testing', 'https://api.launchpad.dev/beta/')
    	The authorization page:
    	   (https://launchpad.dev/+authorize-token?oauth_token=ppd2bRZDnN6VX94lRqrq)
    	should be opening in your browser. After you have authorized
    	this program to access Launchpad on your behalf you should come
    	back here and press <Enter> to finish the authentication process.

    Load the bug:

    	>>> bug = launchpad.bugs[15]

    Search for the attachment containing the string ‘char buf’:

    	>>> results = bug.findAttachments(text=u'char buf')
    	>>> for attachment in results:
    	...     print attachment.title
    	... 
    	Buffer Overflow Intro.txt
    lint

    The changes are lint clean.

Minor Tasks

Around 8 hours were spent trying to find out why on earth my API calls weren’t getting recognized by apidoc or launchpadlib. In the end, Leonard Richardson pointed out that I was missing a make clean for regenerating the WADL cache. Can’t say it feels great to know that overlooking a build command resulted in so many hours of thorough debugging.

I had another issue with Librarian not recognizing the attachment files created during unittests. Luckily it was resolved quickly when Tim Penhey suggested that I need to commit the local transaction in order for the Librarian to see it.

A minor bug with my previous commits was also fixed which resulted in TypeErrors since the arguments weren’t being sent as Unicode strings.

Actions for the Following Report

As outlined earlier on, “Team -> Source Packages -> Bugs -> Attachments” search is the next target.

Tags: , , , , , , , , ,

June 3, 2010

Summer of Code Progress: Bug collections and attachment search

Filed under: Blog — krkhan @ 7:37 pm

Related Links

Summer of Code Archive Inspirated Code
Report Guidelines Ubuntu Wiki
Original Proposal Ubuntu Wiki

Time Spent

11 hours.

Highlights

Storm is by far the most intuitive way I have ever dealt with relational databases. Because of its pretty integration with Zope in Launchpad, I was able to make modifications to the core in a brief, consistent and non-disruptive way.

Concerns

While searching each attachment on the remote server works uber-fast, I don’t think it’ll be easy to have such a feature merged upstream. It may be argued that going through each file at the server side produces unnecessary load. In order to address such concerns I’ll have to generate relevant statistics after testing the changes extensively.

I still don’t understand Launchpad’s branch structure in detail. I’ve noticed that my branches are forked from devel-db instead of devel. I’m confused about the implications this will have on merge proposals and will consult Bryce about it tonight.

Waiting Items

None.

Stalled Items

None.

Accomplishments

  • Patch: Exported bug collection for product.
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    
    <rest-client version="2.3">
    <request>
    <http-version>1.1</http-version>
    <URL>https://api.launchpad.dev/beta/firefox/bugs</URL>
    <method>GET</method>
    <ssl-truststore>/home/krkhan/.keystore</ssl-truststore>
    <ssl-truststore-password>rO0ABXQAB2tyYWNrM2Q=</ssl-truststore-password>
    <ssl-hostname-verifier>ALLOW_ALL</ssl-hostname-verifier>
    <headers>
    <header key="Authorization" value="OAuth realm="OAuth", oauth_nonce="FFFF", oauth_timestamp="1275040360", oauth_consumer_key="just%20testing", oauth_signature_method="PLAINTEXT", oauth_version="1.0", oauth_token="MzwdT4XCNCV2D7m9dK5f", oauth_signature="%260RsqWntpzJWLRnLJV8ML5FBw5700d8qVFFRrNP7mvpbLn2mDCX1DTQXRHTGgPBfx34qMn70fZ4r0DDHM""/>
    <header key="Accept" value="application/json"/>
    <header key="Host" value="api.launchpad.dev"/>
    </headers>
    </request>
    </rest-client>
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    
    <rest-client version="2.3">
    <response>
    <execution-time>1055</execution-time>
    <status code="200">HTTP/1.1 200 Ok</status>
    <headers>
    <header key="Date" value="Thu, 03 Jun 2010 13:08:31 GMT"/>
    <header key="Server" value="zope.server.http (DebugLayerHTTP)"/>
    <header key="X-Powered-By" value="Zope (www.zope.org), Python (www.python.org)"/>
    <header key="Content-Type" value="application/json"/>
    <header key="Content-Length" value="8888"/>
    <header key="Vary" value="Accept,Accept-Encoding"/>
    <header key="Via" value="1.1 launchpad.dev"/>
    <header key="Keep-Alive" value="timeout=15, max=100"/>
    <header key="Connection" value="Keep-Alive"/>
    </headers>
    <body>
    {
       total_size         : 4,
       start              : 0,
       resource_type_link : 'https://api.launchpad.dev/beta/#bug-page-resource',
       entries            : [
          {
             users_unaffected_collection_link          : 'https://api.launchpad.dev/beta/bugs/1/users_unaffected',
             latest_patch_uploaded                     : null,
             users_affected_count_with_dupes           : 0,
             security_related                          : false,
             private                                   : false,
             bug_watches_collection_link               : 'https://api.launchpad.dev/beta/bugs/1/bug_watches',
             date_made_private                         : null,
             linked_branches_collection_link           : 'https://api.launchpad.dev/beta/bugs/1/linked_branches',
             subscriptions_collection_link             : 'https://api.launchpad.dev/beta/bugs/1/subscriptions',
             number_of_duplicates                      : 0,
             id                                        : 1,
             users_unaffected_count                    : 0,
             title                                     : 'Firefox does not support SVG',
             name                                      : null,
             http_etag                                 : '"04c8bb0158a201216e08c5d57e91211a2fa81843-fbd7bf01b3db545656a06d258acd96d362499c81"',
             messages_collection_link                  : 'https://api.launchpad.dev/beta/bugs/1/messages',
             self_link                                 : 'https://api.launchpad.dev/beta/bugs/1',
             who_made_private_link                     : null,
             attachments_collection_link               : 'https://api.launchpad.dev/beta/bugs/1/attachments',
             resource_type_link                        : 'https://api.launchpad.dev/beta/#bug',
             date_last_updated                         : '2006-05-19T06:37:40.344941+00:00',
             description                               : 'Firefox needs to support embedded SVG images, now that the standard has been finalised.\n\nThe SVG standard 1.0 is complete, and draft implementations for Firefox exist. One of these implementations needs to be integrated with the base install of Firefox. Ideally, the implementation needs to include support for the manipulation of SVG objects from JavaScript to enable interactive and dynamic SVG drawings.',
             duplicates_collection_link                : 'https://api.launchpad.dev/beta/bugs/1/duplicates',
             tags                                      : [],
             message_count                             : 2,
             heat                                      : 0,
             bug_tasks_collection_link                 : 'https://api.launchpad.dev/beta/bugs/1/bug_tasks',
             cves_collection_link                      : 'https://api.launchpad.dev/beta/bugs/1/cves',
             users_affected_with_dupes_collection_link : 'https://api.launchpad.dev/beta/bugs/1/users_affected_with_dupes',
             duplicate_of_link                         : null,
             users_affected_count                      : 0,
             owner_link                                : 'https://api.launchpad.dev/beta/~name12',
             date_created                              : '2004-01-01T20:58:04.553583+00:00',
             can_expire                                : false,
             date_last_message                         : null,
             users_affected_collection_link            : 'https://api.launchpad.dev/beta/bugs/1/users_affected'
          },
          <!-- entries omitted -->
       ]
    }
    </body>
    </response>
    </rest-client>
  • Patch: Added findAttachment method for bugs.
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    
    <rest-client version="2.3">
    <request>
    <http-version>1.1</http-version>
    <URL>
    https://api.launchpad.dev/beta/bugs/15?ws.op=findAttachments&amp;text=char%20buf
    </URL>
    <method>GET</method>
    <ssl-truststore>/home/krkhan/.keystore</ssl-truststore>
    <ssl-truststore-password>rO0ABXQAB2tyYWNrM2Q=</ssl-truststore-password>
    <ssl-hostname-verifier>ALLOW_ALL</ssl-hostname-verifier>
    <headers>
    <header key="Authorization" value="OAuth realm="OAuth", oauth_nonce="FFFF", oauth_timestamp="1275040360", oauth_consumer_key="just%20testing", oauth_signature_method="PLAINTEXT", oauth_version="1.0", oauth_token="MzwdT4XCNCV2D7m9dK5f", oauth_signature="%260RsqWntpzJWLRnLJV8ML5FBw5700d8qVFFRrNP7mvpbLn2mDCX1DTQXRHTGgPBfx34qMn70fZ4r0DDHM""/>
    <header key="Accept" value="application/json"/>
    <header key="Host" value="api.launchpad.dev"/>
    </headers>
    </request>
    </rest-client>
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    
    <rest-client version="2.3">
    <response>
    <execution-time>698</execution-time>
    <status code="200">HTTP/1.1 200 Ok</status>
    <headers>
    <header key="Date" value="Thu, 03 Jun 2010 13:08:10 GMT"/>
    <header key="Server" value="zope.server.http (DebugLayerHTTP)"/>
    <header key="X-Powered-By" value="Zope (www.zope.org), Python (www.python.org)"/>
    <header key="Content-Type" value="application/json"/>
    <header key="Content-Length" value="553"/>
    <header key="Vary" value="Accept,Accept-Encoding"/>
    <header key="Via" value="1.1 launchpad.dev"/>
    <header key="Keep-Alive" value="timeout=15, max=100"/>
    <header key="Connection" value="Keep-Alive"/>
    </headers>
    <body>
    {
       total_size : 1,
       start      : 0,
       entries    : [
          {
             title              : 'Buffer Overflow Intro.txt',
             bug_link           : 'https://api.launchpad.dev/beta/bugs/15',
             data_link          : 'https://api.launchpad.dev/beta/bugs/15/+attachment/2/data',
             http_etag          : '"0edda8c433532e60707d34a322ce808dbc508ef1-2e61275e2b0bd7cc15cc6322c5fd26c66272dfa6"',
             message_link       : 'https://api.launchpad.dev/beta/redfish/+bug/15/comments/8',
             type               : 'Unspecified',
             self_link          : 'https://api.launchpad.dev/beta/bugs/15/+attachment/2',
             resource_type_link : 'https://api.launchpad.dev/beta/#bug_attachment'
          }
       ]
    }
    </body>
    </response>
    </rest-client>

Minor Tasks

To simplify the task of debugging Launchpad, I gave my Ubuntu VM remote-access and edited Apache configuration accordingly. Now I can just make changes to the sources using NFS mounted branches and test them in my host browser using launchpad.dev.

Actions for the Following Report

I haven’t yet decided whether I should focus on getting these changes merged into trunk yet or instead move on to launchpadlib and arsenal aspect of my project. Guess I’ll just add the matter to the list of items I need to discuss during tonight’s IRC meeting.

Tags: , , , , , , , , ,

May 28, 2010

Summer of Code Progress: REST collections

Filed under: Blog — krkhan @ 6:44 pm

Starting from this post, I will be documenting my SoC progress according to the Ubuntu guidelines. This will enable me to log my activities in an organized manner. The inaugural report:

Related Links

Summer of Code Archive Inspirated Code
Report Guidelines Ubuntu Wiki
Original Proposal Ubuntu Wiki

Time Spent

8 hours.

Highlights

This is my first experience of developing on a Zope project. The fact that the project is already deployed in a large-scale production environment is daunting as well as exciting at the same time. For Launchpad work, I learned only the bare essentials of Zope, i.e., things that are directly related with my project such as interfaces and adapters. After SoC however, I plan on exploring it in more detail.

Concerns

Testing Launchpad modifications is a beast of a task. After making each set of changes to the code, I have to stop the server and relaunch it again which takes at least 30 seconds. It becomes especially frustrating when I’m in the process of making tiny changes in order to explore the API.

I’ve tried to work around this issue by using pdb.set_trace(). Still, it’s no substitute for being able to edit whole files in one go.

Waiting Items

None.

Stalled Tasks

None.

Accomplishments

  • Test locally running Launchpad with a RESTClient.
  • Go through the code-base and figure out the portions which deal with API services.
  • Export a bug collection for every project:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    
    <rest-client version="2.3"><request>
    <http-version>1.1</http-version>
    <URL>https://api.launchpad.dev/beta/alsa-utils/bugs</URL>
    <method>GET</method>
    <ssl-truststore>/home/inspirated/.keystore</ssl-truststore>
    <ssl-truststore-password>rO0ABXQAB2tyYWNrM2Q=</ssl-truststore-password>
    <ssl-hostname-verifier>ALLOW_ALL</ssl-hostname-verifier>
    <headers>
    <header key="Authorization" value="OAuth realm="OAuth", oauth_nonce="FFFF", oauth_timestamp="1275040360", oauth_consumer_key="just%20testing", oauth_signature_method="PLAINTEXT", oauth_version="1.0", oauth_token="0NWl33zGj3QDtxFWdHCt", oauth_signature="%261XP9XCptHQ3S8g49RjFTr3xFlrK0H73Z5QjbGbxs1pMTc5mzptq0Z5Qs93gjcXDQtqlgFSVVQBPLdK47""/>
    <header key="Accept" value="application/json"/>
    <header key="Host" value="api.launchpad.dev"/>
    </headers>
    </request>
    </rest-client>
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    
    <rest-client version="2.3">
    <response>
    <execution-time>1833</execution-time>
    <status code="200">HTTP/1.1 200 Ok</status>
    <headers>
    <header key="Date" value="Fri, 28 May 2010 12:50:18 GMT"/>
    <header key="Server" value="zope.server.http (DebugLayerHTTP)"/>
    <header key="X-Powered-By" value="Zope (www.zope.org), Python (www.python.org)"/>
    <header key="Content-Type" value="application/json"/>
    <header key="Content-Length" value="10736"/>
    <header key="Vary" value="Accept,Accept-Encoding"/>
    <header key="Via" value="1.1 launchpad.dev"/>
    <header key="Keep-Alive" value="timeout=15, max=100"/>
    <header key="Connection" value="Keep-Alive"/>
    </headers>
    <body>
    {
       total_size           : 15,
       start                : 0,
       resource_type_link   : 'https://api.launchpad.dev/beta/#bug-page-resource',
       next_collection_link : 'https://api.launchpad.dev/beta/alsa-utils/bugs?ws.start=5&amp;ws.size=5',
       entries              : [
          {
             users_unaffected_collection_link          : 'https://api.launchpad.dev/beta/bugs/15/users_unaffected',
             latest_patch_uploaded                     : null,
             users_affected_count_with_dupes           : 0,
             security_related                          : false,
             private                                   : false,
             bug_watches_collection_link               : 'https://api.launchpad.dev/beta/bugs/15/bug_watches',
             date_made_private                         : null,
             linked_branches_collection_link           : 'https://api.launchpad.dev/beta/bugs/15/linked_branches',
             subscriptions_collection_link             : 'https://api.launchpad.dev/beta/bugs/15/subscriptions',
             number_of_duplicates                      : 0,
             id                                        : 15,
             users_unaffected_count                    : 0,
             title                                     : 'Nonsensical bugs are useless',
             name                                      : null,
             http_etag                                 : '"23e175849b60b27f5553ddb56995e044c71d745a-ce4f09c837a78ed5fdf23f759020058f3423a804"',
             messages_collection_link                  : 'https://api.launchpad.dev/beta/bugs/15/messages',
             self_link                                 : 'https://api.launchpad.dev/beta/bugs/15',
             who_made_private_link                     : null,
             attachments_collection_link               : 'https://api.launchpad.dev/beta/bugs/15/attachments',
             resource_type_link                        : 'https://api.launchpad.dev/beta/#bug',
             date_last_updated                         : '2007-12-18T16:31:34.972893+00:00',
             description                               : 'Like this one, natch.',
             duplicates_collection_link                : 'https://api.launchpad.dev/beta/bugs/15/duplicates',
             tags                                      : [],
             message_count                             : 7,
             heat                                      : 0,
             bug_tasks_collection_link                 : 'https://api.launchpad.dev/beta/bugs/15/bug_tasks',
             cves_collection_link                      : 'https://api.launchpad.dev/beta/bugs/15/cves',
             users_affected_with_dupes_collection_link : 'https://api.launchpad.dev/beta/bugs/15/users_affected_with_dupes',
             duplicate_of_link                         : null,
             users_affected_count                      : 0,
             owner_link                                : 'https://api.launchpad.dev/beta/~name16',
             date_created                              : '2007-12-18T16:30:19.103679+00:00',
             can_expire                                : false,
             date_last_message                         : '2007-12-18T16:31:34.790641+00:00',
             users_affected_collection_link            : 'https://api.launchpad.dev/beta/bugs/15/users_affected'
          },
          <!-- entries omitted -->
          {
             users_unaffected_collection_link          : 'https://api.launchpad.dev/beta/bugs/11/users_unaffected',
             latest_patch_uploaded                     : null,
             users_affected_count_with_dupes           : 0,
             security_related                          : false,
             private                                   : false,
             bug_watches_collection_link               : 'https://api.launchpad.dev/beta/bugs/11/bug_watches',
             date_made_private                         : null,
             linked_branches_collection_link           : 'https://api.launchpad.dev/beta/bugs/11/linked_branches',
             subscriptions_collection_link             : 'https://api.launchpad.dev/beta/bugs/11/subscriptions',
             number_of_duplicates                      : 0,
             id                                        : 11,
             users_unaffected_count                    : 0,
             title                                     : 'Make Jokosher use autoaudiosink',
             name                                      : null,
             http_etag                                 : '"8675b8d2649cf0c08d58681036ef1e8bb6a20e78-d7d969f7ed8c2f2def4d9e429f23ac41f7fec4dc"',
             messages_collection_link                  : 'https://api.launchpad.dev/beta/bugs/11/messages',
             self_link                                 : 'https://api.launchpad.dev/beta/bugs/11',
             who_made_private_link                     : null,
             attachments_collection_link               : 'https://api.launchpad.dev/beta/bugs/11/attachments',
             resource_type_link                        : 'https://api.launchpad.dev/beta/#bug',
             date_last_updated                         : '2007-03-15T20:37:51.603369+00:00',
             description                               : 'I\'ve had problems when switching from Jokosher to Totem to play an Ogg.\n\nTotem appears to be playing normally but does not produce any sound.\nIf I close Jokosher then you can hear totem.\n\nI\'ve also had a problem when trying to playback sound within jokosher\nafter switch from another app (I think it was xmms). I get a dialog\nwith the following text:\n\n     Argh! Something went wrong and a serious error occurred:\n\n     Resource busy or not available.\n\n     gstalsasink.c(636): gst_alsasink_open (): /timeline/playbackbin/\n     alsasink0:\n     Device \'hw:0\' is busy\n\nAfter closing and reopening Jokosher, i got the same error again.\nAfter logging out of gnome and logging in again - I still got the same\nerror message.\n\nI had to restart!\n\nIs there some way to reset alsa or the device driver - with out having\nto restart?\n\nRunning on Ubuntu - with Jokosher 0.2 runscript.',
             duplicates_collection_link                : 'https://api.launchpad.dev/beta/bugs/11/duplicates',
             tags                                      : [],
             message_count                             : 7,
             heat                                      : 0,
             bug_tasks_collection_link                 : 'https://api.launchpad.dev/beta/bugs/11/bug_tasks',
             cves_collection_link                      : 'https://api.launchpad.dev/beta/bugs/11/cves',
             users_affected_with_dupes_collection_link : 'https://api.launchpad.dev/beta/bugs/11/users_affected_with_dupes',
             duplicate_of_link                         : null,
             users_affected_count                      : 0,
             owner_link                                : 'https://api.launchpad.dev/beta/~name16',
             date_created                              : '2007-03-15T20:33:56.678930+00:00',
             can_expire                                : false,
             date_last_message                         : null,
             users_affected_collection_link            : 'https://api.launchpad.dev/beta/bugs/11/users_affected'
          }
       ]
    }
    </body>
    </response>
    </rest-client>

Minor Tasks

Having gotten stuck on the task of running RESTClient against my Launchpad instance, I ran into a few issues such as RESTClient’s refusal to accept the local SSL certificate and confusion regarding domain names. The former issue was solved by using keytool to create a trusted keystore while the latter was addressed by using correct domain names as pointed out by William Grant.

Actions for the Following Report

While I have managed to export a web service property for Launchpad projects, I haven’t yet figured out a way to group bugs by projects. This will require a clear understanding of the schemas involved as well as some skills with SQL. My next task therefore will be to export only the bugs that are related to a particular project.

Tags: , , , , , , , ,

May 22, 2010

Summer of Code Progress: Debugging Launchpad’s RESTful API

Filed under: Blog — krkhan @ 8:03 pm

Related Links

Summer of Code Archive Inspirated Code
Original Proposal Ubuntu Wiki

Report

RESTful web services are very fun once they get properly integrated in end-user applications. Testing those services is a whole different thing and I wanted to dive deep into the bare essentials of REST communication performed by Launchpad for its API. The most straightforward way was to write a Python script using urllib2 and/or httplib for the HTTP request/response mantra. Obviously I found it to be a cumbersome solution and then found this little beauty:

RESTClient Screenshot
(Click on the thumbnail for larger version.)

RESTClient does one thing and does it sweetly: it gives you a nice GUI for toying with REST requests. Getting it to work with Launchpad was not as straightforward though as all requests have to be signed to be of any real value. I also had to spent a few hours trying to understand why api.edge.launchpad.net was not recognizing my access tokens until by sheer stroke of luck I used api.staging.launchpad.net and it started working perfectly.

1
2
3
4
5
6
7
8
9
10
11
12
<rest-client version="2.3">
<request>
<http-version>1.1</http-version>
<URL>https://api.staging.launchpad.net/beta/bugs/1</URL>
<method>GET</method>
<headers>
<header key="Authorization" value="OAuth realm="OAuth", oauth_nonce="77848601", oauth_timestamp="1274537900", oauth_consumer_key="just%20testing", oauth_signature_method="PLAINTEXT", oauth_version="1.0", oauth_token="6dwHFn3CPzxjNrrdp3r9", oauth_signature="%26xxxxxx""/>
<header key="Accept" value="application/json"/>
<header key="Host" value="api.staging.launchpad.net"/>
</headers>
</request>
</rest-client>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
<rest-client version="2.3">
<response>
<execution-time>897</execution-time>
<status code="200">HTTP/1.1 200 Ok</status>
<headers>
<header key="Date" value="Sat, 22 May 2010 14:26:32 GMT"/>
<header key="Server" value="zope.server.http (HTTP)"/>
<header key="X-Powered-By" value="Zope (www.zope.org), Python (www.python.org)"/>
<header key="Content-Type" value="application/json"/>
<header key="Content-Length" value="2888"/>
<header key="Etag" value=""d9b357c932ada0a71e96401a8c87368d4704a500-dec69b947b887c4addba8eb4aae5ca64c94f616c""/>
<header key="Vary" value="Accept,Accept-Encoding"/>
<header key="Via" value="1.1 wildcard.staging.launchpad.net"/>
<header key="Keep-Alive" value="timeout=15, max=100"/>
<header key="Connection" value="Keep-Alive"/>
</headers>
<body>
{
   users_unaffected_collection_link          : 'https://api.staging.launchpad.net/beta/bugs/1/users_unaffected',
   latest_patch_uploaded                     : null,
   users_affected_count_with_dupes           : 378,
   security_related                          : false,
   private                                   : false,
   bug_watches_collection_link               : 'https://api.staging.launchpad.net/beta/bugs/1/bug_watches',
   date_made_private                         : null,
   linked_branches_collection_link           : 'https://api.staging.launchpad.net/beta/bugs/1/linked_branches',
   subscriptions_collection_link             : 'https://api.staging.launchpad.net/beta/bugs/1/subscriptions',
   number_of_duplicates                      : 0,
   id                                        : 1,
   users_unaffected_count                    : 6,
   title                                     : 'Microsoft has a majority market share',
   name                                      : 'liberation',
   http_etag                                 : '"d9b357c932ada0a71e96401a8c87368d4704a500-dec69b947b887c4addba8eb4aae5ca64c94f616c"',
   messages_collection_link                  : 'https://api.staging.launchpad.net/beta/bugs/1/messages',
   self_link                                 : 'https://api.staging.launchpad.net/beta/bugs/1',
   who_made_private_link                     : null,
   attachments_collection_link               : 'https://api.staging.launchpad.net/beta/bugs/1/attachments',
   resource_type_link                        : 'https://api.staging.launchpad.net/beta/#bug',
   date_last_updated                         : '2010-05-13T23:24:26.362094+00:00',
   description                               : 'Microsoft has a majority market share in the new desktop PC marketplace.\nThis is a bug, which Ubuntu is designed to fix.\n\nNon-free software is holding back innovation in the IT industry, restricting access to IT to a small part of the world\'s population and limiting the ability of software developers to reach their full potential, globally. This bug is widely evident in the PC industry.\n\nSteps to repeat:\n\n1. Visit a local PC store.\n\nWhat happens:\n2. Observe that a majority of PCs for sale have non-free software pre-installed.\n3. Observe very few PCs with Ubuntu and free software pre-installed.\n\nWhat should happen:\n1. A majority of the PCs for sale should include only free software like Ubuntu.\n2. Ubuntu should be marketed in a way such that its amazing features and benefits would be apparent and known by all.\n3. The system shall become more and more user friendly as time passes.\n\n',
   duplicates_collection_link                : 'https://api.staging.launchpad.net/beta/bugs/1/duplicates',
   tags                                      : [
      'iso-testing',
      'ubuntu'
   ],
   message_count                             : 1198,
   heat                                      : 2138,
   bug_tasks_collection_link                 : 'https://api.staging.launchpad.net/beta/bugs/1/bug_tasks',
   cves_collection_link                      : 'https://api.staging.launchpad.net/beta/bugs/1/cves',
   users_affected_with_dupes_collection_link : 'https://api.staging.launchpad.net/beta/bugs/1/users_affected_with_dupes',
   duplicate_of_link                         : null,
   users_affected_count                      : 378,
   owner_link                                : 'https://api.staging.launchpad.net/beta/~sabdfl',
   date_created                              : '2004-08-20T00:00:00+00:00',
   can_expire                                : false,
   date_last_message                         : '2010-05-13T23:24:21.253673+00:00',
   users_affected_collection_link            : 'https://api.staging.launchpad.net/beta/bugs/1/users_affected'
}
</body>
</response>
</rest-client>

With a local Launchpad ready to do my bidding and a debugging process in place for elaborating its API, I’m looking forward to the coding phase. Which, coincidentally, begins on the same day I give my final exam for the current semester.

Rest REST: the sweet sauce of labor.” — Plutarch

Tags: , , , , , , , , ,
« Previous PageNext Page »