development

Chandler 0.7.4 released

February 14th, 2008 Comments Off

I’m not being paid by OSAF any more, but I’m still following Chandler and am poking at the code (it’s what open source is all about).

Version 0.7.4 of Chandler has been released.

Mac Tiger Intel
Mac Tiger PowerPC
Mac Leopard
Windows
Linux tar.gz
Ubuntu Linux .deb

With Leopard and Ubuntu Chandler is now using the OS’s version of Python, which is interesting.

Chandler 0.7.3 released

December 7th, 2007 No Comments »

Chandler 0.7.3, sporting the month view

It’s official, version 0.7.3 is out. Get your Intel Tiger or Leopard version, or the PPC build.

Let’s not forget those on Windows or Ubuntu 7.10.

Be sure to check out the Release notes!

svn merge problems

October 23rd, 2007 2 Comments »

I seem to be having problems merging from the trunk to the multiweekview branch. Specifically, a file that should be there on the branch, isn’t.

The last time I merged trunk changes into the multiweekview branch was at r15506, when I merged from r15455 through 15504. That went sort of ok (I did the commit of the merge from the multiweekview/chandler directory, and not the multiweekview directory, so I had to do two commits. Oops.).

In r15531, Bryan Stearns checked in a change that deleted a bunch of files and created a bunch of new ones that replaced them. I know, because I reviewed the patch! (r=rae).

I will choose one file as an example: chandler/Chandler.egg-info/resources/images/TriageNow.png. This file existed in r15530, but it was gone in 15531.

On the multiweekview branch, TriageNow.png continued to live, since we had not merged in Bryan’s change. Recently, I wanted to merge in that change, along with all the others that have happened since the last merge. So I executed this command:

% svn merge -r 15505:15564 svn+ssh://svn.osafoundation.org/svn/chandler/trunk

This seemed to work well, until the final line of output from svn, where it prints a series of dots “………….”. My computer hung for about ½ hour.

It was Friday, and I had to get going, and I didn’t want to block Jeffrey from checking in his changes to the branch, so I hit control-C to kill the commit and told Jeffrey to go ahead, resigning myself to figuring it all out later.

Alas, later has lasted a long time.

When I did an svn diff to check how things were, svn said there were no changes. That was a bit strange, since the commit hadn’t finished. Or, maybe it had?

I had checked in all my changes to the branch before merging, so there was no uncommitted work on the branch hanging around, so I was able to freely deleted the entire directory and check out the branch again. After checkout I ran Chandler, and I saw a few changes from the trunk[!]. Specifically, the new wx toolbar appearance.

This was weird. I checked the svn logs as mailed out and didn’t see any record of my merge being committed. Again, “svn diff” reported nothing. So, I did the merge again, same command line as before. This caused output that had errors in it:

A    chandler/Chandler.egg-info/resources/images/Triage.Now.Rollover.Right.png
A    chandler/Chandler.egg-info/resources/images/Markup.Later.StampedRollover.Left.png
A    chandler/Chandler.egg-info/resources/images/Triage.Done.Left.png
A    chandler/Chandler.egg-info/resources/images/Markup.Done.StampedPressed.Right.png
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/MarkupNowStampedPressed.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/MarkupNowStamped.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/TriageLater.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/TriageDone.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/MarkupNowStampedRollover.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/TriageLaterRollover.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/TriageDoneRollover.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/MarkupLaterStampedPressed.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/TriageNowMousedown.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/MarkupDoneStampedPressed.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/MarkupDoneStamped.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/MarkupDoneStampedRollover.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/MarkupLaterStamped.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/TriageDoneMousedown.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/MarkupLaterStampedRollover.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/TriageLaterMousedown.png'
Skipped missing target: 'chandler/Chandler.egg-info/resources/images/TriageNow.png‘
Skipped missing target: ‘chandler/Chandler.egg-info/resources/images/TriageNowRollover.png’
A    chandler/util/triagebuttonimageprovider.py

What this is saying is that svn expected there to be files there, including our trusty TriageNow.png, but they weren’t there. Now, since these files were deleted in r15531, svn should be able to handle that as part of the merge. So I’m thinking the svn database is a bit screwed up.

Then I hear from Bear that my Friday-night-never-ending svn process is still running on the server. He kills it for me, and I try again. Exact same results.

So I **could** try adding the above files myself and see if that makes svn happy. Or maybe we need to do some svn admin work to fix things. I’m not sure.

Chandler Preview is out!

September 12th, 2007 No Comments »

Be sure to get a copy and give it a whirl:

Intel Mac OS X
PPC Mac OS X
Windows
Linux

“repository view” is just a database ref

July 28th, 2007 No Comments »

Something that I find often confuses code I am looking at is the use in Chandler of the term ‘view’ to refer to a database cursor/reference. In most apps out there, ‘view’ refers to a very specific piece of UI.

I wish we could globally substitute something like ‘dbRef’ wherever we use ‘view’ in Chandler’s code. It would make reading the code a lot easier clearer.

And actually, while we’re at it, replace all uses of ‘repository’ with ‘repo’. ‘repository’ doesn’t mean very muchand ‘repo’ stands out more.

Ya, I seem to be in a complaining mood.

Chandler migration - happy surprise

May 22nd, 2007 No Comments »

I contributed to the Chandler migration testing we were doing today and ran across a happy surprise. The idea of the exercise was to migrate data from Chandler 0.6alpha4 to the latest checkpoint build. In the course of this, I exported my calendars both as .ics files and via sharing on the server.

When I loaded up the latest chandler, I first connected to the server and made sure I could see all my calendars, which I could, and then I imported one of the .ics files. Lo and behold, it noticed that I already had the same items from the server and merged the items as they were read in. I ended up not with two copies of every item, but instead, each item was referenced by two calendars. (I had renamed the calendar before importing the .ics file to see if it would create a collection with the same name - and it did).

I was quite pleased at this bit of smarts, as you can tell!

Performance

April 26th, 2007 No Comments »

Last night before heading to bed, I left my Mac running two series of performance tests. One with my wx.GraphicsContext changes turned off, and another with them turned on. The command line was:

date
cp -f parcels/osaf/framework/blocks/calendar/CollectionCanvas{-off,}.py \
&& perftest off \
&& cp -f parcels/osaf/framework/blocks/calendar/CollectionCanvas{-on,}.py \
&& perftest on
date

The tests started at Thu Apr 26 02:59:25 EDT 2007 and finished at Thu Apr 26 05:28:06 EDT 2007. So they ran for about 2½ hours.

You don’t want to do this kind of thing during the day! This ended up being a long post with code, etc, so click to see more.

Performance hints from Andi

April 9th, 2007 No Comments »

This was sent to the chandler-dev list and I wanted to have a nicely-formatter version handy.

Profiling Chandler is tricky because there are lots of moving parts. Here are a few tips and tricks that come to mind:

To compare numbers, always use the same data. Recreating your repository everytime changes the data in subtle ways because you get new UUIDs everytime and the order of things, random in mappings, is conditioned by the hash of their keys, very often UUIDs. Instead, load and prepare your data, backup the repository and always restore it when re-running the testcase.
rc -r ~/Desktop/__repository__.001 (or whatever your path may be)
Preload all items, that way the vaguaries of item loading are out of the way, unless that’s what you’re interested in. Running check() loads all items into the UI view.
rc -r ~/Desktop/__repository__.001 --undo check
–undo check runs check() and undoes bogus versions until check() passes at startup time. If check() passes right away, as I’d expect from a clean restore, –undo will undo nothing and you will have started your chandler will all items pre-loaded.
The -P flag using the perf tests produces profiles that crash hotshot (at least, for me, on my Mac). Also, with -P you cannot be sure how much of the perf test cruft you’re profiling along with it. Know what you’re measuring, and add the profiling hook yourself.
Find the block of code or call you’re interested in and wrap it with a prof.runcall() call, creating a local function for the block of code if need be. For example, to profile import, I just change ImportExport.py around line 267 as follows:
import hotshot prof = hotshot.Profile(’import.log’) collection = prof.runcall(importICalendarFile, fullpath, self.view, coll, filterAttributes, activity, None, logger) prof.close()
Then process the resulting import.log file with the hotshot APIs as is done, for example, in my t82.py file included below. Really, how to process the hotshot output is up to you, whatever works best for the kind of output you’re looking for. t82.py, when run the first time on a .log file, produces a .stats file from it which is vastly faster to reload. This is can be a huge timesaver when profiling things like a large import, for example.
Remove commit() calls. Currently, our code includes commits() in less than ideal places and this has the effect of slowing things down considerably. commit() is an expensive operation that we’d like to do *between* user actions instead of *during* user actions. Measuring it during a user action is pointless unless it is critical to the action.
For example, the criticality of calling commit() at the end of CalendarCanvas.CreateEmptyEvent() is dubious. John has offered to review all such commit calls and remove them if possible, leaving it to the user event that started the action to do the commit, if needed.
Instead of only looking at the amount of time spent doing things, consider how many times they’re done. Currently, I sense that there are a lot of perf gains to be made in the app by doing things less instead of faster. Why are we loading 41 images when creating an event in the calendar, for example ? This leads to the next point.
The app loads things lazily. Items are loaded lazily. Images are loaded lazily. If you see 41 images loaded, the next thing is to profile creating a second event instead. Indeed, the calendar doesn’t load 41 images the second time. Run the action multiple times so that all the init cases are out of the way. But, we’re still calling MenusAndToolbars.appendDynamicBlocks 336 times nonetheless… something to look into there even if we’re only spending 0.007s there, the fact it’s called that often may lead you onto something where significant time is spent overall.
When communicating about profiles, don’t send the .log files around, they can be huge. Use a .stats file instead. As said before, the t82.py file below creates a .stats file from a .log file whenever it is invoked with a .log file. Don’t send a text version out either, the .stats file can be opened with hotshot APIs and queried in various ways to look for callers, callees, different sort orders, etc, etc. The Stats class has several interesting APIs to poke around a .stats file: docs.python.org/lib/profile-stats.html

That’s all I can think of for now. See my t82.py script below.

Andi..

import os
import hotshot, hotshot.stats, pstats

def cum(stats, length=50):
    stats.sort_stats('cum')
    stats.print_stats(length)

def calls(stats, length=50):
    stats.sort_stats('time', 'calls')
    stats.print_stats(length)

def show(file, call='cum', length=50):
    if file.endswith('stats'):
        stats = pstats.Stats(file)
    else:
        stats = hotshot.stats.load(file)
        stats.strip_dirs()
        stats.dump_stats('.'.join((os.path.splitext(file)[0], 'stats')))
    if call == 'cum':
        cum(stats, int(length))
    elif call == 'tot':
        calls(stats, int(length))
    else:
        stats.sort_stats(call)
        stats.print_stats(int(length))

    return stats

if __name__ == "__main__":
    from sys import argv
    show(*argv[1:])

Handy iCal utility

March 1st, 2007 No Comments »

Here’s a little csh script that puts properly-named ics versions of all your ical files on the desktop.

#!/bin/csh -f
set ical_sources_dir=~/Library/"Application Support"/iCal/Sources
foreach dir ("$ical_sources_dir"/*.calendar)
    if(! -r "$dir"/corestorage.ics) continue
    set cal_name=`defaults read "$dir"/Info Title`.ics
    if(-e ~/Desktop/"$cal_name") then
        echo "File ""'""~/Desktop/$cal_name""'"" exists, skipping"
    else
        ln "$dir"/corestorage.ics ~/Desktop/"$cal_name"
    endif
end

Update: Here’s the Python version.

#!/usr/bin/env python
from glob import glob
from os.path import expanduser, expandvars
from os import system, popen, link
from string import strip

ical_sources_dir=expanduser("~/Library/Application Support/iCal/Sources")
for dir in glob("%s/*.calendar" % ical_sources_dir):
    p = popen("defaults read \"%s\"/Info Title 2>/dev/null" % dir)
    cal_name = strip(p.readline()) + ".ics"
    p.close()
    if cal_name != "":
        try:
            link("%s/corestorage.ics" % dir, 
                "%s/%s" % (expanduser("~/Desktop/"), cal_name))
        except:
            pass

When I work on day-to-day Python scripts, I wonder if there are a lot of script writers who start every script with something like:

from sys import * from os import * from os.path import * from string import *

This would make writing Python more Perl-like in its availability of many functions through simple calls.

Conflicting Results

February 28th, 2007 No Comments »

I just made my first successful conflict in Chandler’s PyShell. Woo!

PyShell 0.9.5 - The Flakiest Python Shell Python 2.5 (r25:51908, Jan 31 2007, 21:59:50) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>uuid="27e5e1d4-53ad-40e4-860d-503a137a0806" >>>item = view.findUUID(uuid) >>>from osaf import sharing >>>sharing.SharedItem(item).add() Traceback (most recent call last): File "<input>", line 1, in <module> File "/Users/rae/work/osaf/chandler/chandler/parcels/osaf/pim/stamping.py", line 112, in add "Item %r already has stamp %r" % (self.itsItem, self) StampAlreadyPresentError: Item <Note: 27e5e1d4-53ad-40e4-860d-503a137a0806> already has stamp SharedItem(<Note: 27e5e1d4-53ad-40e4-860d-503a137a0806>) >>>sharing.SharedItem(item).generateConflicts() >>>item <Note: 27e5e1d4-53ad-40e4-860d-503a137a0806> >>>cstates = sharing.SharedItem(item).conflictingStates >>>cstates <DBRefList: It's a New Event.osaf.sharing.shares.SharedItem.conflictingStates<->conflictFor> >>>conflictsGenerator = sharing.SharedItem(item).getConflicts() >>>for conflict in conflictsGenerator: … f = conflict.field … v = conflict.value … p = conflict.peer … print "[%s]=%s (%s)" % (f, v, p) … [location]=San Jose (conflict@example.com) [title]=XYZZY (conflict@example.com) [body]=PLUGH (conflict@example.com) >>>

.. one small step for conflicts..

Chomping at Chandler