Making Our PHP LDAP Capabilities Generally Available
In our web client based on Roundcube, we have had a need to get very large LDAP directories to function as an address book. While "large" is a relatively subjective term, in terms of LDAP "large" simply means running out of time, or meeting the look-through or size limits. Additionally, of course, PHP itself is subject to memory size and maximum execution time restrictions.
We weren't the first to run into these problems, of course, but with a little help we came up with a very, very good solution. Using Virtual List View (VLV) control and Server-Side Sort (SSS) control, browsing very large address books page-by-page takes maybe a tenth of a second per page. Roundcube address book problem solved (thanks Thomas, thanks Robert!)
But are they complex?
VLV and SSS controls are both very complex (the former more than the latter), with configuration on the LDAP server required and client-side BER encoding magic involved. I am not a programmer, let alone traditionally educated, so to me the PHP code (or the code's capabilities, rather) resulting from this effort is simply awesome. To properly function, it requires a patch to PHP to be applied, which I've now submitted for upstream inclusion. Kolab Groupware ships its own PHP packages solely to include this patch. There's many other gotchas to take into account as well.
To use server-side sorting, you have to create an index on the LDAP server with a single attribute containing a list of attribute names using which entries can be sorted. When using LDAP from a client, you have to specify that exact same sort order.
To use Virtual List View controls, you require a successful hit on the server-side sorting (as that is the index). You also have to configure the LDAP server with a search (for which it requires a base, filter and scope). When using LDAP from a client, in addition to hitting the server-side sorting, you have to hit those three exact same search parameters as well.
That may not sound so difficult...
Complications with the User Interface Context
However, an LDAP client like Roundcube lists and/or searches differently based on the context.
One context is the address book. Typically one would browse an addtress book sorted by "Surname, Givenname" or "Givenname Surname". These one or two attributes (displayName can be put to very good use here) would then be included in the server side sorting. But then again, who sensibly browses a very large address book page by page? A 1000 entries with a page size of 40 results in 25 pages already - address book sizes I speak of when I say "large" are literally hundreds of thousands of entries.
So the keyword is searching - but searching is a complex realm in and by itself. If you were allowed to search only on the attributes included in a SSS and VLV, searching is virtually useless (all puns intended). Searching would basically allow you to list VLV entries starting with the first result your search had found.
You would like to search by location, phone number, department, organization, email address and other attributes. This is difficult to do efficiently, but can be done. The show-case is in what I'm sort of announcing in this blog post.
After your search results are in, you would still like them sorted somehow. If you would like to know who lives in Narnia and search for it, your results should not look like so:
but like so (for example):
- Doe, Jane
- Doe, John
A second context is auto-complete. In it's very nature, auto-completion is always a search (and not simply a paginated list of all entries), similar to the search for people living in Narnia. You can imagine how a set of search results could need to be paginated if there were many people living in Narnia.
Furthermore, in the user interface context where you use auto-complete (the compose window, the ACL entry management on folders), no address book interface pop-up is available, as only a small list with hits is displayed. The pages are much smaller, increasing the change the result set needs pagination.
Naturally, auto-completion searches for a limited set of attributes - but also most likely different attributes than an address book listing. Most likely, a search for the purpose of auto-completion looks at a display name ("Doe, J. (Engineer)", perhaps), a given name ("Jane", "John"), a surname ("Doe"), any email address (mail, alias, mailalternateaddress, mailforwardingaddress attributes), etc.
Imagine these searches would occur without the help of pre-sorted indexes. A million-and-one LDAP entries may need to be searched, all results would need to be obtained, the results would then need to be sorted, to then return only the first 10-15 entries. Wouldn't it be great if you could search, sort and paginate with the help of pre-sorted indexes?
But... I did mention earlier, the three magic search parameters need to match the server-side VLV search configured. As it turns out you can actually specify additional search parameters, but only in a very specific way. More awesome stuff I'm announcing in this blog post.
And all of this is only in Roundcube?
Within the Kolab Groupware realm, we have more components that hook into LDAP and perform listings and searches. Two of them are the Kolab Web Administration Panel (or actually, its API) and Syncroton.
So I've had multiple agenda's to attend to; 1) avoid needing to do actual programming for the Web Administration Panel, 2) avoid requiring Roundcube libraries with both the WAP and Syncroton and 3) further enhance our LDAP capabilities for yet even more awesome stuff.
I give to you Net_LDAP3 - not yet a PHP PEAR module, because it's a work in progress, and I don't like their license requirements and I don't like their function naming conventions (camel-case, basically).
My target is to integrate all of the LDAP functionallity required with Roundcube, our Kolab plugins for Roundcube, the WAP and Syncroton.
I've already achieved the following milestones:
- Unless specifically configured, or specifically disabled, or not a supported control on the LDAP server, the Net_LDAP3 module automatically discovers available settings for VLV/SSS listings (method list_entries())
- A method search_entries() with an argument for additional search parameters that automatically uses VLV/SSS configured or discovered but with the additional filter settings,
- Validation of the desired sorting,
- Automatic selection of one sorting configuration out of available sorting configurations,
- Allow a method login() to find a user entry before binding as the user being logged in,
- The execution of a registered hook to obtain or set configuration items using a routine external to Net_LDAP3,
- The execution of a registered hook to use for logging,
- Trace and debug levels
Obviously there's still many things on my TODO list, which ends with patching Roundcube, the WAP and Syncroton to use this module.