Monthly Archives: December 2010

The Future of Sharing (Genealogical Data)

It’s no secret that the current standard for sharing genealogical data, GEDCOM, is woefully out of date. The last official revision to the GEDCOM standard, 5.5, was completed in 1996. A minor update, 5.5.1, was released in 1999 but never officially approved (even though some of its provisions have been adopted by various genealogy programs). Revision 5.5.1 added one very important feature – support for UTF-8 character encoding, which is a form of Unicode, which support multiple character sets (including, for example, Hebrew).

GEDCOM has, for all intents and purposes, been abandoned by the Church of Latter Day Saints (the Mormons) which created and owns the standard. The church has indicated that they will not be updating it, and indeed are replacing the need for it with a new API (Application Programming Interface) which will allow genealogy programs to exchange data with their website (FamilySearch.org). One problem with this approach is the need to go through their website, and the fact that they have not made this API publicly available (i.e. it’s not a public standard, just a private interface to their web site). Another major problem is that there is no data format that allows one to create a family tree that can be shared independently, like GEDCOM is used today. FamilySearch in no way needs such a format, since their mammoth size and importance in the genealogical world will force genealogy program to support its API, as many have already done.

Over the years, there have been many attempts to either upgrade or replace GEDCOM. These efforts have all failed. In general the problem has been that the companies that create genealogy program need to agree to adopt any new standard, and they really haven’t had much incentive to do so. Supporting the import of GEDCOM files allows them to support a basic file interchange, which never will support the full feature-set of their programs which have become much more sophisticated since 1996, but is enough to allow customers to exchange information with their relatives. If they supported a fully-featured GEDCOM replacement (that for example would better support photographs and evidence management), it would only make it easier for customers to try other programs. Thus the disincentive for the companies to support a modern replacement for GEDCOM.

Another problem with replacing GEDCOM has been arguments over the data model used. GEDCOM is based on a nuclear family data model (i.e. one mother, one father and their children). It assumes a nuclear family structure, and other forms of families are harder to support. This problem has caused some to support a data model based not on the family but on the individual. This is philosophical debate, and as you might imagine different people take very strong positions in this battle.

Even with this history, there are a few new initiatives to come up with a replacement for GEDCOM. One initiative that has garnered some attention recently is BetterGEDCOM. The BetterGEDCOM initiative came from the frustration of many genealogists over the lack of updates to GEDCOM and is an attempt to create an open forum for the creation of a new standard. Like many attempts at ‘openness’, however, it has run into its own in-fighting and conflicts. It remains to be seen how successful this attempt with be. Another recent initiative is the International OpenGen Alliance (OpenGen). This effort is a bit more of a top-down approach, being managed by the company that runs AppleTree.com, an online family tree web site. OpenGen is, however, a non-profit organization that is supposed to include more than just the team at AppleTree. There have been some attempts between BetterGEDCOM and OpenGen to coordinate, or at least follow each others’ efforts closely. Time will tell which effort, if either, will be successful in creating a new genealogical data sharing standard.

In case you think it isn’t complicated enough, other web sites beyond FamilySearch.org are also developing their own APIs for exchanging genealogical data. OneGreatFamily.com last year introduced an API called GenealogyCloud. It seems that no third-party applications yet support this API.

Geni.com, which boasts nearly a hundred million profiles on their site, and nearly 50 million that are interconnected in what they call their World Family Tree, just yesterday introduced their own API. Unlike FamilySearch.org, however, they are releasing documentation and sample applications on their web site. This will allow anyone to write applications that interact with Geni.com, similar to the way Facebook allows outside developers to create application that access information on Facebook. This is a very positive step. It’s not coincidence that one of the other large family tree web sites, AppleTree.com, is pushing another initiative to replace GEDCOM (OpenGen). These large sites need to create ways to exchange data and interact with other programs and web sites in order to maintain their growth rates.

MyHeritage.com, another one of the big family tree web sites, has taken a slightly different approach in that they have their own application (Family Tree Builder) that runs on a computer, which can sync data to their web site. While this approach allows them more control over what modifies data on their platform, it has its shortcomings as well, not the least of which it requires Windows to run (this coming from a Mac user). I suspect that MyHeritage.com will release their own public API in the future, if only to compete with Geni.com, their biggest competitor.

We can always hope that FamilySearch.org, Geni.com, MyHeritage.com and AppleTree.com will all come together and create a single API and data format for sharing data, but unfortunately if the past is any guide, this is unlikely to happen.

One indication of the direction the wind is blowing in this regard will be the upcoming RootsTech conference, taking place in February 2011 in Salt Lake City. This conference is the first RootsTech conference, although according to the organizers it replaces three earlier technical conferences – The Conference on Computerized Family History, the Family History Technology Workshop and the FamilySearch Developers Conference. Note that these previous conferences were all connected in some way to the Mormon church. It’s unclear how open this new conference will be to new ideas, or if it is really only looking for input for the existing Mormon church efforts such as FamilySearch.org. I imagine representatives from most of the genealogy software companies and web sites will be in attendance at the conference, as will people associated with the BetterGEDCOM and OpenGen efforts. During the week of the conference there will probably be a lot of blogging about what is going on, but the real test will be after the conference if companies announce intentions to seek a common API or data format to move forward with, or whether everyone will just continue the same disjointed approach that has been pursued for nearly 15 years.

Yad Vashem Pages of Testimony and ShoahConnect.org

A major project of Yad Vashem, the Israel Holocaust museum, since its founding, has been to try to collect information on every victim of the Holocaust. This has taken a number of forms, but for many it is best represented by their Pages of Testimony project, which allows anyone to submit a page of information on someone they knew to have perished in the Holocaust. In many cases, these Pages of Testimony are the only record of existence of someone who died, as many primary records were destroyed in the war. Yad Vashem has made efforts over the years to collect these Pages from survivors and relatives of victims, including a big push in Israel in the 1950s to collect information. I remember over the years when visiting Yad Vashem, part of the visit would include searching their database for names of relatives, and being offered to submit names of people not in the database. In 2004 Yad Vashem finished digitizing all the Pages of Testimony and made the entire database available on the web (with some additional sources of information like transport lists – creating the Shoah Names Database), a major breakthrough for those researching their families, and also enabled a new push from Yad Vashem to gather information on victims not yet in the database. As the number of survivors dwindle, those with memories of relatives and friends who were murdered in the Holocaust, and if Pages of Testimony are not filled out soon, knowledge of some peoples very existence may be forgotten forever.

For the Jewish genealogist, searching the Pages of Testimony has long been a very powerful tool. While not every form was completely filled out, many forms include the name of the person, maiden name for women, the name of the person’s parents, where and when they were born, where they lived during the war, what their profession was, etc. All very important information for someone researching their family. To get the full impact of searching the database, it helps to learn how best to use the Advanced Search interface, which allows you to search by specific fields like town of birth, or maiden name. Searching by town alone is something I highly recommend if you know where someone was born, or where your family was primarily from in Europe, as it may show people that you either didn’t think of, or might show spellings that are different than you are used to seeing. The spelling issue is important since many of the forms were submitted in Hebrew, and had the English spellings transliterated automatically. Thus the spelling may be computer-generated, and not necessarily the same spelling you think it will be.

Another important technique for getting the most out of searching the Pages, is that once you find a Page about someone in your family, you should look to see who submitted the information. That person is likely related to you as well. Depending on when the Page was submitted, you may even be able to track down the person who submitted the Page and find out more, or in some cases track down their descendants. When you find a Page, note that there is a menu called Related Searches. The most important search is ‘Pages of Testimony by submitter(s) with the same name’. Most people who submitted Pages, submitted multiple Pages. By seeing which other Pages were submitted by the same person who submitted a Page on a family member of yours, you will likely find Pages of other family members (although some may be of friends or family members from the other side of their family). This search can also be replicated in the Advanced Search form by entering the name of the submitter, and then you can refine the search by adding other fields.

Some people who first find a record of a family member who died in the Holocaust by searching the Pages are frustrated when they realize that person who submitted the Page of Testimony is a relative that they were not aware of, yet the Page was submitted so long ago that they cannot track down the person (and many submitters are now deceased). In some cases, if the record was submitted by someone who lived in Israel, it is possible to track down the person or their descendant through a database kept by Magen David Adom (Israel’s Red Cross). There are individuals who have access to this same database and can probably search it quicker than going through the normal channels. Some of these people are professional genealogists who charge for this service, but I recommend joining the mailing list of the Israel Genealogical Society (which recently merged with the Jewish Family Research Association) and posting questions about people, as there are people on the list who can and probably will look up people for you for free.

There are other resources for tracking down survivors, or information on victims, such as the International Tracing Service, and connected services run through the American Red Cross and the USHMM. That is a whole different topic, however, perhaps for a later post.

So what happens if you find a Page of Testimony submitted by a relative you can’t track down, and may never be able to find, but through which you think you can reconnect to extended family? Here technology offers an interesting solution. Logan Kleinwaks, who also operates the genealogyindexer.org site, runs a site called ShoahConnect.org which is directly relevant. ShoahConnect does something very powerful – it allows anyone to connect themselves to a Page of Testimony, and if someone else uses the site to connect to the same Page, it will notify each person connected to the same Page that there are other people interested in the person on that Page of Testimony. This allows distant relatives to reconnect through their shared relation to a person recorded on a Page of Testimony on Yad Vashem’s site. As of this writing, there are 744 users of the site, with 10,978 connections. This uses a similar technique to the LostCousins.com web site which connects people through common relatives in census records in the US, Canada and the UK.

ShoahConnect utilizes the Google Toolbar to pull off it’s work, so it’s not quite a simple as selecting a name on a web site. However, once you have Google Toolbar installed (in Firefox for Mac and Windows, or Internet Explorer for Windows), clicking a link on the site adds a C-shaped button to the toolbar. The next time you are viewing a Page of Testimony you just click on the new button to link to that Page of Testimony. Follow the instructions on ShoahConnect to get everything set up, and then head over to Yad Vashem’s site and start adding connections.

One last point to make about Pages of Testimony and ShoahConnect – if you want to make connections to other relatives but can’t find relatives on Yad Vashem that you know perished in the Holocaust, you should submit Pages of Testimony for your relatives to the Yad Vashem site, and then link to the Pages you submitted using ShoahConnect. Another thing to remember is that you can provide your own contact information to the Page of Testimony so that if even if someone is not using ShoahConnect, but finds the Page of Testimony that you submitted, they will still be able to reach you based on the contact information you provided as the submitter of the Page of Testimony. If you have photos of the people you are submitting a Page of Testimony for, make sure to also submit the photos, as your branch of the family may be the only branch to have photos of that relative.

The Encyclopedia of Jewish Life Before and During the Holocaust

It’s hard to do Jewish genealogy and not run into Avotaynu, the long-time and as far as I can tell only publisher dedicated to publishing books on the topic of Jewish genealogy. They also publish a self-titled quarterly journal, Avotaynu, on Jewish genealogy and an e-mail newsletter called Nu, What’s New?.

Avotaynu mostly sells their own books, but occassionally sells other books as well. On set of books they’ve been selling for years is NYU Press and Yad Vashem’s The Encyclopedia of Jewish Life Before and During the Holocaust. It’s a three-volume set that provides details on many of the Jewish communities that existed before the Holocaust. It is essentially an abridgment and translation of Yad Vashem’s much larger Hebrew set called Pinkas HaKehilot, published over many years which covers in much greater detail each of the Jewish communities that existed before the Holocaust and what happened to them during and after the war. I highly recommend this set for anyone researching their families in Europe before the war, as it provides a lot of information on many communities that are hard to find out about elsewhere.

In this weeks’ Nu, What’s New? e-mail, Avotaynu mentions that while the set costs $199 on Amazon, they are currently selling it for only $99 plus shipping, per arrangement with the publisher. Shipping from Avotaynu is $20 in the US. So $119 for the books plus shipping.

NYU Press, the publisher, is also selling it for $99 plus shipping on their own site, with the use of a coupon code (see their site for the code). Oddly they show the $99 price as a discount from $350, yet if you go to the regular page for the set, it sells for $199. In any event, it’s currently $99 plus shipping. I don’t know what the shipping costs are from NYU Press, as you need to become a member of their site to find out.

Interestingly enough, Yad Vashem also sells this set directly from their web site. However, Yad Vashem sells it for only $55. Shipping within Israel is free from Yad Vashem, so if you live in Israel it would seem to be a no-brainer where to order it from. If you live outside of Israel, shipping costs $55 which while high, still brings the total cost in the US to $110 instead of the $119 from Avotaynu.

In short, if you live in Israel, definitely order it from Yad Vashem directly. If you live in the US, perhaps check out how much NYU Press is charging for shipping before deciding where to order from. I highly recommend this set for anyone researching their family in Europe.