Category Archives: Technical

B&F has a new server

The short version of what I’m about to write is that this web site is running on a new web server, which should mean the site will be running faster than ever before. If you see any problems on the site, please contact me so I can fix them. For those interested, I am including more details below.

I am grateful for all the users of this site, and hope everyone who has visited has benefited in some way when they’ve come to the site. This web site has been around since 2010, when it was originally launched as a Blogger site. Back in 2013 I moved the site to self-hosted WordPress, which allowed me to expand the site’s content and functionality, but also meant I had to maintain the site and had to deal with the fact that many other sites were running on the same server. My web host was actually great, and I continue to run other sites on the same host without any problems, but they had no good solution for moving this site to a bigger server when it needed it.

B&F Compendium of Jewish GenealogyWhen I created the B&F Compendium of Jewish Genealogy, I stretched the capacity of the server to its limits. The average WordPress site only has a few Pages (as opposed to Posts, of which there can be many), maybe a few dozen, but the Compendium uses over 25,000 Pages and is continuing to grow. Whether WordPress was the right platform to develop the Compendium on is a different question, but as I had it running in WordPress I had to find a way to increase the capacity of the server without breaking the bank (since this site is a labor of love, and I make no money from it).

For those who use WordPress, think about the fact that on the WordPress editor page, all Pages are loaded into a drop-down menu for selecting a parent page. Imagine now that you have 25,000 pages and that the menu is obviously part of the page that loads.

Amazon LighsailOver the past several months I’ve been working to move my site to a service operated by Amazon called Lightsail. Lightsail is essentially a simplified version of Amazon’s EC2 cloud computing service. You pay a set amount a month and get a VPS server that you control. For $5 a month you get a Linux server with 512MB of RAM and 20GB of storage, and the best part is that you can easily upgrade the server if you need more power. The $5/month is more than I paid before, but it’s still quite reasonable for the added power I get, and if I need to I can move to a more powerful server for simple increments in cost.

Migrating a web site to a new server is never easy, and not to be taken lightly. There are always unexpected problems, and my site had some fairly unique problems. One of the problems I ran into was that the database that holds all the information for the site was so large that it was difficult to even export to a format I could move to the new server. At first the exported database was over 300MB. I looked into the database and saw a lot of the records had to do with plug-ins I used. I had to disable those plug-ins, and remove their data from the database manually, which brought the size down to a still difficult 100MB. When I was finally able to export the database, I found it impossible to import to the new server. The web interface to the database, which was the easiest way to import the data, would run out of memory before completing the import. I tried uploading the file to the server directly and importing the data via the command line, but still had trouble.

Eventually I found a great little piece of code that helped me import the data by breaking it down into smaller chunks and importing it piece by piece (BigDump, which I recommend highly for those who need to import large databases). However, even with this new tool I ended up with an error message I didn’t understand. After asking for help online, I found out that the error was due to the original web site running on an older version of the database software (MySQL) and that the newer version didn’t allow a date field to be empty (set to 0 essentially). When you had a data field in the database that you didn’t have a value for, it was supposed to be set to Jan 1 1970 instead. Go figure, but I had 53 times in the database where I had a zeroed out date, and doing a find and replace in the database fixed the problem and the database finally imported.

Other problems were more mundane. As I tried different parts of the site, I noticed certain things didn’t work properly.

The contact form didn’t work, which it turns out was because the new server didn’t handle mail the same way as the old server. After re-configuring the mail, the contact form began to work.

The maps on the Compendium city pages were not being displayed because for some reason Google thought this was a different site. After setting up new credentials for the the Google Maps API, the maps began to work again as well.

The more insidious problems had to do with file and folder permissions. Having moved much of the files over from my old server, the file permissions on many things were wrong. Now I find file permissions in UNIX to be a form of the dark arts, but slowly I’ve been fixing the problems. I noticed, for example, that I couldn’t upload new images, which is because the server couldn’t create a new folder to upload images to for this month. Sure enough, it was a permissions problem. Upgrading plug-ins has also been difficult due to permission issues. This is probably an issue that will continue to cause issues for some time until I’ve worked out all the bugs.

For the last few days the site has been running completely on the new server. I imagine there will still be problems going forward, and I ask that if you run into anything you think is weird on the site, please please contact me and tell me what you saw. As with the above examples, the problems can be hard to predict, so if you see something odd, such as an error message, or even a missing image, don’t assume I already know about the problem. Please contact me and let me know, so I can fix it for everyone.

Is a third option for transitioning from FTM on the horizon?

I know the transition from FTM doesn’t fall within the primary purpose of this blog, but I’ve always covered the technology of genealogy, and I think the FTM transition has been an important story in genealogy, and has a lot of interesting implications.

I previously reported on the retirement of FTM, what options were out there for transitioning away from FTM, and most recently about Ancestry’s selling of FTM to a third party (Software MacKiev) and licensing it’s APIs to another company (RootsMagic).

In my earliest post on the topic of FTM’s retirement, I mentioned a discontinued product called AncestorSync that had been created to move data between different genealogy programs and services. The need for something like AncestorSync existed (and still exists) because the only way to transfer data between genealogy programs and services has been to use a GEDCOM file, whose standard was last updated back in 1999. GEDCOM hasn’t evolved with genealogy programs, and because of that it cannot transfer everything we collect in our genealogy programs to another program or service without losing some data. Not every program or service interprets GEDCOM the same way either, which leads to other programs like data corruption.

This is why the FTM transition is so interesting. As if to prove how big of a problem all of the above is, we’ve seen multiple genealogy software companies modify their programs to better import GEDCOM files from FTM. So far GEDCOM is the only option available, but by the end of the year there will be the two options I recently mentioned – an updated FTM under new ownership, and a version of RootsMagic that can import FTM files directly. That two companies other than Ancestry will now have software capable of importing FTM files directly, is only because Ancestry no longer views these companies as competitors. Up until now, it wasn’t in Ancestry’s interest to allow any third-party company to be able to read the FTM file format other than themselves.

So we know have two groups of options for former FTM users. We have a slew of genealogy software companies that have updated their programs to better import FTM-generated GEDCOM files, and we have a second smaller group (pair) of companies that will be able to read FTM files directly. That brings us to the third possible option, which I mentioned way back in December, and again just up above – AncestorSync. Years ago when AncestorSync was being introduced I happened to meet the CTO of the company producing it, Dovy Paukstys. We were at a genealogy conference, and he was showing off his product. When FTM made their announcement, I fired off a tweet to Dovy, wondering if AncestorSync might be resurrected. AncestorSync fell off the radar two years ago, and Dovy had moved on to other things, but I figured he would know what happened to the technology.

Back in December I hadn’t heard a response, and didn’t think much about it until a couple of days ago when Dovy finally responded:

dovy tweet
So it seems there might now be another option for FTM users on the way. The goal of AncestorSync, as I recall, was to allow the transfer of data between programs and services while retaining all the information. AncestorSync had modules for each program or service it supposed, and allowed you to move data around between the different modules. Initially the services and programs supported included Geni, MyHeritage, FamilySearch,  ourFamilyology, Legacy Family Tree, RootsMagic, PAF, Ancestral Quest and FTM. I do remember work was underway on The Master Genealogist (since discontinued) and MacFamilyTree.

Of course, if a new standard were to emerge to replace GEDCOM, a program like AncestorSync wouldn’t be necessary. As efforts like GEDCOM X and FHISO haven’t yet managed to come up with a format that can handle better interchange of data and get it accepted by the genealogy software ecosystem, AncestorSync might yet fill the void.

Managing the FTM transition

FTM TRANSITIONIt’s certainly an interesting story about Ancestry dropping their desktop genealogy software Family Tree Maker (FTM). Ancestry themselves claimed the software was “The #1 Selling Family Tree Program”. It would seem unusual that the #1 selling program would be discontinued. It’s possible FTM was some kind of loss-leader to get people to sign up to Ancestry.com, although it seems odd that they would need to lose money on the program. Other genealogy programs seem to make money. It would seem logical then that the transition is strategic, in order to get more people to use their online family tree product, which as part of their overall service, generates much more revenue for them. As a strategic decision, however, I think they made a mistake in not transitioning the features many genealogists rely on in desktop software to their online offering first, features like charts and reports, as well as better backups of data than the GEDCOM available from Ancestry’s online service.

Their initial blog post announcing the ‘retirement’ of FTM has so far generated over 8500 comments. A second follow-up post has generated another 950 comments so far. Cleary, people have concerns about how Ancestry is handling the transition.

Other companies, of course, are not sitting still. Pretty much every other major desktop genealogy software company has made announcements trying to get disaffected FTM users to switch over to their software. Here are the announcements I found:

Ancestral Quest (Win & Mac*): Ancestral Quest competitive upgrade for Family Tree Maker
Family Historian (Win): Family Historian Welcomes Family Tree Maker Users
Heredis (Win & Mac): Important information about genealogy
Legacy (Win): How to import Family Tree Maker into Legacy PLUS your questions answered
MacFamilyTree (Mac): Family Tree Maker discontinued – Switch to MacFamilyTree and Switch from Family Tree Maker to MacFamilyTree and import your family tree
MyHeritage Family Tree Builder (Win & Mac*): FTM Users: Join MyHeritage and get Family Tree Builder with an Unlimited Size Family Site for Free
Reunion (Mac): Moving your tree from Family Tree Maker to Reunion
RootsMagic (Win & Mac*): Family Tree Maker Upgrade

* These Mac versions run in Emulation using CrossOver or similar technology. This means they are essentially the Windows versions running on the Mac, with no special adaptation made to the user interface to fit Macintosh interface guidelines.

My impression was that RootsMagic was the first to come out and announce a transition plan, even launching the site ftmupgrade.com with instructional videos within a day of the announcements. Most companies have offered financial incentives to switch now as well. Ancestral Quest is offering $10 off their normal price, Family Historian is offering 20% off, Heredis is offering 50% off, MacFamilyTree is offering 50% off, MyHeritage is offering an unlimited size family tree (normally their free tree is limited to 250 people), and RootsMagic is offering their full version for $20 (instead of $44.90). Most of these deals are limited in time, so if you’re interested in taking advantage, definitely check out the programs soon.

It seems everyone suggests exporting a GEDCOM from FTM and then importing that GEDCOM. Only FTM 2012 and later support exporting media with the GEDCOM file.

One problem that seems to be common among those transitioning is how FTM handles source citations. FTM allows media to be linked to source citations, which are in turn linked to a master source. Many genealogy programs use a single master source, but not individual source citations for media. This is confusing some imports, and is not being ignored by the other software companies. I’ve noticed Reunion mentioning that they are working on a fix for this in their forum. I’m sure others are also working on this problem.

Do you use FTM? What are your plans for transitioning? Are you planning to switch to Ancestry’s online site, or moving to a different desktop program? Have you already switched? What has been your experience so far?