Category Archives: Technical

B&F has a new server

Technical, This Site,

The short version of what I’m about to write is that this web site is running on a new web server, which should mean the site will be running faster than ever before. If you see any problems on the site, please contact me so I can fix them. For those interested, I am including more details below.

I am grateful for all the users of this site, and hope everyone who has visited has benefited in some way when they’ve come to the site. This web site has been around since 2010, when it was originally launched as a Blogger site. Back in 2013 I moved the site to self-hosted WordPress, which allowed me to expand the site’s content and functionality, but also meant I had to maintain the site and had to deal with the fact that many other sites were running on the same server. My web host was actually great, and I continue to run other sites on the same host without any problems, but they had no good solution for moving this site to a bigger server when it needed it.

B&F Compendium of Jewish GenealogyWhen I created the B&F Compendium of Jewish Genealogy, I stretched the capacity of the server to its limits. The average WordPress site only has a few Pages (as opposed to Posts, of which there can be many), maybe a few dozen, but the Compendium uses over 25,000 Pages and is continuing to grow. Whether WordPress was the right platform to develop the Compendium on is a different question, but as I had it running in WordPress I had to find a way to increase the capacity of the server without breaking the bank (since this site is a labor of love, and I make no money from it).

For those who use WordPress, think about the fact that on the WordPress editor page, all Pages are loaded into a drop-down menu for selecting a parent page. Imagine now that you have 25,000 pages and that the menu is obviously part of the page that loads.

Amazon LighsailOver the past several months I’ve been working to move my site to a service operated by Amazon called Lightsail. Lightsail is essentially a simplified version of Amazon’s EC2 cloud computing service. You pay a set amount a month and get a VPS server that you control. For $5 a month you get a Linux server with 512MB of RAM and 20GB of storage, and the best part is that you can easily upgrade the server if you need more power. The $5/month is more than I paid before, but it’s still quite reasonable for the added power I get, and if I need to I can move to a more powerful server for simple increments in cost.

Migrating a web site to a new server is never easy, and not to be taken lightly. There are always unexpected problems, and my site had some fairly unique problems. One of the problems I ran into was that the database that holds all the information for the site was so large that it was difficult to even export to a format I could move to the new server. At first the exported database was over 300MB. I looked into the database and saw a lot of the records had to do with plug-ins I used. I had to disable those plug-ins, and remove their data from the database manually, which brought the size down to a still difficult 100MB. When I was finally able to export the database, I found it impossible to import to the new server. The web interface to the database, which was the easiest way to import the data, would run out of memory before completing the import. I tried uploading the file to the server directly and importing the data via the command line, but still had trouble.

Eventually I found a great little piece of code that helped me import the data by breaking it down into smaller chunks and importing it piece by piece (BigDump, which I recommend highly for those who need to import large databases). However, even with this new tool I ended up with an error message I didn’t understand. After asking for help online, I found out that the error was due to the original web site running on an older version of the database software (MySQL) and that the newer version didn’t allow a date field to be empty (set to 0 essentially). When you had a data field in the database that you didn’t have a value for, it was supposed to be set to Jan 1 1970 instead. Go figure, but I had 53 times in the database where I had a zeroed out date, and doing a find and replace in the database fixed the problem and the database finally imported.

Other problems were more mundane. As I tried different parts of the site, I noticed certain things didn’t work properly.

The contact form didn’t work, which it turns out was because the new server didn’t handle mail the same way as the old server. After re-configuring the mail, the contact form began to work.

The maps on the Compendium city pages were not being displayed because for some reason Google thought this was a different site. After setting up new credentials for the the Google Maps API, the maps began to work again as well.

The more insidious problems had to do with file and folder permissions. Having moved much of the files over from my old server, the file permissions on many things were wrong. Now I find file permissions in UNIX to be a form of the dark arts, but slowly I’ve been fixing the problems. I noticed, for example, that I couldn’t upload new images, which is because the server couldn’t create a new folder to upload images to for this month. Sure enough, it was a permissions problem. Upgrading plug-ins has also been difficult due to permission issues. This is probably an issue that will continue to cause issues for some time until I’ve worked out all the bugs.

For the last few days the site has been running completely on the new server. I imagine there will still be problems going forward, and I ask that if you run into anything you think is weird on the site, please please contact me and tell me what you saw. As with the above examples, the problems can be hard to predict, so if you see something odd, such as an error message, or even a missing image, don’t assume I already know about the problem. Please contact me and let me know, so I can fix it for everyone.

Is a third option for transitioning from FTM on the horizon?

Software, Technical, , , ,

I know the transition from FTM doesn’t fall within the primary purpose of this blog, but I’ve always covered the technology of genealogy, and I think the FTM transition has been an important story in genealogy, and has a lot of interesting implications.

I previously reported on the retirement of FTM, what options were out there for transitioning away from FTM, and most recently about Ancestry’s selling of FTM to a third party (Software MacKiev) and licensing it’s APIs to another company (RootsMagic).

In my earliest post on the topic of FTM’s retirement, I mentioned a discontinued product called AncestorSync that had been created to move data between different genealogy programs and services. The need for something like AncestorSync existed (and still exists) because the only way to transfer data between genealogy programs and services has been to use a GEDCOM file, whose standard was last updated back in 1999. GEDCOM hasn’t evolved with genealogy programs, and because of that it cannot transfer everything we collect in our genealogy programs to another program or service without losing some data. Not every program or service interprets GEDCOM the same way either, which leads to other programs like data corruption.

This is why the FTM transition is so interesting. As if to prove how big of a problem all of the above is, we’ve seen multiple genealogy software companies modify their programs to better import GEDCOM files from FTM. So far GEDCOM is the only option available, but by the end of the year there will be the two options I recently mentioned – an updated FTM under new ownership, and a version of RootsMagic that can import FTM files directly. That two companies other than Ancestry will now have software capable of importing FTM files directly, is only because Ancestry no longer views these companies as competitors. Up until now, it wasn’t in Ancestry’s interest to allow any third-party company to be able to read the FTM file format other than themselves.

So we know have two groups of options for former FTM users. We have a slew of genealogy software companies that have updated their programs to better import FTM-generated GEDCOM files, and we have a second smaller group (pair) of companies that will be able to read FTM files directly. That brings us to the third possible option, which I mentioned way back in December, and again just up above – AncestorSync. Years ago when AncestorSync was being introduced I happened to meet the CTO of the company producing it, Dovy Paukstys. We were at a genealogy conference, and he was showing off his product. When FTM made their announcement, I fired off a tweet to Dovy, wondering if AncestorSync might be resurrected. AncestorSync fell off the radar two years ago, and Dovy had moved on to other things, but I figured he would know what happened to the technology.

Back in December I hadn’t heard a response, and didn’t think much about it until a couple of days ago when Dovy finally responded:

dovy tweet
So it seems there might now be another option for FTM users on the way. The goal of AncestorSync, as I recall, was to allow the transfer of data between programs and services while retaining all the information. AncestorSync had modules for each program or service it supposed, and allowed you to move data around between the different modules. Initially the services and programs supported included Geni, MyHeritage, FamilySearch,  ourFamilyology, Legacy Family Tree, RootsMagic, PAF, Ancestral Quest and FTM. I do remember work was underway on The Master Genealogist (since discontinued) and MacFamilyTree.

Of course, if a new standard were to emerge to replace GEDCOM, a program like AncestorSync wouldn’t be necessary. As efforts like GEDCOM X and FHISO haven’t yet managed to come up with a format that can handle better interchange of data and get it accepted by the genealogy software ecosystem, AncestorSync might yet fill the void.

Managing the FTM transition

Software, Technical, , , , , , , ,

FTM TRANSITIONIt’s certainly an interesting story about Ancestry dropping their desktop genealogy software Family Tree Maker (FTM). Ancestry themselves claimed the software was “The #1 Selling Family Tree Program”. It would seem unusual that the #1 selling program would be discontinued. It’s possible FTM was some kind of loss-leader to get people to sign up to Ancestry.com, although it seems odd that they would need to lose money on the program. Other genealogy programs seem to make money. It would seem logical then that the transition is strategic, in order to get more people to use their online family tree product, which as part of their overall service, generates much more revenue for them. As a strategic decision, however, I think they made a mistake in not transitioning the features many genealogists rely on in desktop software to their online offering first, features like charts and reports, as well as better backups of data than the GEDCOM available from Ancestry’s online service.

Their initial blog post announcing the ‘retirement’ of FTM has so far generated over 8500 comments. A second follow-up post has generated another 950 comments so far. Cleary, people have concerns about how Ancestry is handling the transition.

Other companies, of course, are not sitting still. Pretty much every other major desktop genealogy software company has made announcements trying to get disaffected FTM users to switch over to their software. Here are the announcements I found:

Ancestral Quest (Win & Mac*): Ancestral Quest competitive upgrade for Family Tree Maker
Family Historian (Win): Family Historian Welcomes Family Tree Maker Users
Heredis (Win & Mac): Important information about genealogy
Legacy (Win): How to import Family Tree Maker into Legacy PLUS your questions answered
MacFamilyTree (Mac): Family Tree Maker discontinued – Switch to MacFamilyTree and Switch from Family Tree Maker to MacFamilyTree and import your family tree
MyHeritage Family Tree Builder (Win & Mac*): FTM Users: Join MyHeritage and get Family Tree Builder with an Unlimited Size Family Site for Free
Reunion (Mac): Moving your tree from Family Tree Maker to Reunion
RootsMagic (Win & Mac*): Family Tree Maker Upgrade

* These Mac versions run in Emulation using CrossOver or similar technology. This means they are essentially the Windows versions running on the Mac, with no special adaptation made to the user interface to fit Macintosh interface guidelines.

My impression was that RootsMagic was the first to come out and announce a transition plan, even launching the site ftmupgrade.com with instructional videos within a day of the announcements. Most companies have offered financial incentives to switch now as well. Ancestral Quest is offering $10 off their normal price, Family Historian is offering 20% off, Heredis is offering 50% off, MacFamilyTree is offering 50% off, MyHeritage is offering an unlimited size family tree (normally their free tree is limited to 250 people), and RootsMagic is offering their full version for $20 (instead of $44.90). Most of these deals are limited in time, so if you’re interested in taking advantage, definitely check out the programs soon.

It seems everyone suggests exporting a GEDCOM from FTM and then importing that GEDCOM. Only FTM 2012 and later support exporting media with the GEDCOM file.

One problem that seems to be common among those transitioning is how FTM handles source citations. FTM allows media to be linked to source citations, which are in turn linked to a master source. Many genealogy programs use a single master source, but not individual source citations for media. This is confusing some imports, and is not being ignored by the other software companies. I’ve noticed Reunion mentioning that they are working on a fix for this in their forum. I’m sure others are also working on this problem.

Do you use FTM? What are your plans for transitioning? Are you planning to switch to Ancestry’s online site, or moving to a different desktop program? Have you already switched? What has been your experience so far?

What DPI should I scan my photos, and in what format do I save them?

Software, Conference, Technical, , , , ,

My lecture Preserving Photographs, Scanning, and Digital Backups at this weeks’ IAJGS International Conference on Jewish Genealogy was well attended with somewhere around 150-200 people. While I can’t post the video of the presentation on my blog, I do want to share some of the information from the lecture here.

The two most common questions I get about scanning photographs are:

1) What DPI do I need to scan my photo?
2) What file format should I save the file in?

DPI stands for dots-per-inch, and refers to how many pixels are present in each inch of the photograph. For example, if you had an 8×10 inch photograph, and you scanned it at 100dpi, you would have a photo that was 800×1000 pixels, or 800,000 pixels altogether. That’s less than a million pixels, or another to say it is it is less than a megapixel. Doubling the DPI to 200dpi, gives you 1600×2000 pixels, or 3,200,000 pixels, or 3.2 megapixels. Note that doubling the DPI effectively quadruples the number of pixels, since the dpi increases in both vertical and horizontal directions.

Here’s another way to look at, in a slide from my presentation:

DPIAnotherWay
Basically, if you look at scanning photographs (or negatives/slides) you can see that scanning it at 300dpi for different sizes will give you much different size images. I have a rule-of-thumb that I use to determine the correct DPI to scan at, and basically it has to do with figuring out the largest size you want to be able to print (printing is usually done at 300dpi) and then adjust your scanning dpi to insure you’ll have enough pixels to print. Here’s the summary:

rule-of-thumb
For people reading this on a small screen where the image is hard to read, the basic rule is:

Minimum resolution (DPI) should be the number of inches of the largest side you want to print, divided by the largest side in inches of what you’re scanning, multiplied by 300.

So if you are scanning a 4×5 print, and want to be able to print at 8×10, you need twice the DPI you’ll print at, so 600dpi. Of course, it doesn’t hurt to scan more than you need, although there are diminishing returns. Not all photographs are high enough quality to give you a better picture when scanned at very high resolution.

A Kodachrome slide supposedly has enough resolution to output about 20 megapixels. That means you can basically max out a 4000dpi slide scanner and get a good result. That said, a small old print with lots of grain probably wouldn’t benefit by going beyond my rule of thumb, and some likely could be safely scanned at a lower resolution.

Storage is cheap though, so I say scan as high a resolution as you want, and use my rule of thumb as the minimum guideline.

So once you’ve figured out what resolution to scan in, what format should you save it in?

The short answer is TIFF. TIFF was actually designed early on for the purpose of scanning photographs. TIFF also, for the most part, does not lose any data in the file format, unlike formats like JPEG which always compress data in a lossy fashion (I say for the most part because it’s technically possible to use JPEG compression in a TIFF file, but it’s rare, and I doubt any scanner software you would use is going to do that). You can scan to TIFF format using LZW compression that is lossless (i.e. does not degrade the photo quality). TIFF is also good because it is so widely supported, and is used by archives and libraries for their own scanning, and is unlikely to become unsupported by future software.

PNG is also a good format for scanning. It’s a more modern format, and offers built-in lossless compression. It’s not as widely supported, but if space is at a premium, it might save you a bit over TIFF.

JPEG is not a good format for scanning, because it a lossy compression format, and you will always lose some data when saving to a JPEG, even if you save it at 100% quality. I sometimes scan to both TIFF and JPEG, as JPEG can be easier to share sometimes, but I am sure to have the TIFF file as well.

PDF is not a good format to scan photographs with, as you have no control over how images are compressed, and editing them is much more difficult than TIFF or PNG. In general, PDF files will actually use JPEG compression anyways, without being able to even set the quality. If you’re scanning a multi-page printed document, you can use PDF as a convenient way of sharing it, but if there are photos and other important content in the document, I would suggest scanning it as a TIFF as well. It’s not well known, but TIFF also supports multi-page documents, just like PDF.

If you have additional questions about scanning photographs, please post them in the comments below.

Using Nikud (Vowels) in Hebrew on a Mac

Software, Technical, ,

I’ve written a couple of articles in the past about using Hebrew on your computer, specifically Finding Hebrew Fonts and the more niche Trick to use Hebrew and Yiddish in Adobe InDesign. Although using Hebrew on one’s computer is fairly simple, one thing that is not so simple is adding Hebrew nikud (vowels) to your text. In Hebrew, unlike in English, vowels are written as a series of marks, generally below the other letters. An example from my article on fonts:

Nikudot-and-Taamim-Example

In the above text, the blue marks are nikud. In general nikud are not needed for advanced readers of Hebrew, and if you were to buy a Hebrew-Language newspaper or a book in a bookstore, none of them would have nikud, except for when the meaning of the word could not be determined otherwise.

Recently, I had reason to add nikud to a document, and I decided to finally figure out how to add them. Keep in mind, I use a Mac, so these are Mac-specific instructions. For general information on nikud, and codes that can be used on Windows, see the Wikipedia article Niqqud.

On the Mac, there are two keyboard layouts you can use for Hebrew.

First, there is the standard Hebrew layout that is what is used in Israel on all computers.

Second, there is something called Hebrew QWERTY, which maps the Hebrew letters to the closest sounding letters in English, so for example Reish (ר) is mapped to the R and Nun (נ) is mapped to the N. There are some useful shortcuts, like end-letters (in Hebrew some letters change form at the end of a word) simply being Shift and the standard key. For someone who works mostly in English and only occasionally needs Hebrew, Hebrew QWERTY is much quicker to learn.

Adding nikud to text can be done with either layout, although there are some differences. In both cases most nikud are added by using a special key combination, usually using Option (Alt) and a second key. In the standard Hebrew layout, most of the nikud map to Option and a number. For example, adding a kubbutz (which looks like three diagonally arranged dots – as in אֻ) is done by typing a letter and then the key combination Option-8. Just like Hebrew QWERTY tries to map the sounds of letters, it also tries to map the sounds of the nikud, so for the example above of the kubbutz, the key combination is Option-U (the kubbutz sounds like a U).

Hebrew QWERTY can use most of the key combinations from the standard Hebrew layout as well, although not all. All the Option-Number combination (i.e. 0-9) can be used on both layouts.

In order to make it easy to learn, I’ve created a chart that lets you figure out which key combination to use for each nikud. You can download it as a PDF and print it out for easy reference. The first ten combinations are shown using the letter Aleph (א) as an example, with the nikud added. When I write Opt-Sh I mean Option-Shift together with the key shown after it. The next two use a vav (ו) as the example, and the last two are specific to the sin/shin (ש). Most of these nikud can be used on many different letters. I have only added the most common nikud, although there are some more rare ones. For those, I suggest taking a look at the Wikipedia article Niqqud.

The chart is below. You can also download a PDF version if you want.