Web Scraping WW2 Landing Ship Data

Quote of the Day

The destinies of two great empires seemed to be tied up in some God
damned things called LSTs.

— Winston Churchill

Introduction

Figure 1: LSTs on Omaha Beach Shortly After D-Day. (Source)

I have been working on improving my web scraping abilities by analyzing WW2 data. I have focused on topics related to how the US took the 14th largest military in the world and in roughly 18 months turned it into a military that could fight anywhere in the world. In this post, I want to look in detail at how war materials were delivered to beaches around the world using a vessel called a Landing Ship Tank (LST). I have wanted to write about the LST for a while, but the web data was distributed on about 1200 separate pages – one for each ship. While a Python script would have worked nicely, I wanted to try gathering the data without doing any programming. I found some software that did a good job automating this task, which I will discuss later in this post.

To understand the role of the LST, you need to understand the US war material supply chain during WW2. To supply its forces, the US needed to:

• manufacture the goods (e.g., tanks)
• transfer the goods to ports (e.g., trains and barges)
• ship the goods on ocean-going transports (e.g., Liberty and Victory ships) to staging areas
• move the goods to a beach using landing ships – the LST is the largest landing ship and it handles the heaviest material (Figure 1).
• move the goods from the beach to the troops using ground transport (e.g. Red Ball Express)

Figure 2: LST Unloading Rail Cars. (Source)

The Landing Ship Tank (LST) was the landing ship class tasked with placing heavy equipment, like tanks and railcars, onto beaches around the world during WW2 (Figure 2). The US and UK jointly developed the LST design during 1941 with production starting in 1942. The Wikipedia provides a comprehensive list of the 1,198 LSTs that were built by the US, but some of these vessels were built after WW2; 1,046 LSTs were built by the US during the war years (Pearl Harbor to VJ day). The US shipyards average 24 LSTs per month throughout the war. Shipyards in Canada and the UK built ~80 LSTs during the war years. The LSTs were so numerous that most were referred to by their hull designations and were not assigned formal names. Because the LSTs were plentiful, some were repurposed as utility vessels. For example, LST-490 was redesignated as USS Agenor, a landing craft repair ship.

While the  LSTs were a critical part of the war supply chain, the ships were not popular with their crews. They were not particularly seaworthy and the crews were often seasick. Because their bow design was focused on opening to deploy tanks on a beach and not for low drag, the ships were slow (maximum speed of ~12 knots)  and were vulnerable to submarine and E-boat attack. The crews often grimly said that LSTs stood for "Large Slow Target" – a sinking LST was not a pretty place. In fact, 40 US LSTs were sunk for all causes during WW2 (see Appendix A).

Background

Two websites have pages assigned to individual LSTs: navsource.org and uboat.net. Because each LST's data was on an individual web page, I needed to scrape 1198 pages. The magnitude of this task meant that I wanted to automate this work.

There are two software tools that I used to gather the data:

• Data Miner
  Data Miner is a chrome app that I used this tool to gather the individual LST URLs from navsource.org and uboat.net.
• Octoparse
  I fed this tool URLs from Data Miner and used it to parse the individual LST URLs. It has a very useful wizard that guides you through its use.

Octoparse produced an Excel spreadsheet with the LST data I was looking for: LST name, date laid, date launched, data commissioned, and manufacturer. There was some cleanup needed, but Power Query handled that nicely. The rest of the work was standard Excel processing – graphs, computing averages, etc.

I should mention that the navsource.org website did not appreciate being scraped and kept asking me if I was a robot. I had to break my parsing up into small bits over a period of days.

My Excel worksheet is here. There is not much there because most of the work was done by Data Miner and Octoparse.

Analysis

Build Time

I am defining build times as the interval between the date the keel was laid to when the LST was commissioned. Table 1 shows how the mean construction time for an LST varied by year. 1942 production began in traditional shipyards, which resulted in relatively short build times.  By 1943, manufacturing was being moved to non-traditional manufacturers and the build times extended. The build times dramatically reduced in 1944 as the manufacturers gained experience. Times grew again in 1945, which I have seen for other WW2 production. The war was clearing drawing to a close by the summer of 1945 and the government was beginning to reduce the priority of war production.

Table 1: Mean LST Build Time By Year.

Monthly Production

Figure 3 shows the number of LSTs commissioned by US manufacturer per month during WW2. Notice how the monthly commissioning rate peaked just before D-Day. This is not a coincidence as the Anzio landing and D-Day landing created a need for LSTs in Europe. Later production during 1944 and 1945 was needed for action in the Pacific theater.

Figure 3: Monthly US LST Production During WW2.

Average LST Production Per Month

Table 2 shows how the mean US monthly LST production varied by year.  Production averaged 24 units per month throughout the war.

Table 2: Monthly Average LST Production.

Shipyards Involved

Table 3 shows the 17 US companies that produced LSTs during WW2. Because the coastal shipyards were busy building warships, the Pentagon awarded the bulk of the LST contracts to companies with no experience building military ships. During peacetime, these companies produced bridges and river craft. They had the welding technology, heavy equipment infrastructure, and management experience needed for switching their manufacturing lines to LSTs and they produced 68% of all the vessels manufactured.

Table 3: US Companies Producing LSTs During WW2.

Conclusion

The LST may not be pretty and its lack of seaworthiness may have sickened its crews, but it was a critical component in bringing supplies to Allied forces during WW2. The first LSTs entered service early in 1942 and the ship class stayed in service with the US Navy until the 1980s.  As with the M4 Sherman tank and the B-24 Liberator bomber, the LST was not loved by its crews. While many historians dwell on its shortcomings, LSTs were produced quickly and in sufficient quantity to make a major impact on both the Atlantic and Pacific battlefields (Figure 4).  The Germans took an alternative approach to war material – they produced some excellent systems but not in time or in numbers sufficient to make a difference (e.g. Type XXI submarine, Me 262).

I am in awe of the fact that in a time before computers, CAD, and robots, over 1000 large ships designed as part of an international partnership could be produced in 44 months that did the job required.

Figure 4: LST Unloading at Iwo Jima. (Source)

Appendix A: US LSTs Lost During WW2.

Table 4 shows the US LSTs lost for all causes during WW2.

Table 4: List of US LSTs Lost During WW2 For All Causes.