recovering postgres from a time machine backup

Last night, I needed to do a full Time Machine restore on my Mac. I messed up installing things and made the problem far worse in my attempt to “fix” it. Then I tried to rollback a fraction of the disk. The laptop predicted 11 hours to rollback the files. It was easier to rollback to the state a few hours earlier. The only time I did a full Time Machine restore was when I got a new computer. That was going forward in calendar time though. This is the first time I went backwards.

For the most part, it was easy. The Mac prompted me on whether it was ok to erase the disk and restore. That took about two hours. My computer was pretty much set up on restore. It remembered lots down to my internet connection and github key. I got re-prompted from my Dropbox credentials (and for two factor.) No big deal.

Then there was Postgres. Every few seconds, I got the message “Do you want the application “postgres” to accept incoming network connections?” The message only appeared for about a second. Even if I was fast enough to click on it, it was back a few seconds later.

I tried signing the app based on some internet posts. No luck.

Temporary Relief

I got temporarily relief while I looked into the problem by blocking all connections:

  • System Preferences
  • Security & Preferences
  • Click the lock to make changes
  • My password
  • Click unlock
  • Firewall Options
  • Click “Block all incoming connections”
  • Ok

This is not helpful in the long run, but it let me look at things without going crazy.

The problem

I checked the database logs in /Library/PostgreSQL/8.4/datapg_log and found a file every few seconds with:

2016-04-25 20:39:15 EDT LOG:  database system was interrupted; last known up at 2016-04-24 09:18:51 EDT
2016-04-25 20:39:15 EDT LOG:  record with zero length at 0/5C42472C
2016-04-25 20:39:15 EDT LOG:  invalid primary checkpoint record
2016-04-25 20:39:15 EDT LOG:  record with zero length at 0/5C4246E8
2016-04-25 20:39:15 EDT LOG:  invalid secondary checkpoint record
2016-04-25 20:39:15 EDT PANIC:  could not locate a valid checkpoint record
2016-04-25 20:39:15 EDT LOG:  startup process (PID 2326) was terminated by signal 6: Abort trap
2016-04-25 20:39:15 EDT LOG:  aborting startup due to startup process failure

Ah ha! That makes perfect sense. I jumped around in time which would confuse a transaction log. Now,t hat’s something I know how to fix. I did a forced reset of the transaction log:

  1. sudo su postgres
  2. cd /Library/PostgreSQL/8.4/bin
  3. ./pg_resetxlog -f /Library/PostgreSQL/8.4/data

And all better! I turned the firewall connections back to the way they were before I started and good as new.

OCP 8 Book Bonus: Creating a Derby Database in Java 8

While it is certainly possible to get all the JDBC questions on the exam without running any code or understanding any SQL, it is nice to be able to follow along. This blog post is meant to help anyone who has purchased our book, OCP: Oracle Certified Professional Java SE 8 Programmer II Study Guide: Exam 1Z0-809, download and run through the examples in the text. It also includes the database setup code so you can simply copy/paste it. The actual book covers what you need to know for the exam.


This blog post assumes you are reading chapter 10 of our OCP 8 book and have gotten up to the part that references this blog post.


Creating your initial database

Apache Derby is an open source database. It is really easy to use and comes with JDK 8. This means you don’t have to install anything special. You can even create and setup the database completely in Java. To start out, copy this code into a file named SetupDerbyDatabase.java.

import java.sql.*;

public class SetupDerbyDatabase {

   public static void main(String[] args) throws Exception {
      String url = "jdbc:derby:zoo;create=true";
      try (Connection conn = DriverManager.getConnection(url); 
           Statement stmt = conn.createStatement()) {
			
	   // stmt.executeUpdate("DROP TABLE animal");
	   // stmt.executeUpdate("DROP TABLE species");
			
	   stmt.executeUpdate("CREATE TABLE species ("
	        + "id INTEGER PRIMARY KEY, "
	 	+ "name VARCHAR(255), "
		+ "num_acres DECIMAL(4,1))");
		
	   stmt.executeUpdate("CREATE TABLE animal ("
		+ "id INTEGER PRIMARY KEY, "
		+ "species_id integer REFERENCES species (id), "
		+ "name VARCHAR(255), "
		+ "date_born TIMESTAMP)");

	   stmt.executeUpdate("INSERT INTO species VALUES (1, 'African Elephant', 7.5)");
   	   stmt.executeUpdate("INSERT INTO species VALUES (2, 'Zebra', 1.2)");

 	   stmt.executeUpdate("INSERT INTO animal VALUES (1, 1, 'Elsa', '2001-05-06 02:15:00')");
	   stmt.executeUpdate("INSERT INTO animal VALUES (2, 2, 'Zelda', '2002-08-15 09:12:00')");
	   stmt.executeUpdate("INSERT INTO animal VALUES (3, 1, 'Ester', '2002-09-09 10:36:00')");
	   stmt.executeUpdate("INSERT INTO animal VALUES (4, 1, 'Eddie', '2010-06-08 01:24:00')");
	   stmt.executeUpdate("INSERT INTO animal VALUES (5, 2, 'Zoe', '2005-11-12 03:44:00')");
			
	   ResultSet rs = stmt.executeQuery("select count(*) from animal");
	   rs.next();
	   System.out.println(rs.getInt(1));
      }
   }
}

Then compile as usual:

javac SetupDerbyDatabase.java

Running it is a bit different as you need to include the Derby jar file in your classpath. If you don’t know how to find it or encounter problems see the next sections of this blog post. Notice the classpath contains the following three things:

  1. The relative or absolute path of the Derby jar file
  2. A separator (semicolon on Windows, colon on Mac/Linux)
  3. A dot (which means the current directory)

For example, on my Mac either of these works:

java -cp "$JAVA_HOME/db/lib/derby.jar:." SetupDerbyDatabase
java -cp "/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/db/lib/derby.jar:." SetupDerbyDatabase

If all goes well, the program will output the number 5.

Alternatively, you could have added Derby to your CLASSPATH environment variable and just run the program as

java SetupDerbyDatabase

How do I set up the classpath to run the Java program?

If you know where the JDK ($JAVA_HOME) is on your computer, you can start there and then look in the db/lib directory to find the derby.jar file. The most likely location for the JDK install is:

Operating System Most likely location
Windows c:\program files or c:\program files (x86)
Mac /Library/Java/JavaVirtualMachinges
Linux /usr

Or you can search for derby.jar to get the exact path. On Mac and Linux, the search command is:

find / -name derby.jar -print 2> /dev/null

What does this program actually do?

The main method starts out by obtaining a connection to the Derby database. It then creates a statement object so it can run updates. it would have been more efficient to use a PreparedStatement, but those aren’t on the exam. We aren’t taking user input here so there is no security risk with SQL Injection.

Then the code runs two SQL statements to create tables in the zoo database. The commands each include:

  • the table name – species and animal
  • the fields in each table along with their type. Integer is like a Java int. Decimal is like a Java double. Timestamp is like a Java LocalDateTime or old java Date. Varchar stands for variable character and is like a String. The variable length part means that the database doesn’t need to allocate space for all 255 characters and should only use the space for the actual length of the string. (This matters when you frequently update the field with values of different lengths)
  • the primary key for each table – this tells the database how to you uniquely identify each row

Then the code runs seven SQL statements to insert rows into these tables. The order of the data matches the order the fields were defined in the create statements.

Finally, the code runs a query to check the rows were added to the database. The count(*) function in SQL always returns a number. For an empty table, this number is zero. Therefore, we can call rs.next() outside of a conditional or loop. We know there is always a number being returned.

Derby will create a “zoo” directory and a derby.log file in whatever directory you ran the program in. The zoo directory is your database.

Frequently Encountered Problems

If you have an error that isn’t here or have trouble with these instructions, feel free to ask a question in the CodeRanch forums

Error #1 – Derby is not in your classpath or points to an invalid location

Exact error message:

Exception in thread "main" java.sql.SQLException: No suitable driver found for jdbc:derby:zoo;create=true

at java.sql.DriverManager.getConnection(DriverManager.java:689)

at java.sql.DriverManager.getConnection(DriverManager.java:270)

at derby.SetupDerbyDatabase.main(SetupDerbyDatabase.java:9)

Solution:

Check you are actually pointing to Derby in your classpath. Also check your classpath has the three required components

Error #2 – The current directory is not in your classpath

Exact error message:

Could not find or load main class derby.SetupDerbyDatabase

Solution:

Check you have the current directory (dot) in your classpath. Also, check you have the correct separator for your operating system (semicolon for Windows, colon for Mac/Linux).

Error #3 – The tables already exist

Exact error message:


Exception in thread "main" java.sql.SQLException: Table/View 'SPECIES' already exists in Schema 'APP'.

at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)

at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)

Solution:

The program can only be run once as is. If you want to run it again, uncomment the two “drop table” lines.

I can’t find derby.jar in my Java install directory?

Make sure you are looking at your JDK and not a JRE. The JRE has less things included. For example, it doesn’t have the javac command. And it doesn’t have Derby.

upgrading from postgres 8.3 to 8.4

I upgraded my postgres today.  (Yes, I know it is an old version – that is what the CodeRanch forums use.)  First I reviewed what I did to install 8.3 originally and how frustrating that was.  Upgrading went smoother.  Here’s what I did:

  1. Download the Mac Postgres 8.4 graphical installer.
  2. Install it.
  3. Create my database and load data
  4. Change context.xml in Tomcat to point to port 5433.  (since I had two postgres installs at the same time, it incremented the port from the default of 5432) (changed this back)
  5. sudo su – root and changed  /Library/PostgreSQL/8.4/data/postgresql.conf to use port 5542.  (If I was doing this again, I’d just uninstall the old version as first step to avoid port conflicts)
  6. In finder: Go > Go to folder > Libraries > Postgres > 8.3 and select the uninstaller app

That’s it.  Nice and easy.  Which is good because the initial install was nothing but easy!

Now I can rewrite this without a stored procedure!