The grogbuild setup basically creates settings backup based in the questions it asks, it then flashes the box with ATV which in turn restores the backup file when it has flashed - the 'Fast Restore' screen, it then reboots the box again.
The fast restore screen can indeed sit there for a good few minutes depending on box before it should reboot and boot fully with Grogbuild settings.
The fact that it keeps returning to the Fast Restore screen suggests ATV is not able to restore the backup for some reason, this may be as
@bazcfc1 has mentioned be due to the USB stick, or for some reason the backup has got corrumpted. Removing the usb just means that ATV doesn't see the backup file so can't restore the Grogbuild files and therefor boots to a plain ATV setup.
Once in ATV you could insert the USB stick again and then try 'restore settings' to see if that works but my bet is the USB stick, make sure it is Initialised on the box and retry