Terrel, I believe what you've described should work. Yes, you'll need to copy files to both servers. As far as I know, Kentico does not store media library files in a database, just some metadata, so after you copy files in the file system, some fields in the database will be invalid (e.g. File Size) - not sure if this is a huge issue, but Kentico should notice this and put exclamation mark next to the file you've copied.
You should have an option to import this file from the Media Lib UI (I believe it will just update metadata in database). You should be able to do butch import, if I'm not mistaken.