File replacing bat/sh file!

Miscellaneous Topics
#1

Hello, everyone!

I have a rather interesting problem. I’m trying to create a self-updating launcher, but I am having some problems, especially on Windows. Windows won’t let me modify any file in any way, so it seems like I’ll need native code for that. The simplest way would be to have the launcher download all updated files with the file extension “.new” added and then execute a small script that scans through the directory and overwrites all old files with their new version. That allows the JVM to unload all files that it used and the script can update everything without any problems. The problem is writing said scripts as I have limited experience with .bat scripts and pretty much no experience with .sh scripts. >___<

I’m currently looking into how to do this with a .bat script, but I will most definitely need help with the .sh version. It’d be great if someone could help me out with it! Help with the .bat file would also be much appreciated!

#2

After much work I’ve come up with this little bat script. It checks all files in the same folder (and all subfolders) for files that have the “.new” extension and overwrites the old files using move commands.


@echo off

echo Checking directory...
call :updateFiles

echo Starting updater!
java -jar updater.jar

exit /b



:updateFiles

for /R %%f in (*.new) do (
    move "%%f" "%%~nf"
    echo Updated %%~nf!
)

for /D %%d in (*) do (
    cd %%d
    call :treeProcess
    cd ..
)

Now to do the same using an .sh script… :emo:

#3

This seemed to work for me.


#!/bin/bash
#copy all files that match any file with the .new extension. Copy the existing files without the .new to .old
find . -type f -name '*.new' | sed 's/\(.*\)\.new$/\1/' | xargs -I '{}' mv {} {}.old
#copy all the .new files removing the .new extension
find . -type f -name '*.new' | sed 's/\(.*\)\.new$/\1/' | xargs -I '{}' mv {}.new {}

Make sure the file you put it in you make executable with chmod +x renamenew.sh or whatever filename you choose.

Explanation:


find . -type f -name '*.new'
#Find starting in the current directory (.) files (-type f) with the name (-name) ending with .new ('*.new')



#sed matches stream input to produce output
sed 's/\(.*\)\.new$/\1/'
#pipe the stream to sed. The '/' is used as a delimeter. The 's' says that this is a substitution. '\(.*\)\.new$' is a regular expression to find any line that ends in new. The '\1' says to output the first group which is the file name without new.



xargs -I '{}' mv {} {}.old
#xargs allows you to pass lines to an executable command. mv is the move command. Basically the same as rename.
#4

Ohhhh!!! Thank you!

One thing though, I want to delete all files that have a .new version of them and replace them with the new version. I don’t want to keep the .old ones. Also, does this script recursively process all subfolders as well?

#5

[quote]Also, does this script recursively process all subfolders as well?
[/quote]
The find command goes to each subdirectory. so now problem about that.

[quote]One thing though, I want to delete all files that have a .new version of them and replace them with the new version. I don’t want to keep the .old ones.
[/quote]
To delete a file on .sh script you use the rm command so to delete the .old files, this should look like this :

find . -type f -name '*.old' -exec rm -f {} \;

ant to delete the .new files :

find . -type f -name '*.new' -exec rm -f {} \;

However don’t play too much around with that command it can really easily cause a lot of damage to the system.

#6

I think you better do this from Java directly, but in another process jar since the OS won’t allow you to modify the files that are in use. Then your scripts will become a lot simpler.


/////// SH version
#!/bin/bash
echo Checking directory
java -jar updateUtil.jar

echo Starting updater
java -jar updater.jar

/////// BAT version
@echo off
echo Checking directory
java -jar updateUtil.jar

echo Starting updater
java -jar updater.jar

Then the main class of the updateUtil.jar will replace all the .new files with .old. Some Java code for this.


public static void main(String[] args)
{
    processDirectory(new File("."));
}

private static void processDirectory(File directory)
{
    for (File file : directory.listFiles())
    {
        if (file.isDirectory())
            processDirectory(file);
        else
        {
            String path = file.getAbsolutePath();
            if (path.endsWith(".new"))
            {
                File oldFile = new File(path.replace(".new", ""));
                oldFile.delete();

                file.renameTo(oldFile);
            }
        }
    }
}

And I think this is the more safe way than relying on complex shell commands. Also this works on any platform that your game runs too.

#7

There is no need for scripting.

Make this launcher jar that is bare bones, that syncs the game-updater, that syncs the game.

This three-stage setup allows you to have a tiny launcher jar (no GUI, tiny chance of bugs) that can update the game-updater (has GUI, will need maintenance) that then spawns the game.

This all happens in the same JVM - use classloaders to load in the jars after the sync.

Keep in mind that ‘downloading’ is actually a hard nut to crack. You will need to handle stalled connections, have retry logic, etc. You cannot rely solely on socket timeout exceptions, you need a thread that monitors the I/O per download and disconnect when throughput stalls for more than N seconds. It took us quite a few iterations to handle all corner cases.

This is how we did it at Puppygames, before ditching it to fully rely on Valve’s CDN.

#8

Life is currently f**king me over and I haven’t gotten any sleep so I’m not fully processing everything right now, but doing the update in a script does have an advantage: It can update the JRE as well.

#9

The find command is always recursive starting from the directory you chose. The .new files will be gone already since the mv is a move not copy. You can use what leSpace gave you for deleting all the .old files when it is done.

These commands can be pretty dangerous. There is no prompting, it just does the work. Since its specific to what you are looking for it should be pretty safe, as long as you don’t make any type-os.

#10

But how would you sync the files without an JVM? You would be moving more and more critical logic into scripts, and those scripts are limited - multi-threading is required in such a sync, and you really want a proper GUI to show while the download is in progress.

Having said that, even with a JVM you can update the JVM by simply putting one along side the current one, and invoke its java-executable.

Anyway, I’m not saying what we did at Puppygames was the holy grail, but it allowed us to update the updater, which turned out to be absolutely necessary, as getting something as simple as downloads to be reliable was truly a headache, given DNS issues (add fallbacks to raw IP-addresses, as some DNS servers will resolve to the wrong IP for sub-domains!), misconfigured firewalls (only use port 80 and 443), crippling anti-virus suites (hash your files, check the hashes regularly - you cannot overwrite a temporarily locked file either), crappy network-connections (people barely inside their wifi-range), full harddisks (if you write until it’s absolutely full, you’re screwed and the player’s OS may get affected), etc, etc. What happens if the player accidentally launches the game twice - and the syncs will run concurrently…

There are a ton of edge cases. You simply cannot handle this all with a .bat / .sh file properly.

#11

I am interested in the process you describe here Riven given the many qualities it has. But I there’s one thing that perturbs me :

[quote]but it allowed us to update the updater, which turned out to be absolutely necessary, as getting something as simple as downloads to be reliable was truly a headache, given DNS issues (add fallbacks to raw IP-addresses, as some DNS servers will resolve to the wrong IP for sub-domains!), misconfigured firewalls (only use port 80 and 443), crippling anti-virus suites (hash your files, check the hashes regularly - you cannot overwrite a temporarily locked file either), crappy network-connections (people barely inside their wifi-range), full harddisks (if you write until it’s absolutely full, you’re screwed and the player’s OS may get affected), etc, etc. What happens if the player accidentally launches the game twice - and the syncs will run concurrently…
[/quote]
When you are talking about the updater you are talking about the game launcher ?
If not then the issues you got with the game launcher might also occurs with the game launcher updater. Which means if the game launcher updater doesn’t work, you can’t do anything and nothing works.

Also isn’t it also better to allow the game launcher to update the game launcher updater ? Which will enable the system to be able to update every single component.

#12

[quote=“leSpace,post:11,topic:55225”]
Ofcourse, at one point, something has to work, but the game-directory must always be in a state that is launchable - that is: no partial updates, ever.

If the launcher (stage 1) doesn’t work, the current version of the updater (stage 2) will be started.
If the updater (stage 2) doesn’t work, the current version of the game will be started.
This implies that your installer is not an ‘empty shell’, you must ship a (supposedly) functioning version of your game in your installer, as people may not have a working internet connection when they first launch the game.

The updater (stage 2) is typically tiny (say, 200KB) so that the launcher (stage 1) can download and store it. The launcher does not extract DLLs or anything too advanced. The ~200KB download is so small, that the ‘logic’ to handle stalled downloads can basically be boiled down to: “if it didn’t finish in 10s, abort and launch updater”.

#13

Thank you. Then i have one last question : Do you check for corrupted files ? At which stage ?

#14

You have to check for corrupted files, deleted files and locked files.

You’d do this during sync (obviously), and immediately before launching the game.

We’d have one hosted ‘index’ file, that listed all relative paths and (post-decompression) hashes, and start the sync from there.

#15

Deploy as HTML5 only and everything is kept server-side. Simples :slight_smile:

#16

Riven, what I had in mind for this was to only move the updating of files that couldn’t be done to a start-up script. That basically means that the launcher is in charge of downloading files, but all files that are locked by the launcher and can’t be updated are simply stored as .new and then updated after the launcher is restarted. The two main files would be the launcher .jar and the JVM’s files.

#17

One ‘problem’ with that approach seems to me, that for the launcher to update itself, it has to be launched twice. If the launcher is broken (which is the reason you want it to update itself) this might be a no-op and the player would be stuck. (short of redownloading the installer, which contains the new launcher)

For me it was all about getting ‘stage 1’ to only download one file, no GUI, no threading, lean and mean - to reduce the chance that it ever needed an update. Once that works reliably you can modify the ‘stage 2’ updater as often as you want, without any fear that your latest updater will include a bug that will ‘brick’ the game and breaks the sync, forcing a redownload of a new installer by the player.

Anyway, I feel like I’m pushing it a bit too much, even when I’m ‘just’ explaining how we did it :point:

#18

Is there even a reliable way with Java to write to the game directory now and in the future?
So will savegames, screenshots, configs and whatnot you just go to the home directory. But for updates…? When not using Steam (yet)…

#19

My experience with the ‘Program Files’ folder (on Windows XP / Vista) is that you may create files and write data into them, but you may not overwrite said files. The OS blocks this action with a permission error. It’s better to keep every file that may change at one point in the far future, either in the home-dir, or in %appdata%.

#20

Which is interesting because my game always had a screenshot function which I very early on moved the path to the home directory due to it not working anymore.
But even if creating works…I guess you could download an update, tag with with date or number and always run the latest… at best