Linux java shutdown io bug

Performance Tuning
#1

I’ve come to suspect there is some kind of shutdown bug at the interface with the operating system on openjdk and shutdownhooks

To be clear i’m not talking about problems from the jdk (there is a hilarious one where if you try to serialize a File that came from a JFileChooser in a shutdownhook it will throw a exception because JFileChooser ‘files’ are a fake flyweight object that depends on a shutdown hook of the jdk to not leak memory)

I’m talking about the jvm exiting - naturally, by escaping the main or run of all threads - before the buffers are committed to disk - a outputstream.flush() is NOT the same as a disk sync.

When i tried to get the file descriptor of the corresponding fileoutputstream and force the issue with the sync method during shutdown, sometimes i would get a SyncFailedException.

I only really suspect this because actually getting information after a certain point in the shutdown turns impossible - all ‘unflushed’ ‘unsynced’ log files start to go nowhere - found this by repeated experimentation and comparing the point where the log stopped making sense - it just disappeared sometimes late in the shutdown

I suspect the shutdown init.d or upstart or systemd or whatever protocol is turning off the discs before the application shutdown hooks flush the data, therefore missing the final ‘sync’.
I also suspect the one sure fire way to prevent this (i already optimized what i have to write to disk by performing amortized writes) is to have a init.d|somethingelse script before umount preventing the normal shutdown sequence from going on if it detects the process still in operation and force a sync in the application at the written files in the shutdownhook.

I’m unhappy with this turn of events and i’d like to rant to whoever designed init.d
And oh, i’d like systemd to do better if it can

#2

It sounds to me like some dependencies are screwed up … nothing should be deactivating the devices until all filesystems on it are unmounted, and those shouldn’t unmount until every process is finished.

I don’t think SysVInit can even control devices like that, but upstart and systemd can. What distribution are you using?

#3

Ubuntu whatever the latest.

Here is the ‘guilty’ code:


    public static void writeObjects(Path objectLocation, Serializable... obj) throws IOException {
        FileOutputStream st = new FileOutputStream(objectLocation.toFile());
        try (ObjectOutputStream s = new ObjectOutputStream(new BufferedOutputStream(st, 20000))) {
            for (Serializable a : obj) {
                s.writeObject(a);
            }
            s.flush();
            
            int counter = 0;
            while (counter++ < 5) {
                try {
                    st.getFD().sync();
                    break;
                } catch (SyncFailedException e) {
                }
            }
        }
    }

I added the sync and changed from Files.newOutputStream to fileoutputstream recently, not even sure if it helps (i know syncfailedexception happens sometimes because ‘paradoxically’ i was able to sometimes write the exception to the log i deleted since, if i tried the sync)

Also the the jvm is not being SIGKILL’ed (or halt’ed) - it’s far too little time, i looked at the killing code in the init.d scripts in sendsigs and it has a 10 seconds maximum tolerance by checking for processes every second. Long before that happens the shutdown is progressing and the jvm dies a natural death.

I would expect umounting would flush the goddamned operating system buffers to disk.

#4

Actually they have a sync just before

I don’t know man… this sucks, it’s a bug i know it.
edit: the code displayed had a sync BEFORE sending SIGTERM, ie: the signal that starts the shutdownhooks.

umounting is in another place than sendsigs

edit2: actually, cat /etc/init.d/umountfs | grep sync
gives nothing and the actual command seems to be
fstab-decode umount -f -r -d $REG_MTPTS

actually pretty suspicious of the force, but i don’t really know anything about umount