You are right that ‘in the general case’ you want some kind of buffer with some size. Syncing with the thread effectively means a buffer with a size of zero: SynchronousQueue. Anyway, way offtopic.
[quote=“nsigma,post:40,topic:35781”]
My signature as AI.
CyanPrime: why don’t you simply display byte[N…M] at a time, instead of the whole file? You can use (another) scrollbar to seek through the file and load/save your current range.
You still haven’t removed line wrapping. I removed line wrapping from your app, inserted a newline after each 10,000 ‘bytes’ added to each text area, along with the other suggestions provided, and things were much more responsive. It took a few seconds to load a 10MB zip file, but the UI was much more responsive the entire time.
[quote]- changing to invokeAndWait() instead of invokeLater() to not spam the EvenQueue (might have caused the freeze)
[/quote]
Yeah, invokeLater() seems to freeze while invokeAndWait() doesn’t freeze.
[quote]You still haven’t removed line wrapping. I removed line wrapping from your app, inserted a newline after each 10,000 ‘bytes’ added to each text area, along with the other suggestions provided, and things were much more responsive. It took a few seconds to load a 10MB zip file, but the UI was much more responsive the entire time.
[/quote]
You’re correct. I am now working on it, and seeing extremely good results. Thank you both for your work.
To anyone who has helped in this thread, do you want to be in the credits?
There is only one queue and with invokeLater() in the loop you are inserting massive amounts of ui updates as fast as you can. So every other component using invokeLater() now has to enqueue it’s work behind maybe 20 (slow) ui updates. By doing so you effectively bring swing to a halt…
It was just to reduce the overall amount of ui updates needed. I don’t think that memory handling or allocation speed has any significant impact in this use case.
In this particular case especially though, I don’t think we’re handing the EDT more than it can handle. I modified the code to not wrap lines, keep each line to 1024 “bytes,” and to update the UI every 36 lines. On my particular machine I can load a 10MB file in 10 seconds consistently with invokeLater(), 15 seconds if I use invokeAndWait(). YMMV, of course.
import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
import java.util.Vector;
import java.io.*;
public class HexEditor extends JFrame{
private static final String HEXES = "0123456789ABCDEF";
JScrollPane hexScroll;
JScrollPane byteScroll;
JPanel panel;
JTextArea hexArea;
JTextArea byteArea;
JFileChooser chooser;
FileInputStream fin;
BufferedInputStream bin;
JMenuBar menuBar;
JMenu file;
JMenuItem load;
JProgressBar progressBar;
public HexEditor(){
super("Cypri's java hex editor");
setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
progressBar = new JProgressBar();
chooser = new JFileChooser();
load = new JMenuItem("Load");
load.addActionListener(new ActionListener(){
public void actionPerformed(ActionEvent event) {
Thread hflt = new Thread(new HexFileLoader());
hflt.start();
}
});
file = new JMenu("File");
file.add(load);
menuBar = new JMenuBar();
menuBar.add(file);
hexArea = new JTextArea(25, 16*3);
hexArea.setFont(new Font("Monospaced", Font.PLAIN, 13));
byteArea = new JTextArea(25, 16);
byteArea.setFont(new Font("Monospaced", Font.PLAIN, 13));
hexScroll = new JScrollPane(hexArea);
byteScroll = new JScrollPane(byteArea);
panel = new JPanel(new BorderLayout());
panel.add(hexScroll, BorderLayout.LINE_START);
panel.add(byteScroll, BorderLayout.LINE_END);
getContentPane().setLayout(new BorderLayout());
getContentPane().add(BorderLayout.NORTH, menuBar);
getContentPane().add(BorderLayout.CENTER, panel);
getContentPane().add(BorderLayout.SOUTH, progressBar);
pack();
}
public static void appendHex(StringBuilder sb, int ch) {
sb.append(HEXES.charAt((ch & 0xF0) >> 4))
.append(HEXES.charAt(ch & 0x0F))
.append(' ');
}
public static void main(String[] args){
SwingUtilities.invokeLater(new Runnable() {
public void run() {
HexEditor app = new HexEditor();
app.setVisible(true);
}
});
}
class HexFileLoader implements Runnable {
public void run() {
try{
chooser.showOpenDialog(HexEditor.this);
fin = new FileInputStream(chooser.getSelectedFile());
bin = new BufferedInputStream(fin);
System.out.println("File length: " + chooser.getSelectedFile().length());
SwingUtilities.invokeLater(new Runnable() {
public void run() {
progressBar.setMaximum((int) chooser.getSelectedFile().length());
progressBar.setValue(0);
}
});
int ch;
System.out.println("Load start.");
long start = System.currentTimeMillis();
StringBuilder sb = new StringBuilder();
StringBuilder sb2 = new StringBuilder();
hexArea.setText("");
byteArea.setText("");
int numOfBytesRead = 0;
int count = 0;
while ((ch=bin.read())!=-1) {
HexEditor.appendHex(sb, ch);
if (ch<0x20 || ch==0x7f) {
ch = ' ';
}
sb2.append((char)ch);
numOfBytesRead++;
count++;
if ((count%1024)==0) {
sb.append('\n');
sb2.append('\n');
}
if (count==1024*36) { // 36 lines
hexArea.append(sb.toString());
byteArea.append(sb2.toString());
sb.setLength(0);
sb2.setLength(0);
count = 0;
final int b = numOfBytesRead;
SwingUtilities.invokeLater(new Runnable() {
public void run() {
progressBar.setValue(b);
}
});
}
}
long time = System.currentTimeMillis() - start;
System.out.println("Load completed in: " + (time/1000f) + " seconds");
} catch(Exception e) {
e.printStackTrace();
}
}
}
}
That’s because without wrapping and overall fewer updates (bigger buffer), the load on the queue is drastically reduced.
If you want to update the GUI more frequently (once per cell or whatever), you can always sleep the Thread and that should keep the queue from getting too big too fast. Obviously the file gets read less quickly, but you’re balancing cosmetics versus functionality at that point.
This is in danger of becoming a camel! :
[1]
OK, understand what you’re getting at, but it’s not like it’s posting a Runnable for every character. If my late night maths is right, it’s a difference of posting 15,000 characters at a time vs 150,000 characters at a time. And can the String get so big that adding it in one event actually freezes the EDT more? Personally, if we were really talking about lots of events on the EDT, I’d find a way to coalesce them, but that would probably be overkill here!
But your append() methods aren’t on the EDT anyway?!
hmm … Riven’s sig is now enlightened with my cod … there’s something fishy going on … ;D
[1] http://en.wiktionary.org/wiki/a_camel_is_a_horse_designed_by_committee
That code doesn’t seem to display anything. Looks like you’re missing the parent HexEditor in your calls in the HexFileLoader thread.
Aye, you’re right, actually though it’s just because I’m batching up the updates to the UI. It’ll miss the last update (only updates on multiples of 1024*36 bytes). I missed it because I only tested on large files. Just add this right before the printing out of the runtime:
if (count>0) {
hexArea.append(sb.toString());
byteArea.append(sb2.toString());
SwingUtilities.invokeLater(new Runnable() {
public void run() {
progressBar.setValue(progressBar.getMaximum());
}
});
}
Yeah, I tested with updating the UI after reading different amounts of bytes, and it never ran poorly (i.e. the UI was always responsive, even updating after every byte), so it just became a matter of how much you wanted to throw at it at a time. I stopped upping the buffer at 1024*36 just because it seemed “fast enough” for a simple test. Cyan can certainly tweak the buffer size to whatever he likes.
Yeah, I cheated a little, because prior to Java 7 JTextArea.append() was listed as threadsafe in its Javadoc. Unfortunately, as of Java 7, that’s no longer the case. I believe they took that promise away from all the various JTextComponents’ methods to modify their text. So you probably should wrap those as well.
@BoBear
Can I ask why you used:
if ((count%1024)==0) {
and not something like:
if (count == 1024) {
They’d be the same thing, right?
[quote=“CyanPrime,post:52,topic:35781”]
Only mostly;
(2048%1024)==0 true
2048==0 false
Ah, alright. Thanks.
Anyway now I am working on synchronizing the two JTextAreas. I think I have the add working correctly, but for the remove I have to override my JTextArea’s Document’s remove method. Anyway this is what I got so far (Modified from BoBear2681 's take on the code from: October 15, 2010, 01:38:18 pm)
import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
import javax.swing.event.DocumentEvent;
import javax.swing.event.DocumentListener;
import javax.swing.event.UndoableEditListener;
import javax.swing.text.AttributeSet;
import javax.swing.text.BadLocationException;
import javax.swing.text.Document;
import javax.swing.text.Element;
import javax.swing.text.Position;
import javax.swing.text.Segment;
import java.util.Vector;
import java.io.*;
public class HexEditor extends JFrame{
private static final String HEXES = "0123456789ABCDEF";
JScrollPane hexScroll;
JScrollPane byteScroll;
JPanel panel;
JTextArea hexArea;
JTextArea byteArea;
JFileChooser chooser;
FileInputStream fin;
BufferedInputStream bin;
JMenuBar menuBar;
JMenu file;
JMenuItem load;
JProgressBar progressBar;
public HexEditor(){
super("Cypri's java hex editor");
setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
progressBar = new JProgressBar();
chooser = new JFileChooser();
load = new JMenuItem("Load");
load.addActionListener(new ActionListener(){
public void actionPerformed(ActionEvent event) {
Thread hflt = new Thread(new HexFileLoader());
hflt.start();
}
});
file = new JMenu("File");
file.add(load);
menuBar = new JMenuBar();
menuBar.add(file);
hexArea = new JTextArea(25, 16*3);
hexArea.setFont(new Font("Monospaced", Font.PLAIN, 13));
byteArea = new JTextArea(25, 16);
byteArea.setFont(new Font("Monospaced", Font.PLAIN, 13));
hexScroll = new JScrollPane(hexArea);
byteScroll = new JScrollPane(byteArea);
hexArea.getDocument().addDocumentListener(new DocumentListener() {
@Override
public void changedUpdate(DocumentEvent e) {
}
@Override
public void insertUpdate(DocumentEvent e) {
try{
byteArea.insert(new String(hexStringToDisplayableByteArray(hexArea.getText(e.getOffset(), e.getLength()).replace(" ","").replace("\n", ""))), e.getOffset()/2);
System.out.println("Added: " + hexArea.getText(e.getOffset(), e.getLength()).replace(" ","").replace("\n", ""));
}
catch(Exception ex){
ex.printStackTrace();
}
}
@Override
public void removeUpdate(DocumentEvent e) {
try{
if(hexArea.getText(e.getOffset(), e.getLength()) != " ")
System.out.println("removed: " + hexArea.getText(e.getOffset(), e.getLength()));
}
catch(Exception ex){
ex.printStackTrace();
}
}
});
panel = new JPanel(new BorderLayout());
panel.add(hexScroll, BorderLayout.LINE_START);
panel.add(byteScroll, BorderLayout.LINE_END);
getContentPane().setLayout(new BorderLayout());
getContentPane().add(BorderLayout.NORTH, menuBar);
getContentPane().add(BorderLayout.CENTER, panel);
getContentPane().add(BorderLayout.SOUTH, progressBar);
pack();
}
public static void appendHex(StringBuilder sb, int ch) {
sb.append(HEXES.charAt((ch & 0xF0) >> 4))
.append(HEXES.charAt(ch & 0x0F))
.append(' ');
}
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
}
return data;
}
public static byte[] hexStringToDisplayableByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
if (data[i / 2]<0x20 || data[i / 2]==0x7f) {
data[i / 2] = ' ';
}
}
return data;
}
public static void main(String[] args){
SwingUtilities.invokeLater(new Runnable() {
public void run() {
HexEditor app = new HexEditor();
app.setVisible(true);
}
});
}
class HexFileLoader implements Runnable {
public void run() {
try{
chooser.showOpenDialog(HexEditor.this);
fin = new FileInputStream(chooser.getSelectedFile());
bin = new BufferedInputStream(fin);
System.out.println("File length: " + chooser.getSelectedFile().length());
SwingUtilities.invokeLater(new Runnable() {
public void run() {
progressBar.setMaximum((int) chooser.getSelectedFile().length());
progressBar.setValue(0);
}
});
int ch;
System.out.println("Load start.");
long start = System.currentTimeMillis();
StringBuilder sb = new StringBuilder();
StringBuilder sb2 = new StringBuilder();
hexArea.setText("");
byteArea.setText("");
int numOfBytesRead = 0;
int count = 0;
while ((ch=bin.read())!=-1) {
HexEditor.appendHex(sb, ch);
if (ch<0x20 || ch==0x7f) {
ch = ' ';
}
sb2.append((char)ch);
numOfBytesRead++;
count++;
if ((count%15)==0) {
sb.append('\n');
sb2.append('\n');
}
if (count==15*36) { // 36 lines
hexArea.append(sb.toString());
byteArea.append(sb2.toString());
sb.setLength(0);
sb2.setLength(0);
count = 0;
final int b = numOfBytesRead;
SwingUtilities.invokeLater(new Runnable() {
public void run() {
progressBar.setValue(b);
}
});
}
}
if (count>0) {
hexArea.append(sb.toString());
byteArea.append(sb2.toString());
SwingUtilities.invokeLater(new Runnable() {
public void run() {
progressBar.setValue(progressBar.getMaximum());
}
});
}
long time = System.currentTimeMillis() - start;
System.out.println("Load completed in: " + (time/1000f) + " seconds");
} catch(Exception e) {
e.printStackTrace();
}
}
}
/*class CypriDoc implements Document{
@Override
public void addDocumentListener(DocumentListener arg0) {
// TODO Auto-generated method stub
}
@Override
public void addUndoableEditListener(UndoableEditListener arg0) {
// TODO Auto-generated method stub
}
@Override
public Position createPosition(int arg0) throws BadLocationException {
// TODO Auto-generated method stub
return null;
}
@Override
public Element getDefaultRootElement() {
// TODO Auto-generated method stub
return null;
}
@Override
public Position getEndPosition() {
// TODO Auto-generated method stub
return null;
}
@Override
public int getLength() {
// TODO Auto-generated method stub
return 0;
}
@Override
public Object getProperty(Object arg0) {
// TODO Auto-generated method stub
return null;
}
@Override
public Element[] getRootElements() {
// TODO Auto-generated method stub
return null;
}
@Override
public Position getStartPosition() {
// TODO Auto-generated method stub
return null;
}
@Override
public String getText(int arg0, int arg1) throws BadLocationException {
// TODO Auto-generated method stub
return null;
}
@Override
public void getText(int arg0, int arg1, Segment arg2)
throws BadLocationException {
// TODO Auto-generated method stub
}
@Override
public void insertString(int arg0, String arg1, AttributeSet arg2)
throws BadLocationException {
// TODO Auto-generated method stub
}
@Override
public void putProperty(Object arg0, Object arg1) {
// TODO Auto-generated method stub
}
@Override
public void remove(int arg0, int arg1) throws BadLocationException {
// TODO Auto-generated method stub
}
@Override
public void removeDocumentListener(DocumentListener arg0) {
// TODO Auto-generated method stub
}
@Override
public void removeUndoableEditListener(UndoableEditListener arg0) {
// TODO Auto-generated method stub
}
@Override
public void render(Runnable arg0) {
// TODO Auto-generated method stub
}*/
}
You’re about to enter a world of pain if you insist on optimizing swing.text.*
There is a reason netbeans rewrote that whole area for their editor - and i that only append to a text area and don’t modify it had to buffer the output of the parsers :
http://code.google.com/p/bookjar-utils/source/browse/BookJar-utils/src/util/swing/BufferedStyledDocumentBuilder.java
(i don’t even know how this code interacts with the document model, i only know the result looks alright and loads faster)
and reimplement the gapbuffer
http://code.google.com/p/bookjar/source/browse/BookJar/src/ui/parsers/SequenceStyledDocument.java
(this implementation is append only, and i doubt some other things work, but removes the need for a huge array - that is a problem when you are appending and need more space, because there is a point there where you have two huge arrays - i had exceptions loading war and peace in my reader ;D - only a 5 mb ascii file. Best is to know the size at the start of course.
If i were you, i’d try to use the netbeans framework. It would help you with normal infrastructure, and at the same time give you efficient text oriented components. Never tried it though, looks hard to learn, and maybe they didn’t optimize large file handling.
Remember the memory problem with char[] in swing.text.* is the gapbuffer (and to a lesser extent, if you’re using them, the AttributeSets), i memory profiled, but you can do the same on the netbeans profiler.
I bet those crazy readers / editors that can handle gigabytes files have parsers/readers so integrated that the model you see is mapped to the file itself. When you do change something, or need to read the text they probably know to send the parser read that part of the file and apply the changes all over again to get the final result to display.
But is it really worth it, reinventing so much of the wheel (with limited features) like that, when what he has now works with files of several MB? It may be worth it in some cases I suppose.
You’re right about the text package not being memory efficient though, and a better solution for a hex editor would probably be to just use a JTable with a custom TableModel backed by the byte array for the file. As of right now, a single byte is represented by 3 chars (2 hex characters and a space), but with the JTable approach each byte would just be the byte itself, and a custom renderer could be used to render if as 6A or FF, for example.
This is exactly why i think keeping Java binary compatibility forever is very, very stupid.
There is lots of very, very bad code in the java std lib. Other languages update their libs so the warts are removed/reworked, but not java, oh no.
f**k it, it’s the one reason i seriously considered using scala. Without projects like swinglabs it would be unbearable. Nothing is going to help you with swing.text.* - it is just not used for anything serious.
[quote]This is exactly why i think keeping Java binary compatibility forever is very, very stupid.
[/quote]
Well seems JavaFX GUI controls are meant to replace SWING.
But this doesn’t help atm…
Okay, I had to override both the PlanDocument class and the JTextArea class, but I think I got it working how I want it. How’s this look?
import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
import javax.swing.event.DocumentEvent;
import javax.swing.event.DocumentListener;
import javax.swing.event.UndoableEditListener;
import javax.swing.text.AttributeSet;
import javax.swing.text.BadLocationException;
import javax.swing.text.Document;
import javax.swing.text.Element;
import javax.swing.text.PlainDocument;
import javax.swing.text.Position;
import javax.swing.text.Segment;
import java.util.Vector;
import java.io.*;
public class HexEditor extends JFrame{
private static final String HEXES = "0123456789ABCDEF";
JScrollPane hexScroll;
JScrollPane byteScroll;
JPanel panel;
CypriTextArea hexArea;
JTextArea byteArea;
JFileChooser chooser;
FileInputStream fin;
BufferedInputStream bin;
JMenuBar menuBar;
JMenu file;
JMenuItem load;
JProgressBar progressBar;
public HexEditor(){
super("Cypri's java hex editor");
setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
progressBar = new JProgressBar();
chooser = new JFileChooser();
load = new JMenuItem("Load");
load.addActionListener(new ActionListener(){
public void actionPerformed(ActionEvent event) {
Thread hflt = new Thread(new HexFileLoader());
hflt.start();
}
});
file = new JMenu("File");
file.add(load);
menuBar = new JMenuBar();
menuBar.add(file);
hexArea = new CypriTextArea(25, 16*3);
hexArea.setFont(new Font("Monospaced", Font.PLAIN, 13));
byteArea = new JTextArea(25, 16);
byteArea.setFont(new Font("Monospaced", Font.PLAIN, 13));
byteArea.setDocument(new CypriDoc());
hexScroll = new JScrollPane(hexArea);
byteScroll = new JScrollPane(byteArea);
hexArea.getDocument().addDocumentListener(new DocumentListener() {
@Override
public void changedUpdate(DocumentEvent e) {
}
@Override
public void insertUpdate(DocumentEvent e) {
try{
byteArea.insert(new String(hexStringToDisplayableByteArray(hexArea.getText(e.getOffset(), e.getLength()).replace(" ","").replace("\n", ""))), e.getOffset()/2);
System.out.println("Added: " + hexArea.getText(e.getOffset(), e.getLength()).replace(" ","").replace("\n", ""));
}
catch(Exception ex){
ex.printStackTrace();
}
}
@Override
public void removeUpdate(DocumentEvent e) {
try{
System.out.println("removed: " + hexArea.doc.removedText);
}
catch(Exception ex){
ex.printStackTrace();
}
}
});
panel = new JPanel(new BorderLayout());
panel.add(hexScroll, BorderLayout.LINE_START);
panel.add(byteScroll, BorderLayout.LINE_END);
getContentPane().setLayout(new BorderLayout());
getContentPane().add(BorderLayout.NORTH, menuBar);
getContentPane().add(BorderLayout.CENTER, panel);
getContentPane().add(BorderLayout.SOUTH, progressBar);
pack();
}
public static void appendHex(StringBuilder sb, int ch) {
sb.append(HEXES.charAt((ch & 0xF0) >> 4))
.append(HEXES.charAt(ch & 0x0F))
.append(' ');
}
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
}
return data;
}
public static byte[] hexStringToDisplayableByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
if (data[i / 2]<0x20 || data[i / 2]==0x7f) {
data[i / 2] = ' ';
}
}
return data;
}
public static void main(String[] args){
SwingUtilities.invokeLater(new Runnable() {
public void run() {
HexEditor app = new HexEditor();
app.setVisible(true);
}
});
}
class HexFileLoader implements Runnable {
public void run() {
try{
chooser.showOpenDialog(HexEditor.this);
fin = new FileInputStream(chooser.getSelectedFile());
bin = new BufferedInputStream(fin);
System.out.println("File length: " + chooser.getSelectedFile().length());
SwingUtilities.invokeLater(new Runnable() {
public void run() {
progressBar.setMaximum((int) chooser.getSelectedFile().length());
progressBar.setValue(0);
}
});
int ch;
System.out.println("Load start.");
long start = System.currentTimeMillis();
StringBuilder sb = new StringBuilder();
StringBuilder sb2 = new StringBuilder();
hexArea.setText("");
byteArea.setText("");
int numOfBytesRead = 0;
int count = 0;
while ((ch=bin.read())!=-1) {
HexEditor.appendHex(sb, ch);
if (ch<0x20 || ch==0x7f) {
ch = ' ';
}
sb2.append((char)ch);
numOfBytesRead++;
count++;
if ((count%15)==0) {
sb.append('\n');
sb2.append('\n');
}
if (count==15*36) { // 36 lines
hexArea.append(sb.toString());
byteArea.append(sb2.toString());
sb.setLength(0);
sb2.setLength(0);
count = 0;
final int b = numOfBytesRead;
SwingUtilities.invokeLater(new Runnable() {
public void run() {
progressBar.setValue(b);
}
});
}
}
if (count>0) {
hexArea.append(sb.toString());
byteArea.append(sb2.toString());
SwingUtilities.invokeLater(new Runnable() {
public void run() {
progressBar.setValue(progressBar.getMaximum());
}
});
}
long time = System.currentTimeMillis() - start;
System.out.println("Load completed in: " + (time/1000f) + " seconds");
} catch(Exception e) {
e.printStackTrace();
}
}
}
class CypriDoc extends PlainDocument{
String removedText;
public String getRemoved(){
return removedText;
}
@Override
public void remove(int offs, int len) throws BadLocationException {
removedText = getText(offs, len);
super.remove(offs, len);
}
}
class CypriTextArea extends JTextArea{
CypriDoc doc;
public CypriTextArea(int rows, int columns){
super(25, 16*3);
doc = new CypriDoc();
super.setDocument(doc);
}
}
}