-
-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Different HDFStores in multiple threads crashes Python #2397
Comments
The underlying storage mechanism, PyTables is inherently not threadsafe for WRITES. HDFStore opens the store file with mode 'a' (append), by default, so this is trying to open 2 writers. Try opening in read mode.
I will add a note to the docs....as this is also a problem in multi-processing (concurrent reads ok, but writing and reading at the same time is a problem) |
Sorry, my original example was a reader. The crash still happens with mode='r'. |
I tried your example (after I created the h5 file), and with mode = 'r'
so in the PyTables layer it is trying to close a file which it thinks is open already. this is a bug in PyTables, see this thread. I guess its not thread-safe even for reads using with doesn't help either
I would just say avoid opening/using the file in multi-threads. I have found no issues using read only in multiple processes however. |
the following example works correctly. I think if you open and close in the main thread, then you can concurrently read w/o a problem in other threads.(still avoid read/writing in more than 1 thread however)
|
We could potentially add locks to HDFStore at some point to prevent multiple threads from accessing the file at once |
Crashes for me (Windows 7 using pytables 2.4.0 and pandas 0.9.1 from http://www.lfd.uci.edu/~gohlke/pythonlibs/). I can't get the stack trace easily, I can try harder if necessary. Simply using tables.openFile and reading a few values seems to work fine.
The text was updated successfully, but these errors were encountered: