RocksDB doesn't like the current re-open process

[bauerj]

Nov 27 14:00:44 daedalus systemd[1]: Started Electrumx.
Nov 27 14:00:44 daedalus electrumx_server.py[7867]: INFO:root:ElectrumX server starting
Nov 27 14:00:44 daedalus electrumx_server.py[7867]: INFO:BlockProcessor:switching current directory to /db
Nov 27 14:00:44 daedalus electrumx_server.py[7867]: INFO:BlockProcessor:reorg limit is 200 blocks
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: INFO:BlockProcessor:successfully opened rocksdb database Bitcoin-mainnet for sync: True
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: INFO:BlockProcessor:closing DB to re-open
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: Traceback (most recent call last):
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "/home/electrumx/electrumx/electrumx_server.py", line 68, in main
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: main_loop()
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "/home/electrumx/electrumx/electrumx_server.py", line 49, in main_loop
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: server = ServerManager(Env())
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "/home/electrumx/electrumx/server/protocol.py", line 221, in __init__
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: self.bp = BlockProcessor(self, env)
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "/home/electrumx/electrumx/server/block_processor.py", line 136, in __init__
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: super().__init__(env)
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "/home/electrumx/electrumx/server/db.py", line 54, in __init__
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: self.reopen_db(True)
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "/home/electrumx/electrumx/server/db.py", line 107, in reopen_db
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: self.reopen_db(self.first_sync)
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "/home/electrumx/electrumx/server/db.py", line 86, in reopen_db
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: self.db = open_db(db_name, self.env.db_engine, max_open_files)
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "/home/electrumx/electrumx/server/storage.py", line 24, in open_db
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: return db_class(name, for_sync)
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "/home/electrumx/electrumx/server/storage.py", line 34, in __init__
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: self.open(name, create=self.is_new, for_sync=for_sync)
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "/home/electrumx/electrumx/server/storage.py", line 110, in open
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: self.db = self.module.DB(name, options)
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "_rocksdb.pyx", line 1346, in rocksdb._rocksdb.DB.__cinit__ (rocksdb/_rocksdb.cpp:21094)
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: File "_rocksdb.pyx", line 82, in rocksdb._rocksdb.check_status (rocksdb/_rocksdb.cpp:2850)
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: rocksdb.errors.RocksIOError: b'IO error: lock Bitcoin-mainnet/LOCK: No locks available'
Nov 27 14:00:45 daedalus electrumx_server.py[7867]: CRITICAL:root:ElectrumX server terminated abnormally

[bauerj]

pyrocksdb doesn't seem to provide a close-method, see stephan-hof/pyrocksdb#32.

[kyuupichan]

Yeah; any good ideas?

The excuse is lame; they don't need to add dynamic checks for closedness; they can repoint the class members like we do, just on open/close

[bauerj]

I agree.

Removing the locks manually seems like a bad idea. I guess the only way would be to properly implement this in pyrocksdb.

[kyuupichan]

OK. The ugly GC thing was suggested on those pages, but it seems insufficient. I'll make the reopen thing somehow a function of the DB impl; it may mean pyrocksdb only has 128 files for sync.

[kyuupichan]

This might be fixed in 0.7.14 - can you check?
Also if fixed, can you try without the gc.collect()? I just want to be sure it's needed; it probably is.

[bauerj]

Yes, it works.