How to share a :memory: database between different process in python using python sqlite3 package


How to share a :memory: database between different process in python using python sqlite3 package



scenario is shown as below,i got many processes to do CPU-bound work and read only on the same database,i know cache and uri key words could be used for sqlite to share database cache between threads,but how about between processes?it's better to be available both for linux and windows, Thank you!


def run(self):
self.phyconn = apsw.Connection(self.fileName)
self.memconn = apsw.Connection(":memory:")
try:#backup.__exit__() just make sure copy if finished,not close backup,so with is good,memconn is still exist when out
with self.memconn.backup("main", self.phyconn, "main") as backup:
# call with 0 to get the total pages
backup.step(0)
total = backup.pagecount

stepped = 0
one_percent = total if total < 100 else total // 100
last_percentage = 0
while stepped <= total:
if self.cancel:
#self.progressCanceled.emit()
self.memconn=None
return
backup.step(one_percent)
stepped = stepped + one_percent
stepped_percentage = stepped*100//total
if stepped_percentage != last_percentage:
last_percentage = stepped_percentage
#self.progressChanged.emit(stepped_percentage)
websocket.UpdateLoadDBProgress(stepped_percentage,self.sid)





What is the actual problem that you're trying to solve? Why can't you use the DB file?
– CL.
Jul 1 at 11:12





get many processes read DB at the same time but not write,so i think if it could just be loaded to memory and let every process share it for io efficiency
– KKG
Jul 1 at 12:11




1 Answer
1



This is not possible.
The whole point if processes is to isolate their memory from each other.
(Most OSes allow shared memory, but there is no portable mechanism for that.)



Even if it were possible, it would not be any faster, because both SQLite and the OS cache the data.





even if sqlite cache the data,is not that per connection?and OS dose not know if you just read,so must get one copy per connection when process is different. after searching , it seems to mmap with vfs is the only wayt,but don't see any lib implement that on sqlite
– KKG
Jul 2 at 1:38






SQLite's per-connection cache is 2 MB by default. The OS certainly knows what accesses you have done.
– CL.
Jul 2 at 6:01






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Popular posts from this blog

List of Kim Possible characters

Audio Livestreaming with Python & Flask

NSwag: Generate C# Client from multiple Versions of an API