[google-appengine] How to delete 800 mln records from Datastore?

[Kuba Włodarczyk]

I've got around 800 mln (!) records in my Datastore entities. How I can
remove them quickly?
I'm trying to delete them via deferred tasks (python script) but it is
extremly slow...
I would appreciate any help. Thanks.

[Vitaly Bogomolov]

Hi Kuba.

Free quota per day for entity deletes is 20K records. So for free you will
deletes data for 80K days

Or you can delete this data in one day and will be charged $1.6K ($0.02 for
every 20K deletes over quota) + additional costs for running instanses.

https://cloud.google.com/appengine/pricing

WBR, Vitaly.

[Kuba Włodarczyk]

Thank you for your answer. That clarifies a lot. But beside costs how I can
do this, let say in one day (expensive option)? I prefer python.

Post by Vitaly Bogomolov
Hi Kuba.
Free quota per day for entity deletes is 20K records. So for free you will
deletes data for 80K days
Or you can delete this data in one day and will be charged $1.6K ($0.02
for every 20K deletes over quota) + additional costs for running instanses.
https://cloud.google.com/appengine/pricing
WBR, Vitaly.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/a57fc329-6aca-4db5-9c55-5650809374fb%40googlegroups.com
<https://groups.google.com/d/msgid/google-appengine/a57fc329-6aca-4db5-9c55-5650809374fb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Vitaly Bogomolov]

something like this. code not tested and constants may be different. except
800M ;)

from google.appengine.api.taskqueue import Queue, Task
from google.appengine.ext import ndb

QUEUE = Queue('default')


def backend_function():
for j in xrange(800000000 / (100 * 20)):
QUEUE.add([Task(url='/remove_100_records_handler') for i in
range(20)])


def remove_100_records_handler():
for i in range(100 / 20):
ndb.delete_multi(YourDatastoreTable.query().fetch(20,
keys_only=True))

WBR, Vitaly

['Mohammad I (Cloud Platform Support)' via Google App Engine]

Hello Kuba,

You can delete entities in bulk from Cloud Datastore using Cloud
Dataflow[1] which is a managed service for developing and executing data
processing workflows. Please look at this section[2] for best practices for
deletion from Cloud Datastore.

[1]https://cloud.google.com/datastore/docs/bulk-delete

[2]https://cloud.google.com/datastore/docs/best-practices#deletions

Post by Kuba Włodarczyk
Thank you for your answer. That clarifies a lot. But beside costs how I
can do this, let say in one day (expensive option)? I prefer python.

Post by Vitaly Bogomolov
Hi Kuba.
Free quota per day for entity deletes is 20K records. So for free you
will deletes data for 80K days
Or you can delete this data in one day and will be charged $1.6K ($0.02
for every 20K deletes over quota) + additional costs for running instanses.
https://cloud.google.com/appengine/pricing
WBR, Vitaly.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/a57fc329-6aca-4db5-9c55-5650809374fb%40googlegroups.com
<https://groups.google.com/d/msgid/google-appengine/a57fc329-6aca-4db5-9c55-5650809374fb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Attila-Mihaly Balazs]

AFAIK the simplest way to delete them which requires *no code to be
written* is the deprecated but still working datastore admin. In your
Google Cloud Console go to Datastore > Admin click on "Open Datastore
Admin", select the entity kind you want to delete and click "Delete
Entities". This will kick off a distributed, fan-out map reduce job which
will delete the entities in a couple of hours.

Of course, as Vitaly said, this will cost you money.

Attila

[Kuba Włodarczyk]

Thanks Attila-Mihaly,

That's not a problem if cost will around what Vitaly said. Regarding your
solution I've tried "Datastore Admin" but delete tasks behave weird.
The job on the list says "(0 steps completed, 1 active) "
Going to details gives me this - please see screenshot attached.

Mohammad, thanks for your suggestions. I've tried this as well, however I
couldn't track progress, so I stoped this task after 7h. Also I wasn't sure
how to set up this task. I couldn't find any guide. I've entered query like
"SELECT * from Transaction" - is that ok? Transaction is my entity I would
like to remove totally.


Jakub



W dniu wtorek, 27 listopada 2018 06:07:50 UTC+1 uÅŒytkownik Attila-Mihaly

Post by Attila-Mihaly Balazs
AFAIK the simplest way to delete them which requires *no code to be
written* is the deprecated but still working datastore admin. In your
Google Cloud Console go to Datastore > Admin click on "Open Datastore
Admin", select the entity kind you want to delete and click "Delete
Entities". This will kick off a distributed, fan-out map reduce job which
will delete the entities in a couple of hours.
Of course, as Vitaly said, this will cost you money.
Attila

['Amit (Google Cloud Support)' via Google App Engine]

Hello Kuba,

As I can see from the link [1] Mohammad shared, it provided the steps on
how to setup the Cloud Dataflow to create a job to delete entities in bulk.
I can see you are on the right track already. After selecting the
‘Datastore to Datastore Delete’ , you need to put that query if
‘Transaction’ is your entity name. You can monitor the progress using Cloud
Dataflow Monitoring Interface. For more details regarding this , please
check this link [2].

[1]
https://cloud.google.com/datastore/docs/bulk-delete#deleting_entities_in_bulk

[2] https://cloud.google.com/dataflow/docs/guides/using-monitoring-intf

Post by Kuba Włodarczyk
Thanks Attila-Mihaly,
That's not a problem if cost will around what Vitaly said. Regarding your
solution I've tried "Datastore Admin" but delete tasks behave weird.
The job on the list says "(0 steps completed, 1 active) "
Going to details gives me this - please see screenshot attached.
Mohammad, thanks for your suggestions. I've tried this as well, however I
couldn't track progress, so I stoped this task after 7h. Also I wasn't sure
how to set up this task. I couldn't find any guide. I've entered query like
"SELECT * from Transaction" - is that ok? Transaction is my entity I would
like to remove totally.
Jakub
W dniu wtorek, 27 listopada 2018 06:07:50 UTC+1 uÅŒytkownik Attila-Mihaly

Post by Attila-Mihaly Balazs
AFAIK the simplest way to delete them which requires *no code to be
written* is the deprecated but still working datastore admin. In your
Google Cloud Console go to Datastore > Admin click on "Open Datastore
Admin", select the entity kind you want to delete and click "Delete
Entities". This will kick off a distributed, fan-out map reduce job which
will delete the entities in a couple of hours.
Of course, as Vitaly said, this will cost you money.
Attila